AI Engineering (6): Outer Loop—Let Runs Hand Off Long-Lived Work

Long-lived work rarely begins and ends with one chat window. A task may wait for CI, review, or approval, and it may be split across multiple runs because time or environments expire. To let a later run truly continue, the system must separate the task from the current conversation.

After this article: you should be able to define the Outer Loop and its minimum parts, then explain recovery, concurrent ownership, retries, and uncertain outside actions.

Evidence boundary: this article abstracts cross-run mechanisms from OpenAI’s Harness engineering and Symphony and Anthropic’s long-running harness and Managed agents material. Community tutorials at the end add teaching perspectives, not product definitions.

1. Understand the Outer Loop through a failure story

1.1 Start with a task that has no Outer Loop

In the first run, an Agent finds the payment-test failure and changes the code, but the run ends before full verification. A second run receives only the original request, so it investigates from the beginning. Then a create-PR request times out; the new run assumes it failed and creates a second PR.

The model is not necessarily less capable. What is missing is cross-run control: how far did the task get, which run currently owns it, what result may finish it, where should recovery begin, and did an outside action actually succeed?

1.2 Learn four small concepts first

Work item: the long-lived task to finish, such as “fix the payment tests and pass review.” It may live for hours or days.
Run: one limited attempt by an Agent to advance the work item. It may finish, stop, or wait.
Agent Loop: the think–act–observe cycle inside one run.
Outer Loop: the control flow around multiple runs that triggers, assigns, verifies, recovers, and retries the same work item.

In one sentence: the Agent Loop advances one run; the Outer Loop keeps the task alive across runs until it finishes or passes to a person.

1.3 Follow one work item from arrival to completion

Create the task: a failing test event or person creates a work item with a goal and completion criteria.
Select the task: the system chooses an item that is ready, appropriately prioritized, and within capacity.
Launch a run: give the Agent the goal, tools, authority, and previous progress.
Persist progress: record confirmed facts, changes, test results, and unfinished work throughout the run.
Verify independently: CI, review, or a business check accepts the result; the Agent's “done” is only a candidate conclusion.
Decide what follows: finish on success, run again with new evidence, wait for authority, or escalate after ineffective attempts.

These six steps are the backbone. Queue, lease, checkpoint, and idempotency are protections added when tasks multiply, runs can stop, or outside actions can be duplicated.

2. Turn the work item into an operating control flow

2.1 Why the task must become a separate work item

Chat history preserves a conversation, but it is not the task itself. A work item first needs only a stable id, goal, completion criteria, current state, completed work, and next step. As the system grows, add priority, dependencies, current owner, attempt count, and outside-object references. A run is one processing record attached to the work item; it should not replace the work item.

Outer loop with trigger, work selection, run launch, verify, persist, retry or next, and an inner agent loop — The inner loop consumes a work item; the outer loop decides when to create it, who gets it, whether to accept the result, and what follows. Persist is drawn as the final handoff, while checkpoints and effect records are written throughout the run.

Task: fix the payment-module regression
State: in progress
Done when: payment tests pass and review accepts the change
Confirmed: the expired-order check is missing
Changed: payment.go
Next: run the payment tests

Minimal work item: first make the goal, progress, and next step understandable to the next run. Add concurrency, retry, and effect fields only when those problems appear.

The authoritative copy of this work item belongs to durable Outer Loop storage, not to one prompt or chat transcript. At launch, the system projects the current goal, authority, and progress to the Agent. The run commits new facts through checkpoints. Only after the verifier passes does the Outer Loop move the task from awaiting validation to completed. Later runs and human reviewers continue from the updated item and its artifact references.

2.2 Turn the six steps into operating responsibilities

These are not a one-pass linear pipeline. Progress recording begins before the run and continues through execution and final handoff; otherwise a crash before verification leaves nothing to recover. Later sections explain concurrent claims, checkpoints, and outside-effect records one at a time. For now, remember the responsibilities in ordinary language.

Phase	Control question	Fact it must leave
Trigger	Which event or person created the task?	Source, duplicate status, time
Select work	Which task can start now?	Selection reason and current worker
Launch run	Which goal, tools, authority, and progress does this Agent receive?	Run configuration, input version, previous progress
Verify	Which independent checks accept the result?	Check evidence and failure reason
Persist	Which progress and outside changes must survive?	Recovery point and action record
Retry / next	Finish, wait, change strategy, or ask a person?	Decision reason and remaining attempts

2.3 When multiple workers are available, who owns the task?

Imagine two workers take the same payment work item from the waiting list and both start editing. A queue can determine who sees the task first, but the system also needs a time-limited claim: one worker holds write ownership for a period, keeps proving it is alive, and loses the claim after expiry so another worker may continue. That time-limited claim is a lease.

Real webhooks can duplicate, schedules can overlap, and workers can crash. Without an ownership protocol, two Agents can change one task at once or a dead worker can hold it forever.

Outer-loop control plane with dedupe, queue, lease, run, verifier, retry budget, and human escalation — Queue controls waiting order, lease controls current ownership, and a fencing token rejects late writes from an old owner. Repeated outside actions are handled separately in Section 3.3.

Deduplication: identical external events create one work item.
Lease: one run owns exclusive writes; read-only investigation may use a separate parallel policy.
Heartbeat: distinguish a long run from a dead worker.
Fencing token: a late write from an old lease cannot overwrite the new owner.
Backpressure: launch according to resources, risk, and human review capacity.

queued
  -- worker-A claim, fence=1 --> running(A)
  -- lease expires -----------> claimable
  -- worker-B claim, fence=2 --> running(B)

complete(worker-A, fence=1)  # rejected: stale owner
complete(worker-B, fence=2)  # accepted

Ownership experiment: the queue decides who is seen first; the lease makes ownership expire; the fencing token deterministically rejects a late write from the old worker.

3. Advanced: recovery, retries, and effects across runs

3.1 How the next run continues after interruption

A checkpoint is a saved starting point from which the next run can recover. Return to the payment story. Before Run 1 ends, it records the goal, confirmed cause, change to payment.go, unrun tests, and log references. Run 2 first verifies that the file and references are still current, then continues from “run the payment tests” instead of investigating again.

Anthropic’s long-running harness experience emphasizes clear progress files, git state, and clean task slices so a later agent can continue. Such a handoff serves machines and people: structured fields make recovery testable, narrative preserves decisions and surprises, and raw artifacts preserve evidence.

Cross-run handoff where Run N passes a checkpoint, effect ledger, and artifacts to Run N plus one — The next run should neither re-guess what happened nor trust an evidence-free summary.

Handoff field	Question	Recovery check
Goal / done	What must be accomplished?	Is it still valid?
Confirmed facts	What is known?	Are sources accessible and fresh?
Effects	Which external state already changed?	Reconcile with the world
Attempts	What failed and why?	Avoid repeating the same strategy
Open / next	What blocks and what is the smallest next step?	Are capability and authority sufficient?
Artifacts	Where are logs, diffs, tests, and screenshots?	Are references complete?

3.2 Failure does not always mean “try again”

An inner-loop tool retry is different from replaying an entire run. The latter costs more and can duplicate effects. Before every relaunch ask: what failed, what changes in the new run, have old effects been reconciled, and how much budget remains? “Try once more” is not a sufficient answer.

Persist retry budget with the work item so worker restarts do not reset it. Vary policy by class: transient infrastructure errors back off; authority blocks wait for approval; repeated invariant failures escalate; an uncertain external write reconciles before replay.

3.3 A timeout does not prove that the outside action failed

Creating a PR, issuing a refund, or sending a notification can succeed while its response is lost. If the outer loop sees only a timeout, it may repeat the effect. Retriable effects need stable idempotency keys and a ledger of proposed, started, committed, and observed states. Recovery queries the world before continuing, compensating, or completing.

effect_key = "issue-1842:create-pr:v3"
ledger: proposed → started → unknown
remote: PR #41 already exists

reconcile(effect_key)
  → find PR #41
  → ledger: committed → observed
  → do not create PR #42

Timeout before and after: a naive flow is timeout → retry → two PRs. A safe flow is timeout → reconcile → reuse the existing PR. Persistence must therefore begin before the effect, not after verification.

Record “the call happened” separately from “the world changed.”
Prefer external API idempotency; otherwise use a business key plus reconciliation query.
Model and audit compensating actions; not every effect is reversible.
An effect with unknown state requires reconciliation; it is not an ordinary failure to retry.

3.4 People should own judgment boundaries, not polling

OpenAI’s Harness engineering starts from scarce human attention. Put people at high-leverage boundaries: conflicting goals, high-risk authority, quality that cannot be automated, and decisions to spend more budget. An escalation package should be compressed but sufficient: what happened, what is verified, options and impact, recommendation, and deadline.

4. Review the Outer Loop before launch

Question	Minimum mechanism	Without it
Can duplicate events create duplicate work?	Dedupe key	Concurrent duplicate effects
Who owns the item now?	Lease plus fencing	Double writes or permanent stall
How is completion accepted?	Independent verifier	Final prose becomes outcome truth
What does the next run continue from?	Checkpoint plus artifacts	Lost state and repeated investigation
What changes on retry?	Error class plus persistent budget	Unbounded spend
What if an effect is uncertain?	Idempotency plus effect ledger	Duplicate refunds, PRs, or messages
When does a person take over?	Escalation policy	No owner or constant interruption

The outer loop turns runs into an operable work system. It still needs external truth for the “verify” step. The final part covers evals and feedback: measure outcomes, localize failures, and turn evidence into safe system change.

Outer Loop: Let Runs Hand Off Long-Lived Work