Rollout and Recovery: Continuing Requires Replay

Imagine a long coding task. The agent has read files, run tools, compacted history, changed model settings, and accumulated enough context that a simple transcript is no longer the whole story. If the process stops, the next runtime must recover more than visible messages. It needs the model-visible history, the current context baseline, session identity, lineage, and the event facts that a client will project back to the user.

That is the job of rollout. Codex writes an append-only JSONL stream of typed RolloutItems, keeps write ordering behind a recorder, and reconstructs the runtime state by replaying those items through session logic. The recovery path is not a convenience wrapper around chat history; it is part of the runtime contract.

This part follows six questions:

1. Why is continuing a thread harder than showing old messages?
2. What does each RolloutItem variant own?
3. How does the writer queue, persist, flush, and recover from failed writes?
4. Why do resume and fork first become InitialHistory?
5. Why does reconstruction scan backward before replaying forward?
6. How do rollback, compaction, and prompt caching meet at this ledger?

1. Continuing Has Three Surfaces

A client needs visible facts. The next model request needs ordered input. The runtime also needs a context baseline: working directory, sandbox, permission profile, model, collaboration mode, and the latest full-context checkpoint. Those surfaces overlap, but they cannot collapse into one saved string.

This is why rollout connects earlier parts of the series. Client projection, multi-agent spawning, context management, and prompt cache discipline all rely on the same durable evidence chain.

2. Five Rollout Items, Five Owners

The protocol defines five rollout variants: SessionMeta, ResponseItem, Compacted, TurnContext, and EventMsg. Each JSONL line also carries a timestamp, so the file can be appended and later read back as a typed stream.

At the shape level, a JSONL fragment can look like this. Each line is an independent fact. Reconstruction does not read it as UI text; it dispatches by item.type.

{"timestamp":"...","item":{"type":"session_meta","thread_id":"t1","source":"cli"}}
{"timestamp":"...","item":{"type":"turn_context","cwd":"workspace","sandbox":"workspace-write"}}
{"timestamp":"...","item":{"type":"event_msg","event":{"type":"turn_started","turn_id":"u1"}}}
{"timestamp":"...","item":{"type":"response_item","item":{"type":"tool_call","name":"shell"}}}
{"timestamp":"...","item":{"type":"compacted","window_id":"w1","summary":"..."}}

TurnContextItem is easy to miss. Its source comment says it is persisted once per real user turn after model-visible context updates are computed, and again after mid-turn compaction when replacement history re-establishes full context. That is how resume and fork recover the latest durable baseline.

3. Writes Queue Before They Become Durable

New and resumed sessions initialize the recorder differently. A new session precomputes rollout path and session metadata, but it can defer file creation until persist(). A resumed session opens the existing rollout for append immediately.

Actual writes go through a background writer task. The recorder sends AddItems, Persist, Flush, and Shutdown commands. Items live in pending_items until they are written successfully. If I/O fails, the writer drops the file handle, keeps the unwritten suffix, and retries after reopening at the next durability barrier.

writer discipline:
record_canonical_items(items) -> queue AddItems
persist()                   -> materialize file + write pending
flush()                     -> durability barrier for rollback/fork/resume
shutdown()                  -> final drain before exit

This explains the flushes in fork and rollback paths. A snapshot is only meaningful if it is based on facts that have reached the durable stream.

4. Resume and Fork Enter Through InitialHistory

Session startup normalizes all starting points into InitialHistory: New, Cleared, Resumed(ResumedHistory), and Forked(Vec<RolloutItem>). New and cleared sessions defer initial context insertion until the first real turn. Resumed and forked sessions create a default turn context and call apply_rollout_reconstruction.

Resume also warns when the last recorded model differs from the current model. Reconstruction can restore the shape of history, but changing models can still change context windows, cache behavior, and runtime performance.

5. Reconstruction Scans Backward, Then Replays Forward

reconstruct_history_from_rollout does not replay the whole file from the first line. It first scans newest-to-oldest. The scan looks for the newest surviving replacement-history checkpoint, previous turn settings, reference context item, and window id. Once those are known, older items cannot affect the rebuilt state.

Rollback becomes a counter during this reverse pass. ThreadRolledBack means “drop the newest N real user turns.” In reverse, that means skipping the next N finalized turn segments that actually contain a user message. Tests cover the distinction: standalone task turns should not consume rollback skips.

After the checkpoint is found, Codex replays the surviving suffix forward. ResponseItems are recorded into ContextManager. Compactions with replacement_history replace history. Legacy compactions without replacement history take a compatibility path. ThreadRolledBack applies drop_last_n_user_turns.

reconstruction:
reverse scan
  find newest surviving replacement_history
  recover previous_turn_settings
  recover reference_context_item
  account for ThreadRolledBack markers

forward replay
  seed ContextManager from replacement_history
  append surviving ResponseItem suffix
  apply rollback markers to the rebuilt history

6. Rollback Adds a Marker, Then Replay Interprets It

Rollback rejects num_turns == 0 and refuses to run while a turn is active. It then requires persisted thread history, flushes pending writes, loads stored history, appends a temporary ThreadRolledBack event, and runs the same reconstruction path. After that, it persists the rollback marker and flushes again.

The older facts remain in the append-only stream. The current world is the replay result of those facts plus the rollback marker. A later resume sees the same marker and computes the same trimmed history.

Integration tests check this through compaction. Rolling back a post-compaction turn should keep the first turn and compaction summary visible while removing the edited post-compaction turn. Another test checks that rolled-back context updates do not linger in the next request.

7. Fork Copies Replayable History

Fork shares the recovery machinery with resume, but lineage is different. Resume keeps the old thread id. Fork creates a new id while recording forked_from_thread_id. The manager reads rollout-backed history, shapes it with fork_history_from_snapshot, and starts a new thread.

Subagent spawning makes the durability requirement explicit: it materializes and flushes the source rollout before reading stored history. The child does not inherit a loose in-memory object; it inherits a segment of rollout items that can be interpreted as InitialHistory::Forked.

8. The Prompt Cache Connection

Part IX argued that prompt cache hits depend on stable request shape. Recovery preserves the precondition for that stability. TurnContextItem tells the runtime what context has already been established. CompactedItem.replacement_history gives history a new base. Rollback markers trim the tail consistently.

If recovery were based only on visible transcript text, Codex could re-inject full context as repeated diffs or carry rolled-back tool output into the next model view. That would be a semantic bug and a prompt-cache-shape bug. The compact/resume/fork tests assert that compacted input prefixes remain predictable across recovery.

9. What to Carry Forward

Rollout/recovery gives a useful agent-design rule: a long-running agent should persist replayable facts, not only final state snapshots.

The internal loop is now almost complete: requests enter the runtime, context forms the model view, tools create side effects, events project to clients, rollout persists the evidence, and recovery turns that evidence back into a usable next turn. The next part should look at the public edge: how SDK and app-server callers enter the same runtime.

Sources

• InitialHistory and ResumedHistory
• RolloutItem and CompactedItem
• TurnContextItem persistence comment and fields
• RolloutRecorder::new create and resume paths
• record_canonical_items, persist, flush, and rollout loading
• writer pending queue, recovery mode, and retry behavior
• writer command loop and SessionMeta write
• record_initial_history and apply_rollout_reconstruction
• reconstruct_history_from_rollout reverse scan and forward replay
• thread_rollback and rollback marker persistence
• resuming a thread from rollout path
• forking a thread and forked_from_thread_id
• test: compact / resume / fork preserve model-visible history prefix
• test: rollback past compaction replays append-only history