How context, memory, and compaction work together

Return to Alice from the previous chapter. She has talked with the agent for two weeks and the transcript still exists. Today she says, “Continue with last week's decision,” but the answer sounds as if the agent has never heard of it. The obvious diagnosis is lost history. Yet the history may still be durable and simply absent from the current model window. The decision may live only in a long transcript rather than long-term memory. Or compaction may have summarized the older exchange without preserving that detail.

All three failures look like forgetting, but they require different repairs. A larger window cannot fix the wrong session route. Telling the model to “remember” does not create durable state. Sending the entire transcript on every run quickly collides with cost and hard capacity. Diagnosis starts by separating record, memory, and runtime view into different owners.

Evidence boundary.This chapter is pinned to commit c549250 and follows the workspace/bootstrap, system-prompt, ContextEngine, and compaction implementation in that snapshot. Memory backends continue to evolve, so the focus is the verifiable runtime contract rather than treating one retrieval implementation as permanent architecture.

1. Context means everything in one model request

In OpenClaw, context is not merely the last few chat messages. It is everything actually sent for one model request: the system prompt, conversation messages, tool calls and results, attachments, and tool definitions. All of them compete for the same context window. There is no permanently reserved space that only conversation history may consume.

This explains a familiar surprise: no extra conversation was added, but installing several verbose tools reduced the history that fit. The visible chat length is not the only input. Images, structured tool results, JSON schemas, and dynamic prompt additions all consume capacity.

model context = system prompt
              + assembled conversation history
              + tool schemas
              + tool results and attachments

memory != model context
transcript != model context

The two inequalities matter more than the formula. A transcript is the durable record of a session. Memory is a set of selected facts intended to survive longer and often cross session boundaries. Model context is the temporary view assembled for one run. The first two can supply the third, but neither automatically or completely becomes it.

2. The workspace is a working directory, not an automatic sandbox

Each agent has a workspace, which becomes the default root for relative tool paths and bootstrap files. workspace.ts creates the directory and starter files; the runtime later reads selected files into the system prompt as Project Context.

“Working directory” must not be read as “filesystem boundary.” Without a sandbox policy, a tool can still access absolute paths outside it. Real filesystem, network, and process isolation belong to sandbox and tool policy, the subject of Part VI. A workspace establishes default ownership and collaboration material, not security by itself.

It is also distinct from OpenClaw's runtime state directory. The workspace contains agent-editable instructions, skills, and memory files. Runtime state contains configuration, credentials, and session storage. Mixing the session database into the workspace—or putting the user project into runtime state—makes permissions, backups, and migration unnecessarily ambiguous.

3. Project Context is a selected bootstrap, not a directory snapshot

The canonical bootstrap set has explicit names: AGENTS.md describes working conventions, SOUL.md carries persona and principles, IDENTITY.md identifies the agent, USER.md holds the user profile, and BOOTSTRAP.md participates during initial setup. Root MEMORY.md enters Project Context only when it already exists and the current session is not a privacy-restricted shared group or channel. The runtime does not recursively read the workspace or inject a file merely because its name looks important.

There are per-file and total bootstrap limits. Oversized files are truncated with a visible marker in the injected content; missing files also remain observable instead of being silently indistinguishable from empty text. That makes “why did the model miss the latter half of the rule?” a context-report question rather than folklore.

TOOLS.md, BOOT.md, and HEARTBEAT.md have useful roles elsewhere, but their names do not put them into the default Project Context list. Source reading should follow the actual bootstrap resolver rather than infer injection semantics from filenames.

4. The system prompt is rebuilt for every run

buildAgentSystemPrompt does not emit one installation-time template. Every run combines the current tools, skill metadata, workspace, runtime and time information, channel capabilities, sandbox state, and bootstrap content. A workspace edit therefore appears on the next construction; no old transcript entry needs to be rewritten.

This also demands a stable/dynamic split. Persona and durable conventions belong in workspace files. Current time, available tools, and sandbox state must be generated at runtime. Persisting every dynamic detail into the transcript creates stale facts; rebuilding all stable rules independently in each channel creates inconsistent ingress.

Slash directives are a preprocessing concern. Controls such as /think, /model, and /queue update runtime or session settings and are removed from the user text sent to the model. The model receives the processed request, not the entire Gateway control protocol embedded in it.

5. Skills enter as a catalog; tools carry description and schema

To control prompt size, the system prompt normally lists only each available skill's name, summary, and location. When the model determines that a capability applies, it reads the full SKILL.md on demand. Injecting every installed skill in full would make adding capabilities reduce the space available for the task and history.

Tool cost is less visible. The system prompt includes a human-readable name and description, while the provider request also includes the JSON schema that constrains callable parameters. The schema may not appear as ordinary chat text, but still consumes context. Tight descriptions and schemas without redundant nesting can save more than removing a few user sentences.

6. ContextEngine is a view-assembly protocol, not another session store

The default legacy context engine preserves established behavior: ingest and afterTurn add little, assemble lets the existing sanitize/validate/limit pipeline shape messages, and compact delegates to built-in summarization. A plugin can occupy the single plugins.slots.contextEngine slot, but that gives it the context lifecycle—not automatic ownership of the canonical transcript database.

The core lifecycle in ContextEngine reads naturally in time:

1. ingest:observe, store, or index a message as it enters the session.
2. assemble:before each model request, return ordered messages within the token budget and optionally a systemPromptAddition.
3. compact:reduce older context near the window limit or when the user invokes /compact.
4. afterTurn:update indexes or durable state and schedule maintenance after a successful run.
5. maintain (optional):perform controlled transcript rewrites through the runtime API, optionally in the background.

The output of assemble is the input view for this run; it is not authority to delete the durable record. Transcript changes must use the safe rewrite interface provided by runtime context. The contract deliberately separates choosing what the model sees from changing historical truth.

7. The hard part of pluggable assembly is the turn fence

A custom engine may mirror transcripts into a vector database and assemble a relevance-ranked view. Retry semantics are more dangerous than retrieval quality: if the first provider request fails and the same logical user turn retries, the engine must not commit it twice or alternately include and exclude it during assembly.

To durably own admitted turns, an engine declares current-turn-fence and atomic-idempotent-advancement semantics, then implements atomic commitTurn keyed by advancementKey. A repeated key must return duplicate instead of appending again. Without the complete declaration, the host conservatively falls back to the legacy path for that logical turn rather than invite double-written history.

This contract transfers to any “external memory plus agent” architecture. Relevance is only the read path. Idempotent commit, repeatable assembly, and a precise visibility boundary for the current turn determine whether the state can be trusted.

8. Transcript, memory, and runtime view have different owners

The transcript records the messages, tool calls, and results that actually occurred under one sessionId. It is the causal ledger. Memory holds selected facts and decisions meant for longer reuse. Runtime View is the temporary message sequence produced by assembly for one run; the view itself does not need to become a new source of truth after the request.

MEMORY.md contains curated long-term memory, while memory/YYYY-MM-DD.md contains daily notes. A private main session can use long-term memory as trusted context or a retrieval source. Group and other sessions must not receive the same private profile unconditionally. Retrieval is a selective bridge with an authorization boundary, not “concatenate all memory files into the prompt.”

Cross-session recall follows the same rule. It may retrieve fragments from other authorized transcripts, but it does not merge session keys or change the owner of either transcript. Otherwise “remember a relevant fact” would silently become “inherit another session's entire authority.”

9. Pruning shrinks the view; compaction persists a summary

Both mechanisms reduce model input, but their durability semantics are different. Pruning omits bulky old tool results during assembly, affecting only the in-memory prompt. The full result remains in the durable transcript for audit or future assembly under another policy. It fits reproducible tool output whose immediate value has decayed.

Compaction summarizes older conversation into one summary entry and persists that entry in the transcript while retaining recent messages verbatim. Future runs see the summary in place of the folded interval. It is therefore a lossy, durable state transition—not merely a cache.

“The model did not see it” therefore does not imply “the data was deleted.” Assembly may not have selected it, pruning may have removed it only from the view, or compaction may have irreversibly generalized the detail.

10. Memory flush rescues durable facts before lossy compression

Before automatic compaction, OpenClaw can run a silent memory-flush turn by default, prompting the agent to append important facts to memory files. The result is not a copy of the compaction summary. A summary exists to continue this session; memory selects information worth retaining for longer and retrieving elsewhere.

The flush may use its own exact model override, which does not inherit the active session's fallback chain. Write paths are constrained so a housekeeping turn cannot modify arbitrary workspace content under the pretext of remembering. If the flush is exhausted or fails, normal reply and compaction paths can degrade; user-visible status requires notifications to be enabled.

The safer practice is still to write durable decisions promptly, not bet everything on one rescue at the edge of the window. The longer organization is delayed, the easier it is for the model to miss the most important decision in the history it is about to compress.

11. Overflow is not “drop the oldest message”

Before a run, OpenClaw estimates proximity to the provider's context limit; during the run, the provider may still report overflow. The legacy engine can compact and retry. A custom engine that owns compaction must honor its own compact contract. Either path must preserve system/tool pairing, the current user turn, and enough recent history to continue coherently.

If compaction times out, the runtime must decide whether it can safely resume from a pre-compaction snapshot rather than continue after a half-written summary. Tool calls make this strict: an assistant tool call and its result cannot be cut into orphans, or the provider may reject the structure and the model may misunderstand whether an external action occurred.

Context governance is consequently more than token arithmetic. It is also transcript-schema repair, provider compatibility, retry idempotency, and side-effect ordering.

12. Inspect the built request instead of guessing from the chat page

/context list shows major contributors, /context detail expands the largest system-prompt and tool-schema entries, and /context map gives a distribution-oriented view. Reports prefer the system prompt captured from the last real embedded run; only when no run report exists do they estimate one on demand.

A reliable diagnostic order is: verify route and sessionId, confirm that the raw fact exists in the transcript, inspect Project Context truncation, check memory write and retrieval, then examine pruning, compaction, and tool-schema share. That sequence prevents every apparent memory issue from becoming an indiscriminate window increase.

/context list
/context detail
/context map
/compact Focus on decisions, constraints, and unfinished work

The focus supplied to /compact should name the structure the summary must preserve, not invite new facts. Decisions, paths, constraints, and unfinished work make useful checkpoints before and after compression.

13. Five kinds of forgetting need five repairs

14. Seven rules to carry from the context system

1. Separate durable record from model view.Successful storage does not mean complete injection on every run.
2. Give each state one source of truth.Transcript, memory, workspace, and runtime view must not impersonate one another.
3. Make bootstrap explicit and bounded.Recursive directory ingestion is expensive and unauditable.
4. Capabilities have two costs.Expand skill manuals on demand; keep callable tool schemas precise and compact.
5. Separate assembly from rewrite authority.Selecting a view is not permission to alter history arbitrarily.
6. Pruning optimizes a view; compaction migrates durable state.Their recovery and audit promises differ.
7. Extract durable facts before lossy compression.Memory flush is a last line of defense, not the only time to remember.

The next chapter changes the question from “what did the model see?” to “why can the model do these things?” We will trace tools, skills, plugins, and hooks as one capability-assembly pipeline, separating description, discovery, registration, policy filtering, and execution.

Source references

• workspace.ts and bootstrap.ts: workspace creation, loading, truncation, and Project Context injection.
• system-prompt.ts: assembling tools, skills, runtime, workspace, and bootstrap into the system prompt.
• context-engine/types.ts and registry.ts: lifecycle contract, slot, and registration.
• compaction-safeguard.ts: safeguards before compaction.
• Context, Context engine, Compaction, and Memory: official semantics and configuration.