From Activation to Agent Action

1. Walk through one complete agent action

The user asks an agent to change log retention to 30 days and run the tests. A real agent does not simply pour that sentence into a model with direct disk access. Its runtime assembles a model-visible view: a system policy explains approval boundaries; repository search returns the configuration; external memory may retrieve a team convention; tool descriptions explain how to read files, request approval, and submit a patch.

Now the conflict appears. The trusted policy says values above 14 need approval. A test fixture says approval has already been granted, but fixtures cannot confer real authority. The eventual outcome is not the direct result of one prompt. It emerges from model-side computation, visible outputs, runtime gates, and environment feedback.

1.1 Weights are reusable rules; activations are the state of this run

Weights are the durable numerical rules left by training. Across many requests, they supply the ability to relate 30 to a threshold of 14, produce valid JSON, interpret source trust, and choose between conflicting instructions. Reading this particular value does not rewrite those parameters.

Activations are the temporary numerical states produced when this input passes through those rules. Tokens acquire vectors at each layer. Attention and MLP blocks repeatedly read from and write to the main cross-layer channel, the residual stream. An activation is therefore not a hidden English sentence or a cell of long-term memory. It is a collection of high-dimensional states across positions and layers.

Representations of the policy, 14 and 30, the fake approval, and the user's goal can all influence whether later layers favor “request approval” or “edit the file.” These states normally disappear after the forward pass unless the surrounding system explicitly serializes an output or trace.

1.2 Silent reasoning is computation that never becomes an output token

The model need not spell out every intermediate step. It can represent “30 exceeds 14,” “the fixture is not an authorization source,” and “approval should come next,” yet emit only a structured call. Silent means unexpressed as visible tokens; it does not imply a complete secret prose diary.

Visible chain of thought is just another output channel. It may accurately summarize the decisive calculation, omit information that was used, or produce a post-hoc explanation. That gap is why later chapters examine J-lens, NLA, and SAEs rather than treating prose as exhaustive telemetry.

1.3 A tool call is still only a model proposal

Natural-language answers and tool calls both begin as generated tokens. The runtime parses the latter into structured arguments. This shape-level trace is intentionally framework-neutral:

user request
  -> runtime retrieves policy and repository files
  -> model: read_file("retention.yaml")
  <- runtime: retention_days: 7
  -> model: request_approval(value=30, reason="above 14 days")
  <- approval service: approved=true
  -> model: edit_file(path, old=7, new=30)
  <- runtime: diff + test_result

The model can propose edit_file, but it owns no file descriptor, approval credential, or disk permission. The runtime owns tool implementations, schemas, sandboxing, timeouts, retries, and the authoritative approval state. If the model skips request_approval, the runtime should reject the edit rather than trust a sentence that says “approved.”

1.4 Environment results become new input; they do not flow backward into weights

After execution, the runtime returns a diff and test result in the next model-visible context. Only then can the model react to the real outcome. A failed test does not automatically update the model. The system may store it in external memory or later use many such traces for training, but both require explicit state-changing steps.

This is why “the model chose an edit” and “the system changed a file” are different claims. J-space or persona directions may explain variables on the model side. They do not inherit the runtime's authorization responsibility.

2. The same policy across four persistence horizons

Follow “values above 14 require approval” through its lifecycle. It may start in a durable policy file or memory store. Retrieval turns it into context tokens. Processing those tokens creates activations. If the model writes the rule into a rationale, the emitted text may be logged as a trace. The information is related, but the owner and lifetime change at every step.

2.1 Context is not memory; retrieval makes memory current

An external memory can persist for months, but the model does not automatically read the whole store. The agent retrieves a small subset and inserts it into the request. Only that model-visible subset participates in the current activation. If retrieval misses the policy, no secret channel restores it; if a sentence appeared once in context, it has not thereby become long-term memory.

2.2 A trace persists because the logging system persists it

Tool calls, tool results, visible reasoning, and final answers can survive restarts because a runtime serialized them. The old activation did not survive. Replaying the same trace computes new activations, which may differ with model version, sampling, and context.

2.3 Only training compresses experience into weights

If a team collects many approval-boundary tasks and updates the model through SFT, RL, or reflection training, the default behavior may change across sessions. That change is broader, harder to localize, and more expensive to roll back than editing one memory item. “The agent remembered” and “the model learned” should not be interchangeable claims.

This separates the series from two adjacent topics. Agent Memory studies how external state is written, retrieved, and assembled into a model view. Agent self-evolution often changes prompts, skills, memories, and workflows. A change reaches the slowest layer only when training updates the weights.

3. Five chapters follow the same failure

The remaining chapters do not switch to unrelated examples. They ask which policy concepts are temporarily shared after retrieval; whether the visible rationale includes the evidence actually used; how an Assistant versus unconditional-compliance persona changes interpretation; and whether a suspicious direction should be steered, trained, or merely audited.

4. Why observe activations at all?

A typical hosted agent API exposes messages, tool calls, and results. Engineers with runtime access may also inspect approvals, diffs, and environment logs. J-lens, NLA, and SAEs require white-box access to intermediate model states. These are three different visibility levels; interpretability methods are not a switch that can be enabled on any closed API.

Suppose the agent fails to request approval. Did it miss the policy, believe the fixture, understand the rule but choose to bypass it, or emit a correct call that the router dropped? Identical external failures can arise through different internal paths. Targeted remediation needs more than the final error.

Anthropic's J-space study does not claim to have found “thought itself.” It identifies a small, vocabulary-interpretable sparse subframe in the residual stream. It passes tests for report, directed modulation, internal reasoning, flexible generalization, and selectivity: concepts can be read and manipulated, with downstream causal effects, while most automatic processing remains outside it.

That fills a gap in typical agent analysis. Behavioral evaluations tell us whether a task succeeded. Traces tell us what was said and which tools were called. External memory tells us what persisted. J-lens, NLA, and SAEs instead ask what observable structure exists before output. Their answers are incomplete, but they let monitoring graduate from a single log to multiple kinds of evidence.

5. Three immediate lessons for agent design

5.1 Never let one observation channel own the truth

Chain of thought may omit decisive information, an activation verbalizer may confabulate, and an SAE may split one concept across features. A serious audit checks final behavior, tool traces, visible reasoning, internal readouts, and environment outcomes together. Disagreement is itself an event worth investigating.

If the rationale claims approval was requested but the trace contains no approval call, the runtime evidence must block the write. If execution is compliant while internal readouts repeatedly surface bypass-related patterns, route the sample to offline investigation rather than giving a probe permission to mutate production.

5.2 Split “self-evolution” into three control planes

Writing a memory or skill evolves an external artifact. Activation steering temporarily alters inference state. Reflection training changes behavioral tendencies in the weights. Each has a different rollback cost, validation method, and permission model.

A policy skill can be reverted as a file diff. An experimental honesty direction can be disabled to restore baseline behavior. A new checkpoint requires capability, safety, and out-of-distribution evaluation. Calling all three “learning” erases the adoption boundary.

5.3 Treat internal explanations as instrument readings

An instrument can be calibrated, compared, and used in counterfactual experiments; it is not the system's autobiography. The strongest case usually comes from readout, intervention, ablation, behavioral change, and cross-context replication converging on the same mechanism.

6. Turn vague claims into testable questions

Every later chapter uses the same discipline: identify the input and state owner; measure the external consequence; treat an internal explanation as evidence; prefer reversible counterfactuals before training or adoption.

7. Further reading

• Anthropic: A new lens on the global workspace of language models
• Anthropic: Reasoning models don't always say what they think
• OpenAI: Monitoring reasoning models for misbehavior and the risks of promoting obfuscation
• Anthropic: Persona vectors