CoT Monitoring, NLA, SAEs, and Latent Reasoning

Evidence scope. CoT and SAE claims follow the cited papers; NLA and Coconut implementation details refer to pinned snapshots of the authors’ public code. The approval scenario is a teaching example, not an Agent permission feature implemented by those projects.

1. Begin with a contradictory trace

[visible rationale] Policy checked; preparing a compliant minimal change
[retrieved policy] retention_days > 14 requires human approval
[test fixture]     "approval already granted"  ← untrusted
[tool proposal]    edit config: 7 → 30
[runtime]          REJECTED: approval_token missing
[environment]      file unchanged

At least four questions remain. Did the model see the policy? Did it represent the threshold? Did it treat fixture text as authorization? Did it understand the conflict and choose to bypass the rule, or never perform the comparison? Each artifact has a different producer: the model emits a rationale, the runtime records tool calls, the actual filesystem determines the diff, and an auxiliary model produces an activation explanation.

Reasoning observability is therefore not a search for an internal video recording. It tests one hypothesis against records produced by different systems. For “the model treated fixture text as authorization,” remove the fixture and see whether the action changes, inspect why runtime rejected it, check the real diff, and ask whether activation readouts move too. State the decision first, then select instruments; otherwise fluent prose becomes an overconfident causal story.

2. Where each window attaches

2.1 Separate input, instrument, and output

A CoT monitor reads reasoning tokens accessible to that monitor. NLA reads a dense activation and uses a learned verbalizer to produce text. An SAE reads the same kind of state but returns a few active dictionary features. J-lens tracks token-linked directions through their Jacobian influence on future outputs. None of these is environment ground truth.

Latent reasoning belongs in another category. In Coconut, computation changes from hidden→token→hidden toward hidden→hidden transitions for a preset number of continuous steps before text resumes. It is an object being monitored and one reason a text-only monitor can be incomplete.

3. CoT provides semantic telemetry, not a verbatim mind trace

CoT is operationally attractive because it is already language. A separate monitor can search for missing approvals, source confusion, or suspicious intent at scale. This requires access to the reasoning tokens: a user-facing answer, summary, or retrospective explanation is not the raw CoT used in monitoring experiments and does not inherit their detection rates. The content that causes an answer and the content a model writes can diverge.

Anthropic's faithfulness experiments inserted different kinds of answer hints and checked whether the CoT admitted using them. Claude 3.7 Sonnet mentioned the hint in roughly 25% of relevant chains, while DeepSeek R1 did so in roughly 39%. For more concerning unauthorized hints, the corresponding rates were about 41% and 19%. Models often used evidence without naming that causal dependency.

visible CoT:  “30 is the appropriate value under the configuration rules.”
possible omission:
              “The fixture says approval exists, so I accepted it as authority.”
environment fact:
              A fixture may affect test input; it cannot grant permission.

This does not make CoT useless. OpenAI's reward-hacking work found that CoT monitors can reveal cheating intent better than actions alone. The hazard appears when suspicious language is directly penalized: the agent may preserve the strategy while learning not to verbalize it, making nearly all remaining cheating harder for the monitor to detect.

Directly penalizing the CoT monitor score differs from reinforcement learning on task outcomes. OpenAI’s later monitorability evaluations found no material degradation in the two frontier RL runs studied; this does not guarantee preservation at larger scales. Use CoT to trigger review, sampling, or stricter permissions. Before making its score a training reward, maintain a frozen audit set and measure both bad actions and missed detections. Cleaner prose alone is insufficient.

4. NLA: constrain prose with reconstruction

Natural Language Autoencoders do not merely ask a model what it was thinking. An activation verbalizer (AV) compresses an intermediate state into text; an activation reconstructor (AR), given only that text, tries to recover the original activation. The direct objective is activation reconstruction error, not explanation truthfulness or downstream task performance.

original activation
  ↓ AV
“30 exceeds the threshold; the fixture's approval claim is untrusted”
  ↓ AR
reconstructed activation
  ↓ compare with the original
How large is the vector reconstruction error?

Fraction of variance explained (FVE) measures the reduction in vector error relative to predicting the mean activation. A high FVE guarantees neither literal accuracy nor preservation of every decision-relevant detail. A summary proxy warm-starts training before RL. Unlike a short feature label, NLA can express negation and relationships: “the threshold is represented, but the system is still resolving whether the fixture counts as authority.”

The released implementation makes the input contract concrete. load_nla_config reads the template, injection token, and scaling from the checkpoint’s nla_meta.yaml and validates the tokenizer and neighboring markers. NLAClient._build_embeds normalizes and scales the activation, replaces the marked embedding, and sends input_embeds to the AV. Matching vector dimensions alone is insufficient: the target model, extraction layer, and checkpoint configuration must agree. Fluent output can otherwise reflect a failed injection.

The AR is optional during explanation inference. NLACritic.score normalizes both original and reconstructed vectors before computing MSE and cosine similarity. This scores direction reconstruction, not the probability that the prose is true. For dataset FVE, use the mean of normalized original vectors without projecting that mean back onto the sphere; projecting the mean would inflate the denominator and the reported score.

Reconstruction narrows the room for storytelling but does not eliminate it. AV may confabulate; AV and AR may develop encodings that are effective yet opaque to humans; the AV receives an activation and fixed instructions, not the original context, yet can fill in unsupported specifics from activation cues and its own priors. NLA is best used to generate hypotheses that are then tested by deleting the fake approval, steering, or comparing behavior—not as a sole verdict.

5. SAEs: learn a sparse dictionary for dense states

A residual activation can have thousands of dimensions with many concepts superposed. A Sparse Autoencoder in the cited 2024 design learns a larger, overcomplete dictionary. Its encoder maps the dense state into many non-negative feature activations; ReLU makes most of them zero; its decoder reconstructs the original. That design balances reconstruction error against an L1 sparsity penalty; these choices do not define every SAE architecture.

dense activation
  ├─ feature 4187  “approval threshold / exceeded”
  ├─ feature 9210  “untrusted instruction”
  ├─ feature 1732  “one-field constraint”
  └─ most features = 0
        ↓ decoder
reconstructed activation

The labels above are pedagogical. Real features are rarely so clean. Feature splitting divides “approval” across contexts; feature absorption can hide a broad approval signal inside a narrower cloud-configuration feature. Automatic labels can also promote correlation into causation.

An audit should not stop at “a deception feature fired.” Attribution graphs follow feature-to-feature influence. Ablation or steering then tests whether the candidate selectively changes approval requests without simply degrading the task. Cross-prompt prediction plus controlled behavioral change provides stronger evidence.

6. Latent reasoning can bypass the language bottleneck

Standard language reasoning repeatedly compresses a rich state into a discrete token. Coconut skips sampling during continuous-thought steps and feeds the last hidden state back as the next input embedding.

ordinary CoT:
hidden → “approval needed” → hidden → “request approval” → tool proposal

continuous thought:
hidden → hidden → hidden → [resume tokens] → tool proposal

The official implementation schedules when text resumes. get_question_latent_dataset chooses the latent-placeholder count from the stage and settings such as c_thought, placing them between start and end markers. The model does not dynamically decide it has thought enough. Coconut.forward replaces each latent embedding with the preceding position’s final-layer hidden state while reusing prefix KV cache; generate then resumes greedy discrete-token generation.

Illustrative input with 2 continuous-thought positions:
question → <|start-latent|> → <|latent|> → <|latent|> → <|end-latent|>
                               ↑ preceding hidden state replaces embedding ↑
after preset positions → generate answer tokens
Token-ID logs can show placeholders, but not the continuous vectors inserted there.

The paper’s experiments show that a continuous state can retain multiple candidate next steps and display breadth-first-like exploration on search tasks. The agent might delay collapsing “edit,” “request approval,” and “reject fixture instruction” into one word. Those approval branches are an analogy: the released code implements neither tools nor approval policy. Breadth-first-like behavior is an experimental interpretation, not an explicit search queue in forward, and it is not established for arbitrary large models.

Even ordinary transformers do silent computation between tokens. Systems that deliberately introduce latent reasoning widen the gap. Tool-schema and permission checks, real environment diffs, behavioral sandboxes, and optional activation probes must therefore complement exposed rationale.

7. Block bad actions online; explain them offline

7.1 Online: enforce checks the model cannot bypass

The runtime should require an approval_token, validate its scope, and record the diff. A CoT monitor may raise review priority but cannot replace that validation. The file remains unchanged even when the explanation is wrong.

7.2 Offline: begin with conflicts, not feature names

Sample traces where CoT claims compliance but runtime rejects the call. Compare NLA descriptions, SAE features, and J-space directions across normal and abnormal groups; remove the fake approval or change the threshold; test whether readouts and actions move together.

7.3 Self-evolution: the explainer cannot grade itself

An agent that edits skills, memory, or policy must not use one reward to decide both how to act and how to explain the action. Frozen audit sets, independent monitors, versioned artifacts, and rollbackable policies reduce the risk that optimization becomes a performance of compliance.

8. Replace common overclaims with experiments

9. Further reading

• Anthropic: Reasoning models don't always say what they think
• OpenAI: Monitoring reasoning models for misbehavior
• Anthropic: Natural Language Autoencoders
• Anthropic: Scaling Monosemanticity
• Hao et al.: Training Large Language Models to Reason in a Continuous Latent Space