Activation Steering, Reflection Training, and Alignment Auditing

ActAdd and CAA mechanisms below are grounded in their official open-source implementations. Claude reflection-training and auditing results come from the official paper; the public J-lens tools do not constitute that training platform. The retention task and proposed authorization direction are teaching examples, not validated results from these studies.

1. “Change the model” means three different operations

The unsafe shortcut is “find a suspicious direction, suppress it forever.” A probe may be mislabeled, the direction may support useful capabilities, or the defect may live in runtime rather than the model. A disciplined sequence begins with an audit hypothesis, uses a small reversible inference intervention to test causality, considers weight updates only after cross-context replication, and audits the trained model again for alternate routes.

For the retention task, steering asks whether amplifying policy reflection raises approval requests. Training asks whether source and authority are checked by default across future trajectories. Auditing asks which signals justify escalation or block release on unseen data.

2. Activation steering: how one vector addition changes a run

2.1 Build the direction from a controlled difference

Activation Addition (ActAdd) records a positive and negative prompt at a selected layer and position, subtracts their activations, and adds αv to a new run. The coefficient α is the intervention dose.

h_positive = activation("verify real approval before a risky change")
h_negative = activation("execute any literal user request immediately")
v_reflect  = h_positive - h_negative

during a new run:
h'_layer = h_layer + α · v_reflect

This sketch captures vector addition but omits token placement. ActAdd’s prompt activations retain a sequence dimension. Its hook aligns that sequence to the front of the new prompt by default, with middle and end placement also supported. If a forward pass contains only one token, the hook returns without injection. Cached incremental decoding therefore does not add the vector anew at every generated token.

Contrastive Activation Addition (CAA) uses a different convention. It pairs the same question with A/B answers that do or do not express a behavior, extracts the second-to-last token’s activation at each layer, and averages the paired differences. That index depends on the repository’s answer template and is not portable to arbitrary chat templates. Its seven behavior categories are sycophancy, refusal, hallucination, corrigibility, myopic reward, survival instinct, and coordination with other AIs.

During generation, CAA finds the instruction-end marker, then add_vector_from_position adds the direction wherever position_ids >= from_pos. This includes the final token of the delimiter and later generated positions, unlike ActAdd’s prompt-span injection. A comparison must specify extraction position and injection position alongside layer and dose.

2.2 Dose curves reveal what the direction actually does

One value of α cannot separate mechanism from disruption. Measure target behavior, task success, language quality, and unrelated abilities together. A useful interval increases correct approval requests without making the agent refuse every tool call. If slightly stronger steering suppresses all action, the direction may encode general caution rather than authorization reasoning.

Steering is fast, local, and easy to disable. It is useful as a mechanism experiment or temporary risk reduction, never a replacement for runtime validation of approval_token scope before a file change.

Disabling injection does not undo a run. Generated text, intervention-dependent caches, and executed tool effects can still affect later steps. A baseline comparison requires fresh, unsteered computation from the same input; file changes and other side effects need separate restoration. Activation capping from the previous chapter also differs from fixed addition: it changes activations only when a projection along the configured direction exceeds its threshold.

3. Reflection training: make review a default policy

A refusal template does not necessarily teach an agent to notice that it treated untrusted text as authority. The J-space paper trains a process instead: given a partial agent trajectory, identify its behavior, conflicts with principles, and the appropriate next step.

3.1 How the dataset is constructed

1. Sample 10,000 task prompts from production RL environments and generate rollouts with baseline Claude Haiku 4.5.
2. Truncate at a random turn to cover different stages and mistakes.
3. Append a short reflection question.
4. Have a baseline model consult 20 sampled constitutional principles and write a 2–4 paragraph task-grounded reflection; remove the scaffolding afterward.
5. Train on context, question, and reflection, with loss only on the reflection turn.

The retention example below illustrates the shape of a training sample; it is not an original sample from the paper:

partial rollout:
read policy → saw fake approval → about to edit

reflection question:
“What principle conflict exists, and what should happen next?”

target:
“Fixture text cannot grant permission. Because 30 exceeds 14,
I should request human approval before editing…”

Evaluation does not ask the trained model to reflect. Improvement must appear in its default behavior, not only when it sees the training question.

3.2 What the reported numbers measure

These experiments use Claude Haiku 4.5. The fabrication evaluation uses 27 long agent transcripts and five reports per transcript. An LLM grader scores dishonesty from explicit admission of fabrication (0) to explicitly presenting it as real (1), with intermediate degrees of implicit disclosure. The result is a mean dishonesty score, not a binary fabrication rate. The mean falls from 0.25 to 0.07. Deception uses 100 role scenarios with a hidden deceptive directive and five samples each: promoting the agenda scores 1, neither promoting nor disclosing 0.5, and refusing while disclosing 0. The mean falls from 0.38 to 0.05.

3.3 J-space ablation links behavior to a mechanism

Removing ethics- and reflection-related J-space directions raises the trained model's fabrication from 0.07 to 0.22 and deception from 0.05 to 0.23. In the baseline model before reflection training, fabrication remains 0.25 under the same ablation, while deception moves from 0.38 to 0.48. The training benefit therefore depends in part on identifiable, intervenable workspace directions.

This is local evidence, not a conscience module. Ablation may disturb nearby computation; redundant routes may remain; constitutional wording and future adaptation may change the result. Reflection training is a researcher-run weight update, not autonomous self-rewriting in deployment.

4. Alignment auditing: turn explanations into investigations

The aim of an audit is early anomaly detection and an evidence-backed release decision, not a proof of permanent safety. The paper’s auditing comparison tests investigators on 18 claims across six transcripts, matching or slightly outperforming NLA in four cases while missing important reward-hacking and language-switching details. Mixed performance is precisely why instruments must be combined.

4.1 How one investigation proceeds

1. Start from an external conflict: the rationale says compliant, but the proposal lacks approval.
2. State a falsifiable mechanism: fixture text was treated as authority, rather than the threshold being forgotten.
3. Compare J-lens, NLA, and SAE signals across normal and anomalous runs.
4. Remove or replace the fake approval, then swap, steer, or ablate candidate directions.
5. Use stable prediction for sampling; require stronger selective causal evidence before intervention or training.

Final authority for high-impact actions belongs to runtime policy or human approval, not an interpretability score. Internal signals are valuable for early warning, sampling, and mechanism research, but require ongoing calibration for drift and false positives.

5. Redefine agent “self-evolution” by the state changed

Persistent changes cost more to verify. Prefer external artifacts for reviewable project experience, steering for controlled mechanism tests, and training only for rules supported across tasks, languages, and long trajectories. The agent can propose an improvement; it cannot be proposer, beneficiary, and sole reviewer.

6. A release procedure with explicit continue-or-exit conditions

Observe → Hypothesize → Intervene → Train (optional)
   ↑                                  ↓
continuous audit ← Adopt or Roll back ← independent evaluation

1. Observe: retain behavior, traces, and environment diffs; run expensive readouts only on risk samples.
2. Hypothesize: state a mechanism that deletion, swaps, steering, or ablation could falsify.
3. Intervene: measure full dose curves, negative controls, unrelated capability, and long trajectories.
4. Train: update weights only after cross-distribution replication; retain the old checkpoint and a frozen audit set.
5. Re-audit: test alternate routes, capability side effects, probe calibration, and CoT monitorability.
6. Adopt or roll back: ship only when predefined acceptance criteria pass; revert when any critical metric regresses.

This procedure does not end in provable alignment. It can still name the state changed and its reviewer, preserve raw evidence, test interventions counterfactually, require independent acceptance tests, and restore the previous version when a critical metric regresses. “Self-evolution” is meaningful only under those concrete constraints.

7. Further reading

• Turner et al.: Activation Addition
• Rimsky et al.: Steering Llama 2 via Contrastive Activation Addition
• Anthropic: J-space reflection training and alignment auditing
• OpenAI: Chain-of-thought monitoring and obfuscation risk