Source Notes and Engineering Essays

Make complex systems readable by path

This blog collects source-reading notes, architecture analysis, and engineering practice. Each series starts with a concrete task, then follows the order in which the program handles messages, state, tool calls, and recovery.

Find an article by keyword →
Rememorio engineering reading path from a concrete problem and source code to execution, stored results, verification, and recovery

Index

Posts

Show

Search includes every article, beyond the first 12.

DeepSeek Harness Source Notes · Part 7When an agent can extend its runtime: from read-only Inspect to persistent plugins

Follow two read-only Inspect tools, plugin_manager, and persistent Bundles to distinguish profile writes, Host activation, and browser verification.

DeepSeek Harness Source Notes · Part 6Multiple Agents Are Not Multiple Messages

Follow spawn/fork, sendMessage, resident capacity, PTC workflows, Ralph, and Team to separate acceptance, completion, and cleanup.

DeepSeek Harness Source Notes · Part 5Compaction Does Not Delete History

Follow text pruning, image offload, and summary replacement to reconstruct requests from system messages, request/header, and cache prefixes.

DeepSeek Harness Source Notes · Part 4A Side Effect Is Not Just a Function Call

Trace tool/call, durability checkpoints, approval, sandboxing, JSONL, and crash repair to separate intent, permission, confinement, and unknown outcomes.

DeepSeek Harness Source Notes · Part 3What Model-visible means logged actually constrains

Trace four message types, stream settlement, replayable projections, and the v3 restoration boundary across model and human views.

DeepSeek Harness Source Notes · Part 2How Everything is a Plugin becomes Cordis

Read patch composition, Context, Service, isolation, Fiber lifecycle, and HMR as one replaceable runtime with ordered ownership.

DeepSeek Harness Source Notes · Part 1 From dsh web to one turn/end

Connect Profile, Bundle, Cordis, ReactLoopAgent, SessionEvent, and Web/ACP into one route before opening projection, side-effect, and recovery boundaries.

Agent Self-Evolution · Part 1 SkillOpt-Lite: When the Coding Agent Becomes the Optimizer

Find recurring failures across several runs, let the coding agent revise skill.md, and rerun the same validation before keeping the change; then inspect which HarnessOpt steps are still disconnected.

OpenClaw Source Notes · Part 8How Heartbeat, Automations, and Recovery Keep an Agent Running

Separate heartbeats, scheduled work, and background tasks; see how schedules are stored, duplicate delivery is avoided, and interrupted work resumes after a restart.

OpenClaw Source Notes · Part 7How Multi-Agent Delegation Inherits and Delivers Work

Separate configured Agents, native subagents, and ACP harnesses; see which context and tools a child receives and how its result returns to the parent task.

OpenClaw Source Notes · Part 6How tool policy, sandbox, approval, and elevated form a security boundary

See which tools are exposed, where commands run, which operations need user approval, and when elevated execution may leave the sandbox.

OpenClaw Source Notes · Part 5How tools, skills, plugins, and hooks become capabilities

See how OpenClaw discovers tools, loads Skills and plugins, filters what is available, validates arguments, and runs hooks before and after each call.

OpenClaw Source Notes · Part 4How context, memory, and compaction work together

Trace workspace bootstrap, the system prompt, and ContextEngine into transcript, memory, pruning, and compaction to see what the model really receives.

OpenClaw Source Notes · Part 3How routing, sessions, and queues assign a message

See how a message selects an Agent and session, then compare how steer, followup, collect, and interrupt queue or stop current work.

OpenClaw Source Notes · Part 2Why the Gateway is a control plane

Separate operator, node, and restricted worker entry paths, then follow handshakes, authorization, and event delivery to see why gaps require a fresh state read.

OpenClaw Source Notes · Part 1 How one message completes an agent turn

Follow one message from ChannelPlugin, sessionKey, and queueing into the embedded agent, then through tool events, completion state, and the final reply.

Model Internals · Part 5Activation Steering, Reflection Training, and Alignment Auditing

Compare ActAdd and CAA intervention positions, separating inference-time steering, reflection training, and independent audits with their validation and rollback conditions.

Model Internals · Part 4Persona, Self-Model, and Post-Training

Trace persona extraction, the Assistant axis, and one-sided capping to see how post-training stabilizes roles and where self-model evidence stops.

Model Internals · Part 3CoT Monitoring, NLA, SAEs, and Latent Reasoning

Compare CoT, NLA, SAE, and J-lens evidence, then trace continuous thought through Coconut.

Model Internals · Part 2J-space and Silent Reasoning

Separate J-lens readouts, sparse reconstruction, and causal intervention through the reference implementation.

Model Internals · Part 1 From Activation to Agent Action

Follow one file change through model computation, a proposed tool call, and runtime execution, separating activations, KV caching, and durable records.

Go from Request to Production · Part 9What Actually Changes When You Upgrade Go?

Separate toolchains, language versions, and compatibility defaults with a Go 1.22 loop experiment, then trace timers, container scheduling, GC, and upgrades.

Go from Request to Production · Part 8 Where does the bottleneck hide after connection reuse?

Trace Transport.getConn, persistConn, bodyEOFSignal, and HTTP/2 stream/flow control, then combine httptrace, metrics, pprof, execution trace, and race.

Go from Request to Production · Part 7 Why does one request land on the heap?

Trace compiler escape graphs, slice backing arrays, mallocgc, each P's mcache, concurrent GC, write barriers, and mark assists to separate allocation rate from live heap.

Go from Request to Production · Part 6 Did the goroutine stop after context cancellation?

Trace cancelCtx, causes, timers, AfterFunc, WithoutCancel, request contexts, and Server.Shutdown to separate signals, cleanup, and joining.

Go from Request to Production · Part 5 Interface, generics, and reflection boundaries

Trace the two interface words, getitab/itabInit, convT, gc shapes and dictionaries, and reflect.Value to separate three kinds of retained type information.

Go from Request to Production · Part 4 When Should We Use Channels or Locks?

Follow hchan, chansend/chanrecv, sudog, selectgo, Mutex, and runtime semaphores through communication, backpressure, and critical sections.

Go from Request to Production · Part 3 Where Does a goroutine Actually Run?

Follow go h.fetch through newproc, local/global run queues, findRunnable, work stealing, preemption, and runnable latency.

Go from Request to Production · Part 2 What Assignment Copies and What a slice Shares

Use the URL slice, result channel, and JSON collector to trace value copies, aliases, append growth, interfaces, and range variables through the spec and runtime.

Go from Request to Production · Part 1 How one request crosses net/http and the runtime

Follow a concurrent fetch request from Server.Serve, request context, and Transport into internal/poll, netpoll, and a runnable goroutine.

AI Engineering · Part 7Evals: How Do We Know the Agent Really Succeeded?

Start with one repeatable task, then add outcome, process, safety, and operations checks.

AI Engineering · Part 6Outer Loop: Let Runs Hand Off Long-Lived Work

A run ends; learn how the task preserves progress, recovers, verifies, and starts another run.

AI Engineering · Part 5Agent Loop: Help the Agent Finish a Task Step by Step

Follow repeated rounds of reading, editing, testing, and deciding inside one run.

AI Engineering · Part 4Harness Engineering: Give the Agent a Safe Place to Act

See how “run the tests” passes through tools, permission, isolation, execution, and records.

AI Engineering · Part 3Context Engineering: Prepare the Right Material for the Next Step

Distinguish the library, retrieved candidates, and the material the model can actually see now.

AI Engineering · Part 2Prompt Engineering: Make the Task Requirements Clear

Start with a vague request and add the goal, background, constraints, done criteria, and output.

AI Engineering · Part 1 How an Agent Task Actually Gets Done

Follow a payment-test task and use Codex source to separate request acceptance, run termination, and verification.

AI Video Production Systems · Part 2 OpenMontage: When a Coding Agent Runs the Video Workflow

Follow manifests, stage directors, checkpoints, render locks, and final review, separating Agent instructions, code-enforced checks, and project-file recovery from automatic execution resumption.

AI Video Production Systems · Part 1 Temporal: When an AI Video Job Must Outlive Its Process

Follow an AI music video through Workflows, Activities, Event History, replay, heartbeats, and external idempotency boundaries.

AI Video Production Systems · Part 0 From Durable Jobs to Delivered Media: Two Kinds of Recovery

Separate Temporal's job identity and event history from OpenMontage's creative stages, artifacts, and human approval before entering both source readings.

Agent Memory · Part 0 Agent Memory Series: Long-term memory is more than vector search

Compare what Mem0, Letta, Graphiti, LangMem, TencentDB, OpenViking, Cognee, and Supermemory store, when they write, and how they retrieve before entering each source reading.

Agent Framework Source Notes · Part 13 How tRPC-Agent-Go Evolution learns and rejects bad Skills

Follow online review and Skill publication separately from GEPA offline optimization and SkillCraft benchmarks, distinguishing candidate edits, validation evidence, and adoption.

Agent Framework Source Notes · Part 12 tRPC-Agent-Go Memory: from experience to searchable knowledge

Follow one environment fact through delta, extraction, reconciliation, Fact/Episode storage, and hybrid retrieval, then test quality, cost, and refusal regressions on LoCoMo-10.

Agent Framework Source Notes · Part 11 How tRPC-Agent-Go runs work and manages context

Follow one request through Runner, Agent, Model, and services, then see when Summary, Context Compaction, and Session Recall compress, rebuild, and recover original events.

Agent Framework Source Notes · Part 10 How Eino compiles Agent apps into Go execution graphs

See how Component, Runnable, and compose Graph compile into executable Go, then follow checkpointing, callbacks, ADK Runner, ChatModelAgent, and AgentTool.

Agent Framework Source Notes · Part 9 CrewAI models Agents as team workflow, not a message bus

Read Agent, Task, Crew, Process, and Flow source to see how CrewAI separates team autonomy from production control.

Agent Framework Source Notes · Part 8 From AutoGen to Microsoft Agent Framework: from multi-agent chat to production orchestration

AutoGen is now in maintenance mode. Read from Core runtime and AgentChat teams to MAF Agent, Workflow, Orchestrations, and Hosting.

Agent Framework Source Notes · Part 7 Agno turns Agents into a platform, not just objects

See how AgentOS registers Agents, Teams, and Workflows, exposes run APIs and storage, and connects approvals, RBAC, scheduling, and external interfaces to the same service.

Agent Framework Source Notes · Part 6 Why ADK Python puts Agent and Workflow together

See how one ADK 2.0 Runner executes autonomous Agents and deterministic Workflows, records progress as Session Events, and exposes long-running work through the Task API.

Agent Framework Source Notes · Part 5 Who does an Agent protocol connect?

Use one contract-renewal task to identify which endpoints MCP, A2A, AG-UI, Agent Client Protocol, and model Provider APIs connect, what they carry, and what they cannot replace.

Agent Framework Source Notes · Part 4 Where does the Agent work?

Use one scripting task to separate Coding / General from Local / Cloud, then see where workspace, sandbox, session, artifact, and memory live—and which survive sandbox deletion.

Agent Framework Source Notes · Part 3 Following AgentScope reply_stream

Expand Pi's minimal baseline by following how AgentScope emits events, records state, approves tools, and adapts OpenAI Responses differently from Chat Completions.

Agent Framework Source Notes · Part 2 Why Pi treats a Coding Agent as an extensible harness

Follow pi-ai, agent-core, and coding-agent to see how a minimal implementation runs the tool loop, stores the session tree, and compacts context while leaving plans, sub-agents, MCP, and permission policy to extensions.

Agent Framework Source Notes · Part 1 What does an Agent framework actually frame?

Use the Claude Code and Codex execution paths as a reference, then compare which components call models, execute tools, store state, and check permissions.

Agent Memory · Part 1 How Mem0's Memory Algorithm Evolved: From Mutable Memories to Graphs, ADD-only Writes, and Multi-signal Retrieval

Read how paper mem0, mem0g, and mem0 v3 move from mutable memories to ADD-only writes, entity linking, and multi-signal retrieval.

Agent Memory · Part 2 Letta / Letta Code: What Returns When an Agent Resumes

Separate archived V1 Block and AgentState recovery from the current Letta Code local backend, which compiles Git-committed memory for later turns.

Agent Memory · Part 3 Graphiti / Zep: Preserve History When Facts Change

Read how episodes, EntityEdge, valid_at / invalid_at, and hybrid retrieval keep facts current without deleting history.

Agent Memory · Part 4 LangMem / LangGraph: When to Write Memory Now or Process It Later

Read how manage_memory, BaseStore, checkpointer, and background manager split memory across execution paths.

Agent Memory · Part 5 TencentDB Agent Memory: From local memory to a team memory server

Read Context Offload, L0-L3, Gateway, Memory Hub, and Memory Proxy as team assets enter agent context.

Agent Memory · Part 6 OpenViking: When Memory, Resources, and Skills Share One Context Tree

Follow one coding-agent task through viking://, L0/L1/L2, find/search, and two-phase session commit, then see which material reaches the next model request.

Agent Memory · Part 7 Cognee / Supermemory: When Multiple Agents Share One Memory Service

Compare Cognee's add / cognify / recall pipeline with Supermemory's add / profile / search / connectors to see how multiple Agents share writes, retrieval, and user profiles.

Claude Code Source Notes · Part 1 Follow one task through the Claude Code runtime

Build the source-reading route first: CLI, REPL, queue, query loop, model calls, tool execution, transcript, and resume.

Claude Code Source Notes · Part 2 Memory is a long-lived instruction layer

Trace CLAUDE.md and memory files through cached reads, invalidation, reloads, and distinct request paths; a file write does not guarantee an immediate request update.

Hermes Agent Source Notes · Part 2 Hermes Agent: Where should information live?

Separate stable, context, volatile, and turn-only input, then decide where facts, evidence, procedures, and temporary state should be stored.

Hermes Agent Source Notes · Part 3 Hermes Agent: From Task Finalization to Background Learning

Follow finalizer, constrained background review, and independent Curator cycles to distinguish candidate proposals, applied writes, and state visible to later turns.

Hermes Agent Source Notes · Part 4 Hermes Agent: How a /learn request becomes a Skill

Trace source reading, normalization, writing, relationship updates, and later loading to see what explicit learning actually changes.

Hermes Agent Source Notes · Part 5 Hermes Agent: How train, validation, and holdout differ

Start with task records, rubrics, and data sources, then build a fair measuring stick before optimizing any Skill.

Hermes Agent Source Notes · Part 6 Hermes Agent: how DSPy and GEPA rewrite a Skill

Make a Skill editable by the optimizer, follow execution, feedback, revision, and reruns, then confirm which Self-Evolution steps are actually connected.

Hermes Agent Source Notes · Part 7 Hermes Agent: what counts as delivery after code changes

Inspect project commands, hermes verify, and opt-in evidence recording. Turn-end checks are off by default and provide bounded nudges when enabled; delivery still depends on actual results.

Claude Code Source Notes · Part 3 Context is not just a summary

Separate memory, transcript, compact, microcompact, and the exact content sent to the model instead of calling everything context.

Claude Code Source Notes · Part 4 Tools are checked before they run

Read the path from model tool_use to validated local work and paired tool_result.

Claude Code Source Notes · Part 5 Permissions are the side-effect brake

Follow allow, deny, ask, hooks, permission requests, and denied tool results through the tool path.

Claude Code Source Notes · Part 6 How commands, Skills, and MCP enter a turn

Read how slash commands, skills, plugins, MCP prompts, and MCP tools enter the current command table and tool pool.

Claude Code Source Notes · Part 7 How Claude Code isolates subagents and forked work

Compare normal subagents with fork paths, worker tool scope, cache-stable prefixes, and context isolation.

Claude Code Source Notes · Part 8 Where Claude Code hooks can change execution

Place hooks back into the turn lifecycle: prompt, tool, stop, compact, session, and subagent boundaries.

Claude Code Source Notes · Part 9 Resume rebuilds runtime state

Separate UI messages, durable transcript, API view, content replacement, and restored runtime state.

Claude Code Source Notes · Part 10 Prompt cache depends on a stable request prefix

Tie cache_control, stable prefixes, fork suffixes, microcompact, replacement state, and cache-break detection into one performance story.

Codex Source Notes · Part 1 Follow one request through the Codex runtime

Follow one request through sessions, step context, optional tool calls, client notifications, and durable recovery records.

Codex Source Notes · Part 2 How session history becomes a model request

See how history and StepContext are prepared separately, with for_prompt, WorldState updates, and compaction shaping the next model input.

Codex Source Notes · Part 3 Protocol and event stream

Distinguish starting or steering a turn, input acceptance and completion, and live notifications from filtered durable records.

Codex Source Notes · Part 4 How a tool request becomes local execution

Follow tool registration and routing into UnifiedExec, separating call results, still-running processes, and output returned to the model.

Codex Source Notes · Part 5 How Codex authorizes and sandboxes a tool

See how approval policy, centralized review, and SandboxAttempt govern execution, rejection, and controlled retries after sandbox denial.

Codex Source Notes · Part 6 How runtime events become client UI

Trace events through core, app-server, and clients, separating streamed views, approval requests, and turn/item reconstruction from rollout.

Codex Source Notes · Part 7 How skills, plugins, MCP, and subagents enter a turn

Separate skill instructions, plugin resources, MCP tool exposure, and child tasks as they enter the current step.

Codex Source Notes · Part 8 When hooks run and what they can change

Follow the separate triggers for prompt, tool, permission, compaction, stop, interrupt, and session-end hooks.

Codex Source Notes · Part 9 How Codex keeps request prefixes reusable

Separate reusable prefixes, cache accounting, and WebSocket continuation across model-specific cache contracts and Responses / Lite request shapes.

Codex Source Notes · Part 10 Rebuilding sessions and forks from rollout records

Follow JSONL writes, reverse scanning, and tail replay to understand session recovery, forks, and compatibility with old rollback records.

Codex Source Notes · Part 11 How SDKs turn external calls into threads and turns

Follow initialization, thread and turn requests into app-server, distinguishing subscriptions, accepted input, and completed work.

Codex Source Notes · Part 12 How Codex extracts and reuses lessons from old threads

Follow two background stages through separate v1/v2 memories, polluted-source removal, and selective reuse in later requests.

Codex Source Notes · Part 13 How Codex limits file writes and network access on Windows

See how restricted accounts, tokens, ACLs, network rules, and service ownership checks constrain Windows command execution.

Hermes Agent Source Notes · Part 1 Hermes Agent: how one task runs and leaves useful experience behind

Follow one message to finalizer: see how the model and tools hand work back and forth, how records are stored and recovered, and how useful lessons remain available later.