Source Notes and Engineering Essays

Make complex systems readable by path

This blog collects source-reading notes, architecture analysis, and engineering practice. Each series starts from one concrete question, builds a map, follows the important runtime paths, and returns to a structure that can be retold.

The newest series is Model Internals: follow the path from activation to agent action through J-space, silent reasoning, personas, steering, and alignment auditing.

Index

Posts

Reading OpenClaw · Part VIIIHow heartbeat, automations, and recovery keep an Agent running

Separate heartbeat, automations, background tasks, and restart recovery; trace durable schedules, task ledgers, idempotent delivery, and bounded replay.

Reading OpenClaw · Part VIIHow multi-agent delegation inherits and delivers work

Separate configured Agents, native subagents, and ACP harnesses; trace child identity, context, tool authority, completion flow, and thread binding.

Reading OpenClaw · Part VIHow tool policy, sandbox, approval, and elevated form a security boundary

Separate capability visibility, execution environment, concrete command consent, and exec host escape, then see how trusted sender and host-local policy narrow them.

Reading OpenClaw · Part VHow tools, skills, plugins, and hooks become capabilities

Trace the candidate pool, skill snapshots, and plugin contracts through layered policy, schema projection, and typed hook checkpoints around every call.

Reading OpenClaw · Part IVHow context, memory, and compaction work together

Trace workspace bootstrap, the system prompt, and ContextEngine into transcript, memory, pruning, and compaction to see what the model really receives.

Reading OpenClaw · Part IIIHow routing, sessions, and queues assign a message

Separate bindings, dmScope, identityLinks, sessionKey/sessionId, and the steer, followup, collect, and interrupt temporal contracts.

Reading OpenClaw · Part IIWhy the Gateway is a control plane

Read connect, hello-ok, method descriptors, operator/node identity, event fanout, gap refresh, and slow-consumer boundaries.

Reading OpenClaw · Part I How one message completes an agent turn

Trace ChannelPlugin, sessionKey, queues, and the embedded agent through tool events, lifecycle termination, and ReplyPayload—the runtime coordinates for the eight-part series.

Model Internals · Five-Part Series From Activation to Agent Action

Start with J-space and silent reasoning; compare CoT, NLA, SAEs, personas, and activation steering; then reconnect the mechanisms to memory, self-evolution, and agent alignment.

Go from Request to Production · Chapter VIII Where does the bottleneck hide after connection reuse?

Trace Transport.getConn, persistConn, bodyEOFSignal, and HTTP/2 stream/flow control, then combine httptrace, metrics, pprof, execution trace, and race.

Go from Request to Production · Chapter VII Why does one request land on the heap?

Trace compiler escape graphs, slice backing arrays, mallocgc, each P's mcache, concurrent GC, write barriers, and mark assists to separate allocation rate from live heap.

Go from Request to Production · Chapter VI Did the goroutine stop after context cancellation?

Trace cancelCtx, causes, timers, AfterFunc, WithoutCancel, request contexts, and Server.Shutdown to separate signals, cleanup, and joining.

Go from Request to Production · Chapter V Interface, generics, and reflection boundaries

Trace the two interface words, getitab/itabInit, convT, gc shapes and dictionaries, and reflect.Value to separate three kinds of retained type information.

Go from Request to Production · Chapter IV When should we use channels or locks?

Follow hchan, chansend/chanrecv, sudog, selectgo, Mutex, and runtime semaphores through communication, backpressure, and critical sections.

Go from Request to Production · Chapter III Where does a goroutine actually run?

Follow go h.fetch through newproc, local/global run queues, findRunnable, work stealing, preemption, and runnable latency.

Go from Request to Production · Chapter II What assignment copies and what a slice shares

Use the URL slice, result channel, and JSON collector to trace value copies, aliases, append growth, interfaces, and range variables through the spec and runtime.

Go from Request to Production · Chapter I How one request crosses net/http and the runtime

Follow a concurrent fetch request from Server.Serve, request context, and Transport into internal/poll, netpoll, and a runnable goroutine.

AI EngineeringAI Engineering VII: Evals—How Do We Know the Agent Really Succeeded?

Start with one repeatable task, then add outcome, process, safety, and operations checks.

AI EngineeringAI Engineering VI: Outer Loop—Let Runs Hand Off Long-Lived Work

A run ends; learn how the task preserves progress, recovers, verifies, and starts another run.

AI EngineeringAI Engineering V: Agent Loop—Help the Agent Finish a Task Step by Step

Follow repeated rounds of reading, editing, testing, and deciding inside one run.

AI EngineeringAI Engineering IV: Harness Engineering—Give the Agent a Safe Place to Act

See how “run the tests” passes through tools, permission, isolation, execution, and records.

AI EngineeringAI Engineering III: Context Engineering—Prepare the Right Material for the Next Step

Distinguish the library, retrieved candidates, and the material the model can actually see now.

AI EngineeringAI Engineering II: Prompt Engineering—Make the Task Requirements Clear

Start with a vague request and add the goal, background, constraints, done criteria, and output.

AI Engineering AI Engineering I: How an Agent Task Actually Gets Done

Follow one task from request to verified result, then name the six engineering problems along the way.

AI Video Production Systems · Chapter II OpenMontage: When the Coding Agent Becomes the Video Studio Control Plane

Follow pipeline manifests, stage directors, checkpoints, ToolRegistry, render locks, and final review from prompt to a resumable, accepted production.

AI Video Production Systems · Chapter I Temporal: When an AI Video Job Must Outlive Its Process

Follow an AI music video through Workflows, Activities, Event History, replay, heartbeats, and external idempotency boundaries.

Agent Memory Agent Memory Series: Long-Term Memory Is More Than Vector Search

Put Mem0, Letta, Graphiti, LangMem, TencentDB, OpenViking, Cognee, and Supermemory on one map before reading each system in depth.

Agent Framework Agent Framework Source Notes: Choose the Runtime Owner Before the Framework Name

Use Claude Code and Codex as a reference point, establish a runtime baseline with Pi and AgentScope, separate local and cloud worksites, then read MCP, A2A, AG-UI, and ACP before comparing ADK, Agno, AutoGen, CrewAI, Eino, and tRPC-Agent-Go.

Long-term memory How Mem0's Memory Algorithm Evolved: From Mutable Memories to Graphs, ADD-only Writes, and Multi-signal Retrieval

Read how paper mem0, mem0g, and mem0 v3 move from mutable memories to ADD-only writes, entity linking, and multi-signal retrieval.

Agent Memory Letta / Letta Code: When Memory Becomes Agent State

Read how memory_blocks, AgentState, Memory.compile, and MemFS put long-term memory inside the agent's own state boundary.

Agent Memory Graphiti / Zep: Memory as a Temporal Context Graph

Read how episodes, EntityEdge, valid_at / invalid_at, and hybrid retrieval keep facts current without deleting history.

Agent Memory LangMem / LangGraph: Hot Path Tools or Background Memory Manager

Read how manage_memory, BaseStore, checkpointer, and background manager split memory across execution paths.

Agent Memory TencentDB Agent Memory: Symbolic Context Offload and Layered Long-Term Memory

Read Context Offload, refs/jsonl/MMD task canvases, and the L0-L3 pipeline that keeps tool logs recoverable.

Agent Memory OpenViking: When Memory, Resources, and Skills Share One Context Tree

Follow one coding-agent task through viking://, L0/L1/L2, find/search, two-phase session commit, and the model-view boundary.

Agent Memory Cognee / Supermemory: When Memory Becomes a Platform Boundary

Read Cognee's add / cognify / recall pipeline beside Supermemory's add / profile / search / connectors contract.

Source notes Claude Code Source Notes I: Follow One Task Through the Runtime

Build the source-reading route first: CLI, REPL, queue, query loop, model stream, tool gates, transcript, and resume.

Source notes Claude Code Source Notes II: Memory Is a Long-Lived Instruction Layer

Trace CLAUDE.md, auto memory, session memory, and user context projection into the next request and prompt-cache prefix.

Hermes Agent · 2 / 7 Hermes Agent: Why Durable Information Needs Different Owners

Separate stable, context, volatile, and turn-only input, then return facts, evidence, procedures, and temporary state to the right owner.

Hermes Agent · 3 / 7 Hermes Agent: How Learning Starts After Delivery

Follow finalizer, background review, nudge, and the two-stage Curator through the runtime path that prepares experience for future turns.

Hermes Agent · 4 / 7 Hermes Agent: How One /learn Request Becomes a Skill

Trace source reading, normalization, writing, relationship updates, and later loading to see what explicit learning actually changes.

Hermes Agent · 5 / 7 Hermes Agent: How Train, Validation, and Holdout Divide Authority

Start with task records, rubrics, and data sources, then build a fair measuring stick before optimizing any Skill.

Hermes Agent · 6 / 7 Hermes Agent: How DSPy and GEPA Rewrite a Skill

Expose a Skill as optimizable instructions, follow execution, feedback, reflection, and reruns, then audit the Self-Evolution prototype's real boundary.

Hermes Agent · 7 / 7 Hermes Agent: What Counts as Delivery After Code Changes

Connect project facts, the evidence ledger, hermes verify, and the finish gate so delivery claims follow fresh passing evidence.

Source notes Claude Code Source Notes III: Context Is Not Just a Summary

Separate memory, transcript, compact, microcompact, and model-visible projection instead of calling everything context.

Source notes Claude Code Source Notes IV: Tools Are Runtime Contracts

Read the path from model tool_use to validated local work and paired tool_result.

Source notes Claude Code Source Notes V: Permissions Are the Side-Effect Brake

Follow allow, deny, ask, hooks, permission requests, and denied tool results through the tool path.

Source notes Claude Code Source Notes VI: Commands, Skills, and MCP

Read how slash commands, skills, plugins, MCP prompts, and MCP tools enter the current command table and tool pool.

Source notes Claude Code Source Notes VII: Subagents and Forks

Compare normal subagents with fork paths, worker tool scope, cache-stable prefixes, and context isolation.

Source notes Claude Code Source Notes VIII: Hooks Are Runtime Gates

Place hooks back into the turn lifecycle: prompt, tool, stop, compact, session, and subagent boundaries.

Source notes Claude Code Source Notes IX: Resume Rebuilds Runtime State

Separate UI messages, durable transcript, API view, content replacement, and restored runtime state.

Source notes Claude Code Source Notes X: Prompt Cache Is Request-Shape Discipline

Tie cache_control, stable prefixes, fork suffixes, microcompact, replacement state, and cache-break detection into one performance story.

Source notes Codex Source Notes I: Follow One Turn Through the Governed Runtime

A source-guided route through entry, typed protocol, session, turn, context, model stream, tool authority, and evidence projection.

Source notes Codex Source Notes II: Context Is a Runtime Ledger, Not Chat History

A source-guided walkthrough of durable history, prompt projection, context diffs, compaction replacement, and rollout recovery.

Source notes Codex Source Notes III: Protocol and Event Stream

A source-guided route through TurnStartParams, Submission, Op, EventMsg, app-server v2 projection, and rollout recovery.

Source notes Codex Source Notes IV: Tools Are Runtime Contracts

Follow ToolSpec, ToolRouter, ToolRegistry, ToolInvocation, hooks, approval, sandboxing, events, and tool output recording.

Source notes Codex Source Notes V: Permission and Sandboxing

Follow turn policy, permission hooks, ToolOrchestrator, SandboxAttempt, and sandbox-denied retry to see how side effects are gated.

Source notes Codex Source Notes VI: Client Projection

Follow EventMsg, TurnItem, ThreadItem, ServerNotification, TUI history cells, rollout, and resume to see how clients observe one fact stream.

Source notes Codex Source Notes VII: Extensions and Multi-Agent

Follow skills, plugins, MCP tool exposure, tool_search, dynamic tools, and AgentControl as extra capability joins one turn.

Source notes Codex Source Notes VIII: Hooks and Lifecycle Slots

Follow HookEventName, hook discovery, prompt gates, tool gates, compaction, and Stop as hooks rejoin the turn runtime.

Source notes Codex Source Notes IX: Performance and Prompt Cache

Follow OpenAI prompt caching, stable prefixes, dynamic tails, prompt_cache_key, compaction, and token usage metrics to see how Codex shapes perceived speed.

Source notes Codex Source Notes X: Rollout and Recovery

Follow RolloutItem, the JSONL writer, InitialHistory, reverse scan, replay suffix, rollback markers, and fork snapshots to see how Codex makes a thread recoverable.

Source notes Codex Source Notes XI: SDK and app-server

Follow app-server initialization, typed ClientRequest, thread/start, turn/start, listener projection, and the Python / TypeScript SDKs to see how external clients enter Codex.

Source notes Codex Source Notes XII: Memory Layer

Follow Feature::MemoryTool, memory_mode, Phase 1 / Phase 2, the memories workspace, read path, citation, and polluted threads as one local recall layer.

Source notes Codex Source Notes XIII: Windows Sandbox

Follow the OpenAI engineering post and openai/codex source to see how sandbox users, ACLs, firewall rules, command runner, and restricted tokens turn runtime authority into local OS boundaries.

Hermes Agent · 1 / 7 Hermes Agent: How One Task Survives and Leaves Experience Behind

Follow one message through finalizer and recoverable delivery, then map Memory, SessionDB, Skills, background review, and multiple runtime entrances.