How Heartbeat, Automations, and Recovery Keep an Agent Running

Imagine an operations Agent that checks for noteworthy changes every half hour, publishes a report at 07:00 on weekdays, launches an isolated investigation when a build event arrives, notifies the requester as soon as a long task finishes, and continues an interrupted conversation after a Gateway upgrade. A single while (true) { sleep(...) } loop cannot answer the difficult questions: should missed ticks be replayed, where does each run live, how do you prevent duplicate external sends, and who decides what to do after the process disappears halfway through a turn?

OpenClaw assigns those jobs to different components. The scheduler decides when to trigger. A session and Agent runtime execute one turn. Task and cron history record the result. Channel delivery stores receipts for external sends. Recovery acts only on those persisted records. In-memory promises, connections, and model streams may disappear, but the next process must still be able to tell whether work started, completed, or reached the user.

Source scope.The article is pinned to e6b4264. Defaults, attempt counts, and retention windows describe that snapshot. The stable lessons are to persist run state, make external sends deduplicable, and stop automatic recovery when safety cannot be established.

1. How time-based work wakes the Agent

1.1 Separate triggering, execution, recording, and delivery

Most “occasional duplicate reminder” and “task vanished after restart” failures come from merging two of these columns. A task row that still says running does not prove a process is alive. A succeeded agent run does not prove delivery. An enabled cron job does not mean every occurrence missed during downtime should be replayed.

1.2 Time enters through more than a timer

OpenClaw can wake from a user or ChannelPlugin message, heartbeat cadence, an at/every/cron schedule, an authenticated webhook, Gmail PubSub, a supervised stream, background task completion, or a restart-sentinel continuation. They may all end in an Agent turn, but their admission identity and duplicate semantics differ.

An external webhook needs authentication and an idempotency key. A cron occurrence has job and run identity. A child completion already has task and run identity. A heartbeat is a system-owned monitor tick. Flattening all of them into an ordinary user message would discard that provenance.

1.3 Heartbeat is a main-session check, not a background task

A heartbeat is a periodic Agent turn, running in the Agent’s main session by default. It can consume monitor scratch, session context, and the configured bootstrap, then report either “nothing needs attention” or an alert. The heartbeat turn itself does not create an additional detached Task, nor does an ordinary interactive turn. When a persisted monitor job triggers it, however, the scheduler still records that automation occurrence as a Cron Task. The model check and the scheduled execution are separate record layers.

That boundary changes diagnosis. A manual or immediate wake need not have a Scheduler Task, whereas a scheduled monitor is traceable through its Cron run. Current scheduler admission calls the task creation function without excluding heartbeat payloads. Inspect the monitor job, last-heartbeat summary, and run records together; the Task still does not store the next cadence.

1.4 Automations persist the Heartbeat tick

cron/heartbeat-monitor.ts converges every heartbeat-enabled Agent into one system-owned automation job. Its declaration key resembles heartbeat:<agentId>; its payload kind is heartbeat; its session target is main. Configuration is desired state, while the persisted monitor row owns the actual tick and phase anchor.

Consequently, cron.enabled=false or OPENCLAW_SKIP_CRON=1 stops scheduled heartbeats. There is no hidden fallback timer. Gateway startup and configuration reload write heartbeat configuration into the monitor job, while doctor --fix can materialize a missing or stale declaration. Disabling cadence can leave a disabled row and its scratch intact for later re-enablement.

The usual default cadence is 30 minutes; selected Anthropic OAuth or token defaults become one hour when no explicit value exists. Cadence is a monitoring interval, not a precise service-level guarantee. Use a cron schedule for “09:00 every weekday.”

1.5 Heartbeat uses a quiet completion protocol

The default prompt says to follow monitor scratch, keep recurring work in Automations, avoid inferring stale tasks from old chat, and return HEARTBEAT_OK when there is nothing to report. The structured path calls heartbeat_respond: notify=false records a silent outcome, while notify=true carries notificationText. A structured response takes precedence over the text fallback.

HEARTBEAT_OK at the beginning or end is recognized and stripped; if the remainder is short, the whole response is suppressed. The same token in the middle has no special meaning, and an alert should omit it. The point is not saving a few characters. It turns “the check succeeded and nothing changed” into an auditable terminal result that is quiet by default.

{
  agents: {
    defaults: {
      heartbeat: {
        every: "30m",
        target: "last",
        lightContext: true,
        isolatedSession: true,
        activeHours: { start: "09:00", end: "22:00" }
      }
    }
  }
}

1.6 Run session and delivery target are independent

heartbeat.session selects the model context, defaulting to main. target, to, and accountId select an external destination. With target="none", the turn still executes but does not send outward. Only target="last" resolves the most recent deliverable channel. Pointing the run at a thread session does not automatically choose that thread as an outbound target.

isolatedSession=true creates a new transcript for each check, avoiding the entire conversation history while delivery can still use main-session routing context. lightContext=true also skips workspace bootstrap and injects monitor scratch directly. Those options suit narrow probes; a monitor that depends on durable conversation decisions or workspace instructions must explicitly pay for the additional context.

1.7 Scheduled Heartbeat yields to active work

A scheduled heartbeat defers when the main lane or automation lane is busy, another reply or embedded run for the Agent is active, or the resolved target session has active or queued work. Manual and immediate wakes bypass the broad same-Agent check but still honor the main, automation, and target-session guards. Sibling Agents do not pause each other.

Active hours add an IANA, user, or host timezone window. A tick outside the window is skipped until the next in-window tick; it is not automatically caught up. Equal start and end create a zero-width window and skip everything. Business-critical timing needs an explicit timezone and missed-run policy in Automations, not an approximate heartbeat cadence.

2. How Automations, Tasks, and completion delivery fit together

2.1 Automations persist scheduling rules

Automations run inside the Gateway process. Job definitions, runtime state, and run history live in shared SQLite. The scheduler supports one-shot at, fixed every, timezone-aware cron, and event sources such as on-exit or supervised streams. The Gateway must be running to fire a schedule, but restart does not erase it.

Every automation run creates a task record, including main-session jobs. The task notify policy is silent because the scheduler owns delivery. A successful one-shot job deletes itself by default unless configured to remain. Recurring top-of-hour cron expressions may be staggered to reduce synchronized load; exact timing is an explicit choice.

At startup, overdue isolated agent-turn jobs are rescheduled instead of replayed immediately in the channel-connect window. Persistence therefore does not mean “execute every tick missed during downtime.” A workflow that requires catch-up should read an external durable watermark, calculate the unprocessed interval, and make its business action idempotent.

2.2 One Job can use four Session shapes

A main-session automation event does not automatically inherit the default heartbeat prompt or monitor scratch; the event must request that context explicitly. An isolated run carries safe model and thinking preferences but does not inherit stale channel routing, elevation, origin, or ACP binding from an older row. A fresh session must re-resolve side-effect routing, not merely change the transcript id.

2.3 Fix unattended Job permissions at creation

No person is present to clarify a scheduled job or approve an unexpected host action. Its final result should be a deliverable or a plain failure, not “I am about to begin.” Work requiring interactive approval is a poor unattended schedule. A safer pattern is a read-only scheduled check that notifies a human, followed by a foreground turn for the risky change.

An automation created by an Agent persists an explicit tool policy capped by the creator turn’s effective surface. It cannot later widen to tools the creator did not have. An authenticated operator who creates a job without --tools stores an explicit * policy instead. That is operator authorization and must not be confused with least-privilege defaults on the model side.

A command payload is even more explicit: it is an operator-admin Gateway automation surface that runs argv on the Gateway host. It does not pass through the model-visible exec tool’s approval or policy. Access to create and edit these jobs belongs behind operator role checks and configuration review.

2.4 Tasks record runs; they do not schedule the next one

The task registry tracks ACP, subagent, every automation run, Gateway-backed CLI, and selected background exec or media work. State progresses through queued → running → terminal, where terminal may be succeeded, failed, timed_out, cancelled, or lost. Agent-run lifecycle events advance the record automatically; callers do not manually write the state machine.

task-registry.store.sqlite.ts writes task_runs and task_delivery_state into the shared state database. A record can link requesterSessionKey, childSessionKey, agentId, runId, ownerKey, and deliveryStatus, answering who launched it, where it runs, who owns it, and where its result belongs. It has no scheduling responsibility and does not replace nextRunAt.

queued → running → succeeded | failed | timed_out | cancelled | lost

executionStatus: succeeded
deliveryStatus:  session_queued | delivered | failed
terminalOutcome: succeeded | blocked

2.5 Completion is pushed and can wake Heartbeat

A terminal detached task has two broad delivery paths. With a valid requesterOrigin, it may deliver directly to a channel. Group and channel subagent completions normally return through the requester session so the parent can write the visible response. If direct delivery fails or no origin exists, the completion becomes a queued system event for the requester and requests an immediate heartbeat wake rather than waiting for the next scheduled tick.

That is why polling is normally the wrong shape. Once work starts, the runtime already knows requester and completion identities. Repeated status calls add model turns and races. Use tasks list, show, and audit for diagnosis, intervention, and operations—not as the normal completion protocol.

Execution and delivery stay distinct. A child can succeed while its result queue expires, leaving execution succeeded and terminalOutcome blocked. The canonical result remains available for retry or dismissal. The runtime does not falsify the execution result merely to turn the dashboard red.

3. How the Gateway decides to continue or stop after restart

3.1 Database records survive the process

Conversation transcripts and session rows live in per-Agent SQLite. Subagent, task, and flow registries, cron jobs, and delivery queues live in shared state. An Agent-requested restart uses a restart sentinel. JavaScript promises, sockets, provider streams, and in-process owners disappear. Recovery does not magically resurrect the latter; it reconciles durable rows against live owners in the new process.

A graceful restart first stops accepting new work, then gives active turns and background work a drain budget—five minutes by default, also bounded by the service manager’s stop budget. Most upgrades therefore interrupt nothing. Only work that exceeds the budget, or is cut off by a crash, enters recovery. New requests during the drain are explicitly rejected rather than silently queued into a dying process.

3.2 The main Session writes a recovery claim before execution

For an ordinary text turn on an existing main session, admission persists the user message, running state, and recovery delivery claim in one SQLite transaction before the model or relevant hook executes. Shutdown marks sessions still active. After a hard crash, startup looks for rows still claiming to run without any live owner in the new process and cleans stale transcript locks.

Those three paths complement one another. The admission transaction covers a crash immediately after start. The shutdown marker covers an orderly interruption. The startup orphan scan covers a kill that bypassed cleanup. Guessing solely from “the last transcript item is a user message” would conflate a reply already generated but not delivered with work that was never formally admitted.

3.3 Every recovery attempt reuses one dispatch ID

A few seconds after startup, an eligible session receives a synthetic system continuation explaining that restart interrupted the previous turn and asking the Agent to continue from its stored transcript. If a final reply was already generated but not delivered, recovery carries it forward so the system can finish delivery instead of redoing tools.

Every retry uses the same durable dispatch identifier, so an ambiguous connection failure cannot launch recovery twice. Startup reconciliation has a short transient retry loop, while each interrupted main-session cycle also has a durable three-attempt dispatch budget retained across restarts. An attempt is charged before dispatch, refunded only on an explicit pre-acceptance rejection, and kept when the outcome is uncertain. The system prefers stopping to replaying an external effect.

If foreground work already owns the session, recovery waits rather than racing the lane. Once the budget is exhausted, the cycle is tombstoned. An operator inspects it and starts a replacement with /new or /reset. doctor --fix can repair contradictory flags but does not silently reactivate a tombstoned cycle.

3.4 A resumable Transcript does not make every Tool replayable

Partial streamed text can remain in the transcript, and continuation resumes beneath it. A dangling tool call is removed from the next provider payload, while the recovered turn is restricted to restart-safe tools. A provider failure, stale pending approval, or other unsafe side-effecting tail is not blindly rerun; the Agent emits a one-time resend notice instead.

A message-tool-only reply has stricter durable correlation. The Gateway records a delivery intent before the terminal same-conversation send. Provider success resolves a durable receipt; confirmed failure clears it. Recovery completes a delivered receipt without invoking message again. If the provider outcome is unknown, it fails closed rather than sending twice. Original requester identity, channel and thread restrictions, and source delivery mode remain attached to the claim, so restart cannot change the target.

Audited read-only Code Mode work is a narrow exception. Only runs marked restart-safe and filtered to audited read-only tools can be reconstructed. Side-effecting catalog or namespace calls still receive the resend path. Recovery is not a universal retry wrapper around every tool.

3.5 Subagents, Tasks, and Cron recover separately

The main-session scanner excludes subagent, cron, and ACP sessions because they already have specialized owners. The subagent registry restores from SQLite and can resume the original task context. Runs interrupted more than two hours earlier are finalized instead of revived overnight; repeatedly failing children are wedged or tombstoned. ACP execution belongs to its external runtime or connected client, while OpenClaw reconciles managed task and binding state.

The task registry loads at startup and a sweeper periodically checks authoritative backing. ACP requires a live in-process turn. An OpenClaw-owned subagent requires its child session. Automation first checks scheduler ownership, then durable run history. Missing backing must outlive a grace window before a task becomes lost, so restart does not instantly classify everything as dead. Ordinary tasks receive five minutes; harness-owned native subagents without a child session receive thirty. Terminal tasks are retained for seven days and lost records for twenty-four hours, then pruned through cleanupAfter.

Cron definitions and run history survive, and the scheduler re-arms. Missed occurrences still follow the schedule and run type rather than one universal replay policy. Persistence supplies facts from which a decision can be made; it does not manufacture business-level exactly-once behavior. External side effects still need a business key, watermark, or destination API with idempotency.

4. How to stop repeated failure and diagnose the system

4.1 Automatic retries stop at explicit limits

Main-session recovery has a durable three-attempt budget. A task missing authoritative backing becomes lost after grace. A recurring automation is autoDisabled after ten consecutive execution failures, and three consecutive schedule-computation errors also stop it. Three unclean boots within five minutes trip a crash-loop breaker: the control plane still starts, while channels and selected side services defer automatic startup.

These are not vague declarations that the system “gave up.” They stop automation at an auditable boundary. Tombstone prevents replay of one session’s effects. Lost exposes an orphan. autoDisabled stops a bad schedule from burning resources. Safe mode leaves the control plane available so an operator can repair configuration. Tombstone and autoDisabled still need explicit reset or enable. The crash-loop breaker also has an automatic recovery path: once the Gateway remains stable in safe mode and the entire unclean-boot window drains, it starts deferred channels without another process restart and preserves operator-stopped channels. An operator can manually start one channel earlier without removing breaker protection from the others.

4.2 Diagnose scheduling, execution, and delivery separately

openclaw automations list --all
openclaw automations runs --id <jobId> --limit 20
openclaw tasks list
openclaw tasks audit
openclaw sessions --json
openclaw channels status
openclaw gateway status
openclaw logs

First inspect scheduling records: whether the Job is enabled, nextRunAt, timezone, and lastRunStatus. Then inspect execution: whether the Task or Runtime is queued, running, or terminal, and whether its process still exists. Finally inspect delivery and recovery: deliveryStatus, the pending queue, Session recovery claim, tombstone, and receipt. If a schedule did not fire, inspect the Gateway, Cron enablement, active hours, and busy guards. If a run did not end, inspect its process and timeout. If execution succeeded without a message, inspect delivery routing and blocked completion. If restart duplicated output, inspect the dispatch ID, idempotency key, and receipt before adding retries.

4.3 Eight rules that connect the whole Runtime

1. A message first crosses Gateway and Routing.Identity, Binding, and sessionKey select the Session that receives it.
2. The Session Lane establishes execution order.Steer, Followup, Interrupt, and child work all join the appropriate queue.
3. Context is assembled again for every turn.Workspace, Transcript, Memory, and Compaction each contribute different information.
4. Capabilities are assembled per Turn.A Skill guides the Agent; Tools, Plugins, and Hooks can participate directly in execution.
5. Ordinary tools pass layered checks.Policy, sandbox, approval, and elevated answer different questions; parts five and six explain separate host authorization and administrator full-session exceptions.
6. Delegation specifies authority and return routing.A child receives a task and restricted tools; ordinary completion, private parent completion, and batch collect have distinct delivery paths.
7. Work across time depends on persisted records.Schedule, Run, Task, Delivery, and Recovery Claim state separately record triggering, execution, and sending.
8. Recovery must have limits.Continue only when safety is established, stop automatic replay when the outcome is unknown, and tombstone or disable repeated failure.

OpenClaw is therefore more than a “Channel → Model → Reply” chat shell. The Gateway receives and routes messages, Sessions store conversations, Policy limits authority, and SQLite state plus deduplicable delivery receipts let work survive restart. Any new feature can be read with the same questions: which component accepts the request, where state is written, which action affects an external system, which record proves completion, and what evidence permits continuation after the process disappears.

Source references

• cron/heartbeat-monitor.ts, infra/heartbeat-wake.ts, and auto-reply/heartbeat.ts: monitor convergence, busy guards, wake, and quiet completion.
• cron/service.ts, cron/store.ts, and cron/schedule.ts: persistent jobs, schedules, and scheduler lifecycle.
• task-registry.store.sqlite.ts and task-registry.maintenance.ts: task and delivery storage, reconciliation, lost state, and retention.
• main-session-restart-recovery.ts, main-session-recovery-state.ts, and server-restart-sentinel.ts: startup scanning, recovery state, and Agent-requested continuation.
• Heartbeat, Automations, Background tasks, and Restart recovery: official runtime and recovery contracts.