How Heartbeat, Automations, and Recovery Keep an Agent Running

Imagine an operations Agent that checks for noteworthy changes every half hour, publishes a report at 07:00 on weekdays, launches an isolated investigation when a build event arrives, notifies the requester as soon as a long task finishes, and continues an interrupted conversation after a Gateway upgrade. A single while (true) { sleep(...) } loop cannot answer the difficult questions: should missed ticks be replayed, where does each run live, how do you prevent duplicate external sends, and who decides what to do after the process disappears halfway through a turn?

OpenClaw answers by separating clock, work, ledger, and delivery. A scheduler owns timing. A session and agent runtime execute one turn. Task and cron history record what happened. Channel delivery owns receipts for external effects. Recovery reconstructs only from those durable facts. In-memory promises, sockets, and model streams may vanish; ownership evidence must not be guessed.

Evidence boundary.The article remains pinned to c549250. Defaults, attempt counts, and retention windows describe that snapshot. The durable lessons are ownership, idempotency fencing, and fail-closed recovery.

1. “Always on” contains four independent questions

Most “occasional duplicate reminder” and “task vanished after restart” failures come from merging two of these columns. A task row that still says running does not prove a process is alive. A succeeded agent run does not prove delivery. An enabled cron job does not mean every occurrence missed during downtime should be replayed.

2. Time enters through more than a timer

OpenClaw can wake from a user or ChannelPlugin message, heartbeat cadence, an at/every/cron schedule, an authenticated webhook, Gmail PubSub, a supervised stream, background task completion, or a restart-sentinel continuation. They may all end in an Agent turn, but their admission identity and duplicate semantics differ.

An external webhook needs authentication and an idempotency key. A cron occurrence has job and run identity. A child completion already has task and run identity. A heartbeat is a system-owned monitor tick. Flattening all of them into an ordinary user message would discard that provenance.

3. Heartbeat is a main-session check, not a background task

A heartbeat is a periodic Agent turn, running in the Agent’s main session by default. It can consume monitor scratch, session context, and the configured bootstrap, then report either “nothing needs attention” or an alert. It does not create a background task record. Ordinary interactive turns do not create one either. Detached ACP, subagent, automation, Gateway-backed CLI, and selected background exec or media work do.

That boundary changes diagnosis. The absence of a heartbeat row in /tasks is expected. Inspect the monitor job, last-heartbeat summary, and heartbeat logs for periodic checks. Use the task board for detached execution, not for storing the next cadence.

4. Automations own the heartbeat tick

cron/heartbeat-monitor.ts converges every heartbeat-enabled Agent into one system-owned automation job. Its declaration key resembles heartbeat:<agentId>; its payload kind is heartbeat; its session target is main. Configuration is desired state, while the persisted monitor row owns the actual tick and phase anchor.

Consequently, cron.enabled=false or OPENCLAW_SKIP_CRON=1 stops scheduled heartbeats. There is no hidden fallback timer. Gateway startup and configuration reload write heartbeat configuration into the monitor job, while doctor --fix can materialize a missing or stale declaration. Disabling cadence can leave a disabled row and its scratch intact for later re-enablement.

The usual default cadence is 30 minutes; selected Anthropic OAuth or token defaults become one hour when no explicit value exists. Cadence is a monitoring interval, not a precise service-level guarantee. Use a cron schedule for “09:00 every weekday.”

5. Heartbeat uses a quiet completion protocol

The default prompt says to follow monitor scratch, keep recurring work in Automations, avoid inferring stale tasks from old chat, and return HEARTBEAT_OK when there is nothing to report. The structured path calls heartbeat_respond: notify=false records a silent outcome, while notify=true carries notificationText. A structured response takes precedence over the text fallback.

HEARTBEAT_OK at the beginning or end is recognized and stripped; if the remainder is short, the whole response is suppressed. The same token in the middle has no special meaning, and an alert should omit it. The point is not saving a few characters. It turns “the check succeeded and nothing changed” into an auditable terminal result that is quiet by default.

{
  agents: {
    defaults: {
      heartbeat: {
        every: "30m",
        target: "last",
        lightContext: true,
        isolatedSession: true,
        activeHours: { start: "09:00", end: "22:00" }
      }
    }
  }
}

6. Run session and delivery target are independent axes

heartbeat.session selects the model context, defaulting to main. target, to, and accountId select an external destination. With target="none", the turn still executes but does not send outward. Only target="last" resolves the most recent deliverable channel. Pointing the run at a thread session does not automatically choose that thread as an outbound target.

isolatedSession=true creates a new transcript for each check, avoiding the entire conversation history while delivery can still use main-session routing context. lightContext=true also skips workspace bootstrap and injects monitor scratch directly. Those options suit narrow probes; a monitor that depends on durable conversation decisions or workspace instructions must explicitly pay for the additional context.

7. Scheduled heartbeat yields to active work

A scheduled heartbeat defers when the main lane or automation lane is busy, another reply or embedded run for the Agent is active, or the resolved target session has active or queued work. Manual and immediate wakes bypass the broad same-Agent check but still honor the main, automation, and target-session guards. Sibling Agents do not pause each other.

Active hours add an IANA, user, or host timezone window. A tick outside the window is skipped until the next in-window tick; it is not automatically caught up. Equal start and end create a zero-width window and skip everything. Business-critical timing needs an explicit timezone and missed-run policy in Automations, not an approximate heartbeat cadence.

8. Automations are the durable scheduler

Automations run inside the Gateway process. Job definitions, runtime state, and run history live in shared SQLite. The scheduler supports one-shot at, fixed every, timezone-aware cron, and event sources such as on-exit or supervised streams. The Gateway must be running to fire a schedule, but restart does not erase it.

Every automation run creates a task record, including main-session jobs. The task notify policy is silent because the scheduler owns delivery. A successful one-shot job deletes itself by default unless configured to remain. Recurring top-of-hour cron expressions may be staggered to reduce synchronized load; exact timing is an explicit choice.

At startup, overdue isolated agent-turn jobs are rescheduled instead of replayed immediately in the channel-connect window. Persistence therefore does not mean “execute every tick missed during downtime.” A workflow that requires catch-up should read an external durable watermark, calculate the unprocessed interval, and make its business action idempotent.

9. One job can use four session shapes

A main-session automation event does not automatically inherit the default heartbeat prompt or monitor scratch; the event must request that context explicitly. An isolated run carries safe model and thinking preferences but does not inherit stale channel routing, elevation, origin, or ACP binding from an older row. A fresh session must re-resolve side-effect routing, not merely change the transcript id.

10. Unattended jobs need authority fixed at creation

No person is present to clarify a scheduled job or approve an unexpected host action. Its final result should be a deliverable or a plain failure, not “I am about to begin.” Work requiring interactive approval is a poor unattended schedule. A safer pattern is a read-only scheduled check that notifies a human, followed by a foreground turn for the risky change.

An automation created by an Agent persists an explicit tool policy capped by the creator turn’s effective surface. It cannot later widen to tools the creator did not have. An authenticated operator who creates a job without --tools stores an explicit * policy instead. That is operator authorization and must not be confused with least-privilege defaults on the model side.

A command payload is even more explicit: it is an operator-admin Gateway automation surface that runs argv on the Gateway host. It does not pass through the model-visible exec tool’s approval or policy. Access to create and edit these jobs belongs behind operator role checks and configuration review.

11. Tasks are an activity ledger, not a scheduler

The task registry tracks ACP, subagent, every automation run, Gateway-backed CLI, and selected background exec or media work. State progresses through queued → running → terminal, where terminal may be succeeded, failed, timed_out, cancelled, or lost. Agent-run lifecycle events advance the record automatically; callers do not manually write the state machine.

task-registry.store.sqlite.ts writes task_runs and task_delivery_state into the shared state database. A record can link requesterSessionKey, childSessionKey, agentId, runId, ownerKey, and deliveryStatus, answering who launched it, where it runs, who owns it, and where its result belongs. It has no scheduling responsibility and does not replace nextRunAt.

queued → running → succeeded | failed | timed_out | cancelled | lost

executionStatus: succeeded
deliveryStatus:  session_queued | delivered | failed
terminalOutcome: succeeded | blocked

12. Completion is pushed and can wake heartbeat

A terminal detached task has two broad delivery paths. With a valid requesterOrigin, it may deliver directly to a channel. Group and channel subagent completions normally return through the requester session so the parent can write the visible response. If direct delivery fails or no origin exists, the completion becomes a queued system event for the requester and requests an immediate heartbeat wake rather than waiting for the next scheduled tick.

That is why polling is normally the wrong shape. Once work starts, the runtime already knows requester and completion identities. Repeated status calls add model turns and races. Use tasks list, show, and audit for diagnosis, intervention, and operations—not as the normal completion protocol.

Execution and delivery stay distinct. A child can succeed while its result queue expires, leaving execution succeeded and terminalOutcome blocked. The canonical result remains available for retry or dismissal. The runtime does not falsify the execution result merely to turn the dashboard red.

13. What actually survives a restart?

Conversation transcripts and session rows live in per-Agent SQLite. Subagent, task, and flow registries, cron jobs, and delivery queues live in shared state. An Agent-requested restart uses a restart sentinel. JavaScript promises, sockets, provider streams, and in-process owners disappear. Recovery does not magically resurrect the latter; it reconciles durable rows against live owners in the new process.

A graceful restart first stops accepting new work, then gives active turns and background work a drain budget—five minutes by default. Most upgrades therefore interrupt nothing. Only work that exceeds the budget, or is cut off by a crash, enters recovery. New requests during the drain are explicitly rejected rather than silently queued into a dying process.

14. The main session writes a recovery claim before execution

For an ordinary text turn on an existing main session, admission persists the user message, running state, and recovery delivery claim in one SQLite transaction before the model or relevant hook executes. Shutdown marks sessions still active. After a hard crash, startup looks for rows still claiming to run without any live owner in the new process and cleans stale transcript locks.

Those three paths complement one another. The admission transaction covers a crash immediately after start. The shutdown marker covers an orderly interruption. The startup orphan scan covers a kill that bypassed cleanup. Guessing solely from “the last transcript item is a user message” would conflate a reply already generated but not delivered with work that was never formally admitted.

15. Recovery reuses one dispatch identity

A few seconds after startup, an eligible session receives a synthetic system continuation explaining that restart interrupted the previous turn and asking the Agent to continue from its stored transcript. If a final reply was already generated but not delivered, recovery carries it forward so the system can finish delivery instead of redoing tools.

Every retry uses the same durable dispatch identifier, so an ambiguous connection failure cannot launch recovery twice. Startup reconciliation has a short transient retry loop, while each interrupted main-session cycle also has a durable three-attempt dispatch budget retained across restarts. An attempt is charged before dispatch, refunded only on an explicit pre-acceptance rejection, and kept when the outcome is uncertain. The system prefers stopping to replaying an external effect.

If foreground work already owns the session, recovery waits rather than racing the lane. Once the budget is exhausted, the cycle is tombstoned. An operator inspects it and starts a replacement with /new or /reset. doctor --fix can repair contradictory flags but does not silently reactivate a tombstoned cycle.

16. A resumable transcript does not make every tool replayable

Partial streamed text can remain in the transcript, and continuation resumes beneath it. A dangling tool call is removed from the next provider payload, while the recovered turn is restricted to restart-safe tools. A provider failure, stale pending approval, or other unsafe side-effecting tail is not blindly rerun; the Agent emits a one-time resend notice instead.

A message-tool-only reply has stricter durable correlation. The Gateway records a delivery intent before the terminal same-conversation send. Provider success resolves a durable receipt; confirmed failure clears it. Recovery completes a delivered receipt without invoking message again. If the provider outcome is unknown, it fails closed rather than sending twice. Original requester identity, channel and thread restrictions, and source delivery mode remain attached to the claim, so restart cannot change the target.

Audited read-only Code Mode work is a narrow exception. Only runs marked restart-safe and filtered to audited read-only tools can be reconstructed. Side-effecting catalog or namespace calls still receive the resend path. Recovery is not a universal retry wrapper around every tool.

17. Subagents, tasks, and cron recover under different owners

The main-session scanner excludes subagent, cron, and ACP sessions because they already have specialized owners. The subagent registry restores from SQLite and can resume the original task context. Runs interrupted more than two hours earlier are finalized instead of revived overnight; repeatedly failing children are wedged or tombstoned. ACP execution belongs to its external runtime or connected client, while OpenClaw reconciles managed task and binding state.

The task registry loads at startup and a sweeper periodically checks authoritative backing. ACP requires a live in-process turn. A subagent requires its child session. Automation first checks scheduler ownership, then durable run history. Missing backing must outlive a grace window before a task becomes lost, so restart does not instantly classify everything as dead. Terminal tasks are normally retained for seven days, lost records for a shorter window, then pruned through cleanupAfter.

Cron definitions and run history survive, and the scheduler re-arms. Missed occurrences still follow the schedule and run type rather than one universal replay policy. Persistence supplies facts from which a decision can be made; it does not manufacture business-level exactly-once behavior. External side effects still need a business key, watermark, or destination API with idempotency.

18. Bounded failure keeps “always on” from becoming “retry forever”

Main-session recovery has a durable three-attempt budget. A task missing authoritative backing becomes lost after grace. A recurring automation is autoDisabled after ten consecutive execution failures, and three consecutive schedule-computation errors also stop it. Three unclean boots within five minutes trip a crash-loop breaker: the control plane still starts, while channels and selected side services defer automatic startup.

These are not vague declarations that the system “gave up.” They stop automation at an auditable boundary. Tombstone prevents replay of one session’s effects. Lost exposes an orphan. autoDisabled stops a bad schedule from burning resources. Safe mode leaves the control plane available so an operator can repair configuration. After fixing the cause, the operator explicitly enables, resets, retries, or starts the affected component.

19. Diagnose always-on work with three ledgers

openclaw automations list --all
openclaw automations runs --id <jobId> --limit 20
openclaw tasks list
openclaw tasks audit
openclaw sessions --json
openclaw channels status
openclaw gateway status
openclaw logs

The first ledger is schedule: enabled state, nextRunAt, timezone, and lastRunStatus. The second is execution: queued/running/terminal task state and whether its live owner exists. The third is delivery and recovery: deliveryStatus, pending queue, session recovery claim, tombstone, and receipt. If a schedule did not fire, inspect Gateway, cron enablement, active hours, and busy guards. If a run did not end, inspect runtime ownership and timeout. If it succeeded without a message, inspect delivery routing and blocked completion. If restart duplicates output, inspect dispatch, idempotency, and receipts before adding retries.

20. The eight-part runtime model

1. A message first crosses Gateway and routing.Identity, binding, and sessionKey establish ownership.
2. The session lane establishes time order.Steer, followup, interrupt, and child trees still obey ownership.
3. Context is a projection for one turn.Workspace, transcript, memory, and compaction have distinct durability.
4. Capabilities are assembled per turn.A skill guides; tools, plugins, and hooks enter the executable surface.
5. Security is an intersection.Policy, sandbox, approval, and elevated answer different questions.
6. Delegation only narrows authority.A child receives a task contract; the parent keeps final delivery.
7. Always-on behavior needs durable owners.Schedule, run, task, delivery, and recovery claims each own facts.
8. Recovery must be bounded.Resume only when safe, fail closed when outcome is unknown, and tombstone or disable repeated failure.

OpenClaw is therefore more than a “Channel → Model → Reply” chat shell. It is an Agent runtime with Gateway as its control plane, sessions as conversation ownership, policy as an authority ceiling, and SQLite state plus idempotent receipts as the bridge across time. Any new feature can be read with the same questions: who owns ingress, who owns state, who may create a side effect, who proves completion, and what evidence permits continuation after the process is gone?

Source references

• cron/heartbeat-monitor.ts, infra/heartbeat-wake.ts, and auto-reply/heartbeat.ts: monitor convergence, busy guards, wake, and quiet completion.
• cron/service.ts, cron/store.ts, and cron/schedule.ts: persistent jobs, schedules, and scheduler lifecycle.
• task-registry.store.sqlite.ts and task-registry.maintenance.ts: task and delivery storage, reconciliation, lost state, and retention.
• main-session-restart-recovery.ts, main-session-recovery-state.ts, and server-restart-sentinel.ts: startup scanning, recovery state, and Agent-requested continuation.
• Heartbeat, Automations, Background tasks, and Restart recovery: official runtime and recovery contracts.