Temporal: When an AI Video Job Must Outlive Its Process

Consider a concrete product request. A user supplies a song and a visual direction. The system writes a treatment, splits it into eight shots, submits those shots to video models or GPU workers, retries failures, normalizes color, adds subtitles, composes the final cut, uploads it, and notifies the user. The path may take twenty minutes, or hours once quotas, queues, and human review enter the picture.

1. First, watch an ordinary backend lose the music video

The most direct implementation starts a goroutine after the business API receives an HTTP request: call the LLM, submit eight shots, poll for results, and compose the final cut. As long as the request and process remain alive, that code can work. The trouble begins halfway through.

• After a browser refresh, the generation may still be running, but the user no longer knows where to find it.
• After a pod restart, the current shot, retry count, and waiting state disappear with process memory.
• If the video provider accepted a request but its response vanished, the backend sees only a timeout; submitting again may pay for a duplicate job.

These failures look like separate frontend, process, and network problems. They share one gap: progress exists only inside this request and process. More error branches can handle failures the function observes; they cannot give state back to a process that no longer exists.

2. Temporal in plain language: the job can live without a living process

One-sentence definition: Temporal is a workflow platform that records long-running job progress outside the process so a new process can continue it. It does not generate video, and it is not another agent framework. It remembers which job this is, what has finished, what it is waiting for, and what should happen next.

Start with only four building blocks:

• Job card (Workflow Execution): the durable identity of one concrete job—here, “generate this music video for user 42,” not an abstract code definition.
• Fact log (Event History): the job's ordered record of what started, what was scheduled, and which step completed.
• Flow rules (Workflow): the decisions about what comes next, such as “compose only after all eight shots finish.” They read history but do not call the outside world directly.
• Real action (Activity): a call to the LLM, video provider, object store, or notification API whose result goes back to the job card.

Put those blocks around one concrete object and the following teaching record emerges. It is not a Temporal SDK request; it is the unit we will keep carrying through the article:

Job identity: mv-20260710-42
Raw input: song://summer.wav + “neon highway, eight shots”
Flow rules: treatment -> parallel shots -> acceptance -> compose -> notify
Confirmed fact: treatment at object://mv-42/treatment.json
Waiting on: provider job vid-8831 for scene-03
Final artifact: not produced yet
User-visible state: 6 / 8 shots complete

The original audio, treatment, and video bytes belong in object storage. Identity and progression facts belong in Temporal. The UI derives “6 / 8” from Workflow state. These locations should not collapse into one giant state object: large artifacts do not belong in history, and UI state must not become the authority for completion.

Business-hosted Worker processes run Workflow and Activity code. A Worker is not a fifth piece of durable state; it is a replaceable executor. A useful memory aid is: Temporal Service keeps the job card and fact log; a Worker is only a temporary person doing the work. Durable does not mean that one goroutine lives forever. It means the goroutine may disappear while the job identity and completed facts remain. That is the persistence described by the official Workflow Execution overview.

2.1 Follow the same music video from start to finish

1. Create the job: the business API starts a Workflow Execution with a stable business ID, then it can return the HTTP response.
2. Record the starting point: Temporal writes “this music video started” before handing a small decision task to a Worker.
3. Decide the next step: the Worker runs the Workflow; seeing no treatment yet, the Workflow schedules a “write treatment” Activity.
4. Touch the real world: an Activity Worker runs the Activity and calls the LLM or video provider; the returned result is recorded in history.
5. Wait and hand off: separating submission from waiting leaves no Activity Worker occupied during remote generation. When a callback, timer, or poll result arrives, a compatible Worker polling the right queue can continue.
6. Finish and observe: composition writes the video to object storage, a notification Activity sends the message, and a reopened browser queries the same execution ID.

Those six steps are the main route. Task Queues, replay, Signals, heartbeats, and idempotency keys are protections added when Workers can disappear and external calls can time out.

2.2 Three overloaded uses of “frontend” and “worker”

With the main route established, the remaining name collision is easier to remove. The browser frontend is the user interface. Temporal's Frontend Service receives cluster API calls. A Temporal Worker is a user-hosted process that runs Workflow and Activity code. That worker is also different from a child agent that an agent framework may call a worker.

3. How a start request becomes a recoverable workflow

Now map the job card to implementation. The start request does not send that whole teaching record verbatim. It answers three ordinary questions: which business job is this, which workflow rules govern it, and which group of Workers can handle it. The SDK expresses those answers as Workflow ID, Workflow Type, Task Queue, and input before calling StartWorkflowExecution. The repository README is explicit that this repository contains the Temporal Server; language SDKs implement Workflows, Activities, and Workers. Temporal is a platform plus SDKs, not an imported library that silently takes over arbitrary goroutines.

3.1 Frontend creates an execution instead of holding a connection

The service entry point WorkflowHandler.StartWorkflowExecution prepares the request, resolves the namespace, and delegates creation to History Service. Its comment says that the call creates a WorkflowExecutionStarted event and schedules the first Workflow Task. Once Start succeeds, a connected browser is no longer a condition for execution.

Business identity differs from one run: Workflow ID and Run ID identify the business flow and a particular run respectively. Within a Namespace, only one execution with a given Workflow ID can be open at once; Continue-As-New or a Workflow Retry creates a new Run ID. For duplicate start requests, Conflict Policy handles an open execution, while Reuse Policy and retention constrain reuse after closure. A stable business ID matters, but it is not a permanent deduplication table.

3.2 History records facts before Workers advance them

History Service maintains mutable state and an append-only Event History for each Workflow Execution. The architecture document states the central design directly: complete workflow state can be rebuilt through replay; Workflow code must be deterministic and side-effect free, while Activity code must be idempotent or explicitly non-retryable. See docs/architecture/README.md.

Let the same music video advance several steps. Fields and intermediate events that do not affect this explanation are simplified, but the order among actors, Commands, and accepted events is the important part:

Client -> Temporal Service
  StartWorkflowExecution(workflow_id=mv-20260710-42, input_ref=object://mv-42/input.json)
Temporal Service -> Event History
  WorkflowExecutionStarted

Workflow Worker -> Temporal Service
  Command: ScheduleActivity(write-treatment)
Temporal Service -> Event History
  ActivityTaskScheduled
Activity Worker -> Temporal Service
  RespondActivityTaskCompleted(result_ref=object://mv-42/treatment.json)
Temporal Service -> Event History
  ActivityTaskCompleted(result_ref=object://mv-42/treatment.json)

Workflow Worker -> Temporal Service
  Command: ScheduleActivity(submit-scene-03)
Activity Worker -> Temporal Service
  RespondActivityTaskCompleted(provider_job_id=vid-8831)
Temporal Service -> Event History
  ActivityTaskCompleted(provider_job_id=vid-8831)
Webhook API -> Temporal Service
  SignalWorkflowExecution(scene-03-ready, artifact_ref=object://mv-42/scene-03.mp4)
Temporal Service -> Event History
  WorkflowExecutionSignaled(scene-03-ready, artifact_ref=object://mv-42/scene-03.mp4)

API calls and history events come from different actors. An Activity Worker reports completion; the server's RespondActivityTaskCompleted handler appends the completion event and requests another Workflow Task. Completion of a submission Activity means it obtained a provider job ID, not that the video exists. Likewise, recording a Signal in history acknowledges delivery; processing the message, validating the artifact, and advancing Workflow state happen afterward.

A Worker takes the Workflow Task, replays the Workflow function until it waits on an Activity, Timer, Signal, or Child Workflow, and returns Commands describing what should happen next. Matching's PollWorkflowTaskQueue shows how replaceable Workers poll Matching and receive queued work.

The easy-to-miss distinction is: a Worker proposes Commands; Temporal Service accepts state transitions and appends facts. A proposal that never reached the Service may disappear with the Worker. A fact already in Event History becomes input to the next Worker.

4. Workflows decide the next step; Activities touch the world

Why not call the LLM, video provider, storage service, and notification API directly inside the Workflow function? Recovery depends on replay. The same Workflow code may run many times and must emit the same Commands for the same history. Reading wall-clock time, choosing random values, or making a direct provider call would make that path drift. Temporal SDKs therefore push non-determinism and side effects into Activities.

4.1 Where each music-video step should run

Temporal's own Activity overview names media transcoding, LLM calls, and large downloads as representative use cases. It recommends splitting larger functionality into smaller Activities for tighter recovery, timeouts, and idempotency. See Temporal Activities.

4.2 Waiting is not a sleeping goroutine

After the video provider returns a job ID, a Workflow may wait on a Signal, Timer, or Activity Future. The wait is reconstructible from server state and needs no pinned application thread. A webhook can Signal the Workflow; a user style change or cancellation can alter later control flow through Signals, Updates, or cancellation requests. The documentation explains what Queries can read and how Signals and Updates can change a Workflow: Workflow Message Passing.

5. Recovery starts after the last recorded fact

Event History does not serialize a Worker stack. It records state transitions accepted by the Service. A new Worker starts the Workflow function from the beginning; the SDK supplies recorded Activity results, Timers, and Signals and checks generated Commands against history. Completed Activity results are reused; running steps and pending retries continue according to their own recorded state.

Return to the eight parallel shots: six are complete, scene-03 is queued at the provider, and scene-08 awaits retry. A new Worker neither resubmits all eight nor resumes only one “next step.” It reuses six results, keeps waiting for scene-03, and lets scene-08 follow its retry policy; composition starts after every shot passes acceptance. Losing a Workflow Worker alone does not require recreating an Activity that was already scheduled and is still pending.

5.1 Browser, Workflow Worker, and Activity Worker failures differ

For a long Activity, heartbeats provide liveness, cancellation delivery, and application-level progress checkpoints. A compositor processing 8,000 frames may heartbeat its latest completed chunk so the next attempt can continue after a Worker loss. The official failure guide explains Start-To-Close timeout and heartbeat payload recovery: Detecting Activity Failures. The server entry point documents the same liveness/progress split in RecordActivityTaskHeartbeat.

A heartbeat is a recovery hint, not a promise that every frame is durable. SDKs may coalesce reports, so a later attempt can read only details the Service actually received. The compositor must still safely repeat its last chunk. Record live UI progress separately; a heartbeat is not a Workflow business-completion event.

5.2 Processes may change; the Workflow semantics behind old executions cannot change arbitrarily

“Any Worker can take over” has one more condition: its Workflow code must still be able to interpret existing history. Suppose old history says “compose after a shot becomes ready,” while a new build unconditionally inserts a moderation Activity at that point. During replay, the new Command sequence may no longer match history. Process replacement solves machine failure; it does not automatically solve an incompatible code rollout.

Old execution history: scene-ready -> compose
Unsafe direct replacement: scene-ready -> moderate -> compose

A safe rollout must answer:
1. Does compatible code keep handling old executions, or are they pinned by version routing?
2. Is the new branch introduced through the SDK's patch / version mechanism?
3. Can real Event Histories pass replay tests before rollout?
4. How do old and new executions roll back if the release fails?

Temporal's current Worker Deployments guide recommends Worker Versioning as the default for safely deploying new Workflow code and keeps patching as a compatibility path when versioned deployments are unavailable. For AI jobs that span tens of minutes or hours, this is part of durable execution across releases, not a deployment footnote.

6. The dangerous gap: the provider succeeded, but the reply vanished

A common misreading is that Temporal makes every step exactly once. Workflow control can produce an effectively-once execution effect through history, but an Activity still enters an uncertain interaction with an external system.

1. submit_scene(scene-03) creates a paid provider job.
2. The Worker or network fails before receiving the job ID.
3. Temporal only knows that the Activity did not complete and retries it.
4. Without a stable external identity, the second request may create another job.

6.1 Bind idempotency to the logical action

A practical design derives a stable key from workflowId + sceneId + operation and asks the provider, or an internal submission table, to deduplicate on it. Persist the returned provider job ID as the Activity result and perform later waits and reads against that ID. Without provider idempotency, a local submission record narrows the window but cannot erase the “remote success before local record” ambiguity on its own.

{
  "workflow_id": "mv-20260710-42",
  "scene_id": "scene-03",
  "operation": "submit-video",
  "idempotency_key": "mv-20260710-42:scene-03:submit-video"
}

6.2 Cancellation is a cooperation protocol, not remote kill

Workflow cancellation can stop unscheduled work and propagate cancellation to running Activities. The remote long-running Activities used here receive server cancellation through heartbeats; business code must then observe its context and exit. Checking a local context alone does not replace heartbeating, and heartbeat throttling may delay delivery. The provider must also expose a cancellation API before business code can stop the remote GPU job. “Canceled in Temporal” therefore does not automatically mean “all external compute stopped instantly.”

6.3 Provider success, Workflow completion, and product delivery are three states

Another common mistake is to treat the provider's succeeded status as product completion. A callback proves only that the provider claims the job ended. Business code still needs to verify that the object is readable, format and duration are valid, every requested shot exists, cancellation did not win the race, and the accepted artifact reference was recorded in the Workflow. Only then should the product mark the music video downloadable or publishable and decide when to notify the user.

provider produced
-> Activity verifies artifact and cost
-> Temporal records artifact_ref
-> Workflow satisfies every completion condition
-> product marks the music video deliverable
-> notification is sent idempotently

if any acceptance gate fails
-> do not claim delivery
-> retry, compensate, or route to human review

This separates “an external file exists,” “the flow has completion evidence,” and “the product authorizes delivery.” Temporal can preserve and recover these checks, but business code still defines quality rules, budget limits, and final publication authority.

7. tRPC-Agent-Go runs the Agent; Temporal lets the job outlive a Worker

Temporal is not another GraphAgent. tRPC-Agent-Go runs models, tools, subagents, event streams, and graph nodes. Temporal preserves long-running identity, state, timers, retries, messages, and recovery after a process disappears.

7.1 The durable-looking pieces already in tRPC-Agent-Go

First separate what is actually connected in the fixed snapshot. agent/taskrun defines a replaceable Controller interface, while the bundled implementation used by the examples is inprocess.Service. Products must implement a distributed durable controller separately; the interface alone does not make the main path distributed. Graph checkpoints and detached cancellation are implemented, but each handles only the cases below.

The source makes these conclusions explicit. WithDetachedCancel changes parent-context cancellation propagation only. Dynamic Workflow still describes execution as foreground and one-shot; the Graph documentation presents checkpoints as the basis for explicit recovery and time travel; taskrun says a multi-node controller needs external storage, queues, leases, and cross-node cancellation; and the bundled FileStore's normalizeLoadedRuns marks unfinished runs as interrupted by a prior runtime restart instead of resuming them.

For example, the current SandboxRunner.ExecuteWorkflow starts a fresh isolated Python guest for each call and passes an optional timeout context to the guest and host callbacks. This improves generated-code isolation without turning Python control flow into replayable history. Persisted child Agent events in a Session do not let a restarted process resume at the next Python line; external side effects already performed do not roll back automatically either. Treat script isolation, conversation persistence, and execution recovery as separate capabilities when deciding which layer remains missing.

7.2 Coarse and fine-grained integrations

A coarse integration runs an entire runner.Run as one Activity. It is easy to adopt, but a mid-run Worker crash may repeat the whole invocation, so LLM and tool effects need strong idempotency. A fine-grained integration models LLM calls, tool calls, video submission, and human waits as Activities or Child Workflows. It provides better recovery points and visibility, but turns the Agent loop into a durable state machine and costs much more to build.

Put the same scene-03 through both designs to see exactly which calls repeat after a Worker crash. This is a design comparison, not a Temporal adapter already bundled with tRPC-Agent-Go:

The safer division of responsibilities is not to rebuild Temporal inside the Agent runtime. Keep a replaceable durable-controller interface: use in-process taskrun for simple deployments and attach Temporal, a cloud state machine, or an internal scheduler when cross-node recovery becomes a product requirement.

8. The industry has no single answer; choose by recovery pressure

Temporal is a mature durable-execution platform descended from Uber Cadence, but a long job does not automatically require Temporal. A single idempotent async video API may need only a database job row and queue. Kubernetes GPU DAG pressure may favor Argo Workflows. An AWS-native organization may prefer Step Functions to avoid operating a separate workflow service.

Useful primary comparisons include Restate, DBOS, Inngest, AWS Step Functions, and Azure Durable Functions. They agree on the pressure—state, checkpoints, retries, recovery—and differ in programming model, deployment model, and who operates the engine.

9. Compress the music video into transferable rules

“Building a flow inside a Worker” and “letting the flow outlive the Worker” are different jobs. The first decides how models, tools, and subagents run while the process is alive. The second decides who can prove what already happened and what should happen next after that process disappears. Temporal's value is precisely this move: a short-lived control flow becomes an execution identity driven by durable history and recoverable by a new Worker.

A reliable AI video system still combines several components. A durable orchestrator preserves job identity and schedules recovery; a GPU or Kubernetes platform schedules compute; object storage keeps large artifacts; an Agent runtime performs reasoning; provider deduplication and submission records prevent duplicate external work. Once every action has an executor and every record has a storage location, “the frontend died but the backend is still running” becomes testable system behavior instead of a vague requirement.

Sources

• temporalio/temporal fixed source snapshot
• Temporal Server architecture
• Temporal Workflow Execution
• Temporal Activities
• Activity failure detection and heartbeat
• Signals, Queries, and Updates
• Temporal Worker Deployments
• trpc-group/trpc-agent-go fixed source snapshot