Durability & replay

Durability comes from checkpoint + deterministic replay — the same model Temporal and DBOS use. Understanding it is the one thing worth reading before you build.

The mechanismCopy Anchor Link

Each step records its result in the store at a deterministic position (seq). Recovery works by re-running the workflow function from the top — but a step that already has a completed checkpoint returns its saved result instead of executing again:

run abc
  step[0] reserveStock  → execute → checkpoint {seq:0, completed, output}
  step[1] chargeCard     → execute → checkpoint {seq:1, completed, output}
  💥 crash
  --- engine restarts, resumes run abc ---
  step[0] reserveStock  → completed checkpoint → return saved output (NOT re-run)
  step[1] chargeCard     → completed checkpoint → return saved output (NOT re-run)
  step[2] ship           → no checkpoint → execute for real

Only a completed checkpoint short-circuits replay. A non-terminal checkpoint — a remote step's pending (dispatched, awaiting its worker) or a local step's in-flight running (see trackStepStart) — does not: the step is re-awaited (remote) or re-run (local). So a crash mid-step is safe — there's no half-finished result to honour.

You write ordinary linear code; the engine guarantees each step runs exactly once, logically, even across crashes and deploys.

The execution model: start enqueues, a worker runs the bodyCopy Anchor Link

start does not run the workflow inline. engine.start(workflow, input, runId, opts?) creates the run as 'pending' and returns immediately with { runId, status: 'pending' } (a delayed startAt run returns suspended). Starting is idempotent by runId: a second start with the same id returns the existing run's state instead of creating a duplicate. The caller — an HTTP handler, say — never blocks on workflow logic; the body is dispatched to a worker.

start throws, before any run is created, when the workflow isn't registered (workflow checkout is not registered, or workflow checkout@2 is not registered for a start pinned with opts.version) or when the input fails the workflow's inputSchema (a SchemaValidationError). A bad payload never becomes a run.

start('checkout', order, runId)  → run created, status: 'pending', returns now
  (a worker picks the run up)    → status: 'running', body executes
  ctx.waitForSignal('approve')   → status: 'suspended'
  signal('approve')              → resumes → completes

• 'pending' is a run status: created and enqueued, not yet picked up by a worker.

• Where a run executes is a runDispatcher. On a transport with a run queue (bullmq, queue, db) the default enqueues the run there, and whichever driving instance takes it executes it (see Where a run executes). Otherwise the default is in-process — the run executes on the same instance, asynchronously on a microtask — so a single-process app still runs workflows with zero setup; it just doesn't block the caller. runDispatcher in config/durable.ts accepts any { dispatch(runId) } object, so a broker-backed dispatcher can enqueue run ids for a worker pool that calls engine.runOne(runId) per consumed id (lease + resume one run; returns null if another instance holds the lease or the engine is draining). drive: false swaps in a no-op dispatcher: the process creates runs but leaves them pending for a durable:work poll.

• Await the outcome with waitForRun when you need it inline: engine.waitForRun(runId, opts?) resolves once the run settles — a terminal state (completed/failed/cancelled/dead) or suspended. Pass { terminal: true } to keep waiting through suspensions until the run actually ends, and { timeoutMs } to bound the wait (it rejects with a waitForRun(<id>) timed out error; the run itself is untouched). It resolves with the RunResult ({ runId, status, output?, error? }) and does not throw for a failed run, so check status.

  Copy Text

  import engine from '@adonis-agora/durable/services/main'
  
  await engine.start('checkout', order, runId)
  const settled = await engine.waitForRun(runId) // completed, failed, … or suspended
  const done = await engine.waitForRun(runId, { terminal: true, timeoutMs: 60_000 }) // ignores suspensions

  BaseWorkflow's Wf.start(input) is this pair with { terminal: true } built in; see Running a workflow.

1// a worker leases the pending run and runs the body:2async run(ctx: WorkflowCtx, order: Order) {3  await ctx.step(reserveStock, order)4  await ctx.waitForSignal(`approve:${order.id}`)5  await ctx.step(shipOrder, order)6}

Workers: a process that drains pending runsCopy Anchor Link

For scale you can split an app into web processes (handle HTTP, the dashboard) and worker processes (execute workflows) sharing only the database. A worker is the durable:work ace command, which on an interval calls the engine's worker-side primitives:

• engine.runPending() — lease and run all pending runs.
• engine.recoverIncomplete() — reclaim runs a crashed worker left running.
• engine.resumeDueTimers() — wake runs whose sleep or retry timer has come due.
• engine.sweepTimeouts() — fail runs that blew their execution timeout.
• Fire any due recurring schedules (from config/durable.ts and colocated static schedule), then any due persisted schedules when persistedSchedules: true.
• engine.sweepRetention() — delete terminal runs past their retention age (a no-op without a retention config; at most one pass a minute).
• engine.sweepStalled() — notify engine.onStalled(listener) subscribers about stranded runs (a no-op without listeners; at most one pass a minute).

Each phase is isolated: one failing phase is logged and the tick continues with the next. --interval sets the poll interval (default 1000 ms) and --drain-timeout the shutdown drain bound (default 10000 ms):

node ace durable:work --interval=500 --drain-timeout=30000

An instance configured with role: 'control-plane' in config/durable.ts gets a no-op run dispatcher: its start only enqueues, leaving the run pending for a durable:work process to pick up over the database. An explicit runDispatcher overrides the role default. See Roles & config.

The default (no explicit worker) is the in-process dispatcher — so single-process apps need none of this. Runs simply execute on the same instance, just asynchronously. Run a dedicated durable:work process once you outgrow that.

If your background work does not justify a second container, the provider can run the same loop inside the web process instead — worker: { embedded: true } in config/durable.ts (with optional intervalMs and drainTimeoutMs, same defaults as the command). It starts in the web environment only (so a node ace command, including durable:work itself, never runs a second loop), and its shutdown is wired to app.terminate(), so a deploy drains exactly like a dedicated worker.

The one rule: workflows are deterministicCopy Anchor Link

Because the body re-runs on recovery, it must be deterministic. All non-determinism — network calls, queries, Date.now(), randomness, IO — must live inside a step. The workflow body is just orchestration: call a step, use its result, decide the next.

// ✗ wrong — non-determinism in the workflow body
engine.register('checkout', '1', async (ctx, order) => {
  if (Math.random() < 0.1) { /* re-runs differently on replay */ }
  const now = Date.now() // different on every replay
})

// ✓ right — side effects inside steps, deterministic sources from ctx
engine.register('checkout', '1', async (ctx, order) => {
  const quote = await ctx.localStep('quote', () => pricing.fetch(order)) // checkpointed
  const issuedAt = await ctx.now() // recorded once, replayed verbatim
  await ctx.localStep('charge', () => billing.charge(quote, issuedAt))
})

ConsequencesCopy Anchor Link

• Idempotency. The engine guarantees logical exactly-once, but if the process dies after a remote worker ran and before its result was recorded, the step may physically run twice. Steps receive a stable stepId (runId:seq) — make handlers idempotent or dedupe on it. The same holds for ctx.localStep: a crash inside the body re-runs it on replay, so key its side effects on something stable such as ctx.runId plus the step name (see What gets persisted).
• Retries. Configure per step: ctx.localStep(name, fn, { retries: 3, backoff: 'exp' }) for an in-process step, or ctx.step(ref, input, { retries: 3, backoff: 'exp' }) for a dispatched one. Throw a FatalError to stop retrying a business failure outright. See Retries & backoff.
• Self-healing recovery. The engine resumes every run a previous process left running (engine.recoverIncomplete(), called every tick by the durable:work loop). This runs both on boot and periodically — so a run orphaned by a crashed worker is reclaimed within ~leaseMs, not only on the next deploy. While a run executes, its worker renews the recovery lease (every leaseMs / 2), so a live worker keeps a long-running run while a crashed worker's lease still expires and another instance takes over.

Deploys, versioning & multiple instancesCopy Anchor Link

Because the engine is embedded in your app, deploys and replicas need a word.

• The state survives the deploy. Checkpoints live in the database, so killing the old instance loses nothing. The new instance recovers running runs — and any other instance reclaims orphaned runs within ~leaseMs via periodic recovery — while the timer poller resumes due sleeps. Suspended and pending runs don't even notice a deploy; a worker picks them up afterward.
• Version-pinned replay (skew protection). Replay is positional — changing a workflow's body (reordering/inserting steps) while runs are in flight would corrupt them. So a workflow is registered with a version, and a run resumes on the version it started on (workflowVersion on the run). Register the old and new versions side by side during a breaking change: in-flight runs drain on the old version, new runs start on the newest. A run whose version is no longer registered fails loudly rather than corrupting. See Versioning & determinism.
• One instance per run (recovery lease). With several replicas, each would try to recover the same running runs. The engine takes an atomic, self-expiring lease on a run before resuming it (StateStore.tryLockRun), so a run is picked up by exactly one instance, and renews the lease while it runs (StateStore.renewRunLock) so a long-running run isn't reclaimed out from under a live worker. A crashed worker stops renewing, so its lease expires and another instance recovers the run. A turn releases its lease only after the state it settled on (completed, failed, suspended, blocked) is written, so a run never sits unleased while it still reads running; and recovery re-reads a run once it holds the lease, leaving alone anything that is no longer running (a turn that settled, or a run a fast step result already resumed, between the orphan listing and the lease). Set leaseMs (config leaseMs, default 30s) above how long a single resume step runs.
• Remote workflow turns. A workflow whose body runs on another worker (a thin worker or the Python SDK) advances one turn at a time, and any replica may apply the turn's decision: it is applied by run id, durably, only while the run still awaits that turn. A lost turn is re-driven after remoteAdvanceSilenceMs. See Remote workflow turns.
• Graceful shutdown. On SIGINT/SIGTERM the durable:work loop drains: it stops picking up new runs and waits for in-flight ones to settle (bounded by --drain-timeout, default 10s), then releases the leases of any run still in flight so the next instance recovers it immediately instead of waiting for lease expiry. The embedded worker drains the same way on app.terminate(), and so does every other store-backed process (a web pod without the embedded worker, a control plane): the provider calls engine.drain(drainTimeoutMs) (default 10s, falling back to worker.drainTimeoutMs) before closing the transport and control plane.