Failure modes & recovery

The other pages in this section explain each reliability primitive on its own — retries, sagas, flow control, the dead-letter queue. This page is the other direction: you're staring at a suspended run that isn't moving, and you need to know which failure that is and which knob heals it. Each entry names the symptom you can actually observe — a status, a checkpoint field, a dashboard panel — and the one configuration or command that resolves it.

Where a run actually executesCopy Anchor Link

Before diagnosing anything, know which of the three runDispatcher modes your app runs — it changes who is on the hook when a run doesn't move:

The matrixCopy Anchor Link

Worker crashes mid-run (leased run)Copy Anchor Link

Covered in depth by Durability & replay and the dead-letter queue — the short version, for this matrix: a live worker renews a recovery lease while it executes a run; a crashed worker stops renewing, so engine.recoverIncomplete() (called every tick of the worker loop — durable:work or the embedded one, starting with the first tick after boot) reclaims the orphaned run within ~leaseMs. Every reclaim increments the run's recoveryAttempts before resuming — that field is the trace of how many times this run has been picked back up. Set maxRecoveryAttempts in config/durable.ts so a genuine poison pill (a run that crashes the process every time it's resumed) moves to the terminal dead status instead of crash-looping the instance forever.

This net catches any crash while a run is actively executing. Recovery re-reads the run once it holds the lease and only acts on one that still reads running, so a run that settled after the orphan listing is left alone and not counted.

One shape is deliberately not counted as a crash: a run with a remote step still pending on the transport. Its work is with a worker that holds no run lease, so a free lease proves nothing. Recovery parks such a run suspended on the step's deadline (or the reconcileMs wake) instead, without touching recoveryAttempts, and the worker's result resumes it. A slow remote step therefore can't be dead-lettered while its worker is busy.

A timeoutMs step whose coordinator crashedCopy Anchor Link

A timeoutMs step is awaited in memory by the process that dispatched it, and its liveness timer is a setTimeout there. It still writes a pending checkpoint before it dispatches. If the coordinating process dies, the run reads running with a stale lease, and recovery finds that pending step — so it parks the run suspended on the reconcileMs wake (default 5 minutes) rather than counting a recovery attempt. When the wake comes due, the replay reaches the step and dispatches it again from its first attempt (the step is idempotent by its stepId). If the worker finished in the meantime, its result completes the checkpoint and resumes the run first. With reconcileMs: 0 there is no wake, and the run strands until the stalled-run pager flags it as wakeForever or someone runs engine.redispatchPending(runId).

A remote workflow's run found this way (running, free lease, a call still pending) is different: it reads running only while a turn is being computed or applied, so its turn was lost — an inline advance hit remoteAdvanceSilenceMs, or the replica applying a decision died. Recovery parks it on a wake about a second out instead of the reconcileMs window, and the timer pass then dispatches a fresh turn with the same history. The call already in flight is not dispatched again: its pending checkpoint dedupes it, and its result resumes the run as usual.

Lost remote dispatch: the run suspends and nothing auto-redrives itCopy Anchor Link

This is the failure mode that actually costs an on-call engineer a night: a dispatched ctx.step (no timeoutMs) persists a pending checkpoint and suspends the run durably — it is not held in memory. If the worker that claimed the job then crashes before producing a result, or the transport drops the job outright (a Redis eviction, a broker moving a stalled job to failed and removing it), that checkpoint just sits pending forever. The run's status reads suspended, not failed — there's no error to alert on.

By design, this does not self-heal by default. The run does periodically wake — a suspend with no natural timer gets a fallback wakeAt (reconcileMs, default 5 minutes) so it isn't literally invisible — but the reconcile re-drive replays straight back into the same guard, finds the step still pending, and re-suspends rather than re-dispatching. That's deliberate: the engine cannot tell "the worker died" apart from "the worker is just slow," and re-dispatching on every 5-minute reconcile would double-run a slow-but-live worker's step. So by default a lost dispatch cycles quietly through suspended → reconcile → suspended forever, never erroring, never progressing.

There are exactly three nets, in order of preference:

1. Per-step timeoutMs (+ retries/backoff). Opts the step out of the durable-suspend path entirely into an in-memory await with heartbeat-rearmed liveness: if the worker produces neither a result nor a heartbeat within timeoutMs, the engine presumes it dead and retries. The catch: the timer is setTimeout inside the coordinating process — if that process (not the worker) dies, the timer dies with it, and the step is only re-dispatched once recovery has parked the run and its reconcileMs wake comes due (see above). Reach for this only when you genuinely need to detect and replace a stuck worker.
2. remoteRedispatchMs / remoteRedispatchMax (engine config). The store-driven net for a no-timeoutMs step. Unset by default — opt in explicitly. When set, a dispatched remote step's pending checkpoint carries a lease in its wakeAt: the run suspends on that deadline, and only a lapsed lease re-dispatches the same stepId instead of re-suspending, bumping attempts each time, bounded by remoteRedispatchMax (default 10) before it gives up and fails the step with a RemoteStepError (stepError.code: 'remote_step_lost'). A worker that is still holding the step renews the lease with its heartbeats (throttled to one write per half-window, so a hot beat loop is not an UPDATE storm), so a long-but-live step is not re-dispatched just for outliving the window. A step fan writes its pending checkpoints with the lease already stamped and the run parks on it, and the instance that sent the fan answers its steps' pickup beats from memory until the half-window, so a fan of thousands of steps costs no extra checkpoint reads or writes on pickup. A replay re-reads a pending step from the store before stamping or re-driving it, so a result that landed after the replay took its checkpoint snapshot is replayed, never overwritten by the stale pending row. This covers both paths a remote step reaches the broker by — a native ctx.step and a polyglot workflow's call fan-out (ctx.gather_calls) — and a re-drive is marked as such: the step.started event carries redispatched: true and the checkpoint's own event trail gets a warn step.redispatched (step.lost at the bound), so it reads differently from a failure retry. Two requirements are still non-negotiable: the window must exceed the longest legitimate step (or you double-run a slow-but-live worker that isn't beating), and steps must be idempotent (re-dispatch can genuinely double-execute a step whose original job merely arrives late).
3. engine.redispatchPending(runId) — the operator escape hatch. Re-enqueues every pending remote checkpoint on a run right now, regardless of how long they've been sitting. It's explicitly documented as safe to call on a healthy run — it only touches checkpoints already pending, and a dispatched step is idempotent by its stable stepId (runId:seq) by contract, so a redundant re-dispatch just produces a duplicate (dedupable) attempt rather than corrupting state. This is what you reach for at 2am on a run you've confirmed is stuck, without waiting for remoteRedispatchMs to be configured or to elapse.

The stranded signatureCopy Anchor Link

Query for this pattern to confirm "lost dispatch" before reaching for redispatchPending:

• The run's status is 'suspended' and its updatedAt is stale (older than a few reconcile cycles).
• Its most recent checkpoint has kind: 'remote', status: 'pending', and an enqueuedAt far older than the step should ever take.
• attempts on that checkpoint is 1 (never re-dispatched) if remoteRedispatchMs is unset, or climbing but still pending if it is set and still under remoteRedispatchMax.
• The checkpoint's wakeAt is NULL when remoteRedispatchMs is unset (nothing is leasing the step); with it set, a NULL wakeAt on a pending remote checkpoint means the engine has not seen this step since it was dispatched, and a lease in the past means the re-drive is due on the next timer tick.

That combination — suspended run, one pending remote checkpoint, old enqueuedAt, stale updatedAt — is the signature. A run that's merely waiting on a slow-but-live worker looks identical for a while, which is exactly why the engine won't auto-redispatch it; the operator's judgment call ("this has been pending for way longer than the step ever legitimately takes") is what redispatchPending exists to encode manually. You don't have to run this query by hand, though — the stalled-run pager below watches for exactly this signature and pages you.

The stalled-run pager — onStalled + stalledAfterCopy Anchor Link

engine.onStalled(listener) turns the stranded signature into an alert instead of a query you remember to run at 2am. Register a listener (exactly like engine.onDead) and the worker tick sweeps for stranded in-flight runs — self-throttled to one pass a minute, skipped while the engine drains, and a no-op when no listener is registered:

import engine from '@adonis-agora/durable/services/main'

engine.onStalled(({ run, ageMs, wakeForever, stalePending }) => {
  pager.alert(`durable run ${run.id} (${run.workflow}) stalled for ${Math.round(ageMs / 60_000)}min`, {
    wakeForever,          // suspended with NO wake timer — nothing will ever re-drive it
    stalePending,         // the oldest silent pending remote step, when that's the cause
  })
})

A run is paged when it has sat untouched past stalledAfter (config, duration string or ms, default '15m') and matches one of the two stranded shapes — disambiguated from healthy long work:

• wakeForever — suspended with no wake timer at all, so nothing will ever re-drive it (the reconcileMs: 0 hazard).
• stalePending — an old pending remote step whose worker heartbeat has gone silent past the threshold; a long step that is still heartbeating is not stalled, and neither is a long ctx.sleep (its wakeAt is in the future).

Each stranded episode pages once: a run you were already paged about is skipped until it makes progress and strands again, so a wedged run doesn't re-fire your pager every minute. The listener receives a StalledRunInfo — the run, its ageMs since last activity, the wakeForever flag, and the stalePending detail (seq, name, attempts, ageMs, heartbeatAgeMs) when a silent remote step is the cause. A listener that throws never breaks the sweep, and onStalled returns an unsubscribe function.

Where it runs matters:

• Only in the worker loop. The sweep is a phase of durable:work (or the embedded loop), so register the listener in a preload the worker processes load. A listener on a web-only process never fires.
• Per instance. Each worker sweeps, and remembers what it already paged, on its own (in memory). With several workers holding a listener, one stranded episode pages once per worker, and a restart forgets what was paged. Register it on one process, or de-duplicate in the pager by run.id + run.updatedAt.
• Bounded and namespaced. Each pass looks at up to 100 running/suspended runs untouched for longer than stalledAfter, in the engine's own namespace.

export default defineConfig({
  stalledAfter: '30m', // page only after 30 minutes of silence — above your longest legitimate step
})

Set stalledAfter above your longest legitimate remote step, for the same reason --stale and remoteRedispatchMs need headroom: a slow-but-healthy step under the threshold is exactly the case the engine refuses to guess about.

Queue-transport specificsCopy Anchor Link

The queue driver (transports.queue(...)) rides an @adonisjs/queue adapter. Two recovery mechanisms live in the transport itself, below the engine:

• Failed job processing: a step handler that throws does not fail the job — the worker reports a failed result and the engine's retry policy decides. What the transport redelivers (one poll interval later) is a job whose processing threw: the result could not be pushed, or an engine-side result or decision consumer hit a store error.
• Stalled claims: a coarse background sweep (default every 30s) calls adapter.recoverStalledJobs for every queue this instance pops from — a worker that crashes silently mid-job (no throw, just gone) stops renewing its claim, and once the claim has gone unrenewed for stalledThresholdMs (default 2min on an adapter that can renew claims — every @adonisjs/queue 0.6/0.7 driver; 30min on a custom adapter without renewJobs, whose claims are never renewed) the job is re-delivered. A live worker renews every claim it holds each half threshold, so a long step is never mistaken for a dead one — only a worker that is gone, or whose event loop is blocked past the threshold. A step job reclaimed more than maxStalledCount times is failed and, on @adonisjs/queue 0.7, bridged to a retryable failed step result, so the engine's retry policy (then the workflow's error path) takes over instead of the run waiting forever (0.6 fails and removes the job itself, with no result bridged). Tune stalledCheckIntervalMs (0 disables the sweep), stalledThresholdMs (raise it above the longest event-loop block), and maxStalledCount (default 3 — bounds a poison job) on transports.queue(...); see the queue transport. Adapters that don't implement recoverStalledJobs are detected and skipped — on those, a silently-dead worker's job is simply lost, and engine.redispatchPending(runId) is the fix, since the checkpoint (not the queue job) is the source of truth the engine acts on.

Deploys: drain before you exitCopy Anchor Link

A process killed mid-run leaves its runs to lease expiry: nothing is lost, but each one waits ~leaseMs (default 30s) before another instance's recovery picks it up. There is one exception. A process that restarts under the same stable instanceId (the pod name) reclaims the runs still leased to its id on its first recovery passes after boot. It never takes a run it is executing itself, and other instances still wait for expiry. That only holds if no two live processes share an id. Draining avoids the wait altogether:

• durable:work traps SIGINT/SIGTERM, stops leasing new work, waits for the tick in flight, then calls engine.drain(). --drain-timeout (default 10 000 ms) is the budget for that whole sequence, the tick wait included.
• The embedded loop (worker: { embedded: true }) drains the same way on app shutdown, bounded by worker.drainTimeoutMs (default 10 000).
• Every other store-backed process — a web process whose in-process dispatcher may be mid-turn, a drive: false pod, a control plane — is drained by the provider's shutdown() before the transport closes, bounded by drainTimeoutMs (default 10 000; worker.drainTimeoutMs when set). See Drain on deploy.

drain(timeoutMs) stops recovery, timer and pending pickups and waits for in-flight executions — including continue-as-new links and deferred children — to settle. It never cancels anything. When the budget runs out, it releases the leases this process still holds, so the next instance's recoverIncomplete() replays those runs immediately instead of after leaseMs. A hot continue-as-new loop keeps the drain busy for its whole budget, so size the timeout for your longest in-flight turn, and keep it below your orchestrator's kill grace period (Kubernetes' terminationGracePeriodSeconds, 30s by default).

Namespaces: the classic two-pools mistakeCopy Anchor Link

namespace (set on the engine in config/durable.ts) partitions a deployment: it's stamped on every run the engine creates, scopes runPending / recoverIncomplete / resumeDueTimers / sweepTimeouts to that namespace, and (for the queue/eventEmitter transports) folds into every queue/channel name. Two standalone engines that both leave namespace unset both default to 'default' — if they share one state store or one broker, they are, by construction, the same pool: pool B's worker can lease and execute a run pool A created, because nothing distinguishes them.

That's the classic mistake: standing up a second, supposedly-isolated worker pool (a second environment, a second tenant) against the same Postgres/Redis without giving it its own namespace. The symptom isn't an error — it's runs quietly executing on the "wrong" pool, or (calling resume directly rather than through the poll paths) a NamespaceMismatch thrown for a run that belongs to another pool. The fix is a one-line config change, not a topology change: set a distinct namespace per pool. (Only a control-plane with no namespace drives every pool; a standalone engine never does — it is in 'default' when unset.) See Namespaces — sharing one broker across pools for the full mechanics and the exact queue-name scheme.

Testing failure modesCopy Anchor Link

The two in-process transports behave differently on an unregistered handler, which matters when you're writing a test for any of the above:

• InMemoryTransport (@adonis-agora/durable/testing, the default in createTestEngine()) drives dispatch straight into runStepHandler. With no handler registered for the step name, runStepHandler returns immediately with status: 'failed', error: { message: 'no handler for <name>', retryable: false } — the step fails, synchronously and non-retryably. Good for testing "the worker doesn't have this handler."
• EventEmitterTransport (the production in-process transport, transports.eventEmitter()) is closer to a real broker: dispatch emits an event, and if nothing is listening for that step name, nothing answers and no result is ever emitted. The step's pending checkpoint sits there exactly like a lost broker job. Use this transport in a test to reproduce the "lost remote dispatch" failure mode above (e.g. to exercise remoteRedispatchMs or redispatchPending against a step that will never complete on its own), by never registering a handler for the step you want to strand.