Failure modes & recovery

The other pages in this section explain each reliability primitive on its own — retries, sagas, flow control, the dead-letter queue. This page is the other direction: you're staring at a suspended run that isn't moving, and you need to know which failure that is and which knob heals it. Each entry names the symptom you can actually observe — a status, a checkpoint field, a dashboard panel — and the one configuration or command that resolves it.

Where a run actually executes

Before diagnosing anything, know which of the three runDispatcher modes your app runs — it changes who is on the hook when a run doesn't move:

The matrix

Worker crashes mid-run (leased run)

Covered in depth by Durability & replay and the dead-letter queue — the short version, for this matrix: a live worker renews a recovery lease while it executes a run; a crashed worker stops renewing, so engine.recoverIncomplete() (called on boot and every durable:work tick) reclaims the orphaned run within ~leaseMs. Every reclaim increments the run's recoveryAttempts before resuming — that field is the trace of how many times this run has been picked back up. Set maxRecoveryAttempts in config/durable.ts so a genuine poison pill (a run that crashes the process every time it's resumed) moves to the terminal dead status instead of crash-looping the instance forever.

This net catches any crash while a run is actively executing, including a timeoutMs remote step whose in-memory liveness timer died with the coordinating process (see below) — the run is still running/leased the whole time it awaits, so it's still reachable by the same lease-expiry path.

Lost remote dispatch: the run suspends and nothing auto-redrives it

This is the failure mode that actually costs an on-call engineer a night: a dispatched ctx.step (no timeoutMs) persists a pending checkpoint and suspends the run durably — it is not held in memory. If the worker that claimed the job then crashes before producing a result, or the transport drops the job outright (a Redis eviction, a broker moving a stalled job to failed and removing it), that checkpoint just sits pending forever. The run's status reads suspended, not failed — there's no error to alert on.

By design, this does not self-heal by default. The run does periodically wake — a suspend with no natural timer gets a fallback wakeAt (reconcileMs, default 5 minutes) so it isn't literally invisible — but the reconcile re-drive replays straight back into the same guard, finds the step still pending, and re-suspends rather than re-dispatching. That's deliberate: the engine cannot tell "the worker died" apart from "the worker is just slow," and re-dispatching on every 5-minute reconcile would double-run a slow-but-live worker's step. So by default a lost dispatch cycles quietly through suspended → reconcile → suspended forever, never erroring, never progressing.

There are exactly three nets, in order of preference:

1. Per-step timeoutMs (+ retries/backoff). Opts the step out of the durable-suspend path entirely into an in-memory await with heartbeat-rearmed liveness: if the worker produces neither a result nor a heartbeat within timeoutMs, the engine presumes it dead and retries. The catch: the timer is setTimeout inside the coordinating process — if that process (not the worker) dies, the timer dies with it, and the run just sits running until recoverIncomplete reclaims it on lease expiry (see above). Reach for this only when you genuinely need to detect and replace a stuck worker.
2. remoteRedispatchMs / remoteRedispatchMax (engine config). The store-driven net for a no-timeoutMs step. Unset by default — opt in explicitly. Once a pending checkpoint has sat past remoteRedispatchMs, the next reconcile re-drive re-dispatches the same stepId instead of re-suspending, bumping attempts each time, bounded by remoteRedispatchMax (default 10) before it gives up and fails the step with a RemoteStepError (code: 'remote_step_lost'). Two requirements are non-negotiable: the window must exceed the longest legitimate step (or you double-run a slow-but-live worker), and steps must be idempotent (re-dispatch can genuinely double-execute a step whose original job merely arrives late).
3. engine.redispatchPending(runId) — the operator escape hatch. Re-enqueues every pending remote checkpoint on a run right now, regardless of how long they've been sitting. It's explicitly documented as safe to call on a healthy run — it only touches checkpoints already pending, and a dispatched step is idempotent by its stable stepId (runId:seq) by contract, so a redundant re-dispatch just produces a duplicate (dedupable) attempt rather than corrupting state. This is what you reach for at 2am on a run you've confirmed is stuck, without waiting for remoteRedispatchMs to be configured or to elapse.

The stranded signature

Query for this pattern to confirm "lost dispatch" before reaching for redispatchPending:

• The run's status is 'suspended' and its updatedAt is stale (older than a few reconcile cycles).
• Its most recent checkpoint has kind: 'remote', status: 'pending', and an enqueuedAt far older than the step should ever take.
• attempts on that checkpoint is 1 (never re-dispatched) if remoteRedispatchMs is unset, or climbing but still pending if it is set and still under remoteRedispatchMax.

That combination — suspended run, one pending remote checkpoint, old enqueuedAt, stale updatedAt — is the signature. A run that's merely waiting on a slow-but-live worker looks identical for a while, which is exactly why the engine won't auto-redispatch it; the operator's judgment call ("this has been pending for way longer than the step ever legitimately takes") is what redispatchPending exists to encode manually. You don't have to run this query by hand, though — the stalled-run pager below watches for exactly this signature and pages you.

The stalled-run pager — onStalled + stalledAfter

engine.onStalled(listener) turns the stranded signature into an alert instead of a query you remember to run at 2am. Register a listener (exactly like engine.onDead) and the worker tick sweeps for stranded in-flight runs — self-throttled to one pass a minute, and a no-op when no listener is registered:

engine.onStalled(({ run, ageMs, wakeForever, stalePending }) => {
  pager.alert(`durable run ${run.id} (${run.workflow}) stalled for ${Math.round(ageMs / 60_000)}min`, {
    wakeForever,          // suspended with NO wake timer — nothing will ever re-drive it
    stalePending,         // the oldest silent pending remote step, when that's the cause
  })
})

A run is paged when it has sat untouched past stalledAfter (config, duration string or ms, default '15m') and matches one of the two stranded shapes — disambiguated from healthy long work:

• wakeForever — suspended with no wake timer at all, so nothing will ever re-drive it (the reconcileMs: 0 hazard).
• stalePending — an old pending remote step whose worker heartbeat has gone silent past the threshold; a long step that is still heartbeating is not stalled, and neither is a long ctx.sleep (its wakeAt is in the future).

Each stranded episode pages once: a run you were already paged about is skipped until it makes progress and strands again, so a wedged run doesn't re-fire your pager every minute. The listener receives a StalledRunInfo — the run, its ageMs since last activity, the wakeForever flag, and the stalePending detail (seq, name, attempts, ageMs, heartbeatAgeMs) when a silent remote step is the cause. A listener that throws never breaks the sweep, and onStalled returns an unsubscribe function.

export default defineConfig({
  stalledAfter: '30m', // page only after 30 minutes of silence — above your longest legitimate step
})

Set stalledAfter above your longest legitimate remote step, for the same reason --stale and remoteRedispatchMs need headroom: a slow-but-healthy step under the threshold is exactly the case the engine refuses to guess about.

Queue-transport specifics

The queue driver (transports.queue(...)) rides an @adonisjs/queue adapter. Two recovery mechanisms live in the transport itself, below the engine:

• Thrown handlers: a job whose handler threw is redelivered one poll interval later, so a transient failure inside a step handler does not need the engine's retry policy to notice it.
• Stalled claims: a coarse background sweep (default every 30s) calls adapter.recoverStalledJobs for every queue this instance pops from — a worker that crashes silently mid-job (no throw, just gone) leaves its claim's acquiredAt frozen, and once it exceeds stalledThresholdMs (default 30min) the job is re-delivered. The threshold is deliberately generous because the claim is never renewed while a worker processes: a legitimately long step holds one claim for its whole duration, so a threshold below your longest step would double-run a merely-slow worker. Tune stalledCheckIntervalMs (0 disables the sweep), stalledThresholdMs (raise it above your longest step), and maxStalledCount (default 3 — bounds a poison job) on transports.queue(...); see the queue transport. Adapters that don't implement recoverStalledJobs are detected and skipped — on those, a silently-dead worker's job is simply lost, and engine.redispatchPending(runId) is the fix, since the checkpoint (not the queue job) is the source of truth the engine acts on.

Namespaces: the classic two-pools mistake

namespace (set on the engine in config/durable.ts) partitions a deployment: it's stamped on every run the engine creates, scopes runPending / recoverIncomplete / resumeDueTimers / sweepTimeouts to that namespace, and (for the queue/eventEmitter transports) folds into every queue/channel name. Two engines that both leave namespace unset both default to 'default' — if they share one state store or one broker, they are, by construction, the same pool: pool B's worker can lease and execute a run pool A created, because nothing distinguishes them.

That's the classic mistake: standing up a second, supposedly-isolated worker pool (a second environment, a second tenant) against the same Postgres/Redis without giving it its own namespace. The symptom isn't an error — it's runs quietly executing on the "wrong" pool, or (calling resume directly rather than through the poll paths) a NamespaceMismatch thrown for a run that belongs to another pool. The fix is a one-line config change, not a topology change: set a distinct namespace per pool. See Namespaces — sharing one broker across pools for the full mechanics and the exact queue-name scheme.

Testing failure modes

The two in-process transports behave differently on an unregistered handler, which matters when you're writing a test for any of the above:

• InMemoryTransport (@adonis-agora/durable/testing, the default in createTestEngine()) drives dispatch straight into runStepHandler. With no handler registered for the step name, runStepHandler returns immediately with status: 'failed', error: { message: 'no handler for <name>', retryable: false } — the step fails, synchronously and non-retryably. Good for testing "the worker doesn't have this handler."
• EventEmitterTransport (the production in-process transport, transports.eventEmitter()) is closer to a real broker: dispatch emits an event, and if nothing is listening for that step name, nothing answers and no result is ever emitted. The step's pending checkpoint sits there exactly like a lost broker job. Use this transport in a test to reproduce the "lost remote dispatch" failure mode above (e.g. to exercise remoteRedispatchMs or redispatchPending against a step that will never complete on its own), by never registering a handler for the step you want to strand.