Reliability

A durable workflow's whole reason to exist is that the world it talks to is unreliable: APIs time out, workers crash mid-step, downstream systems rate-limit you, and the occasional run is a poison pill that takes the process down with it. The engine treats each of those as a first-class concern rather than something you hand-roll on top.

A worker crashing mid-run is handled automatically by self-healing recovery: while a run executes its worker renews a recovery lease, and a crashed worker stops renewing, so its lease expires. Recovery (engine.recoverIncomplete()) runs both on boot and periodically — the durable:work loop calls it every tick — so an orphaned running run is reclaimed by another instance within ~leaseMs, not only on the next deploy. See Durability & replay.

The building blocks compose: a remote step can retry with backoff, register a compensation to undo itself, be admitted through a rate-limited queue, and — if recovery can never make it past a crash — land in the dead-letter state for a handler to deal with.

The four primitives

At a glance

• Retries — both ctx.step (dispatched) and ctx.localStep (in-process) retry on failure. A local step retries the in-process function; a dispatched step has a durable retry path (a failed dispatch re-dispatches by suspending the run on a persisted wakeAt, so retries survive a crash) plus a timeoutMs heartbeat-liveness path for presumed-dead workers.
• Sagas — attach a compensate closure to a ctx.localStep, or a compensate step ref to a dispatched ctx.step; the engine runs registered undos in reverse order when the run fails. compensationRetries retries a transient undo, and you can trigger the saga deliberately with engine.cancel(runId, { compensate: true }).
• Flow control — register a queue with engine.registerQueue and reference it from ctx.step(step, input, { queue }). A call that can't be admitted re-suspends with a retry time. For a global cross-process cap, use the @adonis-agora/durable/admission-redis backend.
• Dead-lettering — maxRecoveryAttempts caps how many times crash-recovery retries a run before moving it to the terminal dead status, where it stays inspectable and retriable. engine.onDead routes the dead run to a handler.
• Retention — the retention config hard-deletes terminal runs past a per-status age, swept by the worker tick; engine.onEvict archives each run (with its checkpoints) before deletion, and a throwing hook skips the delete.
• The stalled-run pager — engine.onStalled + stalledAfter turn the stranded signature into a page instead of a query you remember to run: once per stranded episode, a listener receives the run and why it looks stuck.