Reliability

A durable workflow's whole reason to exist is that the world it talks to is unreliable: APIs time out, workers crash mid-step, downstream systems rate-limit you, and the occasional run is a poison pill that takes the process down with it. The engine treats each of those as a first-class concern rather than something you hand-roll on top.

A worker crashing mid-run is handled automatically by self-healing recovery: while a run executes its worker renews a recovery lease, and a crashed worker stops renewing, so its lease expires. Recovery (engine.recoverIncomplete()) runs periodically: the worker loop — durable:work, or the embedded loop you get with worker: { embedded: true } — calls it every tick, so an orphaned running run is reclaimed by another instance within ~leaseMs (default 30s), not only on the next deploy. A run whose remote step is still in flight is not an orphan: recovery parks it suspended and lets the worker's result resume it. See Durability & replay.

The building blocks compose: a remote step can retry with backoff, register a compensation to undo itself, be admitted through a rate-limited queue, and — if recovery can never make it past a crash — land in the dead-letter state for a human or a handler to deal with.

The building blocksCopy Anchor Link

At a glanceCopy Anchor Link

• Retries — both ctx.step (dispatched) and ctx.localStep (in-process) retry on failure. A local step retries the in-process function; a dispatched step has a durable retry path (a failed dispatch re-dispatches by suspending the run on a persisted wakeAt, so retries survive a crash) plus a timeoutMs heartbeat-liveness path for presumed-dead workers. Throw a FatalError (or report a retryable: false error from the worker) to opt a deterministic business failure out of retrying. A dispatch that is lost outright is re-driven by engine.redispatchPending(runId) or the opt-in remoteRedispatchMs self-heal.
• Sagas — attach a compensate closure to a ctx.localStep, or a compensate step ref to a dispatched ctx.step; the engine runs registered undos in reverse order when the run fails. compensationRetries retries a transient undo, every compensation surfaces as a compensate:<step> event so a stranded undo is visible, and you can trigger the saga deliberately with engine.cancel(runId, { compensate: true }).
• Flow control — register a queue with engine.registerQueue and reference it from ctx.step(step, input, { queue }). A call that can't be admitted re-suspends with a retry time, and the timer poller retries admission when it comes due, so the limit survives a crash. priority and fairnessKey on the call decide who gets the next contended slot. For a global cross-process cap, use the @adonis-agora/durable/admission-redis backend.
• Concurrency quotas — concurrency (on the workflow config, engine.register, or per start via StartOptions.concurrency) caps how many runs sharing a key are in flight across workflows; an over-limit start is rejected with ConcurrencyLimitError before anything is created.
• Singleton workflows — singleton: { key, limit?, maxQueueDepth? } serializes runs of one workflow that share a key: excess runs wait suspended and are admitted in creation order, and maxQueueDepth rejects a start with SingletonQueueFullError once the backlog is full.
• onError routing — onError: [{ code, workflow }] hands a run that failed with a matching error code to a handler workflow, and the run settles with the handler's outcome, so a fan-out over it keeps going when one item fails.
• Dead-lettering — maxRecoveryAttempts caps how many times crash-recovery retries a run before moving it to the terminal dead status, where it stays inspectable and retriable from the dashboard. engine.onDead routes the dead run to a handler.
• Retention — the retention config hard-deletes terminal runs past a per-status age, swept by the worker tick; engine.onEvict archives each run (with its checkpoints) before deletion, and a throwing hook skips the delete. engine.purgeRuns(scope) deletes a scope on demand.
• The stalled-run pager — engine.onStalled + stalledAfter turn the stranded signature into a page instead of a query you remember to run: once per stranded episode, a listener receives the run and why it looks stuck.
• Multiple instances — web and worker processes compete for the same result queue (and, on queue/db, heartbeat queue), so each delivery lands on exactly one of them. Steps without timeoutMs resume on whichever instance gets the result; a pure producer (on the queue, db or bullmq transport) can opt out of the race with consumers: 'never'.
• Deploys — on SIGTERM the worker loop stops picking up work and engine.drain() waits for in-flight executions, bounded by one shutdown budget (--drain-timeout, default 10s); leases still held when it runs out are released so another instance replays those runs immediately. See Drain on deploy.