Delivery under multiple instances
Web and worker instances compete for the SAME result and heartbeat queues — each delivery lands on exactly one of them. What that means for steps with and without timeoutMs, why a persisted heartbeat doesn't re-arm a timer, and when to run a pure producer with consumers: 'never'.
Web + workers consume the SAME result and heartbeat queues, in competition. Each delivery lands on exactly one instance:
- It landed on the dispatcher: the fast in-memory path — the result resolves (or the heartbeat re-arms the liveness timer) directly.
- It landed on another instance: that instance writes the checkpoint (
completeRemoteResult) and tries to resume the run — which fails while the dispatcher still holds the lease (the lease covers the whole execution). The dispatcher stays deaf until its timer fires.
Practical rules:
- Steps WITHOUT
timeoutMssuspend durably: any instance can resume the run from the checkpoint. This is the dead-pod-proof mode (scale-down/crash/deploy). Combine it withremoteRedispatchMs+remoteRedispatchMaxagainst a lost dispatch, and withreconcileMs(default 5 min) against a lost wake. - Steps WITH
timeoutMsget fast dead-worker detection, but depend on the in-memory re-arm: use them for SHORT steps (minutes). When the timeout fires, the engine re-reads the checkpoint — if it completed, it resolves with the recorded output; if still pending, it fails as before. - Never assume "heartbeat in the database = timer re-armed": the pulse persists on whichever instance received it, but it only re-arms the timer on the instance that dispatched the step.
- Processes that never execute steps should set
consumers: 'never'(a pure producer): no point-to-point delivery lands on them, so the race with the worker fleet is over.durable:workre-enables consumption for itself viaengine.startConsumers().
Failure modes & recovery
An operator-facing map from symptom to knob — where a run actually executes (runDispatcher), what reclaims a worker that crashed mid-run, why a lost remote dispatch does NOT auto-redrive by design, the three nets that catch it (timeoutMs, remoteRedispatchMs, redispatchPending), the stalled-run pager (engine.onStalled + stalledAfter), queue-transport specifics, namespaces, and how to reproduce each failure in a test.
Overview
How remote steps travel to workers. Transports are config-driven drivers selected by name in config/durable.ts — from the in-process memory driver for zero-infra single-process handlers, to the queue driver over @adonisjs/queue, and a broker-less SQL driver that rides the database you already run.