Agora
Reliability

Delivery under multiple instances

Web and worker instances compete for the SAME result and heartbeat queues — each delivery lands on exactly one of them. What that means for steps with and without timeoutMs, why a persisted heartbeat doesn't re-arm a timer, and when to run a pure producer with consumers: 'never'.

Web + workers consume the SAME result and heartbeat queues, in competition. Each delivery lands on exactly one instance:

  • It landed on the dispatcher: the fast in-memory path — the result resolves (or the heartbeat re-arms the liveness timer) directly.
  • It landed on another instance: that instance writes the checkpoint (completeRemoteResult) and tries to resume the run — which fails while the dispatcher still holds the lease (the lease covers the whole execution). The dispatcher stays deaf until its timer fires.

Practical rules:

  1. Steps WITHOUT timeoutMs suspend durably: any instance can resume the run from the checkpoint. This is the dead-pod-proof mode (scale-down/crash/deploy). Combine it with remoteRedispatchMs + remoteRedispatchMax against a lost dispatch, and with reconcileMs (default 5 min) against a lost wake.
  2. Steps WITH timeoutMs get fast dead-worker detection, but depend on the in-memory re-arm: use them for SHORT steps (minutes). When the timeout fires, the engine re-reads the checkpoint — if it completed, it resolves with the recorded output; if still pending, it fails as before.
  3. Never assume "heartbeat in the database = timer re-armed": the pulse persists on whichever instance received it, but it only re-arms the timer on the instance that dispatched the step.
  4. Processes that never execute steps should set consumers: 'never' (a pure producer): no point-to-point delivery lands on them, so the race with the worker fleet is over. durable:work re-enables consumption for itself via engine.startConsumers().