Retries & backoff

Transient failures are the normal case for anything that crosses a network boundary, so every step is retryable. The mechanics differ between an in-process ctx.localStep — work that runs inside the engine — and a dispatched ctx.step routed to a worker, because the engine can re-run a local function in place but a dispatched step has to survive the worker (and even the orchestrator) disappearing mid-flight.

Local-step retriesCopy Anchor Link

ctx.localStep(name, fn, opts?) runs a unit of work in-process and checkpoints its result. If fn throws, the engine retries it up to retries attempts, spacing the attempts with the step's backoff configuration:

const quote = await ctx.localStep(
  'quote',
  () => pricing.fetch(order),
  { retries: 5, backoff: 'exp', backoffMs: 200, backoffMaxMs: 10_000, jitter: true },
)

The StepOptions retry fields are:

• retries — maximum number of attempts before the step (and the run) fails. Defaults to 1 (a single try).
• backoff — how the delay between attempts grows: 'fixed' keeps it constant, 'exp' doubles it each attempt.
• backoffMs — the base delay in ms. Omit (or set to 0) to retry with no delay.
• backoffMaxMs — an upper bound, so an exponential backoff doesn't grow without limit.
• jitter — adds random jitter (the delay becomes 50–100% of the computed value) so a fleet of runs retrying the same downstream don't synchronize into a thundering herd.

timeoutMs and pickupTimeoutMs are accepted by the same StepOptions type but are ignored on a local step: they are liveness windows for a dispatched step (see below). A local step that can hang needs its own timeout inside fn (an AbortSignal.timeout(...) on the call it makes, for example).

With backoff: 'exp', backoffMs: 200 the delays before attempts 2, 3, 4… are 200ms, 400ms, 800ms… until they hit backoffMaxMs. With jitter: true each of those is scaled to a random point in its top half, so two runs that failed at the same instant don't retry in lockstep. Every computed delay is also clamped to Node's setTimeout ceiling (2³¹−1 ms, about 24.8 days), so a very large exponential never overflows into an immediate retry.

A local step's retries are in memory: the backoff is a timer inside the turn that is executing the run, which keeps holding (and renewing) the run's lease while it waits. Only the outcome is checkpointed — the step's output once an attempt succeeds, or its error once the last attempt fails — so if the process dies between attempts, recovery replays the step from attempt 1. Keep local-step backoffs short; a long wait between attempts belongs on a dispatched step, whose retries suspend durably.

Every error a local step throws is retried except a FatalError (below). A retryable: false property on a thrown error is a worker-side verdict for dispatched steps and has no effect here.

FatalError — never retriedCopy Anchor Link

Not every failure is worth retrying. A declined card or invalid input will fail the same way on every attempt. Throw a FatalError to fail the run immediately, regardless of the step's retries:

import { FatalError } from '@adonis-agora/durable'

await ctx.localStep('charge', async () => {
  const res = await stripe.charge(order)
  if (res.declined) throw new FatalError('card declined', 'declined')
  return res
})

The optional second argument is a machine-readable code (here 'declined') that ends up on the run's structured error. Use FatalError for deterministic business verdicts; let ordinary throws (a timeout, a 502, a dropped connection) flow through the retry path. When the verdict is an expected outcome rather than a reason to fail the run, give the workflow an onError handler for that code.

Dispatched-step retriesCopy Anchor Link

A dispatched step (ctx.step) runs on a worker — possibly in another process. There are two distinct paths for handling its failures, chosen by whether the step sets timeoutMs.

The durable path (no timeoutMs)Copy Anchor Link

This is the default and the recommended one. When you dispatch a step that has no timeoutMs, the engine dispatches the task, persists a pending checkpoint, and suspends the run durably — it is not held in memory awaiting the result. Whichever instance receives the worker's result resumes the run, so the worker (and even the dispatching orchestrator) can scale down or crash mid-step without losing the run or re-running completed work.

Declare the durable-retry policy on the step's @Step/defineStep config (or override it per call in the ctx.step options):

export const chargeCard = defineStep(
  'payments:charge-card',
  async (input: { orderId: number; amountCents: number }) => ({ chargeId: await stripe.charge(input) }),
  {
    input: z.object({ orderId: z.number().int(), amountCents: z.number().int() }),
    output: z.object({ chargeId: z.string() }),
    retries: 4,
    backoff: 'exp',
    backoffMs: 500,
    backoffMaxMs: 30_000,
    jitter: true,
  },
)

// in the workflow:
const charge = await ctx.step(chargeCard, { orderId: order.id, amountCents: order.total })

Every retry field is also a per-call override on ctx.step(handler, input, opts): the effective policy is the declared one with any opts field winning, so a single call site can tighten or relax the default without touching the step definition.

// same step, a more patient policy for this one call site
await ctx.step(chargeCard, input, { retries: 8, backoffMaxMs: 120_000 })

When a worker reports a failed result for a durable step, the engine consults retries. If the budget remains, it re-dispatches — but durably, not in a loop. The first time it sees the failed checkpoint it computes the next retry deadline as now + backoffDelay(attempt) and stamps it on the checkpoint's wakeAt, then suspends. Because that deadline is persisted on the checkpoint rather than living in a timer in memory, it is replay-stable and crash-safe (a process that dies before the retry fires picks it back up when the timer poller sees the wakeAt come due). Once the deadline passes, the poller resumes the run, the call re-dispatches with an incremented attempt (and the same stepId), and the cycle continues until the result lands or retries is exhausted. Then the step's error is thrown into the workflow as a RemoteStepError (exported from @adonis-agora/durable), whose stepError holds the worker's message, code and details.

A step that names a flow-control queue goes back through admission on every retry: the slot an attempt held is released when its result lands, so a step waiting out its backoff doesn't keep a slot.

Retrying after the budget is spentCopy Anchor Link

Exhausting retries fails the run (unless a saga, an onError route or the workflow's own catch handles it). A retry of the run — the dashboard's retry, node ace durable:retry <runId>, or engine.requeue(runId) — resets every failed checkpoint to a fresh first attempt (attempts back to 0, retryable forced back on, due immediately), so the replay re-dispatches the failed step with its full budget instead of re-throwing the recorded failure. Completed steps are not re-run.

1@Workflow({ name: 'checkout', version: '1' })2export default class CheckoutWorkflow {3  constructor(4    private pricing: PricingSteps,5    private payments: PaymentSteps,6    private email: EmailSteps,7  ) {}8 9  async run(ctx: WorkflowCtx, order: Order) {10    const quote = await ctx.step(this.pricing.fetchQuote, order)11    const charge = await ctx.step(this.payments.chargeCard, order)12    await ctx.step(this.email.confirm, { order, charge }, { retries: 5 })13    return charge.id14  }15}

Opting out: retryable: falseCopy Anchor Link

A durable remote step retries on a failed worker result unless the worker marks the error as non-retryable. A worker that reports an error has returned a deterministic verdict — a declined card, a validation failure — so re-dispatching it just hammers the worker for the same answer. Throw an error with retryable: false and the engine surfaces it to the workflow immediately instead of retrying:

import engine from '@adonis-agora/durable/services/main'

// A handler served by name on the engine's transport (an `app/steps` handler works the same way):
engine.handle('payments:charge-card', async (input) => {
  const res = await stripe.charge(input)
  if (res.declined) {
    // a deterministic verdict — don't make the engine retry it
    throw Object.assign(new Error('card declined'), { code: 'declined', retryable: false })
  }
  return { chargeId: res.id }
})

The default is retryable !== false, i.e. an ordinary error is retried; only an explicit retryable: false opts out. This is the worker-side counterpart of throwing FatalError in a local step — and FatalError itself carries retryable: false, so throwing one from a TypeScript step handler (throw new FatalError('card declined', 'declined')) has the same effect without the Object.assign. A worker with no handler for the step's name reports a non-retryable no handler for <name> failure.

The in-memory liveness path (timeoutMs)Copy Anchor Link

Setting timeoutMs on a dispatched step opts it into a different path. timeoutMs is a liveness window: if the worker produces neither a result nor a heartbeat within that many ms, the engine presumes it dead, fails the dispatch with a RemoteStepTimeout, and — because that timeout is retryable — re-dispatches it up to retries. Each heartbeat the worker emits (the automatic pickup beat, and every log.heartbeat() the handler calls) rearms the window, so a long but healthy step that keeps beating stays alive well past timeoutMs. Before it times a step out, the engine re-reads the checkpoint: a result another instance already recorded is used instead of failing the step, and a heartbeat another instance persisted keeps the window open (see Delivery under multiple instances).

The re-dispatches on this path happen back to back, inside the turn that awaits the step: backoff/backoffMs are not applied between timeout retries. When the last attempt times out, the RemoteStepTimeout is thrown into the workflow. A step fan (ctx.all(step, inputs)) does not accept a step with timeoutMs; fans need durable steps.

export const renderVideo = defineStep(
  'media:render',
  async (input: { assetId: string }) => ({ url: await render(input) }),
  {
    input: z.object({ assetId: z.string() }),
    output: z.object({ url: z.string() }),
    timeoutMs: 60_000, // presume the worker dead after 60s of silence (no result, no heartbeat)
    retries: 3,
  },
)

The crucial difference between the two paths: the durable path retries on a worker reporting a failure and suspends between attempts (crash-safe, not in memory); the liveness path retries on a worker going silent and awaits the result in memory between attempts. They also disagree on what a reported error means: the durable path retries it (subject to retryable), while a timeoutMs step only re-dispatches on a timeout and surfaces any reported error to the workflow on the first attempt. Reach for timeoutMs only when you genuinely need to detect and replace a stuck worker; otherwise leave it off and get the durable, crash-safe path.

The liveness timer lives in the process that dispatched the step. If that process dies, the step's pending checkpoint stays behind: once the run's lease expires, recovery parks the run suspended on the reconcileMs wake (default 5 minutes) without counting a recovery attempt, and the replay then dispatches the step again from its first attempt. A result the worker publishes in the meantime still completes the checkpoint and resumes the run. See Failure modes.

Queue wait vs. silence while running — pickupTimeoutMsCopy Anchor Link

A single window has to cover two very different stretches of a step's life, and they want very different numbers:

1. Waiting in the queue. Nobody has claimed the job yet, so nobody is beating for it. During a backlog this can legitimately run into minutes.
2. Running on a worker. Somebody owns it and is expected to beat. Silence here is genuinely suspicious within seconds.

Set timeoutMs tight enough to catch a wedged worker and a queue backlog trips it; set it loose enough to survive the backlog and a wedged worker goes unnoticed for just as long. pickupTimeoutMs splits the two:

await ctx.step(harvestBatch, input, {
  timeoutMs: 30_000,        // once running: 30s of silence means the worker is gone
  pickupTimeoutMs: 600_000, // before pickup: 10 minutes of queue wait is fine
  retries: 3,
})

A worker emits an automatic beat the moment it claims the task, and that first beat is what switches the window from pickupTimeoutMs to timeoutMs. So the two read as: pickupTimeoutMs is "how long may it stay queued", timeoutMs is "maximum silence while running".

It defaults to timeoutMs, so leaving it off preserves the single-window behaviour. It is only consulted when timeoutMs is set — on a step with no timeoutMs there is no in-memory window at all, and pickupTimeoutMs does nothing.

An expiry of either window raises the same retryable RemoteStepTimeout, so retries applies identically to both. To make either window meaningful, have the handler call log.heartbeat(progress?) as it works.

A third case: the dispatch itself gets lostCopy Anchor Link

Neither path above covers a step whose job never produces a result at all: no failure, and no timeoutMs window to expire, because the worker crashed mid-step or the transport dropped the job (a Redis flush or eviction, or a stalled job moved to failed and removed). The reconcile re-drive is deliberately conservative here: it re-suspends a still-pending step rather than re-dispatching it, so a merely slow worker is never double-run. Without a timeoutMs, that means the run sits on pending until something re-dispatches it:

• The BullMQ transport closes this gap for its own consumers: when a crashed or stalled task job ends failed, the worker publishes a synthetic, retryable failed result for it, so the checkpoint fails and the durable retry path above takes over. A worker on another SDK only gets this if that SDK implements the same bridge.

• For any transport, engine.redispatchPending(runId) (or the dashboard's redispatch action) re-enqueues a run's stuck pending remote steps by hand.

• remoteRedispatchMs, an opt-in setting in config/durable.ts, makes this self-healing. A dispatched remote step carries a lease on its pending checkpoint (wakeAt), and the run suspends on it (never later than the reconcileMs wake would have woken it anyway; an early wake just re-parks on the unexpired lease). A result, or a step-scoped heartbeat, must land before the lease lapses; a heartbeat renews it, throttled to one write per half-window, so a worker beating every few seconds does not cause a write storm. A step fan's checkpoints are written with the lease already on them, and the run parks on it at send; the instance that sent the fan answers pickup beats from memory until the half-window. A lapsed lease re-dispatches the same stepId, bounded by remoteRedispatchMax (default 10), after which the step fails with a RemoteStepError (stepError.code: 'remote_step_lost') and reaches the workflow's own error path. It is off by default because a re-dispatch can double-run a step that is merely slow and does not beat: the window must exceed the longest legitimate duration of a non-heartbeating step, and the step must be idempotent.

  The same policy covers a polyglot workflow's call fan-out, not just ctx.step.

  A re-drive is observably different from a retry: the step.started event carries redispatched: true, and the checkpoint's own event trail gets a warn-level step.redispatched (step.lost when the bound runs out). Alert on those to spot a worker dying under you, as opposed to steps that merely fail.

export default defineConfig({
  // ...
  // Longer than your longest remote step. A step that heartbeats renews its own lease,
  // so this window only has to cover steps that go silent.
  remoteRedispatchMs: 45 * 60_000,
  remoteRedispatchMax: 3,
})

See Lost remote dispatch for the operator view, and Recovering a lost remote step dispatch for the dashboard side.