Retries & backoff

Transient failures are the normal case for anything that crosses a network boundary, so every step is retryable. The mechanics differ between an in-process ctx.localStep — work that runs inside the engine — and a dispatched ctx.step routed to a worker, because the engine can re-run a local function in place but a dispatched step has to survive the worker (and even the orchestrator) disappearing mid-flight.

Local-step retries

ctx.localStep(name, fn, opts?) runs a unit of work in-process and checkpoints its result. If fn throws, the engine retries it up to retries attempts, spacing the attempts with the step's backoff configuration:

const quote = await ctx.localStep(
  'quote',
  () => pricing.fetch(order),
  { retries: 5, backoff: 'exp', backoffMs: 200, backoffMaxMs: 10_000, jitter: true },
)

The StepOptions retry fields are:

• retries — maximum number of attempts before the step (and the run) fails. Defaults to 1 (a single try).
• backoff — how the delay between attempts grows: 'fixed' keeps it constant, 'exp' doubles it each attempt.
• backoffMs — the base delay in ms. Omit (or set to 0) to retry with no delay.
• backoffMaxMs — an upper bound, so an exponential backoff doesn't grow without limit.
• jitter — adds random jitter so a fleet of runs retrying the same downstream don't synchronize into a thundering herd.
• timeoutMs — bound a single attempt; on a local step it caps how long fn may run.

With backoff: 'exp', backoffMs: 200 the delays before attempts 2, 3, 4… are 200ms, 400ms, 800ms… until they hit backoffMaxMs. With jitter: true each of those is scaled to a random point in its top half.

FatalError — never retried

Not every failure is worth retrying. A declined card or invalid input will fail the same way on every attempt. Throw a FatalError to fail the run immediately, regardless of the step's retries:

import { FatalError } from '@adonis-agora/durable'

await ctx.localStep('charge', async () => {
  const res = await stripe.charge(order)
  if (res.declined) throw new FatalError('card declined', 'declined')
  return res
})

The optional second argument is a machine-readable code (here 'declined') that ends up on the run's structured error. Use FatalError for deterministic business verdicts; let ordinary throws (a timeout, a 502, a dropped connection) flow through the retry path.

Dispatched-step retries

A dispatched step (ctx.step) runs on a worker — possibly in another process. There are two distinct paths for handling its failures, chosen by whether the step sets timeoutMs.

The durable path (no timeoutMs)

This is the default and the recommended one. When you dispatch a step that has no timeoutMs, the engine dispatches the task, persists a pending checkpoint, and suspends the run durably — it is not held in memory awaiting the result. Whichever instance receives the worker's result resumes the run, so the worker (and even the dispatching orchestrator) can scale down or crash mid-step without losing the run or re-running completed work.

Declare the durable-retry policy on the step's @Step/defineStep config (or override it per call in the ctx.step options):

export const chargeCard = defineStep(
  'payments:charge-card',
  async (input: { orderId: number; amountCents: number }) => ({ chargeId: await stripe.charge(input) }),
  {
    input: z.object({ orderId: z.number().int(), amountCents: z.number().int() }),
    output: z.object({ chargeId: z.string() }),
    retries: 4,
    backoff: 'exp',
    backoffMs: 500,
    backoffMaxMs: 30_000,
    jitter: true,
  },
)

// in the workflow:
const charge = await ctx.step(chargeCard, { orderId: order.id, amountCents: order.total })

When a worker reports a failed result for a durable step, the engine consults retries. If the budget remains, it re-dispatches — but durably, not in a loop. It computes the next retry deadline as now + backoffDelay(attempt) and stamps it on the failed checkpoint's wakeAt, then suspends. Because that deadline is persisted on the checkpoint rather than living in a timer in memory, it is replay-stable and crash-safe (a process that dies before the retry fires picks it back up when the timer poller sees the wakeAt come due). Once the deadline passes, the poller resumes the run, the call re-dispatches with an incremented attempt, and the cycle continues until the result lands or retries is exhausted.

1@Workflow({ name: 'checkout', version: '1' })2export default class CheckoutWorkflow {3  constructor(4    private pricing: PricingSteps,5    private payments: PaymentSteps,6    private email: EmailSteps,7  ) {}8 9  async run(ctx: WorkflowCtx, order: Order) {10    const quote = await ctx.step(this.pricing.fetchQuote, order)11    const charge = await ctx.step(this.payments.chargeCard, order)12    await ctx.step(this.email.confirm, { order, charge }, { retries: 5 })13    return charge.id14  }15}

Opting out: retryable: false

A durable remote step retries on a failed worker result unless the worker marks the error as non-retryable. A worker that reports an error has returned a deterministic verdict — a declined card, a validation failure — so re-dispatching it just hammers the worker for the same answer. Throw an error with retryable: false and the engine surfaces it to the workflow immediately instead of retrying:

// inside the worker handler:
transport.handle('payments:charge-card', async (input) => {
  const res = await stripe.charge(input)
  if (res.declined) {
    // a deterministic verdict — don't make the engine retry it
    throw Object.assign(new Error('card declined'), { code: 'declined', retryable: false })
  }
  return { chargeId: res.id }
})

The default is retryable !== false, i.e. an ordinary error is retried; only an explicit retryable: false opts out. This is the worker-side counterpart of throwing FatalError in a local step.

The in-memory liveness path (timeoutMs)

Setting timeoutMs on a dispatched step opts it into a different path. timeoutMs is a liveness window: if the worker produces neither a result nor a heartbeat within that many ms, the engine presumes it dead, fails the dispatch with a RemoteStepTimeout, and — because that timeout is retryable — re-dispatches it up to retries. Each heartbeat the worker emits (via transport.onHeartbeat) rearms the window, so a long but healthy step that keeps beating stays alive well past timeoutMs.

export const renderVideo = defineStep(
  'media:render',
  async (input: { assetId: string }) => ({ url: await render(input) }),
  {
    input: z.object({ assetId: z.string() }),
    output: z.object({ url: z.string() }),
    timeoutMs: 60_000, // presume the worker dead after 60s of silence (no result, no heartbeat)
    retries: 3,
  },
)

The crucial difference between the two paths: the durable path retries on a worker reporting a failure and suspends between attempts (crash-safe, not in memory); the liveness path retries on a worker going silent and awaits the result in memory between attempts. Reach for timeoutMs only when you genuinely need to detect and replace a stuck worker; otherwise leave it off and get the durable, crash-safe path.

Queue wait vs. silence while running — pickupTimeoutMs

A single window has to cover two very different stretches of a step's life, and they want very different numbers:

1. Waiting in the queue. Nobody has claimed the job yet, so nobody is beating for it. During a backlog this can legitimately run into minutes.
2. Running on a worker. Somebody owns it and is expected to beat. Silence here is genuinely suspicious within seconds.

Set timeoutMs tight enough to catch a wedged worker and a queue backlog trips it; set it loose enough to survive the backlog and a wedged worker goes unnoticed for just as long. pickupTimeoutMs splits the two:

await ctx.step(harvestBatch, input, {
  timeoutMs: 30_000,        // once running: 30s of silence means the worker is gone
  pickupTimeoutMs: 600_000, // before pickup: 10 minutes of queue wait is fine
  retries: 3,
})

A worker emits an automatic beat the moment it claims the task, and that first beat is what switches the window from pickupTimeoutMs to timeoutMs. So the two read as: pickupTimeoutMs is "how long may it stay queued", timeoutMs is "maximum silence while running".

It defaults to timeoutMs, so leaving it off preserves the single-window behaviour. It is only consulted when timeoutMs is set — on a step with no timeoutMs there is no in-memory window at all, and pickupTimeoutMs does nothing.

An expiry of either window raises the same retryable RemoteStepTimeout, so retries and backoff apply identically. To make either window meaningful, have the handler call log.heartbeat(progress?) as it works.