Retries & backoff

Transient failures are the normal case for anything that crosses a network boundary, so every step is retryable. A step is always dispatched to a worker and the engine handles its failures along one durable retry path: the run suspends between attempts rather than holding a worker (or the orchestrator) in memory, so a step survives the worker — and even the orchestrator — disappearing mid-flight.

Retries & backoffCopy Anchor Link

A step's retry policy is declared on its @Step handler and applies wherever the step is dispatched. If the handler throws, the engine retries it up to retries attempts, spacing the attempts with the step's backoff configuration:

@Step({ retries: 5, backoff: 'exp', backoffMs: 200, backoffMaxMs: 10_000, jitter: true })
async fetchQuote(order: Order): Promise<Quote> {
  return this.pricing.fetch(order);
}

// in the workflow — the declared policy applies:
const quote = await ctx.step(this.pricing.fetchQuote, order);

1@Workflow({ name: 'checkout', version: '1' })2export class CheckoutWorkflow {3  constructor(4    private readonly pricing: PricingService,5    private readonly payments: PaymentsService,6    private readonly email: EmailService,7  ) {}8 9  async run(ctx: WorkflowCtx, order: Order) {10    const quote = await ctx.step(this.pricing.fetchQuote, order);11    const charge = await ctx.step(this.payments.chargeCard, order);12    await ctx.step(this.email.confirm, { order, charge }, { retries: 5 });13    return charge.id;14  }15}

And as a live system — watch the backoff double after each failed attempt while the run sits suspended:

The options are:

• retries — maximum number of attempts before the step (and the run) fails. Defaults to 1 (a single try).
• backoff — how the delay between attempts grows: 'fixed' keeps it constant, 'exp' doubles it each attempt.
• backoffMs — the base delay in ms. Omit (or set to 0) to retry with no delay at all.
• backoffMaxMs — an upper bound, so an exponential backoff doesn't grow without limit.
• jitter — adds random jitter (50–100% of the computed delay) so a fleet of runs retrying the same downstream don't synchronize into a thundering herd.

Every one of these is also a per-call override on ctx.step(handler, input, opts): the effective policy is the @Step-declared one with any opts field winning, so a single call site can tighten or relax the default without changing the handler.

With backoff: 'exp', backoffMs: 200 the delays before attempts 2, 3, 4… are 200ms, 400ms, 800ms… until they hit backoffMaxMs. With jitter: true each of those is then scaled to a random point in its top half, so two runs that failed at the same instant don't retry in lockstep.

FatalError — never retriedCopy Anchor Link

Not every failure is worth retrying. A declined card or invalid input will fail the same way on every attempt, so burning the retry budget on it just delays the inevitable. Throw a FatalError inside the handler to fail the run immediately, regardless of the step's retries:

import { FatalError } from '@dudousxd/nestjs-durable-core';

@Step()
async charge(order: Order) {
  const res = await this.stripe.charge(order);
  if (res.declined) throw new FatalError('card declined', 'declined');
  return res;
}

The optional second argument is a machine-readable code (here 'declined') that ends up on the run's structured error. Use FatalError for deterministic business verdicts; let ordinary throws (a timeout, a 502, a dropped connection) flow through the retry path.

The durable retry pathCopy Anchor Link

A step runs on a worker — possibly in another process or another language — so the engine handles its failures durably rather than by re-running a function in a loop. When a step has no timeoutMs, the engine dispatches the task, persists a pending checkpoint, and suspends the run durably — it is not held in memory awaiting the result. Whichever instance receives the worker's result resumes the run, so the worker pod (and even the dispatching orchestrator) can scale down or crash mid-step without losing the run or re-running completed work.

@Injectable()
export class PaymentsService {
  @Step({
    name: 'payments.charge-card',
    input: z.object({ orderId: z.string(), amountCents: z.number().int() }),
    output: z.object({ chargeId: z.string() }),
    retries: 4,
    backoff: 'exp',
    backoffMs: 500,
    backoffMaxMs: 30_000,
    jitter: true,
  })
  async chargeCard(input: { orderId: string; amountCents: number }): Promise<{ chargeId: string }> {
    return { chargeId: await this.stripe.charge(input) };
  }
}

// in the workflow:
const charge = await ctx.step(this.payments.chargeCard, { orderId: order.id, amountCents: order.total });

When a worker reports a failed result for a step, the engine consults retries. If the attempt budget remains, it re-dispatches — but durably, not in a loop. The first time it sees the failed checkpoint it computes the next retry deadline as now + backoffDelay(attempt) and stamps it on the failed checkpoint's wakeAt, in clock-space, then suspends. Because that deadline is persisted on the checkpoint rather than living in a timer in memory, it is replay-stable (a resume recomputes the same decision) and crash-safe (a process that dies before the retry fires picks it back up when the timer poller sees the wakeAt come due). Once the deadline passes, the poller resumes the run, the call re-dispatches with an incremented attempt, and the cycle continues until the result lands or retries is exhausted.

Opting out: retryable: falseCopy Anchor Link

A dispatched step retries on a failed worker result unless the worker marks the error as non-retryable. A worker that reports an error has returned a deterministic verdict — a declined card, a validation failure — so re-dispatching it just hammers the worker for the same answer. Set retryable: false on the StepError the worker returns and the engine surfaces it to the workflow immediately instead of retrying:

// inside the worker handler (TypeScript or, symmetrically, the Python SDK):
@Step('payments.charge-card')
async charge(input: { orderId: string; amountCents: number }) {
  const res = await this.stripe.charge(input);
  if (res.declined) {
    // a deterministic verdict — don't make the engine retry it
    throw Object.assign(new Error('card declined'), { code: 'declined', retryable: false });
  }
  return { chargeId: res.id };
}

The default is retryable !== false, i.e. an ordinary error is retried; only an explicit retryable: false opts out. This is the worker-side counterpart of throwing FatalError in a step handler.

The in-memory liveness path (timeoutMs)Copy Anchor Link

Setting timeoutMs on a step opts it into a different path entirely. timeoutMs is a liveness window: if the worker produces neither a result nor a heartbeat within that many ms, the engine presumes it dead, fails the dispatch with a RemoteStepTimeout, and — because that timeout is retryable — re-dispatches it in-place up to retries. Each heartbeat the worker emits (via transport.onHeartbeat) rearms the window, so a long but healthy step that keeps beating stays alive well past timeoutMs.

@Step({
  name: 'media.render',
  input: z.object({ assetId: z.string() }),
  output: z.object({ url: z.string() }),
  timeoutMs: 60_000, // presume the worker dead after 60s of silence (no result, no heartbeat)
  retries: 3,
})
async render(input: { assetId: string }): Promise<{ url: string }> {
  return { url: await this.encoder.render(input.assetId) };
}

The crucial difference between the two paths: the durable path retries on a worker reporting a failure and suspends between attempts (crash-safe, not in memory); the liveness path retries on a worker going silent and awaits the result in memory between attempts. They also disagree on what a reported error means — the durable path retries it (subject to retryable), while a timeoutMs step only re-dispatches on a timeout, surfacing any reported error to the workflow on the first attempt. Reach for timeoutMs only when you genuinely need to detect and replace a stuck worker; otherwise leave it off and get the durable, crash-safe path.

A third case: the dispatch itself gets lostCopy Anchor Link

Neither path above covers a step whose job never produces a result at all — no failure, no timeoutMs heartbeat to time out — because the worker crashed mid-step or the transport dropped the job (a Redis flush/eviction, or a stalled BullMQ job moved to failed and removed). Reconcile re-drives are deliberately conservative here: they re-suspend a still-pending step rather than re-dispatching it, so a merely-slow worker is never double-run. Without a timeoutMs, that means the run hangs on pending forever unless something re-dispatches it:

• The BullMQ transport closes this gap for the consumers that implement the bridge: a crashed/stalled task job's terminal failure is turned into a synthetic failed StepResult, so the checkpoint fails and the normal durable retry path above kicks in. The TypeScript transport and the Python durable-worker both do this; a worker on any other SDK (or an older durable-worker) does not, and its steps stay pending.

• For any transport, engine.redispatchPending(runId) (or the dashboard's Re-dispatch action) manually re-enqueues a run's stuck pending remote steps.

• remoteRedispatchMs — an opt-in engine option — makes this self-healing: a dispatched remote step carries a lease on its pending checkpoint (wakeAt), and the run suspends on it (never later than the reconcileMs sweep would have woken it anyway). A result — or a step-scoped heartbeat, which renews the lease (throttled to one write per half-window, so a worker beating every couple of seconds is not an UPDATE storm) — must land before it lapses; a lapsed lease re-dispatches the same stepId, bounded by remoteRedispatchMax (default 10) before failing the step as remote_step_lost (which reaches the workflow's own error path). Off by default, because re-dispatch can double-run a step whose job is merely slow and does not beat: the window must exceed the longest legitimate duration of a non-heartbeating step, and the step must be idempotent.

  This covers both shapes of remote step: an in-process ctx.step and a polyglot workflow's call fan-out (ctx.gather_calls). Before @dudousxd/nestjs-durable-core 0.71, the fan-out path ignored the option entirely — a lost job there was skipped on every subsequent turn and the run waited forever. If you wire the engine through DurableModule, note that the option is only forwarded from @dudousxd/nestjs-durable 0.71 as well.

  A re-drive is observably different from a retry: the step.started event carries redispatched: true, and the checkpoint's own event trail gets a warn step.redispatched (step.lost when the bound runs out). Alert on those to see a worker that is dying under you rather than steps that merely fail.

DurableModule.forRoot({
  // Longer than your longest remote step. A step that heartbeats renews its own lease, so this
  // window only ever has to cover steps that go silent.
  remoteRedispatchMs: 45 * 60_000,
  remoteRedispatchMax: 3,
});

See Recovering a lost remote step dispatch for the dashboard side of this (the stale-pending flag and the Re-dispatch button).