Retries & backoff

Transient failures are the normal case for anything that crosses a network boundary, so every step is retryable. A step is always dispatched to a worker and the engine handles its failures along one durable retry path: the run suspends between attempts rather than holding a worker (or the orchestrator) in memory, so a step survives the worker — and even the orchestrator — disappearing mid-flight.

Retries & backoff

A step's retry policy is declared on its @Step handler and applies wherever the step is dispatched. If the handler throws, the engine retries it up to retries attempts, spacing the attempts with the step's backoff configuration:

@Step({ retries: 5, backoff: 'exp', backoffMs: 200, backoffMaxMs: 10_000, jitter: true })
async fetchQuote(order: Order): Promise<Quote> {
  return this.pricing.fetch(order);
}

// in the workflow — the declared policy applies:
const quote = await ctx.step(this.pricing.fetchQuote, order);

1@Workflow({ name: 'checkout', version: '1' })2export class CheckoutWorkflow {3  constructor(4    private readonly pricing: PricingService,5    private readonly payments: PaymentsService,6    private readonly email: EmailService,7  ) {}8 9  async run(ctx: WorkflowCtx, order: Order) {10    const quote = await ctx.step(this.pricing.fetchQuote, order);11    const charge = await ctx.step(this.payments.chargeCard, order);12    await ctx.step(this.email.confirm, { order, charge }, { retries: 5 });13    return charge.id;14  }15}

And as a live system — watch the backoff double after each failed attempt while the run sits suspended:

The options are:

• retries — maximum number of attempts before the step (and the run) fails. Defaults to 1 (a single try).
• backoff — how the delay between attempts grows: 'fixed' keeps it constant, 'exp' doubles it each attempt.
• backoffMs — the base delay in ms. Omit (or set to 0) to retry with no delay at all.
• backoffMaxMs — an upper bound, so an exponential backoff doesn't grow without limit.
• jitter — adds random jitter (50–100% of the computed delay) so a fleet of runs retrying the same downstream don't synchronize into a thundering herd.

Every one of these is also a per-call override on ctx.step(handler, input, opts): the effective policy is the @Step-declared one with any opts field winning, so a single call site can tighten or relax the default without changing the handler.

With backoff: 'exp', backoffMs: 200 the delays before attempts 2, 3, 4… are 200ms, 400ms, 800ms… until they hit backoffMaxMs. With jitter: true each of those is then scaled to a random point in its top half, so two runs that failed at the same instant don't retry in lockstep.

FatalError — never retried

Not every failure is worth retrying. A declined card or invalid input will fail the same way on every attempt, so burning the retry budget on it just delays the inevitable. Throw a FatalError inside the handler to fail the run immediately, regardless of the step's retries:

import { FatalError } from '@dudousxd/nestjs-durable-core';

@Step()
async charge(order: Order) {
  const res = await this.stripe.charge(order);
  if (res.declined) throw new FatalError('card declined', 'declined');
  return res;
}

The optional second argument is a machine-readable code (here 'declined') that ends up on the run's structured error. Use FatalError for deterministic business verdicts; let ordinary throws (a timeout, a 502, a dropped connection) flow through the retry path.

The durable retry path

A step runs on a worker — possibly in another process or another language — so the engine handles its failures durably rather than by re-running a function in a loop. When a step has no timeoutMs, the engine dispatches the task, persists a pending checkpoint, and suspends the run durably — it is not held in memory awaiting the result. Whichever instance receives the worker's result resumes the run, so the worker pod (and even the dispatching orchestrator) can scale down or crash mid-step without losing the run or re-running completed work.

@Injectable()
export class PaymentsService {
  @Step({
    name: 'payments.charge-card',
    input: z.object({ orderId: z.string(), amountCents: z.number().int() }),
    output: z.object({ chargeId: z.string() }),
    retries: 4,
    backoff: 'exp',
    backoffMs: 500,
    backoffMaxMs: 30_000,
    jitter: true,
  })
  async chargeCard(input: { orderId: string; amountCents: number }): Promise<{ chargeId: string }> {
    return { chargeId: await this.stripe.charge(input) };
  }
}

// in the workflow:
const charge = await ctx.step(this.payments.chargeCard, { orderId: order.id, amountCents: order.total });

When a worker reports a failed result for a step, the engine consults retries. If the attempt budget remains, it re-dispatches — but durably, not in a loop. The first time it sees the failed checkpoint it computes the next retry deadline as now + backoffDelay(attempt) and stamps it on the failed checkpoint's wakeAt, in clock-space, then suspends. Because that deadline is persisted on the checkpoint rather than living in a timer in memory, it is replay-stable (a resume recomputes the same decision) and crash-safe (a process that dies before the retry fires picks it back up when the timer poller sees the wakeAt come due). Once the deadline passes, the poller resumes the run, the call re-dispatches with an incremented attempt, and the cycle continues until the result lands or retries is exhausted.

Opting out: retryable: false

A dispatched step retries on a failed worker result unless the worker marks the error as non-retryable. A worker that reports an error has returned a deterministic verdict — a declined card, a validation failure — so re-dispatching it just hammers the worker for the same answer. Set retryable: false on the StepError the worker returns and the engine surfaces it to the workflow immediately instead of retrying:

// inside the worker handler (TypeScript or, symmetrically, the Python SDK):
@Step('payments.charge-card')
async charge(input: { orderId: string; amountCents: number }) {
  const res = await this.stripe.charge(input);
  if (res.declined) {
    // a deterministic verdict — don't make the engine retry it
    throw Object.assign(new Error('card declined'), { code: 'declined', retryable: false });
  }
  return { chargeId: res.id };
}

The default is retryable !== false, i.e. an ordinary error is retried; only an explicit retryable: false opts out. This is the worker-side counterpart of throwing FatalError in a step handler.

The in-memory liveness path (timeoutMs)

Setting timeoutMs on a step opts it into a different path entirely. timeoutMs is a liveness window: if the worker produces neither a result nor a heartbeat within that many ms, the engine presumes it dead, fails the dispatch with a RemoteStepTimeout, and — because that timeout is retryable — re-dispatches it in-place up to retries. Each heartbeat the worker emits (via transport.onHeartbeat) rearms the window, so a long but healthy step that keeps beating stays alive well past timeoutMs.

@Step({
  name: 'media.render',
  input: z.object({ assetId: z.string() }),
  output: z.object({ url: z.string() }),
  timeoutMs: 60_000, // presume the worker dead after 60s of silence (no result, no heartbeat)
  retries: 3,
})
async render(input: { assetId: string }): Promise<{ url: string }> {
  return { url: await this.encoder.render(input.assetId) };
}

The crucial difference between the two paths: the durable path retries on a worker reporting a failure and suspends between attempts (crash-safe, not in memory); the liveness path retries on a worker going silent and awaits the result in memory between attempts. They also disagree on what a reported error means — the durable path retries it (subject to retryable), while a timeoutMs step only re-dispatches on a timeout, surfacing any reported error to the workflow on the first attempt. Reach for timeoutMs only when you genuinely need to detect and replace a stuck worker; otherwise leave it off and get the durable, crash-safe path.

A third case: the dispatch itself gets lost

Neither path above covers a step whose job never produces a result at all — no failure, no timeoutMs heartbeat to time out — because the worker crashed mid-step or the transport dropped the job (a Redis flush/eviction, or a stalled BullMQ job moved to failed and removed). Reconcile re-drives are deliberately conservative here: they re-suspend a still-pending step rather than re-dispatching it, so a merely-slow worker is never double-run. Without a timeoutMs, that means the run hangs on pending forever unless something re-dispatches it:

• The BullMQ transport closes this gap automatically: a crashed/stalled task job's terminal failure is bridged into a synthetic failed StepResult, so the checkpoint fails and the normal durable retry path above kicks in — no configuration needed.
• For any transport, engine.redispatchPending(runId) (or the dashboard's Re-dispatch action) manually re-enqueues a run's stuck pending remote steps.
• remoteRedispatchMs — an opt-in engine option — makes this self-healing: a reconcile re-drive that finds a remote step still pending past the window re-dispatches it, bounded by remoteRedispatchMax (default 10) before giving up and failing the step as remote_step_lost. Off by default, because re-dispatch can double-run a step whose job is merely slow — the window must exceed the longest legitimate duration and the step must be idempotent.

See Recovering a lost remote step dispatch for the dashboard side of this (the stale-pending flag and the Re-dispatch button).