Dead-letter queue

Most failures are transient and recover cleanly. But occasionally a run is a poison pill: it crashes the process every time recovery picks it up — a deserialization bug, a non-deterministic change, an infinite loop in a step. Left alone, crash-recovery would resume it, crash, resume it again on the next boot, and loop forever, taking the whole instance down with it each time. The dead-letter queue breaks that loop.

The walkthrough below follows one poison-pill run the whole way: it crashes, crash-recovery keeps resuming it until it hits maxRecoveryAttempts, the engine moves it to the terminal dead status — and then a @DeadLetter() handler picks it up as a durable run of its own to page and file a ticket. Note the two lanes: the poisoned run ends on its lane — dead, in red — while the DLQ handler is a separate run living on the lane below.

1// a poison-pill run, and the handler that catches it2@Workflow({ name: 'pipeline', version: '3' })3export class PipelineWorkflow {4  constructor(5    private readonly alerts: AlertsService,6    private readonly tickets: TicketService,7  ) {}8 9  async run(ctx: WorkflowCtx, input: PipelineInput) {10    const extracted = await ctx.step('pipeline.extract', input);11    // a deserialization bug here throws on every recovery — a poison pill12    return ctx.step('pipeline.transform', extracted);13  }14 15  // Auto-registered as 'pipeline.dlq' — a durable workflow of its own16  @DeadLetter()17  async onDead(ctx: WorkflowCtx, dl: DeadLetter<PipelineInput>) {18    await ctx.step(this.alerts.page, { runId: dl.deadRunId, error: dl.error?.message });19    const ticket = await ctx.step(this.tickets.create, { payload: dl.input });20    return { ticketId: ticket.id };21  }22}

And as a live system — inject a poison pill and watch it crash-loop, get dead-lettered, and hand off to the DLQ handler while the healthy traffic never stops:

Capping recovery: maxRecoveryAttempts

Crash-recovery is self-healing and periodic: while a run executes its worker renews a recovery lease, so a crashed worker's lease expires and recovery (engine.recoverIncomplete(), called every tick by the TimerPoller) reclaims the orphaned run within ~leaseMs — on any instance, not just on the next boot. That's what makes a genuine poison pill visible: it gets picked up again and again. Every time crash-recovery picks up a still-running run, the engine increments that run's recoveryAttempts before resuming it (so a crash mid-resume still advances the counter). Set maxRecoveryAttempts — an engine/module option — and once a run exceeds it, instead of resuming yet again the engine moves the run to the terminal dead status with a max_recovery_attempts error, releases its lease, and stops touching it:

DurableModule.forRoot({
  store,
  transport,
  maxRecoveryAttempts: 5, // after 5 crash-recoveries, dead-letter the run instead of looping
});

Omit maxRecoveryAttempts for unlimited recovery (the default). A dead run is terminal but not lost: it stays fully inspectable in the dashboard — its history, its checkpoints, its error — and it can be retried from there once you've shipped a fix. The point is that one poison pill no longer keeps the process crashing for every other run.

Handling dead runs: engine.onDead

Parking a dead run is the floor, not the ceiling. Subscribe with engine.onDead to be notified the moment a run is dead-lettered — the listener receives the dead WorkflowRun (status dead, with its error) — so you can do something active: page an on-call engineer, push to a real message queue, or kick off a workflow that handles it.

@Injectable()
export class DeadLetterAlerts implements OnModuleInit {
  private readonly logger = new Logger(DeadLetterAlerts.name);

  constructor(
    private readonly engine: WorkflowEngine,
    private readonly pager: PagerService,
  ) {}

  onModuleInit() {
    this.engine.onDead((run) => {
      this.logger.error(`run ${run.id} (${run.workflow}) dead-lettered: ${run.error?.message}`);
      void this.pager.alert('durable-poison-pill', { runId: run.id, workflow: run.workflow });
    });
  }
}

onDead returns an unsubscribe function. It fires on the instance that dead-letters the run.

Routing to a DLQ workflow

The most powerful option is to route dead runs into a workflow of their own — a durable handler that gets all the reliability machinery (retries, steps, its own observability) to triage the failure. The handler is started with a typed DeadLetter<TInput> payload, idempotent by a dlq:<runId> id so a run is never double-handled:

import type { DeadLetter } from '@dudousxd/nestjs-durable';

interface DeadLetter<TInput = unknown> {
  deadRunId: string; // the dead run — inspectable + retriable in the dashboard
  workflow: string; // the workflow whose run died
  input: TInput; // the original input it was started with
  error?: StepError; // the failure that moved it to `dead`
}

Dead-lettering is naturally per-workflow — a dead checkout run wants a different handler than a dead pipeline run — so the target is resolved per dead run, in this order:

1. an inline @DeadLetter() method on the workflow class — the co-located default;
2. the workflow's @Workflow({ deadLetterWorkflow }) reference to another registered workflow;
3. the module-level deadLetterWorkflow default, for everything that declares neither.

Inline @DeadLetter() — one file

The simplest case: a method on the workflow class, marked @DeadLetter(). It runs as a durable workflow in its own right (auto-registered as <name>.dlq, with checkpointing and a dashboard run), and it shares the class's injected dependencies — so the handler lives right next to the workflow it protects:

@Workflow({ name: 'pipeline', version: '3' })
export class PipelineWorkflow {
  constructor(
    private readonly alerts: AlertsService,
    private readonly tickets: TicketService,
  ) {}

  async run(ctx: WorkflowCtx, input: PipelineInput) {
    const extracted = await ctx.step('pipeline.extract', input);
    const transformed = await ctx.step('pipeline.transform', extracted);
    return ctx.step('pipeline.load', transformed);
  }

  // Runs when a `pipeline` run is dead-lettered. Typed to the workflow's own input, same deps.
  @DeadLetter()
  async onDead(ctx: WorkflowCtx, dl: DeadLetter<PipelineInput>) {
    await ctx.step(this.alerts.page, {
      event: 'durable-dead-letter',
      runId: dl.deadRunId,
      workflow: dl.workflow,
      error: dl.error?.message,
    });

    const ticket = await ctx.step(this.tickets.create, {
      title: `Dead-lettered run ${dl.deadRunId} (${dl.workflow})`,
      body: dl.error?.message ?? 'unknown error',
      payload: dl.input, // typed PipelineInput, ready to replay after a fix
    });

    return { ticketId: ticket.id };
  }
}

@Workflow({ deadLetterWorkflow }) — a separate workflow

When the DLQ handler is large, reused across workflows, or just deserves its own file, point at another registered workflow — by class (refactor-safe) or by name (for a cross-runtime handler). Register it as a normal provider like any other workflow:

@Workflow({ name: 'pipeline', version: '3', deadLetterWorkflow: PipelineDlqWorkflow })
export class PipelineWorkflow {
  /* ...run... */
}

@Workflow({ name: 'pipeline-dlq', version: '1' })
export class PipelineDlqWorkflow {
  async run(ctx: WorkflowCtx, dl: DeadLetter) {
    /* same handler body as above */
  }
}

Module-level default

Set the module's deadLetterWorkflow to catch every workflow that declares neither of the above — a single backstop DLQ for the whole app:

DurableModule.forRoot({
  store,
  transport,
  maxRecoveryAttempts: 5,
  deadLetterWorkflow: 'global-dlq', // fallback for workflows without their own
});

A workflow that declares both an inline @DeadLetter() method and a deadLetterWorkflow reference is a configuration error — two targets for one workflow is ambiguous, so the module fails fast at boot rather than silently picking one. Pick one.

Because every start is idempotent by dlq:<runId>, a run that gets dead-lettered (and re-detected) more than once still triggers exactly one DLQ run.

The dashboard links the two ends of this relationship: from a dead run you can see the DLQ run it spawned, and the DLQ run carries the originating deadRunId — so you can walk from a poison pill to its handler and back, retry the original once it's fixed, and confirm the DLQ run did its job.