Dead-letter queue

Most failures are transient and recover cleanly. But occasionally a run is a poison pill: it crashes the process every time recovery picks it up — a deserialization bug, a non-deterministic change, an infinite loop in a step. Left alone, crash-recovery would resume it, crash, resume it again on the next boot, and loop forever, taking the whole instance down with it each time. The dead-letter queue breaks that loop.

Capping recovery: maxRecoveryAttemptsCopy Anchor Link

Crash-recovery is self-healing and periodic: while a run executes its worker renews a recovery lease, so a crashed worker's lease expires and recovery (engine.recoverIncomplete(), called every tick by the durable:work loop, or by the embedded loop of worker: { embedded: true }) reclaims the orphaned run within ~leaseMs — on any instance, not just on the next boot. That's what makes a genuine poison pill visible: it gets picked up again and again. Every time crash-recovery picks up a run that still reads running once it holds the lease (it re-reads the run under the lease, so a run that settled after the orphan listing is left alone and never counted), the engine increments that run's recoveryAttempts before resuming it (so a crash mid-resume still advances the counter).

A run whose remote step is still in flight is not counted. A free run lease doesn't mean anything crashed in that case: the step is on the transport with a worker that holds no run lease. Recovery parks such a run suspended on the step's deadline or the reconcileMs wake, and the worker's result resumes it. A slow remote step therefore can't burn the budget and be dead-lettered while its worker is still busy.

Set maxRecoveryAttempts in config/durable.ts, and once a run exceeds it (the pickup that would be attempt maxRecoveryAttempts + 1), instead of resuming yet again the engine moves the run to the terminal dead status with a max_recovery_attempts error, releases its lease, and stops touching it:

export default defineConfig({
  // … plus your transport/store (see Getting Started)
  maxRecoveryAttempts: 5, // after 5 crash-recoveries, dead-letter the run instead of looping
})

Omit maxRecoveryAttempts for unlimited recovery (the default). A dead run is terminal but not lost: it stays fully inspectable in the dashboard — its history, its checkpoints, its error — and it can be retried from there (or via durable:retry) once you've shipped a fix. A retry (engine.requeue) of a dead or failed run resets its recoveryAttempts to 0, so the retried run gets the full budget again. Requeueing a run that is still live keeps its count. The point is that one poison pill no longer keeps the process crashing for every other run.

Routing to a DLQ workflowCopy Anchor Link

The most useful thing to do with a dead run is hand it to a workflow of its own: a durable handler that gets all the reliability machinery (retries, steps, its own observability) to triage the failure. Declare it and the engine starts it for you.

An inline deadLetter() methodCopy Anchor Link

The shortest form lives on the workflow class itself. Define a deadLetter(ctx, dead) method on a BaseWorkflow:

import { inject } from '@adonisjs/core'
import { BaseWorkflow } from '@adonis-agora/durable'
import type { DeadLetter, WorkflowCtx } from '@adonis-agora/durable'
import AlertService from '#services/alert_service'
import TicketService from '#services/ticket_service'

@inject()
export default class PipelineWorkflow extends BaseWorkflow {
  static workflow = { name: 'pipeline', version: '1' }

  constructor(
    private alerts: AlertService,
    private tickets: TicketService,
  ) {
    super()
  }

  async run(ctx: WorkflowCtx, input: PipelineInput) {
    // …
  }

  async deadLetter(ctx: WorkflowCtx, dl: DeadLetter<PipelineInput>) {
    await ctx.step(this.alerts.page, { runId: dl.deadRunId, workflow: dl.workflow })
    const ticket = await ctx.step(this.tickets.create, {
      title: `Dead-lettered run ${dl.deadRunId} (${dl.workflow})`,
      payload: dl.input, // the original input, ready to replay after a fix
    })
    return { ticketId: ticket.id }
  }
}

Discovery registers the method as a separate workflow named <name>.dlq (here pipeline.dlq), with the same version and origin, and routes this workflow's dead runs to it. It runs as a durable workflow of its own (ctx is the handler run's context), and it shares the instance's constructor-injected services.

A deadLetterWorkflow referenceCopy Anchor Link

To point at a handler that lives elsewhere, or to share one across workflows, name it. deadLetterWorkflow takes a workflow name, a workflow class, or a ref, and is accepted in three places:

export default defineConfig({
  maxRecoveryAttempts: 5,
  deadLetterWorkflow: 'global-dlq', // fleet default for every workflow
})

export default class CheckoutWorkflow extends BaseWorkflow {
  static workflow = { name: 'checkout', version: '1', deadLetterWorkflow: CheckoutDlqWorkflow }
  // …
}

engine.register('import', '1', importFn, { deadLetterWorkflow: 'import-dlq' })
// remote workflows take it too: engine.registerRemote(name, version, { …, deadLetterWorkflow })

The handler is an ordinary workflow, so write it like any other (run(ctx, dl: DeadLetter)). The first match wins: the inline deadLetter() method, then the workflow's own deadLetterWorkflow, then the config default. A class that declares both a deadLetter() method and a deadLetterWorkflow option throws at registration ("declares both a deadLetter() method and a deadLetterWorkflow option").

What the handler run getsCopy Anchor Link

• Run id dlq:<deadRunId>. start is idempotent by run id, so a run that is dead-lettered (and re-detected) more than once still triggers exactly one handler run. The dashboard links the pair by that id (see below).
• Input DeadLetter, exported from @adonis-agora/durable: { deadRunId, workflow, input, error? } — which run died, of which workflow, with what input, and why (max_recovery_attempts). DeadLetter<TInput> types input.
• The dead run's namespace. Tags are not copied.
• Only dead runs are routed — runs past maxRecoveryAttempts, so it must be set. An ordinary failed run is onError's job.
• No chains. A handler run (a dlq: id) that itself goes dead is not routed again.
• A failed start is warned, not thrown. If the handler is not registered on the instance that dead-letters the run, the engine logs a console.warn and recovery carries on.

Handling dead runs manually: engine.onDeadCopy Anchor Link

Subscribe with engine.onDead to be notified the moment a run is dead-lettered. The listener receives the dead WorkflowRun (status dead, with its error), so you can page an on-call engineer or push to a real message queue:

import logger from '@adonisjs/core/services/logger'
import engine from '@adonis-agora/durable/services/main'

engine.onDead((run) => {
  logger.error(`run ${run.id} (${run.workflow}) dead-lettered: ${run.error?.message}`)
  void pager.alert('durable-poison-pill', { runId: run.id, workflow: run.workflow })
})

onDead returns an unsubscribe function. It fires on the instance that dead-letters the run, which is whichever worker's recovery pass picked the run up — so register the listener in a preload that every worker process loads (start/durable.ts), not only in the web process. It fires before the declared dead-letter workflow is started.

The listener is called synchronously and its return value is not awaited. A throw is swallowed so that a broken handler can't break recovery, and a rejected promise goes unobserved, so catch inside it (void pager.alert(...).catch(...)). The hook is in-memory, and so is the declarative routing above: if the process dies between the dead write and the call, nothing replays it. For a handler that must not be missed, alert on the run.failed event or on dead runs in the dashboard as well.

onDead stays the escape hatch for routing the declarative form can't express. To start a handler from it yourself, keep the dlq:<runId> id (and don't also declare a deadLetterWorkflow, since both would claim the same id):

import type { DeadLetter } from '@adonis-agora/durable'

const handlers: Record<string, string> = { pipeline: 'pipeline-dlq', checkout: 'checkout-dlq' }

engine.onDead((run) => {
  const input: DeadLetter = { deadRunId: run.id, workflow: run.workflow, input: run.input, error: run.error }
  void engine.start(handlers[run.workflow] ?? 'global-dlq', input, `dlq:${run.id}`)
})

Dead-lettering also goes through the engine's event stream: the transition emits a run.failed event whose error.code is max_recovery_attempts, so events and interceptors, OpenTelemetry and Telescope see a dead-lettered run as a failure without any extra wiring. The dead status and that error code are what tell it apart from an ordinary step failure.

The handler run is a normal durable run. It shows up in the dashboard with its own history, so you can confirm it did its job (alerted, opened a ticket, compensated), then retry the original once it's fixed.

1// config/durable.ts caps recovery: after 5 crash-recoveries → terminal 'dead'2//   defineConfig({ store, transport, maxRecoveryAttempts: 5 })3 4// route every dead run into a durable DLQ workflow — idempotent by dlq:<runId>5engine.onDead((run) =>6  engine.start('dlq', { deadRunId: run.id, input: run.input, error: run.error }, `dlq:${run.id}`),7)8 9engine.register('dlq', '1', async (ctx, dl: DeadLetter) => {10  await ctx.localStep('page', () => alerts.page({ runId: dl.deadRunId, error: dl.error?.message }))11  const ticket = await ctx.localStep('open-ticket', () => tickets.create({ payload: dl.input }))12  return { ticketId: ticket.id }13})

Dead runs in the dashboardCopy Anchor Link

The dashboard links the two ends of a dead-letter by the dlq:<runId> id:

• A run whose id starts with dlq: gets a red dlq chip in the runs list. Its detail page shows a Dead-letter handler banner that links back to the dead run it is handling.
• A dead run's detail page looks up dlq:<its id>. If that handler run exists, the page shows a Dead-lettered banner with an open DLQ handler link. If it doesn't, no link is shown.

From either end you can walk from a poison pill to its handler and back, check that the handler did its job, and retry the original once it's fixed.

Retrying a dead run with corrected inputCopy Anchor Link

When the cause was bad input, engine.retryWithInput(runId, input, newRunId?) starts a fresh run with the same workflow but corrected input — useful for replaying a dead run once you've fixed the data that poisoned it:

const { runId } = (await engine.retryWithInput(deadRunId, { ...originalInput, fixed: true }))!

It is a new run with a clean history (newRunId defaults to <runId>~retry~<8 hex chars>), started with the original run's tags and namespace; the dead run stays as it is, inspectable. It resolves to { runId }, or null when deadRunId is unknown. When the input was fine and the fix was a deploy, retry the run itself instead (engine.requeue(runId) / durable:retry), which replays its completed steps and resets recoveryAttempts.