Sagas & compensation

A durable run often performs several irreversible side effects in sequence — reserve inventory, charge a card, allocate a shipment. If a later step fails, the earlier effects are still out there in the world, and "the run failed" is not an acceptable end state when money has moved. The saga pattern handles this: alongside each step that does something, you register how to undo it, and when the run fails the engine runs those undos in reverse order. @adonis-agora/durable builds this in via a compensate option — a closure on ctx.localStep for in-process work, or a step ref on the dispatched ctx.step.

Local compensation — a closure on ctx.localStepCopy Anchor Link

Attach a compensate callback to an in-process ctx.localStep. The callback is registered when the step completes; if the run later fails, the engine runs every registered compensation in reverse order (last completed step undone first), restoring the world before failing the run.

import engine from '@adonis-agora/durable/services/main'

engine.register('checkout', '1', async (ctx, order: Order) => {
  const reservation = await ctx.localStep(
    'reserve-inventory',
    () => inventory.reserve(order.items),
    { compensate: () => inventory.release(order.items) },
  )

  const charge = await ctx.localStep(
    'charge-card',
    () => payments.charge(order.customerId, order.totalCents),
    { compensate: () => payments.refund(order.customerId, order.totalCents) },
  )

  // If this step throws, the engine runs the two compensations above in reverse:
  // refund the card, then release the inventory — then fails the run.
  const label = await ctx.localStep('allocate-shipment', () => shipping.allocate(order, reservation.id))

  return { chargeId: charge.id, tracking: label.tracking }
})

If allocate-shipment throws (and exhausts its retries, or throws a FatalError), the run fails — but before it does, the engine refunds the card and releases the inventory. The saga is reconstructed from the run's history on replay, so it works correctly even after a crash: the steps that completed are the ones whose compensations get registered.

Concretely: on resume, a completed localStep checkpoint returns its saved output without re-running fn — but its compensate closure is still pushed onto the compensation stack, exactly as it was the first time. That is what makes registration crash-safe: after a mid-run crash, the resumed execution rebuilds the full stack of undos for every step that had actually completed, not just the ones this pass happened to run.

Dispatched compensation — a step ref on ctx.stepCopy Anchor Link

A dispatched ctx.step undoes itself with a compensate step ref — a @Step/defineStep handler (or its name) that runs as its own dispatched step at unwind time. The undo handler receives a StepUndo envelope — { input, output } — of the step it compensates, so it can reverse exactly what ran:

import { Step } from '@adonis-agora/durable'
import type { UndoOf } from '@adonis-agora/durable'

export default class BookingSteps {
  @Step('flights:book')
  async book(input: { flightId: string; pax: number }) {
    return { bookingRef: await airline.book(input) }
  }

  // The undo — typed to the step it compensates via UndoOf<...>.
  @Step('flights:cancel')
  async cancel(undo: UndoOf<BookingSteps['book']>) {
    await airline.cancel(undo.output.bookingRef)
  }
}

// in the workflow (step class injected as `this.booking`):
const booking = await ctx.step(this.booking.book, { flightId, pax }, {
  compensate: this.booking.cancel, // dispatched in reverse if the run later fails
})

The undo is dispatched to whatever worker serves its name and called with one argument: the StepUndo<TInput, TOutput> envelope — { input, output } of the call it is undoing (book's input and its { bookingRef } result here). The original handler is never re-invoked in-process. That is the whole contract, and it is why a worker in another runtime can serve the undo: it just reads { input, output } off the task it received.

UndoOf<H> derives that envelope type from the original step method, so the ref form is compile-checked against the call it undoes instead of hand-written and driftable:

type UndoOf<H> = H extends (input: infer I, ...rest: never[]) => infer R
  ? StepUndo<I, Awaited<R>>
  : StepUndo<unknown, unknown>

The string form — compensate: 'flights:cancel' — works the same way for a handler with no JS reference to import, e.g. an undo served by a Python worker's @worker.step("flights:cancel"). Routing is by name either way. From the undo's own @Step config the engine takes its routing (name and partition), its input schema and its liveness timeoutMs; its retry budget is the engine-level compensationRetries, the same for every undo.

Both kinds of compensation live in the same reverse-order stack: whether a step registered a local closure or a dispatched undo ref, the engine unwinds them last-completed-first when the run fails.

A dispatched step is already deduplicated by its deterministic stepId (runId:seq), which workers can use as an idempotency key — so there's no separate idempotency-key option.

1@Workflow({ name: 'book-trip', version: '1' })2export default class BookTripWorkflow {3  constructor(private trips: TripSteps) {}4 5  async run(ctx: WorkflowCtx, trip: TripRequest) {6    const flight = await ctx.step(this.trips.bookFlight, trip, {7      compensate: this.trips.cancelFlight,8    })9    const hotel = await ctx.step(this.trips.bookHotel, trip, {10      compensate: this.trips.cancelHotel,11    })12    // the deposit registers no compensate — nothing of its own to undo13    await ctx.step(this.trips.chargeDeposit, { trip, flight, hotel })14    return { flight, hotel }15  }16}

What triggers the sagaCopy Anchor Link

Any error that escapes the workflow body triggers compensation — not just a business-logic throw. A ctx.step or ctx.localStep that exhausts its retries, or throws a FatalError, propagates out of the body like an ordinary exception, and the engine treats that the same way whatever the reason the step gave up. The saga does not run on a suspension: a durable ctx.sleep, a pending ctx.step, a signal wait or a continue-as-new all leave the compensation stack untouched — it is only consulted when the run's execution ends in an unhandled error (or on a compensating cancel).

The unwind is a strict LIFO over the compensations registered so far: the engine walks the stack from the most recently completed step back to the first, undoing exactly the steps that finished before the failure. A step that never completed never registered a compensation, so there is nothing to undo for it. In the checkout above, charge-card's refund runs before reserve-inventory's release.

Hand-rolled compensation: the control-flow-signal pitfallCopy Anchor Link

The compensate option is the engine's own machinery and is already safe — reach for it whenever a step's undo is a static ref or closure. When the undo doesn't fit (it is conditional, or needs values only known deep in a catch), you write the catch yourself:

import { isWorkflowControlFlowSignal } from '@adonis-agora/durable'

try {
  await ctx.step('payments:charge-card', input)
} catch (error) {
  if (isWorkflowControlFlowSignal(error)) throw error // suspend / continue-as-new: rethrow as-is
  await ctx.step('payments:refund', input) // a REAL failure — safe to run cleanup here
  throw error
}

The guard and rethrow are not optional. A suspend (a durable sleep, a step still pending) and a continue-as-new unwind the turn by throwing internally — the same JS exception mechanism a real failure uses to reach your catch. If the catch runs its cleanup step on one of those, that step is recorded as an extra command during what a later replay will see as a plain suspend, and the resumed replay dies with a NonDeterminismError. The predicate checks a stamped marker (CONTROL_FLOW_SIGNAL), not instanceof; it is deliberately false for a cancelled run and for a real step failure, which you may compensate on directly. See Catching errors, and the rethrow-control-flow-signals lint rule that flags an unguarded catch.

Retrying a transient undoCopy Anchor Link

A compensation can itself fail transiently — the refund API might be momentarily unreachable. The engine retries each compensation up to compensationRetries times. This is an engine-level option (it applies to every compensation), set in config/durable.ts, and it defaults to 1, i.e. a single attempt with no retry:

export default defineConfig({
  // … plus your transport/store (see Getting Started)
  compensationRetries: 5, // up to 5 attempts per saga undo before giving up on it
})

Attempts run back to back: there is no backoff between them, for a local closure or a dispatched undo. Because a compensation may run more than once, compensations should be idempotent — releasing an already-released reservation or refunding an already-refunded charge must be a no-op. A compensation that keeps failing past compensationRetries is skipped rather than allowed to throw: a permanently-failing undo must not mask the original failure or strand the remaining compensations. The run still settles failed with the error that triggered the saga, not the compensation's.

A dispatched undo whose step definition carries no liveness timeoutMs is additionally bounded by compensationTimeoutMs (engine config, default 300 000 — 5 minutes; values under 1 000 are raised to 1 000): if the worker produces no result within the window, that attempt fails like any other undo failure — retried up to compensationRetries, then skipped loudly — instead of the unwind awaiting a lost job forever.

export default defineConfig({
  compensationRetries: 5,
  compensationTimeoutMs: 120_000, // give up on one dispatched undo attempt after 2 minutes
})

The unwind itself is durableCopy Anchor Link

Compensations are not just dispatched — they are checkpointed, at reserved negative sequence numbers (-2 - idx, in registration order) so they can never collide with the body's own checkpoints. Each undo persists a pending checkpoint before it dispatches and settles it on the outcome, which buys the unwind the same crash-safety as the run body:

• A crash mid-unwind doesn't restart it from scratch. A re-driven unwind reads the checkpoints, skips undos already completed (no double refunds), and resumes each remaining undo's attempt count where it left off.
• Multi-pod deployments work. On a shared results queue, a dispatched undo's worker result can land on another pod; because the undo is a persisted checkpoint, whichever pod consumes the result completes the unwind instead of dropping it.

Two ways a dispatched undo deliberately differs from an ordinary ctx.step dispatch:

• It never waits behind flow control. An undo is handed straight to the transport, skipping the admission/flow-control gates an ordinary step goes through: the saga is often unwinding because the system is under pressure, and the very limit that throttled the forward work must not starve its own undo. Plan undo-handler capacity accordingly if a saga can fire under load.
• It is awaited inside the unwind, not suspended on: the engine waits for the worker's result (or the settled checkpoint, when another pod consumed it), bounded by the undo's own timeoutMs heartbeat window or else by compensationTimeoutMs.

You don't configure any of this — it's how the saga executes. It's why the "compensations should be idempotent" rule above is about your undo handlers, not about the engine re-running undos it already knows finished.

Compensations are visibleCopy Anchor Link

Every compensation surfaces as a compensate:<step> event, emitted as a step.completed (the undo ran) or step.failed (it exhausted its retries) lifecycle event. The dashboard and the Telescope view render these, so a stranded undo is visible rather than silently swallowed. For the checkout above you'd see compensate:charge-card and compensate:reserve-inventory appear in the timeline as the saga unwinds. Each undo is also a real checkpoint at its reserved negative seq, and the dashboard's run view splits those out of the body timeline into a separate compensation section, listed in unwind order.

Compensating cancellationCopy Anchor Link

The saga also runs when you deliberately cancel a run with compensation. A plain engine.cancel(runId) is immediate: it marks the run cancelled right away and broadcasts the cancellation so a worker actually running it can abort cooperatively — but it does not undo completed steps. Passing { compensate: true } instead runs the saga first:

// Immediate cancel — mark cancelled, abort in-flight work, but leave completed side effects in place:
await engine.cancel(runId)

// Compensating cancel — undo the completed steps in reverse, THEN mark the run cancelled:
await engine.cancel(runId, { compensate: true })

Terminal runs are immutable: cancel on a run that is already completed, failed, cancelled or dead is a no-op that returns its current state. It never relabels a failed run as cancelled and never cascades to the run's children, which may be unrelated live work (for example the next queued turn an agent queue started as a child of the turn that failed).

Likewise, a failed run is only re-driven by an explicit retry (engine.requeue(runId) / the dashboard retry, or a direct engine.resume(runId)). The engine's implicit wake-ups (a late signal or step result, a wait recheck, a due timer, a redelivered dispatch) leave it failed.

{ compensate: true } applies to a run that is running or suspended. A run that is still pending (or parked blocked) is cancelled immediately, without the unwind, even with the option set.

A compensating cancel returns immediately (non-blocking), reporting the run's current status rather than cancelled, and runs the undo in the background: the run is resumed so the replay re-registers the saga from history, the engine runs the compensations in reverse, and marks the run cancelled. For the checkout example, cancelling a run that had already charged the card and reserved inventory with { compensate: true } issues the refund and releases the reservation before the run becomes cancelled — leaving the world clean, exactly as a failure would.

The request itself is durable: cancel({ compensate: true }) persists a cancel marker alongside broadcasting the cancellation, so the undo survives a crash or a pod handoff — whichever pod next drives the run honors the marker and runs the saga, even if the pod that took the cancel request couldn't run the workflow (or died before it did).

The cancel is honored however the turn ends. If the run is mid-turn when you cancel it, that turn (or the resume queued behind it) checks for the cancel when it returns, throws, continues-as-new or suspends. In each case it runs the compensations and settles cancelled, instead of completing, failing or forking a continuation. A body with no suspension point left can't slip past a compensating cancel this way.

Cancelling a live run also cancels its children — awaited (ctx.child) and fire-and-forget (ctx.startChild) alike, recursively — with the same { compensate } option, so each child runs its own saga when asked to. Children that already finished are left alone.

To cancel many runs at once, engine.cancelWhere(filter, opts?) cancels every run matching a run query (with the same { compensate } option).

Saga vs. retries vs. dead-letterCopy Anchor Link

These three mechanisms answer different questions about a failure, and one run often uses all three. Retries & backoff answer "can this exact step just be tried again?" — reach for them first, on the step itself, when a failure is transient (a timeout, a 502). Sagas answer "this run is not going to complete — what do I need to undo?" — reach for compensate when a step has an irreversible side effect an earlier step would otherwise leave stranded. They compose: a step exhausts its own retries (or throws FatalError) before its failure ever reaches the saga, so compensation is what happens after retrying has given up. The dead-letter queue answers a third, orthogonal question: "this run keeps crashing the process itself" — a poison pill a saga can't help with, because the body never gets far enough to fail cleanly. Cap it with maxRecoveryAttempts and handle it with engine.onDead; use sagas for a run that fails cleanly but has already done things in the world.