Sagas & compensation
Undo a dispatched step's side effects with a compensate ref/name that receives a StepUndo<TInput, TOutput> envelope, retried per the undo's own @Step config and checkpointed at negative seqs for crash-safe resume — plus the ctx.localStep closure form, compensationRetries, compensate:<step> visibility, and compensating cancellation via engine.cancel(runId, { compensate: true }).
A durable run often performs several irreversible side effects in sequence — book a flight, book a hotel,
charge a deposit. If a later step fails, the earlier effects are still out there in the world, and "the run
failed" is not an acceptable end state when money has moved. The saga pattern handles this: alongside each
step that does something, you register how to undo it, and when the run fails the engine runs those
undos in reverse order. nestjs-durable builds this in via a compensate option — on a dispatched ctx.step
call (the common case: the undo runs on a worker, same as the step it undoes) or on ctx.localStep (an
in-process closure).
Below is a trip booking that reserves a flight and a hotel before charging a deposit. Each booking step
names its own undo as a compensate ref; when the deposit is declined, watch the compensations unwind in
reverse — hotel, then flight — before the run settles failed:
1@Workflow({ name: 'book-trip', version: '1' })2export class BookTripWorkflow {3 constructor(private readonly trips: TripService) {}4 5 async run(ctx: WorkflowCtx, trip: TripRequest) {6 const flight = await ctx.step(this.trips.bookFlight, trip, {7 compensate: this.trips.cancelFlight,8 });9 const hotel = await ctx.step(this.trips.bookHotel, trip, {10 compensate: this.trips.cancelHotel,11 });12 // the deposit registers no compensate — nothing of its own to undo13 await ctx.step(this.trips.chargeDeposit, { trip, flight, hotel });14 return { flight, hotel };15 }16}Compensating a dispatched step
Pass compensate alongside a ctx.step call. The undo is itself an ordinary @Step — pass its reference
(compile-time checked) or its name (for a cross-runtime undo, e.g. a Python @step):
@Workflow({ name: 'book-trip', version: '1' })
export class BookTripWorkflow {
constructor(private readonly trips: TripService) {}
async run(ctx: WorkflowCtx, trip: TripRequest) {
const flight = await ctx.step(this.trips.bookFlight, trip, {
compensate: this.trips.cancelFlight,
});
const hotel = await ctx.step(this.trips.bookHotel, trip, {
compensate: this.trips.cancelHotel,
});
// charge-deposit registers no compensate — nothing of its own to undo
await ctx.step(this.trips.chargeDeposit, { trip, flight, hotel });
return { flight, hotel };
}
}
// trip.service.ts — an undo is an ordinary @Step, typed with UndoOf
@Step()
async cancelFlight(undo: UndoOf<TripService['bookFlight']>) {
await this.flightsApi.cancel(undo.output.bookingId);
}The undo is dispatched to whatever worker serves its name, called with one argument: the
StepUndo<TInput, TOutput> envelope — { input, output } of the call it's undoing (bookFlight's trip
input and its { bookingId } result here), never the original handler re-invoked in-process. That's the
whole contract, and it's why a Python worker can serve the undo by name with no side-channel lookup: it just
reads { input, output } off the task it received.
UndoOf<H> derives that envelope type from the original step method, so the ref form is compile-checked
against the call it undoes instead of hand-written and driftable:
type UndoOf<H> = H extends (input: infer I, ...rest: never[]) => infer R
? StepUndo<I, Awaited<R>>
: never;
// cancelFlight's argument is exactly StepUndo<TripRequest, { bookingId: string }> —
// derived from bookFlight's own signature, not restated by hand.
async cancelFlight(undo: UndoOf<TripService['bookFlight']>) { ... }The string form (compensate: 'TripService.cancelFlight') works the same way for a handler with no JS
reference to import — routing is by name either way.
The undo's retry policy is its own. A dispatched compensation is dispatched exactly like any other
ctx.step call — its retries/backoff/backoffMs/backoffMaxMs/jitter come from its own
@Step-declared config, never from the compensated call's per-call opts. It's a separately-declared
handler with its own failure policy, not an extension of the step it undoes.
The unwind is checkpointed, not just an in-memory pass. Every undo — dispatched or local — gets a
checkpoint at a reserved negative seq: the first one undone lands at seq -1, the next at -2, and so on,
named compensate:<originalStepName>. Negative seqs can never collide with a body position (those start at
0), so the saga's undo trail composes onto the run's ordinary checkpoint history with no schema change,
and shows up in run detail exactly like any other step. Because compensate:<originalStepName> is
deterministic and rebuilt identically on every replay, a crash mid-unwind resumes cleanly: an already-
completed undo (dispatched or local) is skipped rather than re-run, and the next one in the stack still
gets its turn — exactly once. A crash while a dispatched undo is pending just re-suspends the run on it,
the same as a crash mid-dispatch of an ordinary step.
The in-process alternative: ctx.localStep
ctx.localStep's compensate is the in-process form — the undo runs as a plain closure instead of being
dispatched to a worker. Reach for it when the undo is cheap in-process work (an in-memory cache release, a
call into a service already injected into the workflow class) rather than something that belongs on a
worker:
const reservation = await ctx.localStep(
'reserve-inventory',
() => this.inventory.reserve(order.items),
{ compensate: () => this.inventory.release(order.items) },
);The callback is registered when the step completes; if the run later fails, the engine runs every
registered compensation — local and dispatched together — in one strict reverse order. The saga is
reconstructed from the run's history on replay: on resume, a completed localStep checkpoint returns its
saved output without re-running fn — but its compensate closure still gets pushed onto the
compensation stack, exactly as it was the first time. That's what makes it crash-safe at the registration
level: after a mid-run crash, the resumed execution reconstructs the full stack of undos for every step that
had actually completed, not just the ones this pass happened to run.
Its unwind is checkpointed too, the same as the dispatched form's: a crash mid-unwind no longer re-runs a local undo that already completed, and doesn't restart its retry budget from attempt 1 — the checkpoint at its negative seq is what a later re-drive checks before running it again.
Because a compensation can still run more than once in the field regardless — a transient retry that succeeded but whose result got lost before it checkpointed, for instance — compensations should be idempotent: releasing an already-released reservation or refunding an already-refunded charge should be a no-op either way.
What triggers the saga
Any error that escapes the workflow body triggers compensation — not just a business-logic throw. A ctx.step
or ctx.localStep that exhausts its retries, or throws a FatalError directly, propagates out of run()
exactly like an ordinary exception, and the engine treats that the same way regardless of why the step gave
up. The saga does not run on every suspension: ContinueAsNew and an ordinary waitForSignal/ctx.step
suspension both leave the run's compensation stack untouched — it's only consulted when the run's execution
ends in an unhandled error.
Mechanically, a suspend or continueAsNew unwinds the current turn by throwing internally
(WorkflowSuspended / ContinueAsNew) — the exact same JS exception mechanism a real step failure uses to
reach your catch. The engine tells the two apart with a shared marker,
CONTROL_FLOW_SIGNAL, and the exported
isWorkflowControlFlowSignal(error) predicate. This only matters to you if you write your own try/catch
around a ctx.step/ctx.sleep/ctx.waitForSignal/etc. instead of relying on compensate — see the pitfall
below.
Hand-rolled compensation: the control-flow-signal pitfall
The compensate option (above) is the engine's own machinery for registering an undo, and it's already
safe — reach for it whenever a step's undo is a static ref. But some compensation logic doesn't fit a static
compensate ref (it's conditional, needs values only known deep in a catch, or you're porting saga code
that predates compensate), so you write the catch yourself:
try {
await ctx.step(this.payments.chargeCard, input);
} catch (error) {
if (isWorkflowControlFlowSignal(error)) throw error; // suspend/continue-as-new: rethrow as-is
await ctx.step(this.payments.refund, input); // a REAL failure — safe to run cleanup here
throw error;
}The isWorkflowControlFlowSignal(error) check and rethrow are not optional. WorkflowSuspended and
ContinueAsNew are thrown internally to unwind a turn — on a durable sleep, an awaited-but-still-pending
step, ctx.continueAsNew, and so on — and they are not real failures. If your catch doesn't rethrow them
untouched and instead unconditionally runs its cleanup/compensation step, that compensation step gets
recorded as an extra history command during what a later replay will see as a plain suspend, not a
failure. The resumed replay then dies with a NonDeterminismError, because the code path it's replaying no
longer matches what was checkpointed. See
Troubleshooting: a caught suspend runs compensation and corrupts the run.
Deliberately not control-flow, and safe to compensate on directly — no guard needed: a cancelled run
(Cancelled, thin worker) is a terminal outcome you may legitimately want to react to, and a ctx.step /
ctx.all rejection that reaches your catch (a StepFailed/GatherError or an ordinary thrown error) is a
real failure your saga is meant to compensate for.
Import isWorkflowControlFlowSignal from @dudousxd/nestjs-durable-core (or @dudousxd/nestjs-durable,
which re-exports it) — see API reference.
The reverse order is a strict LIFO unwind of the compensations registered so far: the engine walks the stack
from the most recently completed step to the first, undoing exactly the steps that finished before the
failure (a step that never completed never registered a compensation, so there's nothing to undo for it).
For the trip-booking example, that means bookHotel's undo runs before bookFlight's, because the hotel
booking completed second.
Retrying a transient undo
A compensation can itself fail transiently — the cancel API might be momentarily unreachable. Where that retry policy comes from depends on the form:
- Dispatched (
ctx.step'scompensate): retried per the undo handler's own@Step-declared policy (retries/backoff/backoffMs/backoffMaxMs/jitter) — the same durable retry/backoff every other dispatched step gets, resumable across a crash. - Local (
ctx.localStep'scompensate): retried in-process, back-to-back, up to the engine/module-levelcompensationRetriesoption (applies to every local compensation; defaults to1— a single attempt, no retry):
DurableModule.forRoot({
store,
transport,
compensationRetries: 5, // retry each LOCAL saga undo up to 5 times before giving up on it
});Either way, a compensation that keeps failing past its retry budget is skipped rather than allowed to
throw or stall the rest of the unwind: the engine records the failure as a compensate:<step> event and
moves on to the next one in the stack. One stuck undo never halts the unwind — every remaining compensation
still gets its turn — and it never masks the original failure: the run still settles failed with the error
that triggered the saga in the first place, not the compensation's.
Design notes
Two ways a dispatched compensation deliberately differs from an ordinary ctx.step dispatch:
- It never waits behind a queue. A compensation dispatch skips flow-control on purpose: the saga is
often unwinding because the system is under pressure, and the very
queuethat throttled (or failed) the forward work must not be able to starve its own undo. Compensations always dispatch directly, by name. - It is always a durable dispatch. An ordinary step's
timeoutMsswitches it onto the in-memory liveness path (held await + heartbeats — traded against durable suspension). An unwind must never make that trade: a compensation is dispatched durably and waited on exactly like a step with notimeoutMs, so a crash mid-undo resumes instead of losing the wait.
Either way the unwind is checkpointed and crash-safe — but plan undo-handler capacity accordingly if a saga can fire under load.
Compensations are visible
Every compensation surfaces as a compensate:<step> event, emitted as a step.completed (the undo ran) or
step.failed (it exhausted its retries) lifecycle event, and as a real checkpoint at its reserved negative
seq. The dashboard renders the whole unwind as an amber Compensation section on the run's timeline, so a
stranded undo is visible rather than silently swallowed. For the trip-booking example you'd see
compensate:TripService.bookHotel and compensate:TripService.bookFlight appear as the saga unwinds.
Compensating cancellation
The saga also runs when you deliberately cancel a run with compensation. A plain engine.cancel(runId) is
immediate: it marks the run cancelled right away and broadcasts the cancellation so a worker actually
running it can abort cooperatively — but it does not undo completed steps. Passing { compensate: true }
instead runs the same unwind used on failure, dispatched and local compensations alike:
// Immediate cancel — mark cancelled, abort in-flight work, but leave completed side effects in place:
await engine.cancel(runId);
// Compensating cancel — undo the completed steps in reverse, THEN mark the run cancelled:
await engine.cancel(runId, { compensate: true });A compensating cancel works by resuming the run with a cancellation pending: the replay re-registers the
saga from history, and at the run's suspension point the engine runs the compensations in reverse and marks
the run cancelled (rather than re-suspending). While the undo runs — including waiting on an in-flight
dispatched compensation — the run's status stays cancelling rather than suspended: unlike the failure
path (where re-suspending between undo dispatches is fine, a resumable state no different from an ordinary
in-flight ctx.step), a compensating cancel needs cancelling to survive a crash and be recognized as
still-in-progress by waitForRun, recovery, and a repeat cancel call. It flips to cancelled once every
undo resolves — so a compensating cancel is visible and crash-durable rather than looking like a
still-running run.
Saga vs. retries vs. dead-letter
These three mechanisms answer different questions about a failure, and a single run typically uses all
three at once. Retries & backoff answer "can this exact step just be tried
again?" — reach for them first, on the step itself, whenever a failure is transient (a timeout, a 502) and
the step is naturally retriable. Sagas answer a different question: "this run is not going to complete —
what do I need to undo?" — reach for compensate whenever a step has an irreversible real-world side effect
that an earlier step in the same run would otherwise leave stranded. They compose: a step still exhausts
its own retries (or throws FatalError) before its failure ever reaches the saga, so compensation is what
happens after retrying has given up. Dead-letter queues answer a third,
orthogonal question: "this run keeps crashing the process itself" — a poison pill that a saga can't help
with, because the run's body never gets far enough to fail cleanly and trigger its compensations. Route a
poison-pill run to a @DeadLetter() handler to break the crash loop; use sagas for a run that fails cleanly
but has already done things in the world.
Retries & backoff
Durable step retries with fixed/exponential backoff and jitter (a failed step re-dispatches on a persisted wakeAt), FatalError and worker-side retryable:false to opt out, and the in-memory timeoutMs + heartbeat liveness path for presumed-dead workers.
Flow control
Every knob that throttles or prioritizes dispatched steps: durable queues (concurrency caps + fixed-window rate limits) via engine.registerQueue, per-call priority + fairnessKey on ctx.step, worker/transport concurrency (fixed or adaptive), and RedisAdmissionBackend for a fleet-wide global cap.