Running in production
The operational checklist for taking @adonis-agora/durable from a single dev process to a real deployment — picking a store and a transport, choosing a topology, who drives timers and recovery, leases and the dead-letter cap, draining on deploy, replay-safe releases, schema ownership, retention, consumers, encryption at rest, and observability.
Everything works out of the box in one dev process with zero infrastructure — that is the point of the durability model. This page is the other half: what to change, and what to double-check, before that process becomes a real deployment with several replicas, real deploys, and someone else's uptime on the line. Each item says why it matters, shows the config/durable.ts change, and links to the page with the full detail.
1. Swap the dev defaults
A config/durable.ts that names no store and no transport boots the engine on two in-memory defaults. Neither survives a restart.
- Store. With no
store, the provider usesInMemoryStateStore— aMapin this process. A crash or redeploy loses every run, anddurable:runs/ the dashboard only ever see this process's runs. Select the Lucid store before anything durability-sensitive ships. - Transport. With no
transport, the provider uses the testmemorytransport, which runs a handler inline inside the dispatch call. Pick a real driver:eventEmitterfor in-process steps, orqueue/db/bullmqonce steps must run in another process. See Transports.
import { defineConfig, stores, transports } from '@adonis-agora/durable'
export default defineConfig({
store: 'lucid',
stores: { lucid: stores.lucid() },
transport: 'event-emitter',
transports: { 'event-emitter': transports.eventEmitter() },
})2. Pick a topology
Which transport you need depends on where step bodies run, not on how many replicas you have. The durable state — runs, checkpoints, leases, signal waiters, timers — always lives in the shared store. What the transport and the control plane add is cross-process delivery.
| Topology | Store | Transport | Control plane | Who drives timers, recovery and schedules |
|---|---|---|---|---|
| (a) One process | lucid | eventEmitter | none | worker: { embedded: true } in that process, or a durable:work sidecar |
| (b) Several web replicas, steps run in the pod that started them | lucid | eventEmitter | optional (see what you lose without it) | durable:work process(es), or worker.embedded on the replicas |
| (c) Steps run in a separate worker fleet | lucid | queue, db or bullmq | controlPlanes.redis | durable:work pods |
(a) Single process — eventEmitter + lucid
The simplest real deployment: one Adonis process that serves HTTP, runs workflow bodies, and serves app/steps over an in-process EventEmitter. Checkpoints are in your database, so a crash mid-run resumes from the last completed step.
The one thing you must not forget: something has to tick the worker loop. Durable sleeps, retry backoffs, crash recovery, execution timeouts, schedules and retention are all driven by that loop, and a plain web process does not run it. Either embed it:
export default defineConfig({
store: 'lucid',
stores: { lucid: stores.lucid() },
transport: 'event-emitter',
transports: { 'event-emitter': transports.eventEmitter() },
// Tick pending/recovery/timers/timeouts/schedules/retention inside the web process.
worker: { embedded: true },
})or run node ace durable:work next to the web server. The embedded loop only starts in the web environment, so it never doubles up with durable:work or turns node ace migration:run into a worker. See Running the loop in the web process.
(b) Several web replicas, steps stay in-process — eventEmitter is still enough
If every replica serves the same app/steps and your workflows do their I/O in ctx.localStep (or in ctx.step, which on eventEmitter is handled in the pod that dispatched it), you do not need a broker to scale horizontally. Correctness never depended on the transport, because every cross-replica decision goes through the shared store:
- One executor per run. Every execution — a fresh start, a poll pickup, a signal, a timer, recovery — first takes an atomic lease on the run row (
tryLockRun) and renews it everyleaseMs / 2. A replica that fails to take the lease does not run the body. - Signals.
engine.signal(token, payload)looks up the waiter row in the store, writes the signal checkpoint, and resumes the run under a lease on whichever replica received the call. If no run is waiting yet, the signal is buffered in the store and consumed by the nextwaitForSignal(token). If the run is mid-turn on another replica, the checkpoint is already durable and the owner re-checks it when its turn suspends (with thereconcileMswake as the fallback). - Cancellation. A plain
cancel(runId)writescancelledto the run row directly. A turn still executing on another replica cannot overwrite that: every settle is a conditional write that is refused once the run is terminal, and any later resume sees the terminal status and stops. Acancel(runId, { compensate: true })also buffers a durablecancel:<runId>marker in the store, so the replica that runs the run's next turn runs the saga unwind, even after a crash. That includes a turn already executing on another replica: when it returns, throws, continues-as-new or suspends, it finds the marker and compensates instead of settling. - Timers, recovery, schedules. These are store queries (
listDueTimers, orphaned leases, schedule windows) that any ticking process can answer, with the lease again deciding who executes.
What you give up without a control plane is everything that is broadcast between processes rather than written to the store:
Without controlPlane | Effect |
|---|---|
| Cross-pod lifecycle events | The dashboard's live tail and engine.subscribe only see runs executing on the pod that serves them. Anything that waits on events — engine.waitForRun(runId) and the blocking BaseWorkflow.start(...) built on it — never hears about a run that settles on another process (for example a run that suspended on ctx.sleep and was resumed by durable:work), so it waits until its timeout. Use dispatch + polling getRun, or add a control plane. |
| Instant cancellation | The cancel is durable either way, but the replica currently executing the run is not told. Its in-flight turn (including the ctx.localStep it is running) keeps going until the turn ends and its settle is refused; engine.onCancel listeners there never fire. |
| Schedule pause fan-out | A runtime pause/resume from the dashboard (engine.setSchedulePaused) applies only to the pod that served the request; other ticking pods keep firing. Pin paused in config to pause fleet-wide. |
enqueued nudge | Other pods are not told a run was enqueued. Nothing in the library subscribes to it by default — it only matters if you wired engine.onEnqueued yourself. |
And someone must still drive the loop: run one or more durable:work processes, or set worker.embedded on the replicas. Several ticking processes are safe — every phase leases before it executes, and schedule windows are idempotent by run id — but each one polls the store.
ctx.step on eventEmitter in a fleet
A ctx.step dispatched over eventEmitter is served by the dispatching pod only. If that pod dies mid-step, nothing else holds the job: the run is recovered elsewhere, but the step's pending checkpoint has no worker. That is the lost-dispatch failure mode — set remoteRedispatchMs, or keep in-process work in ctx.localStep, which simply re-runs on replay.
(c) Remote steps across processes — a broker + controlPlanes.redis
Once step bodies must run somewhere other than the pod that dispatched them — a dedicated worker fleet, a GPU pool, a Python worker — pick a cross-process transport and add the Redis control plane so events and cancellation reach every pod:
import { controlPlanes, defineConfig, stores, transports } from '@adonis-agora/durable'
export default defineConfig({
store: 'lucid',
stores: { lucid: stores.lucid() },
transport: 'queue',
transports: { queue: transports.queue({ connection: 'redis' }) },
// Live tail, cancellation and schedule pauses fan out across every pod.
controlPlane: controlPlanes.redis({ connection: 'main' }),
})Run the step handlers on node ace durable:work pods, which also tick the loop. The queue driver rides @adonisjs/queue; db needs no broker at all; bullmq is the one to pick for a cross-ecosystem fleet. For a store-less worker fleet, see roles.
Keep workflow bodies off the web pods if they are heavy. By default a pod executes the runs it starts (and the ones it requeues or recovers) right away, in-process. A web pod that should only enqueue sets drive: false, and consumers: 'never' so it claims no broker jobs either:
export default defineConfig({
// …store, transport, controlPlane as above
drive: process.env.DURABLE_DRIVE !== 'false', // 'false' on web pods: start() persists, durable:work executes
consumers: 'never', // web pods never claim step jobs or results; durable:work consumes regardless
})A started run then stays pending until a durable:work pod's next poll picks it up. durable:work drives through that poll whatever drive says, so one config file serves both kinds of pod. What still executes on a drive: false pod is a resume it is asked for directly — an engine.signal or webhook it handles, a compensating cancel — under the run's lease, like on any instance. drive: false together with worker: { embedded: true } is a contradiction and throws at boot, and an explicit runDispatcher still wins over it. role: 'control-plane' never drives in-process either, and also stops serving app/steps in that process.
3. Crash recovery is a lease
Running more than one process is safe by design:
- A crashed executor's lease expires. While a run executes, its process renews a recovery lease every
leaseMs / 2. A crashed process stops renewing, the lease lapses, and the nextrecoverIncomplete()— every worker-loop tick, on any process — reclaims the run and replays it from its last checkpoint.leaseMsdefaults to 30 000. - Exactly one instance per run.
tryLockRunis atomic, so several processes racing to recover the same run produce one owner. A lease renewal that reports a takeover fences the old executor: its eventual settle becomes a no-op instead of clobbering the new owner's writes. - A lost wake still wakes. A run that suspends with no timer of its own gets a fallback wake
reconcileMslater (default 5 minutes, set inconfig/durable.ts;0disables it), so a signal or child completion that raced the suspend is picked up on the next pass. - A dispatched step is not a crash. A run found
runningwith a free lease while one of its remote steps is stillpendingon the transport is not counted as a recovery attempt. Its work is with a worker that holds no run lease. Recovery parks itsuspendedon the step's deadline or thereconcileMswake instead, and the worker's result resumes it. Only runs with no step in flight count towardmaxRecoveryAttempts. - One bad run doesn't stop the sweep. A run that throws while a recovery, timer or pending pass resumes it (a workflow this pod doesn't have mid-deploy, a store error on one row) is logged with its id and
workflow@versionand skipped. The rest of the batch still runs, and the skipped run is picked up again once its lease expires.
Give each process a stable, unique instanceId. Leases are held by it, so two processes sharing one id could execute the same run at once. And it is how you find a pod again: the run's lease owner (lockedBy), the worker heartbeats, the dashboard's Workers table and the control-plane events all name it, so a random id that changes on every restart turns each deploy into a fresh set of anonymous workers. The orchestrator's pod name is a good fit:
export default defineConfig({
leaseMs: 60_000, // above how long a single resume step runs
instanceId: process.env.HOSTNAME, // unique per process
})A stable id also shortens crash recovery. A process killed without draining (SIGKILL, an OOM kill) leaves its runs running with leases that name it. When it comes back under the same id, its first recovery passes also pick up the runs still leased to that id and replay them at once, instead of waiting up to leaseMs for those leases to expire. A run this process is executing is never taken this way. The engine only releases a lease under its own id when nothing in the process is executing the run and the lease was written before the engine started. Other instances still wait for the lease to expire, as before. The extra lookup only runs during the first leaseMs after boot, because no lease from the previous process can outlive that.
This is why the id must be unique per live process, not just stable. If two live processes share an id, each reads the other's leases as its own: the leases no longer keep them apart, and a restart of one can reclaim runs the other is still executing. One id per pod (the pod name) is right. One id for a whole deployment, or a pod running several engines under one id, is not.
None of this needs a control plane. See Durability & replay and Delivery under multiple instances.
4. Cap poison pills with the dead-letter state
Self-healing recovery loops forever on a run that crashes the process every time it is resumed — a deserialization bug, an infinite loop. Cap it:
export default defineConfig({
maxRecoveryAttempts: 5, // past 5 crash-recoveries, the run moves to `dead`
})maxRecoveryAttempts is unlimited by default. Past the cap the run moves to the terminal dead status with a max_recovery_attempts error: still inspectable and retriable (node ace durable:retry <runId>), but no longer crash-looping. A retry resets the run's recoveryAttempts to 0, so the retried run gets a fresh budget instead of being dead-lettered again on the next recovery pass.
Hand dead runs to a workflow that triages them — pages someone, opens a ticket, compensates. Name a fleet-wide default in config, and override it per workflow:
export default defineConfig({
maxRecoveryAttempts: 5,
deadLetterWorkflow: 'global-dlq', // a name, or the workflow class
})import { BaseWorkflow } from '@adonis-agora/durable'
export default class PipelineWorkflow extends BaseWorkflow {
static workflow = { name: 'pipeline', version: '1', deadLetterWorkflow: 'pipeline-dlq' }
// …or handle it inline instead (registered as `pipeline.dlq`, sharing this class's services):
// async deadLetter(ctx: WorkflowCtx, dead: DeadLetter) { … }
}A run that goes dead starts its handler as dlq:<runId> (idempotent, so a run dead-lettered twice is handled once; the dashboard links the pair by that id), in the dead run's namespace, with a DeadLetter input: { deadRunId, workflow, input, error }. The inline deadLetter() method wins over static workflow.deadLetterWorkflow, which wins over the config default; declaring both the method and the option throws at boot. A handler run that itself goes dead is not routed again. engine.onDead still fires first, for alerting or routing of your own. See Dead-letter queue.
5. Drain on deploy
On SIGTERM you want a process to stop picking up new runs and let in-flight ones finish, rather than die mid-step and leave the run to a leaseMs timeout.
durable:workhandlesSIGINT/SIGTERMitself: it stops leasing new work at once, waits for the tick in flight, then callsengine.drain(--drain-timeout)(default 10 000 ms) before exiting. The drain timeout is the budget for the whole shutdown: the time spent waiting for the tick is deducted from the drain (engine.beginDrain(spentMs)), and a tick still running when the budget is spent is no longer waited for. From the stop request to the end of the drain takes at most--drain-timeout.worker: { embedded: true }drains the same way from the provider'sshutdown(), bounded as a whole byworker.drainTimeoutMs(default 10 000). The loop is stopped and awaited before the transport and control plane close, so an in-flight run can still dispatch and await a remote step while it drains.- Every other process drains from the provider's
shutdown()too — a web process (whose in-process dispatcher may be mid-turn), adrive: falsepod handling a signal, a control plane. Onapp.terminate()(the web environment turnsSIGTERMinto it) the provider drains the engine before it closes the transport and control plane, bounded bydrainTimeoutMs(default 10 000;worker.drainTimeoutMswhen set). A process running the embedded loop is drained once, by the loop.
export default defineConfig({
drainTimeoutMs: 15_000, // above your typical in-flight turn
})drain(timeoutMs) stops recovery, timer and pending pickups, then waits for every in-flight execution — including continue-as-new links and deferred children — to settle. It does not cancel anything: if the timeout wins, it releases the leases this process still holds, so the next process's recoverIncomplete() replays those runs from their last checkpoint immediately instead of after leaseMs. Size the timeout above your typical in-flight turn, and below the orchestrator's grace period (Kubernetes' terminationGracePeriodSeconds, 30s by default). A role: 'tenant' pod binds no engine and has nothing to drain.
6. Replay-safe deploys
Replay is positional, and a deploy is exactly when in-flight runs meet new code. Three layers catch a breaking change at different points:
- Lint at author time.
@adonis-agora/durable-eslint-pluginflagsDate.now(),performance.now(),Math.random(),new Date(),crypto.randomUUID()in a workflow body, acatchthat swallows control-flow signals, and globalfetch(...)/ rawengine.*calls in the body. Wire it into CI. - Regression-test replay.
node ace durable:export <runId> --out tests/fixtures/x.jsoncaptures a real run's history;assertReplayablereplays it against your current code in CI and throwsNonDeterminismErroron divergence. See Replay CI. - Version breaking changes. A run always resumes on the version it started on. When a change inserts, removes or reorders steps, register the new version alongside the old one and keep both deployed until the old runs drain — or guard a surgical change with
ctx.patched(id). A run whose version is no longer registered fails withworkflow <name>@<version> is not registered — keep the prior version deployed. See Versioning & determinism.
Roll out workers and web together when a workflow or its steps changed. On a shared broker, a worker still running an old build can pick up a run whose history no longer matches its code — see the stale-worker pitfall in Tenancy.
7. Own the schema
By default (autoSchema: true) the provider calls the store's ensureSchema() at boot — for Lucid, the same idempotent, additive createDurableTables the published migration runs. If the database user may not run DDL at boot, or every schema change must be a reviewed migration, turn it off:
export default defineConfig({
autoSchema: false, // the published migration is now the only thing that creates or repairs tables
})With autoSchema: false, run node ace migration:run after every upgrade. With it on, watch for the one-shot repaired the durable schema in place warning — it means the database was behind the library and your migration history does not know. See Lucid → autoSchema.
The boot check is cheap and safe to run on every pod. A fingerprint row in durable_schema_meta (the library's DURABLE_SCHEMA_REVISION plus your collation) answers a current schema in two statements; only a missing or stale marker — a fresh database, or an upgrade that changed the DDL — runs the full check, under pg_advisory_xact_lock on Postgres or GET_LOCK on MySQL, so pods booting together don't race the same CREATE TABLE. A table you changed by hand behind a current marker needs createDurableTables(db, connectionName, { force: true }).
On MySQL, plan the first upgrade to a release with longtext columns. The JSON columns are longtext there (MySQL text stops at 64KB), and an older table's text columns are widened with an ALTER TABLE … MODIFY that rebuilds the table and locks its writes. On a large durable_step_checkpoints, run node ace migration:run in a maintenance window instead of letting the first booting pod do it. See MySQL: longtext and collation.
8. Retention
Without a policy, terminal runs accumulate in durable_workflow_runs and its child tables forever, and the worker loop's status scans grow with them. Set an age per terminal status, plus scoped policies where one kind of run deserves its own window:
export default defineConfig({
retention: {
completed: '14d',
cancelled: '14d',
failed: '90d', // keep failures longer for debugging
policies: [
{ statuses: ['completed'], maxAge: '1d', scope: { tags: ['chat'] } },
// Bound by count instead of (or as well as) age: keep acme's 200 newest settled runs.
{ statuses: ['completed', 'cancelled'], maxAge: '30d', maxCount: 200, scope: { namespace: 'acme' } },
],
},
})maxCount keeps the newest runs (by start time) of the policy's whole status set within its scope and deletes the older ones; with maxAge as well, a run goes once it is past either bound. A policy needs at least one of the two, or the app fails at boot. The sweep runs in the worker loop (durable:work or the embedded loop), throttled to one pass a minute, and deletes a run together with its child runs. Pair it with engine.onEvict to archive a run before it is deleted. See Retention.
When data has to go now rather than age out — a tenant deleted their account — engine.purgeRuns(scope) hard-deletes everything matching a scope in bounded batches (cancelling live runs first unless cancelLive: false), and engine.purgeNamespace(namespace) is the shortcut for one pool. See Purging on demand.
9. Stop web pods from racing the worker fleet
Broker queues are point-to-point: every process that consumes competes for each job. On the queue transport, a web fleet that never executes steps should say so, so step jobs and results stop landing on pods that only enqueue:
export default defineConfig({
consumers: 'never', // pure producer: dispatch runs and read the store, never claim broker jobs
})durable:work re-enables consumption for itself, so you can ship one config to both. The default 'auto' already keeps node ace console/REPL processes from consuming. The queue, db and bullmq transports all honour this gate; the in-process transports have no broker to compete for. See Delivery under multiple instances and consumers.
10. Isolate environments and pools
A staging deploy, a second pool, or a developer laptop pointed at the same database or broker is not an observer — with the same namespace it polls the same rows and the same queues, and it will execute that pool's runs. Give each pool its own namespace (and a partition for workers sharing handler names):
export default defineConfig({
namespace: process.env.DURABLE_NAMESPACE, // unset = the 'default' pool
})Unset, a standalone engine is in the 'default' pool: it drives only runs stamped 'default' (and rows written before namespaces existed, which read back as 'default'), never another pool's. Only a role: 'control-plane' engine with no namespace is the operator that drives every pool — that is how one control plane serves many tenant pools.
See Tenancy and the two-pools mistake.
11. Flow control across replicas
Two different knobs limit how much work runs at once, and they answer different questions:
- Worker concurrency — how many steps one worker process runs at a time. It is a transport option on
bullmq,queueanddb: a number (fixed, no control loop), or'adaptive'to let the worker grow its limit while steps saturate it at a steady latency and shrink it on queueing, errors, stalls, the CPU ceiling or RSS. It protects the worker: its memory, its CPU, its event loop. - Admission queues — how many calls to one downstream may be in flight, or how fast, across every workflow that uses it. Declare them in config and reference one from a step with
ctx.step(step, input, { queue: 'emails' }). They protect the dependency: a rate-limited API, a database that falls over past N connections.
import { admissions, defineConfig, transports } from '@adonis-agora/durable'
export default defineConfig({
transport: 'bullmq',
transports: {
bullmq: transports.bullmq({ connection: { host: 'redis', port: 6379 }, concurrency: 'adaptive' }), // per worker
},
queues: [
{ name: 'emails', concurrency: 10, rateLimit: { limit: 1000, periodMs: 3_600_000 } },
{ name: 'reports', concurrency: 2 },
],
// Without this the queue caps count per engine instance: `concurrency: 10` admits ten per replica.
admission: admissions.redis({ connection: 'main' }),
})queues is the declarative form of engine.registerQueue(...), with the same fields (name, concurrency, rateLimit, retryMs, fairness, order); naming one queue twice throws at boot. A call the queue does not admit is not dispatched: the run re-suspends on a short wake and asks again. Worker concurrency multiplies by replicas (eight pods at concurrency: 4 run 32 steps at once); an admission cap backed by Redis does not.
When thousands of runs come due at once (recovery after an outage, a fan's children waking together), each timer pass loads at most timerBatchSize of them (default 500), most overdue first, shuffled so instances don't contend row by row. See Flow control.
Wide fans
A workflow that fans out over thousands of items is where the defaults stop being enough. In order:
- Fan out over a step, not a child workflow, when each item is one unit of work.
ctx.all(processRecord, records, { concurrency: 500 })costs one checkpoint and one dispatch per item; a child fan (ctx.all(ItemWorkflow, items)) costs a whole run per item — its row, a turn, a lease, a completion signal. Use child runs when each item is a workflow of its own (several steps, its own retries, its own timeline). See Fanning out over a step. - Cap the fan with
concurrency. Unset, every step is dispatched at once. A step fan sends waves of that size and resumes the workflow once per settled wave, not once per result. Never fan out withPromise.alloverctx.step, which replays the workflow once per result. - Size the workers. Steps in flight is worker
concurrency× replicas. Start from what the downstream and the worker's memory allow, or use'adaptive'. - Results settle in bulk on the engine side. Results come back as fast as the workers produce them. The
bullmq,queueanddbtransports hand each instance at least 512 results at once (resultConcurrencyonly raises that), and one run's results in hand are settled in batches of up to 256: one batched checkpoint read, one conditional write and one resume per batch. The fan's barrier count is skipped while more of the same fan's results are queued behind the batch, so a 20,000-step fan asks it a few dozen times rather than once per batch. The dispatching turn writes the fan's pending checkpoints and sends its tasks in bulk too (saveCheckpoints,dispatchMany). - Children spread over the instances. On
bullmq,queueanddb, a child fan's starts go out through the run queue in batches and every driving instance executes its share; each instance takes at most 64 runs from the queue at once and runs at most 256 in-process. A child whose completion never reached its parent (a crash between the two writes) is settled by the parent's next turn. See Where a run executes. - Let the dashboard read aggregates. On the Lucid store the run detail is built from per-group checkpoint counts and a few paged reads instead of every checkpoint, and it carries an
ETag: a poll that finds nothing changed gets a bare304. See Polling the run detail.
See wide fans for the transport side.
Remote work that goes quiet
Two opt-in self-heals cover work a remote process dropped without a word:
remoteRedispatchMs— a remote step with notimeoutMsstillpendingpast this window is dispatched again (up toremoteRedispatchMax, default 10). The step must be idempotent. See Failure modes.remoteAdvanceSilenceMs— a remote workflow turn (a Python or NestJS workflow advanced by a worker) with no decision and no run-scoped heartbeat for this long is re-driven: a fresh turn is dispatched with the same history, and a late decision of the old one is dropped. Workers beat every few seconds while a turn runs, so a long replay is not re-driven. See Remote workflow turns.
On bullmq, two worker-side guards complete the picture. stepTimeoutMs puts a wall-clock ceiling on each step handler: a step still running at the ceiling publishes a retryable failure, so the engine's retry dispatches it again. And a watchdog PINGs every pub/sub connection each pingIntervalMs (default 30s) and reconnects one that stopped answering, so a half-open connection can't silently stop heartbeats and events. See Wedged steps and Subscriber health.
12. Encrypt payloads at rest
Run and step inputs and outputs are stored as JSON. To keep them out of the database in the clear, wrap the store in CodecStateStore with your own PayloadCodec — and know that the default coverage leaves step events, error messages and heartbeat progress unencoded unless you opt into coverage: 'extended'. See Encrypting payloads at rest.
13. Guard each run at the boundary
Two registration options catch incidents before they become stuck runs:
executionTimeout('72h'or ms, onengine.registeroptions or a class'sstatic workflow) bounds a run's wall-clock life. The worker loop'ssweepTimeoutsmoves an overrunning run tocancelledwith codeexecution_timeoutand cancels its child subtree.inputSchema(any Standard Schema — zod, valibot, arktype — onstatic workfloworengine.registeroptions) andvalidateInput(a function) reject a bad payload atstart, before a run is created. The schema runs first andvalidateInputthen sees its output (parsed, coerced, defaulted), so keep the structural checks in the schema and the cross-field or lookup checks invalidateInput. The run stores the input as sent, and the body receives the parsed value, re-parsed before every turn — keep the schema deterministic and bumpversionwhen it changes. A schema failure throws aSchemaValidationError(aFatalError, codeinvalid_input).
Both depend on the worker loop running somewhere, like timers and retention.
14. Observe it
- The dashboard is the operating surface — run list, step graph, retry/cancel/fix-and-replay. It fails closed by default: set
DURABLE_DASHBOARD_TOKEN, or supply your ownauthorizeguard or the session layer. - Page on stranded runs.
suspendedis not an error, so a lost dispatch alerts nobody. Registerengine.onStalled(and tunestalledAfter, default'15m') so the worker loop pages you once per stranded episode;node ace durable:runs --staleis the same query by hand. See the stalled-run pager. - Dead runs.
engine.onDeadfires for every run that crossesmaxRecoveryAttempts. - Traces. OpenTelemetry turns runs into traces — a root span per run, a child span per step — and continues them on remote workers through
traceparent. One caveat before you build a latency dashboard on it: the root span ends when the run first suspends and is not reopened on resume, so steps that settle after a sleep, a signal wait or a remote dispatch show up as spans without that parent. - Metrics.
attachDurableMetrics(engine)(from@adonis-agora/durable/otel, no OTel SDK needed) keeps run and step counters with p50/p95/max durations, andcollectMetricsserves a Prometheus scrape. Both count only the events this instance produced, so a fleet behind a control plane is not double-counted. See Metrics. - Telescope. Register
durableTelescopeExtension()under Telescope'sextensionsfor a Workflows dashboard (golden signals, current-state gauges, recent failures, the worker fleet) next to your requests and jobs. With@adonis-agora/diagnosticsinstalled the provider bridges engine events onto the diagnostics bus automatically. See Telescope and lifecycle events.
When something does go wrong, start from the symptom: Troubleshooting and Failure modes & recovery.
CLI
Ace commands for durable — durable:work runs the store-backed worker loop, durable:worker runs a store-less thin worker, durable:runs lists runs (including the --stale view), durable:retry re-enqueues a run, durable:export captures a replay fixture for CI, and make:workflow scaffolds a workflow.
Troubleshooting
Symptom-first answers to the errors and stuck states you will actually hit — NonDeterminismError, NestedWorkflowCallError, a caught suspend that corrupts a run, runs wedged in pending, running or suspended, timers and schedules that never fire, waitForRun hanging across pods, jobs stuck in active on the queue transport, steps that run twice, node ace processes that hang or steal work, schema and dashboard surprises, and the failures fixed by upgrading to 0.50 or 0.51.