CLI

@adonis-agora/durable ships six ace commands: two long-running workers (durable:work for a store-backed pod, durable:worker for a store-less thin pod), a run lister, a retry, a replay-fixture exporter, and a workflow generator. Except for durable:worker, they all resolve the WorkflowEngine bound by the provider, so they operate on the same engine (and store) your app is configured with.

node ace configure @adonis-agora/durable

configure registers the commands barrel (@adonis-agora/durable/commands) in adonisrc.ts so the commands become available to ace — there's no separate package to install.

durable:work — the worker loopCopy Anchor Link

The long-running worker. On an interval it picks up pending runs, recovers crashed runs, resumes due timers, sweeps execution timeouts, and fires any due scheduled workflows. It stays alive until SIGINT/SIGTERM, then drains in-flight executions so a deploy hands off cleanly.

node ace durable:work
node ace durable:work --interval=500 --drain-timeout=15000

Each tick runs engine.runPending() → engine.recoverIncomplete() → engine.resumeDueTimers() → engine.sweepTimeouts() → fire due schedules (the schedules from config/durable.ts merged with any colocated static schedule, config winning on a key collision, then the persisted schedules when persistedSchedules is on) → engine.sweepRetention() (retention eviction, self-throttled to once a minute) → engine.sweepStalled() (the stalled-run pager, same throttle, a no-op without listeners). Each phase is isolated: one that throws (a transient store hiccup) is logged as tick phase <name> failed: … and the rest of the tick still runs.

On start the loop also calls engine.startConsumers(). Broker transports (BullMQ, DB, queue) defer their consumer subscriptions when the app boots in the console or repl environment (the default consumers: 'auto' in config/durable.ts), so a one-shot node ace command (a migration, durable:runs, node ace repl) stays a pure producer and never claims jobs. durable:work is the process that declares itself a worker and flushes them. consumers: 'always' keeps consumers eager everywhere; consumers: 'never' defers them in every environment, web included. A custom loop built on runWorkerLoop gets the same behaviour; one built on runTick alone must call engine.startConsumers() itself.

On SIGINT/SIGTERM the loop stops picking up new runs, waits for the tick in flight, then calls engine.drain() with whatever is left of --drain-timeout. A tick that outlives the whole budget is abandoned with a warning instead of holding the pod past its termination grace period, so keep --drain-timeout below your orchestrator's grace period.

Run this as a dedicated process (or several) when you split web from worker; in a single-process app the in-process dispatcher already runs workflows, so this is optional but recommended for recovery and timers.

durable:worker — the store-less workerCopy Anchor Link

The worker for a role: 'tenant' pod: it serves step bodies and never touches a state store. It boots the container-bound WorkerRuntime, registers every handler under stepsPath (default app/steps; every directory when it's an array), advertises its descriptor and heartbeat so the control plane can route to it, and drains on SIGINT/SIGTERM.

node ace durable:worker

It takes no flags — everything comes from config/durable.ts (partition, which falls back to namespace and then 'default'; namespace; transport; stepsPath; workflowsPath, whose workflow names it advertises in the descriptor). On a role: 'tenant' pod it uses the runtime the provider bound, which publishes the descriptor and heartbeat on the shared Redis. On any other role it builds a self-contained runtime from the configured transport whose advertising is a no-op, and it throws when that transport cannot serve handlers (no handle()). Running it on a store-backed role logs a warning and points you at durable:work; see Thin workers for the full topology.

durable:runs — list runsCopy Anchor Link

Lists recent runs, optionally filtered by status and workflow, reading through engine.listRuns.

node ace durable:runs
node ace durable:runs --status=failed
node ace durable:runs --workflow=checkout --limit=100

The table has the columns RUN, WORKFLOW, STATUS, UPDATED, RECOVERY and PENDING:

RUN        WORKFLOW  STATUS     UPDATED  RECOVERY  PENDING
ord-42     checkout  suspended  4h32m    -         4h32m (attempt 1)
ord-43     checkout  suspended  41m3s    -         41m3s (attempt 1, hb 12s)
ord-40     checkout  completed  2d6h     -         -

Every row also carries liveness signals — how long ago the run was updated, how many times recovery has picked it up, and the age of its oldest pending remote step. That matters because suspended alone cannot tell a run mid-step apart from a run whose dispatch was lost.

--stale is the query that separates them. It narrows the listing to running/suspended runs whose oldest pending remote step is older than the threshold — the stranded signature — and prints a recovery hint for each one (engine.redispatchPending(runId) or node ace durable:retry <runId>). A step whose worker heartbeat is younger than the threshold is excluded however old the step is: a beating worker is mid-flight, and re-driving it would run it twice.

# What has been sitting on a remote step for over an hour?
node ace durable:runs --stale=1h

The threshold is a single-unit duration: ms, s, m, h or d (90s, 15m, 4h, 2d). Anything else (--stale=15mins) silently falls back to the 15-minute default rather than failing. Without --status, --stale scopes the query to running/suspended so newer terminal runs do not push stale candidates out of the --limit window.

Set the threshold above your longest legitimate step that does not heartbeat, or a slow-but-healthy run shows up as stranded.

For the listing to show runs across processes, the engine must be backed by a persistent store (config/durable.ts) — the default in-memory store only sees this process's runs.

durable:retry — re-enqueue a runCopy Anchor Link

Re-enqueues a run for a worker to (re-)execute. It calls engine.requeue(runId): the run goes back to pending, any stale lease is cleared, and a worker resumes it (replaying its checkpoints, re-attempting the failed step).

node ace durable:retry checkout:1234

An unknown run id exits 1 with Run <id> not found.. Run durable:work (or any worker) to pick it up afterward. This is the terminal equivalent of the dashboard's retry button — handy for replaying a dead run once you've shipped a fix.

durable:export — capture a replay fixtureCopy Anchor Link

Exports a run's replayable history — the run row plus its full checkpoint timeline — as JSON, for the replay-CI loop. Commit the file and assert it with assertReplayable(register, parseRunHistory(json)): a code change that renames, reorders or removes a step at a position the history already recorded then fails the build instead of corrupting an in-flight run on deploy.

node ace durable:export ord-42                                       # JSON to stdout
node ace durable:export ord-42 --out tests/fixtures/checkout.json    # or to a file

An unknown run id exits 1 with an error. In code, the same capture is captureHistory(engine, runId) from @adonis-agora/durable/testing.

make:workflow — scaffold a workflow classCopy Anchor Link

Scaffolds a BaseWorkflow subclass under app/workflows/, the parallel to @adonisjs/queue's make:job. The generated class is auto-registered on the engine at boot — see app/workflows & make:workflow.

node ace make:workflow order

The name may carry the Workflow suffix or not: order, Order, OrderWorkflow and order_workflow all write app/workflows/order_workflow.ts with the class OrderWorkflow and the registered name 'order'. The class extends BaseWorkflow with a static workflow = { name, version } identity and a run(ctx, input) method; standard generator flags (such as --force) are passed through to the stub.

Building your ownCopy Anchor Link

The pieces these commands are made of are exported from @adonis-agora/durable, so a custom worker process or admin script composes them instead of re-implementing the loop:

For example, an ops endpoint that reports stranded runs without shelling out to ace:

import type { HttpContext } from '@adonisjs/core/http'
import { DEFAULT_STALE_MS, attachLiveness, filterStale, listRuns } from '@adonis-agora/durable'
import engine from '@adonis-agora/durable/services/main'

export default class OpsController {
  async strandedRuns({ response }: HttpContext) {
    const runs = await listRuns(engine, { statuses: ['running', 'suspended'], limit: 200 })
    const stale = filterStale(await attachLiveness(engine, runs), DEFAULT_STALE_MS)

    return response.ok({ stranded: stale.map(({ run }) => run.id) })
  }
}

What the CLI does not doCopy Anchor Link

There is no single-run inspector and no cancel command. To look at one run's step timeline, open it in the dashboard or read engine.listCheckpoints(runId); to cancel, call engine.cancel(runId) from a script or use the dashboard's cancel action.