Dashboard

The dashboard mounts an operations console into your AdonisJS app: a React single-page app plus the JSON API it runs on. It reads straight from the state store through the engine's read API, so it works without an OpenTelemetry collector. It ships in the main @adonis-agora/durable package — no extra install.

node ace configure @adonis-agora/durable

configure registers @adonis-agora/durable/dashboard_provider in adonisrc.ts and publishes config/durable_dashboard.ts. The provider registers the routes on boot — nothing else to wire.

ConfigurationCopy Anchor Link

config/durable_dashboard.ts controls where it mounts and who may reach it:

import { defineConfig } from '@adonis-agora/durable/dashboard'

export default defineConfig({
  // enabled: true,
  // path: '/durable',
  // authorize: (ctx) => ctx.auth.user?.isAdmin === true,
})

AuthorizationCopy Anchor Link

The default authorize fails closed in every environment: it requires a bearer token equal to the DURABLE_DASHBOARD_TOKEN env var, and denies everything when that var is unset. (Before 0.37 it was open outside NODE_ENV=production — which left dev boxes, staging, and any deployment with a misspelled/unset NODE_ENV serving an unauthenticated console.) The token is accepted from an Authorization: Bearer <token> header or an x-durable-token header, compared in constant time; the ?token= query param works on GET and HEAD only (it exists for the SSE live-tail, whose EventSource cannot set headers) — never on a mutating route, so a leaked/logged URL can't act as a destructive magic link.

Three ways to open the console up, in order of preference:

1. configure dashboardAuth — the session guard becomes the gate (the default token guard steps aside);
2. set DURABLE_DASHBOARD_TOKEN (or supply your own authorize);
3. for local development only, allowUnauthenticated: true serves the console with no auth at all — it exists so the old open-by-default behavior has to be spelled out in config (and greppable at review time). It logs a warning at boot.

Two request-level guards apply on top of whatever gate you pick:

• Cross-site rejection — a browser-originated cross-site mutating request (by Sec-Fetch-Site, else Origin vs Host) gets 403 { "error": "cross-site request rejected" } before any handler runs, closing the drive-by CSRF shape (the mutating endpoints need no body, so they used to be firable as simple cross-origin POSTs). Sec-Fetch-Site: same-site counts as cross-site on purpose, so a sibling subdomain cannot drive the console's actions either; an Origin: null is rejected too. Non-browser clients send neither header and pass.
• Session roles — a dashboardAuth session whose roles list is non-empty must include operator or admin (compared case-insensitively) to hit a mutating route; everything else is a viewer (403). An empty/absent roles list keeps full access, so existing hooks that never set roles are unchanged.

Every mutating attempt is also audited before its handler runs — actor (session name/id, token, or anonymous), method and path without the query string — through the app logger, or your own audit hook on the config. Reads are not audited. Payloads an operator submits (fix-and-replay input, signal/update/task bodies, webhook callbacks) are capped at 1 MiB (413). A handler that throws answers a generic 500 { "error": "internal error" }; the detail goes to the app logger, never to the client.

Override authorize with your own session/role/IP guard — a denied request gets a 403: JSON for an API call, the access-denied page for a browser:

export default defineConfig({
  authorize: async (ctx) => {
    await ctx.auth.check()
    return ctx.auth.user?.isAdmin === true
  },
})

If you use @adonis-agora/authz for RBAC, the shared authorizeByRoles helper reads your user and roles for you — same semantics as the requireRole middleware, no ctx narrowing:

import { authorizeByRoles } from '@adonis-agora/authz'

export default defineConfig({
  authorize: authorizeByRoles({ roles: ['ADMIN'] }),
})

The user is resolved from ctx.auth.getUser() (authkit) or ctx.auth.user (any guard); roles is any-of over effectiveRoles (token claims ∪ DB roles). Anonymous requests are denied (false). The same helper works across every @adonis-agora dashboard — durable, telescope, media and agent — so one RBAC gate config reads the same everywhere.

Prefer a real sign-in page, or to open the console from an app where the operator is already signed in? Opt into the built-in dashboard auth (dashboardAuth), which layers a signed session cookie on top of authorize.

The access-denied pageCopy Anchor Link

A refused API request gets JSON (403 { "error": "forbidden" }, or 401 { "error": "unauthorized", "auth": { "modes": [...] } } without a session) — that is what the console's own fetch calls expect. A refused page navigation — the console shell, its assets, or the Mode-A-only "open this from your app" case — gets a real page instead: a dark card in the console's visual language showing the status, a sentence explaining the refusal, a "Back to app" link and, when dashboardAuth.login is configured, a "Sign in" button. It carries no inline script, so a nonce'd script-src CSP cannot break it (its one inline <style> picks up @adonisjs/shield's request nonce).

Tweak it with accessDenied — every field optional:

export default defineConfig({
  accessDenied: {
    brand: 'Entre Textos',           // eyebrow + <title>; default "Durable"
    title: 'Sem acesso',             // default depends on the refusal
    message: 'Peça ao admin para liberar o console de workflows.',
    homeHref: '/admin',              // "Back to app"; default "/", `false` hides it
    homeLabel: 'Voltar',
    loginHref: '/entrar',            // default: the built-in login page when one exists
    loginLabel: 'Entrar',
    accent: '#f59e0b',               // any CSS colour; default: the console's lime
  },
})

Or replace it. Pass a function and it receives the refusal (status, reason — 'forbidden', 'unauthenticated' or 'session-required' — basePath, loginHref, and the CSP nonce when there is one) plus the HttpContext. Return an HTML string to have it served with the right status; answer the request yourself and return nothing to make the provider stand down:

export default defineConfig({
  accessDenied: (info, ctx) => {
    if (info.reason === 'unauthenticated') {
      ctx.response.redirect(`/login?next=${encodeURIComponent(info.basePath)}`)
      return
    }
    return `<!doctype html><title>${info.status}</title><h1>Sem acesso</h1>`
  },
})

An authorize hook that already wrote a redirect still wins — the provider never overwrites a location header, with or without accessDenied.

Redacting payloadsCopy Anchor Link

The store may hold payloads encrypted (a codec-wrapped store), but the console reads through the codec, so operators always see them decoded. redact keeps a card number or PII out of the operator's browser. Each hook receives the serialized shape and returns what to send. It runs on the runs list, the run detail and fan member pages:

export default defineConfig({
  redact: {
    run: (run) => ({ ...run, input: undefined }),
    checkpoint: (cp) => ({ ...cp, output: '[redacted]' }),
  },
})

Where it mountsCopy Anchor Link

All routes are relative to path (default /durable), and every one of them runs behind the authorize guard.

PagesCopy Anchor Link

The console routes client-side on the URL hash (#/run/<id>), so a deep link needs no server-side rewrite.

Reading runsCopy Anchor Link

GET /api/runs accepts two spellings of the same filter, both served by the console's RunFilter class (a BaseModelFilter over a run-query draft):

• the flat spelling the console has always sent: ?status=failed&tag=etl (repeat a key for a set: ?tag=etl&tag=nightly), attr as key:op:value repeats;
• the structured envelope @adonis-agora/filter-client builds: filter[status]=failed, filter[tag][]=etl&filter[tag][]=nightly, filter[attr]=tier:eq:enterprise.

The attr ops are eq, ne, gt, gte, lt, lte, in — opaque through both spellings (filter[attr.<key>][<op>] takes the filter lib's canonical names: equals, notEquals, gt, gte, lt, lte, and in/isAnyOf). Values are coerced: true/false become booleans, anything numeric becomes a number, everything else stays a string. Only the first two colons delimit, so a value may contain colons of its own. in takes a |-separated set (tier:in:pro|enterprise) and matches it as OR inside the one predicate — two eq predicates on one key are ANDed like every other pair, which no run can satisfy.

GET /durable/api/runs?namespace=eu-west&attr=amount:gte:5000&attr=tier:eq:enterprise&status=failed

A refused structured filter — unknown field, unsupported operator, a group the draft cannot express — answers 400 instead of silently widening. Unknown flat params stay ignored, so old callers (and endpoint mechanics like page) keep working. (static throwOnInvalid on the filter class is what makes the envelope strict.)

Paging is the same offset shape every Agora library takes (@adonis-agora/filter's page/size): a 1-based page number and a page size, ?page=2&size=100. The 0-based offset it resolves to is the store's business, never the wire's.

The response is { runs, meta: { page, size, count }, statuses }. The envelope key is meta, the same name AdonisJS/Lucid's own .paginate() uses — every @adonis-agora/* listing spells it that way. meta.count is how many rows this page returned, not a total — count === size is the "there may be more" signal.

The picker enumerationCopy Anchor Link

GET /api/runs/values answers what the console's tag, tenant and attribute pickers list: the distinct values one axis takes across the runs matching every OTHER active predicate, with counts — most common first, engine-minted tags (singleton:<key>) last. Served through the filter lib's groupByCountFromRequest with the console's adapter. The axis rides groupByCount[field] (workflow, status, namespace, tag, attr for the keys in use, attr.<key> for the values under one key — top-level field works too); the scope rides the same params as GET /api/runs (minus status, which is deliberately dropped so the offers don't collapse to the status being viewed). limit bounds the rows (default 100, capped at 200), offset pages them, search narrows them server-side — this endpoint keeps limit/offset on purpose: it is filter's own group-by-count aggregation (groupByCount[limit]/groupByCount[offset]), not a run listing, so limit/offset IS the aligned spelling here — a rare value outside the first page stays reachable by typing. The response is a bare [{ value, count }] array.

There is no separate checkpoints route: a run's step timeline comes back as the timeline array of GET /api/runs/:id, each entry carrying seq, name, kind, status, attempts, workerGroup, input/output/error, events, parallelGroup, and the derived durationMs / queueMs.

A fan with more than 50 members (a ctx.all over children or over a step, a polyglot gather) is summarized: its members leave timeline, and the fan comes back as one entry of fans with its group, label, seq range, size (members started so far), total (its declared size: a ctx.all group carries it as all:<seq>:n<size>, a step fan as steps:<seq>:<size>), notStarted, failFast, counts by status (a failed child counts as failed), start and finish, and its first 20 failed members. A fan declared wider than 50 is summarized even while only a few of its waves have been sent. children then lists the first 200 child run ids and childCount carries the total. GET /api/runs/:id/fans/:group pages the members on demand. ?full=true returns the detail unsummarized. On a store-less tenant pod the control plane summarizes before the detail crosses the wire.

Polling the run detail: ETag and 304Copy Anchor Link

On a store with the checkpoint aggregates (checkpointStats + queryCheckpoints, which the Lucid and in-memory stores have), the summarized detail is built from one GROUP BY over the run's checkpoints plus a few targeted reads — the rows outside wide fans, each fan's first 20 failures, the folded checkpoints and the first child edges — instead of every checkpoint. A 40,000-member fan is drawn from a handful of rows.

Those responses carry an ETag (derived from the run row and the checkpoint buckets) and Cache-Control: private, no-cache. A request whose If-None-Match still names the ETag gets a bare 304, answered after reading the run and the buckets only. The browser adds If-None-Match to the console's polls by itself, so an unchanged run costs a 304 instead of a body. GET /api/runs/:id/fans/:group is versioned the same way, per page. ?full=true, and a store without the aggregates, answer in full without an ETag, as before. The options cross the tenant proxy unchanged.

Live-tailing one runCopy Anchor Link

GET /api/runs/:id/stream is an SSE endpoint. Each frame is a default message event whose data is one serialized engine lifecycle event for that run:

const source = new EventSource(`/durable/api/runs/${runId}/stream`)

source.onmessage = (msg) => {
  const event = JSON.parse(msg.data)
  // event.type is one of run.started / run.completed / run.failed / run.suspended /
  // run.routed / run.unrouted / step.started / step.completed / step.failed /
  // step.progress (a live step event, never persisted) / capability.unavailable / protocol.incompatible
  console.log(event.type, event.name, event.durationMs)
}

Open streams are capped at 100 per instance and 10 per client address. Past either cap the route answers 429 { "error": "too many open event streams" }.

The tail is cross-pod when the engine has a control plane. Every instance broadcasts the lifecycle events it produces over the control plane, and the others deliver them to their local subscribers. A dashboard served from a web process can therefore live-tail a run that is executing on a durable:work pod. Without a control plane the stream still works, but only for runs executing in the same process as the dashboard.

Acting on runsCopy Anchor Link

Every one of these is non-blocking: they enqueue work and return, never replaying a workflow inline in the HTTP request.

Fix and replayCopy Anchor Link

A run that failed on a bad input cannot be "retried" into success — a retry replays the same input. retry-with-input instead starts a brand-new run with a corrected input:

const res = await fetch(`/durable/api/runs/${runId}/retry-with-input`, {
  method: 'POST',
  headers: { 'content-type': 'application/json', ...headers },
  body: JSON.stringify({ input: { orderId: 'ord_42', amount: 1999 } }),
})

const { result } = await res.json()
result.runId // 'run-abc~retry~1f4c9e02'

The new run inherits the original's workflow, tags and namespace, and starts with a clean history — no checkpoints are replayed. The original run is left completely intact, keeping its status, its error and its full timeline as the record of what went wrong.

The link between the two is the run id itself: the new id is <originalRunId>~retry~<short-uuid>, and the console parses that suffix to draw the lineage between them. There is no separate parent column.

Continuing a breakpointCopy Anchor Link

ctx.breakpoint() parks a run indefinitely at a known point, with a visible breakpoint checkpoint in its timeline and no wake timer — a deliberate manual gate rather than a failure. POST /api/runs/:id/continue releases it, which is the console's counterpart to calling engine.continue(runId) from code.

It answers 404 with run <id> is not paused at a breakpoint when the run has no pending breakpoint checkpoint, so the call is safe to fire blindly.

Human-in-the-loop: signals, updates and tasksCopy Anchor Link

The runs list already names what a suspended run is parked on (the waiting column); these three verbs let the operator act on it from the console instead of dropping into a REPL:

• POST /api/runs/:id/signal delivers { token, payload? } to a ctx.waitForSignal rendezvous. It is guarded: unless you pass force: true, the token must be one the run is currently waiting on — a typo'd token would otherwise buffer a stray payload silently instead of resuming anything. A mismatch answers 409 carrying the waitingOn token list, so a client can offer the real choices; force: true buffers the payload anyway (the reliable-delivery path).
• POST /api/runs/:id/update/:name delivers { arg? } to a ctx.onUpdate(name) point. The workflow's registered validator arbitrates server-side: a rejection answers 422 with the reason, and nothing is delivered.
• POST /api/runs/:id/tasks/:name/complete (body { result? }) and .../fail (body { error }, a string; it defaults to failed from console) settle an external ctx.task — the console equivalents of engine.completeTask / engine.failTask. The response is honest about delivery: { result, delivered: true } when a live waiter consumed it, { result: null, delivered: false } when no waiter was live yet and the completion was buffered for the run to consume when it reaches the task's wait — buffered is success, not a 404.

On a store-less tenant pod these verbs answer 404: they need the engine's signal surface. retry, retry-with-input and continue do work there; they ride the wire to the control plane (see the gateway verbs).

Bulk actionsCopy Anchor Link

POST /api/bulk/:action applies one action to everything a filter matches, where :action is retry or cancel (anything else is a 400). The selection comes from the query string, using the same filters as GET /api/runs (status, workflow, tag, namespace, origin, createdAfter / createdBefore, attr, in either spelling) — there is no id list to assemble:

POST /durable/api/bulk/retry?status=failed&workflow=charge-order&namespace=eu-west
POST /durable/api/bulk/cancel?status=suspended&tag=stuck&compensate=true

?compensate=true runs the saga compensations and is honoured for cancel only.

Asking for the saga undoCopy Anchor Link

Both cancel routes read compensate from either the query string or the JSON body, so ?compensate=true and { "compensate": true } are equivalent. A value spelled true / 1 / yes / on (or the bare ?compensate) means yes; false / 0 / no / off means no; absent means no.

Anything else is a 400. That is deliberate: the two ways to guess at an unreadable value are "skip an undo the operator asked for" and "run an undo they did not", and both would be silent. An explicit rejection is the only answer that cannot mislead.

The response is { matched, applied }. matched is how many runs the filter returned; applied is how many the action actually changed. They differ when a run has already moved on — a run that throws or answers "nothing to do" is skipped rather than aborting the batch.

Live queries, webhooks and metricsCopy Anchor Link

GET /api/runs/:id/events/:key is a side-effect-free read of a live (or finished) run, the HTTP form of engine.getEvent(runId, key). It answers { value }, or {} when the run never published that key, and 404 for an unknown run. The value goes through your redact.checkpoint hook as the output of the event:<key> checkpoint, since that is where it is stored.

POST /api/webhooks/:token resumes a run parked on ctx.webhook().wait(). It only takes webhook tokens (wh:<runId>:<seq>), so it is not a second signal endpoint. A callback for a run waiting on that token resumes it and answers { result }. One that lands before the run reaches wait() — the url was handed out, then the run kept going — is buffered and answers 202 { buffered: true }, as long as the run exists, is not terminal and has not consumed that webhook yet; the later wait() consumes it. A callback for an unknown or terminal run, or a duplicate of one already delivered, answers 404 and buffers nothing. Past the 1 MiB payload limit it answers 413.

GET /api/metrics serves text/plain; version=0.0.4 for a Prometheus scrape:

# TYPE durable_runs_total counter
durable_runs_total{event="completed"} 1284
durable_runs_total{event="failed"} 7
# TYPE durable_steps_total counter
durable_steps_total{event="completed"} 9921
...
# TYPE durable_pending_runs gauge
durable_pending_runs 12
# TYPE durable_running_runs gauge
durable_running_runs 3
# TYPE durable_dead_runs gauge
durable_dead_runs 0
# TYPE durable_group_queue_depth gauge
durable_group_queue_depth{group="reports"} 4
# TYPE durable_group_live_workers gauge
durable_group_live_workers{group="reports"} 0

• The counters are the collectMetrics counters of the process that served the scrape, started when the dashboard boots. They count only the events that process produced, so scrape every pod and sum: each event is counted once across the fleet. A tenant pod has no engine and serves no counters.
• The gauges are read at scrape time: durable_pending_runs (the dispatch backlog), durable_running_runs, durable_dead_runs (the dead-letter queue), and per worker group its queue depth and live-worker count. depth > 0 with 0 live workers is the alert. On a tenant pod the run gauges count the tenant's own runs through the control plane, and are left out when the control plane is too old to answer.

SchedulesCopy Anchor Link

These are the console's Schedules tab, riding engine.listSchedules / setSchedulePaused / triggerSchedule. The listing returns { schedules }, each entry carrying the cadence (cron/everyMs/timezone), the effective paused state with a pausedAtRuntime flag telling you whether a console override (which wins over config until the next deploy) is in force, the current window's lastFireAt/nextFireAt/currentWindowRunId, and that run's lastRunStatus when it exists. trigger is idempotent by the window's deterministic run id — triggering a window that already fired returns the existing run rather than forking a duplicate. An unknown :key answers 404; an action other than pause/resume/trigger is a 400.

Fleet and topologyCopy Anchor Link

GET /api/workers returns the raw GroupHealth[] — for each routing token, its queue depth and the individual live workers with their instanceId and lastBeatAt. That is what tells "one worker is wedged" apart from "the whole group is down".

On the BullMQ, queue and db transports, each worker's heartbeat also carries a live status telemetry snapshot (WorkerStatus) that the console renders on its worker cards: the concurrency mode and limit (fixed or adaptive, with min/max), inFlight, RSS (rssBytes, plus rssLimitBytes/rssPct when a ceiling is known), cpuPct, throughputPerMin, p95Ms, and the last adaptive concurrency adjustment with its reason. Workers that beat without one (a thin worker runtime's compact beat) omit status — the cards degrade to liveness only.

GET /api/health is the older, compact view of the same data: it collapses liveWorkers to a count and adds stalled: depth > 0 && liveWorkers === 0. It stays for hand-written clients and probes; prefer /api/workers for anything new.

GET /api/compat powers the compatibility panel described below: per group, each pod's protocol range and capabilities with a compatible / degraded / incompatible outcome and the exact reason, alongside every blocked run and why it parked.

The routes are all named (durable_dashboard.index, durable_dashboard.runs.show, durable_dashboard.runs.retry_with_input, durable_dashboard.bulk, …), so you can reference them from your own code or wrap them in middleware.

What it showsCopy Anchor Link

• Runs — every run with its status, filterable by status, workflow, tag, namespace, origin, creation time and search attribute. The tag, tenant and attribute filters are value pickers: they list what the runs actually contain (counted server-side over the rest of the active filter, searchable, paged), take several values at once, and still accept a typed value. pending is a run created and enqueued but not yet picked up; a dead run carries a distinct badge so a poison pill is easy to spot.
• What a suspended run is waiting on — a suspended run parked on a signal, a webhook callback, a child run or a breakpoint is surfaced as waiting, with what it is blocked on right in the row (signal approve, child ord-42.child.0, breakpoint). See below.
• Step timeline — the selected run's steps in order, tagged by kind (local / remote / sleep / signal), with worker group, queue wait and duration. In-flight steps appear while they run, not only on completion — a remote step in flight reads as pending, and a local step's executing body as running (the trackStepStart option).
• Live tail — the open run streams its events over SSE, so a long run updates without a reload.
• Children and lineage — a run's child workflows are linked from its detail, as is the original of a fix-and-replay run.
• Wide fans — a fan of more than 50 members draws as one node in the graph and one row in the timeline, labelled label ×total, with a segmented progress bar against its declared size, a legend (done, failed, running, pending, not started), 600 of 40000 started, and its sampled failures. Once the run has finished (completed, failed, cancelled or dead), members still running or pending read as interrupted and the ones never sent as never ran, marked (failFast) when a failFast fan stopped early. A blocked or suspended run is not finished, so its fans stay in flight. Expanding it lists the members in a virtualized list that pages them in as you scroll, with a failed-only toggle. The run header lists the first 12 children and counts the rest, and only the first 50 children's details are fetched to label their rows.
• Actions — retry, cancel, cancel + undo (saga compensation), fix-and-replay, redispatch, continue, the human-in-the-loop verbs (deliver a signal, a validated update, or a task completion), and the bulk equivalents of retry and cancel.
• Schedules — the ticked scheduled workflows with their fire windows and effective pause state, plus pause / resume / run-now controls that apply fleet-wide at runtime.
• Worker cards — on the BullMQ, queue and db transports, each worker's live telemetry: concurrency, in-flight, RSS/CPU, throughput and p95.

The waiting stateCopy Anchor Link

waiting is not a ninth run status — it is a derived view of a suspended run. The engine records one generic suspended status; the listing joins it against the signal-waiter table (listSignalWaiters) and reports what the run is parked on:

The kind comes from the waiter token's prefix, so no extra store read is needed: wh: is a webhook, child: a child run, bp: a breakpoint (named breakpoint rather than its raw bp:<runId>:<seq> token), an event: token a ctx.waitForEvent (named by its decoded event name), and anything else a signal named by the token itself. engine.continue resumes a breakpoint through the same signal machinery.

A suspended run with no waiter gets no waiting value, and that absence is itself informative: a bare ctx.sleep, an in-flight remote step and a run whose wake was lost all fall into that bucket. Use durable:runs --stale to separate the last one from the first two.

waiting appears on the rows of GET /api/runs only, and only when the store exposes listSignalWaiters — a store-less tenant pod skips it silently.

Display statesCopy Anchor Link

The waiting value is one input to a richer display state that the runs list and the run detail header both derive the same way, so they always agree. The stored status never changes. The console refines what it shows:

no-worker is gated on a real backlog, not on zero live workers alone. A worker only heartbeats for a group while it serves it, so an idle group (a run parked on its reconcile timer with nothing enqueued, a schedule between fires) legitimately reports no live workers without anything being stuck. Worker health comes from GET /api/workers, so on transports without groupHealth (anything but BullMQ, queue and db) nothing ever reads no-worker.

Filter pickersCopy Anchor Link

The tag, tenant and search-attribute filters list the values the runs actually carry, each with a count, and take several values at once. A free-text box is usable only by an operator who already knows the tenant name or tag, and a typo there returns an empty list that reads exactly like "no runs match".

• Options are counted over the runs the other filters already select. Pick a tenant and the tag picker offers that tenant's tags, so a picker never offers a value whose result set is empty. Status is deliberately left out of that scope, so the options don't collapse to the status being viewed.
• Several values on one axis are ORed, and the axes are ANDed. For example: these two tenants, tagged either of these two ways. On search attributes that needs the in operator (shown as is any of), because two equality predicates on one key are ANDed like every other pair and no run has two values for one attribute.
• Typed text still works. The offered list is a bounded top-N (tag cardinality grows with the data, since a singleton:<key> tag is minted per key), so a rare value can be real and absent from it. Engine-minted tags are listed last.

A tag or tenant chip on a run row adds to the selection instead of replacing it, so a "these two kinds of run" view can be built by clicking.

Tenant and origin filtersCopy Anchor Link

Tenant matches the run's namespace exactly and is pushed down to the store as the namespace param on GET /api/runs and POST /api/bulk/:action. Its default is every namespace. The dashboard's read paths are not namespace-scoped, so this is a filter the operator chooses, not a scope imposed on them, and not an authorization boundary either. Selecting several tenants compares them side by side.

Origin (which package declared the run's workflow) is a row of facet chips above the list: all, one chip per package, and app when any loaded run has none, each with a count. A run with no origin is the app's own workflow (one registered without a package origin), as are runs created before the field existed. The row only appears once at least one loaded run has a named package origin: on an app where no workflow declares one, every run is the app's own and there is nothing to choose between. An exact-match store filter can't express "no origin", so the chips are counted over the rows the console has already loaded rather than pushed down. That keeps the app count on screen while a package is selected, so the app's own runs never look like they don't exist.

At scale: paging and list rowsCopy Anchor Link

A deployment accumulates runs for as long as retention keeps them, so the console treats the list as pages:

• The console asks GET /api/runs for 100 rows at a time and fetches the next page as the last loaded rows scroll into view. The server defaults size to 50 and caps it at 200.
• The list is virtualized: only the rows in the viewport are mounted, so a deep page costs the same DOM as a shallow one.
• GET /api/runs returns list rows, not whole runs: id, workflow and version, status, namespace, origin, tags, timestamps, recoveryAttempts and waiting. input, output and error are left out, because no row renders them and on a busy deployment they are most of the bytes. GET /api/runs/:id still carries all three.

Dead-letter runs and their linksCopy Anchor Link

A run that crash-recovery gives up on after maxRecoveryAttempts moves to the terminal dead status. The console shows it as a first-class status, with a dead filter chip and a distinct badge. A dead run stays inspectable and can still be retried from the UI once you've fixed the cause.

When your onDead handler starts a handler run under the id dlq:<deadRunId>, the console links the two both ways. A dlq: run gets a red dlq chip in the list and a banner linking back to the dead run. A dead run's detail looks up dlq:<its id> and, when that run exists, shows a Dead-lettered banner with an open DLQ handler link.

Recovering a lost remote step dispatchCopy Anchor Link

A remote step with no timeoutMs dispatches its job, persists a pending checkpoint and suspends until the worker's result resumes it. If the worker crashed mid-step or the transport dropped the job, the result never comes. By design the reconcile re-drive re-suspends a still-pending step instead of re-dispatching it, so a slow worker is never double-run, and the run waits indefinitely.

The timeline flags this instead of drawing it as a healthy in-flight step. A remote step pending for more than 10 minutes is checked against its group's worker heartbeats:

• If a worker beat for the group in the last 90 seconds, the row reads being worked by <instanceId> — heartbeat Ns ago (plus the worker's in-flight count when it reports one). The dispatch reached a live worker, so wait.
• If no worker did, the row reads awaiting worker result — dispatched Nm ago, no live worker (possibly lost).

A live heartbeat proves the worker process is alive, not that the step is progressing. From a lost row you have two ways to recover:

• Re-dispatch — POST /api/runs/:id/redispatch (the console's Re-dispatch action) re-enqueues every pending remote step on the run, bumping its attempts, so the idempotent step re-runs and its result resumes the run. It is the same as engine.redispatchPending(runId).
• Broker bridge (BullMQ and queue) — when a task job reaches a terminal failed state with no result (its worker crashed, or it stalled past the broker's stalled-count limit), the transport publishes a retryable failed result for the step, so the engine settles the checkpoint and its normal retry re-dispatches. That bridge lives in the consumer: a worker on another SDK that does not implement it still leaves the step pending.
• Self-heal — set remoteRedispatchMs (and optionally remoteRedispatchMax, default 10) in config/durable.ts, and a reconcile pass re-dispatches a remote step still pending past the window by itself, recording a warn-level step.redispatched event on the checkpoint; the step.started event of that re-drive carries redispatched: true, so an alert can tell "re-driven after a lost worker" apart from "retried after a failure". Once the bound is spent, the step fails with remote_step_lost. This is off by default. See Failure modes for the window-sizing rules.

Fleet healthCopy Anchor Link

When you run a split or polyglot cluster, the console grows a health and compatibility panel fed by the handshake descriptors and diagnostics events:

• Per pod — its protocol version, negotiated level (compatible / degraded / incompatible), and a red flag with the exact reason on a mismatch (e.g. "no common protocol major: local speaks [1, 1], remote speaks [2, 2]").
• Blocked runs — runs parked because no live worker advertises a required capability, so a stuck fleet is diagnosable at a glance rather than a silent hang.

A store-less tenant api/dashboard pod serves this same view — its reads proxy to the control plane over the wire, so the panel looks identical whether or not the pod owns a store. retry (an in-place retry), retry-with-input, continue, the value pickers and the list's waiting column all ride the wire too. Against a control plane too old to know those verbs, each falls back to what it did before: retry re-dispatches the run's pending steps, retry-with-input and continue answer 404, the pickers count a bounded listing, and rows carry no waiting. Some verbs have no wire equivalent yet and always answer 404 on such a pod: the human-in-the-loop verbs (signal, update, tasks/*), events/:key and webhooks/:token, and the schedule controls.

Engine health and admin surfaceCopy Anchor Link

The console's worker cards and its actions are a thin layer over methods WorkflowEngine exposes directly. Import the engine from @adonis-agora/durable/services/main when you're scripting an operational task (an ace command, a REPL session) instead of clicking through the console.

Worker liveness — engine.workerHealth(extra?: string[]) returns one GroupHealth per worker group. It is the same call behind GET /api/workers and the Telescope worker panel:

interface GroupHealth {
  group: string
  /** Outstanding jobs in the group's task queue: waiting + active + delayed + prioritized on BullMQ,
   *  pending only on the queue transport, pending + claimed task rows on the db transport. */
  depth: number
  /** Workers with a non-expired heartbeat for this group. */
  liveWorkers: WorkerHeartbeat[]
}

interface WorkerHeartbeat {
  group: string
  /** Stable per-process id (host + pid), so N replicas of a group each show as a distinct worker. */
  instanceId: string
  lastBeatAt: number
  /** BullMQ, queue and db workers: concurrency, in-flight, RSS/CPU, throughput, p95, last adjustment. */
  status?: WorkerStatus
}

depth > 0 && liveWorkers.length === 0 is the actionable alert: work is queued and nothing is consuming it. Pass extra to include groups the engine can't derive from its registrations. Only the BullMQ, queue and db transports implement groupHealth (queue with a worker-key store), so on other transports the list is empty.

Bulk and tree operations:

• engine.cancelWhere(filter, opts?) cancels every run matching a run query (workflow, status, tag, namespace, search-attribute predicates), each through the normal cancel path: child cascade, optional { compensate: true } saga undo, cancel listeners and the control-plane broadcast. It is the scriptable form of the console's bulk cancel, without the 500-run cap. It returns one RunResult per run it cancelled (a run that no longer exists is skipped).
• engine.deleteRun(runId) hard-deletes a run and its whole child subtree (checkpoints, signal waiters, search-attribute rows) and returns the number of runs deleted. Unlike cancel, the run disappears from the listing. Cancel a live run first, since deleting one mid-flight orphans its worker. For deleting by scope, see engine.purgeRuns.
• engine.getRunChildren(parentRunId) returns the ids a run spawned via ctx.child or ctx.startChild. The console's run tree and the cancel cascade walk the same parent-to-children edge.

Graceful shutdown — engine.drain(timeoutMs = 10_000) stops the engine picking up new runs and waits for in-flight executions to settle, up to timeoutMs, then releases the recovery leases of anything still in flight so the next pod can take over without waiting for them to expire. durable:work already drains on stop, so you only call it yourself when you run the engine some other way.

Cross-instance hooks — onEnqueued and onCancel let an instance react to activity from other instances over the control plane, without polling:

import engine from '@adonis-agora/durable/services/main'

// Pick up a run the instant another pod enqueues it.
const stopEnqueued = engine.onEnqueued((runId) => engine.runOne(runId))

// Cooperative cancellation: abort long in-flight work instead of finishing it just to discard it.
const stopCancel = engine.onCancel((runId) => myWorkerBridge.abort(runId))

Both return an unsubscribe function. onEnqueued is only useful on instances that should execute runs.

Calling the API directlyCopy Anchor Link

The JSON API is a small, framework-light surface — listRuns, getRun, runValues, retryRun, retryWithInputRun, continueRun, redispatchPendingRun, cancelRun, bulkAction, getRunEvent, deliverWebhook, metricsText, health, workers, topology and compat, all exported from @adonis-agora/durable/dashboard — so you can build your own UI or operational scripts on the same routes. The full wire contract, human-in-the-loop and schedule verbs included, is published as the OpenAPI document:

// Retry every run that failed on a transient charge error, in one call.
const res = await fetch('/durable/api/bulk/retry?status=failed&workflow=charge-order', {
  method: 'POST',
  headers,
})

const { matched, applied } = await res.json()
if (matched === 500) {
  // The cap was hit — narrow the filter and repeat.
}

The OpenAPI contractCopy Anchor Link

GET /durable/api/openapi.json serves a hand-written OpenAPI 3.1 document describing every route above — paths, request bodies, response shapes, the run and checkpoint schemas. It is the single machine-readable statement of the wire contract: generate a typed client from it, diff it in CI to catch a response reshaping, or import it into an API console. It is kept in lockstep with the handlers by a spec that asserts the document matches the registered route table, so it cannot silently drift.

For deep operational health views (success rate, throughput, top failures) inside your existing observability UI, see the Telescope integration.