Agora
Observability

Dashboard

The embedded operations console — a React SPA plus a JSON API mounted into your AdonisJS routes. Filter runs, live-tail a run's timeline, fix-and-replay a bad input, deliver signals/updates/task completions, pause and trigger schedules, act on runs in bulk, and read the worker fleet's health — with the whole wire contract published as OpenAPI 3.1.

The dashboard mounts an operations console into your AdonisJS app: a React single-page app plus the JSON API it runs on. It reads straight from the state store through the engine's read API, so it works without an OpenTelemetry collector. It ships in the main @adonis-agora/durable package — no extra install.

node ace configure @adonis-agora/durable

configure registers @adonis-agora/durable/dashboard_provider in adonisrc.ts and publishes config/durable_dashboard.ts. The provider registers the routes on boot — nothing else to wire.

Configuration

config/durable_dashboard.ts controls where it mounts and who may reach it:

config/durable_dashboard.ts
import { defineConfig } from '@adonis-agora/durable/dashboard'

export default defineConfig({
  // enabled: true,
  // path: '/durable',
  // authorize: (ctx) => ctx.auth.user?.isAdmin === true,
})
OptionDefaultDescription
enabledtrueWhen false, no routes are registered at all.
path'/durable'Route prefix the console mounts under. '/' mounts at the root.
authorizesee belowPer-request guard: (ctx: HttpContext) => boolean | Promise<boolean>.
dashboardAuthabsentOpt-in session layer on top of authorize.

Authorization

The default authorize fails closed in every environment: it requires a bearer token equal to the DURABLE_DASHBOARD_TOKEN env var, and denies everything when that var is unset. (Before 0.37 it was open outside NODE_ENV=production — which left dev boxes, staging, and any deployment with a misspelled/unset NODE_ENV serving an unauthenticated console.) The token is accepted from an Authorization: Bearer <token> header or an x-durable-token header; the ?token= query param works on GET only (it exists for the SSE live-tail, whose EventSource cannot set headers) — never on a mutating route, so a leaked/logged URL can't act as a destructive magic link.

Three ways to open the console up, in order of preference:

  1. configure dashboardAuth — the session guard becomes the gate (the default token guard steps aside);
  2. set DURABLE_DASHBOARD_TOKEN (or supply your own authorize);
  3. for local development only, allowUnauthenticated: true serves the console with no auth at all — it exists so the old open-by-default behavior has to be spelled out in config (and greppable at review time). It logs a warning at boot.

Two request-level guards apply on top of whatever gate you pick:

  • Cross-site rejection — a browser-originated cross-site mutating request (by Sec-Fetch-Site, else Origin vs Host) gets a 403 before any handler runs, closing the drive-by CSRF shape (the mutating endpoints need no body, so they used to be firable as simple cross-origin POSTs). Non-browser clients send neither header and pass.
  • Session roles — a dashboardAuth session whose roles list is non-empty must include operator (or admin) to hit a mutating route; everything else is a viewer (403). An empty/absent roles list keeps full access, so existing hooks that never set roles are unchanged.

Every mutating attempt is also audited — actor (session name/id, token, or anonymous), method and path — through the app logger, or your own audit hook on the config. Payloads an operator submits (fix-and-replay input, signal/update/task bodies) are capped at 1 MiB (413).

Override authorize with your own session/role/IP guard — a denied request gets a 403: JSON for an API call, the access-denied page for a browser:

export default defineConfig({
  authorize: async (ctx) => {
    await ctx.auth.check()
    return ctx.auth.user?.isAdmin === true
  },
})

If you use @adonis-agora/authz for RBAC, the shared authorizeByRoles helper reads your user and roles for you — same semantics as the requireRole middleware, no ctx narrowing:

import { authorizeByRoles } from '@adonis-agora/authz'

export default defineConfig({
  authorize: authorizeByRoles({ roles: ['ADMIN'] }),
})

The user is resolved from ctx.auth.getUser() (authkit) or ctx.auth.user (any guard); roles is any-of over effectiveRoles (token claims ∪ DB roles). Anonymous requests are denied (false). The same helper works across every @adonis-agora dashboard — durable, telescope, media and agent — so one RBAC gate config reads the same everywhere.

The console exposes retry, cancel, replay and bulk actions that mutate runs. Always front it with a real guard in production — the default token gate is a floor, not a substitute for your app's auth.

Prefer a real sign-in page, or to open the console from an app where the operator is already signed in? Opt into the built-in dashboard auth (dashboardAuth), which layers a signed session cookie on top of authorize.

The access-denied page

A refused API request gets JSON (403 { "error": "forbidden" }, or 401 { "error": "unauthorized", "auth": { "modes": [...] } } without a session) — that is what the console's own fetch calls expect. A refused page navigation — the console shell, its assets, or the Mode-A-only "open this from your app" case — gets a real page instead: a dark card in the console's visual language showing the status, a sentence explaining the refusal, a "Back to app" link and, when dashboardAuth.login is configured, a "Sign in" button. It carries no inline script, so a nonce'd script-src CSP cannot break it (its one inline <style> picks up @adonisjs/shield's request nonce).

Tweak it with accessDenied — every field optional:

config/durable_dashboard.ts
export default defineConfig({
  accessDenied: {
    brand: 'Entre Textos',           // eyebrow + <title>; default "Durable"
    title: 'Sem acesso',             // default depends on the refusal
    message: 'Peça ao admin para liberar o console de workflows.',
    homeHref: '/admin',              // "Back to app"; default "/", `false` hides it
    homeLabel: 'Voltar',
    loginHref: '/entrar',            // default: the built-in login page when one exists
    loginLabel: 'Entrar',
    accent: '#f59e0b',               // any CSS colour; default: the console's lime
  },
})

Or replace it. Pass a function and it receives the refusal (status, reason'forbidden', 'unauthenticated' or 'session-required'basePath, loginHref, and the CSP nonce when there is one) plus the HttpContext. Return an HTML string to have it served with the right status; answer the request yourself and return nothing to make the provider stand down:

config/durable_dashboard.ts
export default defineConfig({
  accessDenied: (info, ctx) => {
    if (info.reason === 'unauthenticated') {
      ctx.response.redirect(`/login?next=${encodeURIComponent(info.basePath)}`)
      return
    }
    return `<!doctype html><title>${info.status}</title><h1>Sem acesso</h1>`
  },
})

An authorize hook that already wrote a redirect still wins — the provider never overwrites a location header, with or without accessDenied.

Where it mounts

All routes are relative to path (default /durable), and every one of them runs behind the authorize guard.

Pages

MethodRoutePurpose
GET/durablethe React console (index.html, with the base path and API base injected)
GET/durable/assets/:filethe console's hashed JS/CSS bundle, served immutable

The console routes client-side on the URL hash (#/run/<id>), so a deep link needs no server-side rewrite.

Reading runs

MethodRoutePurpose
GET/durable/api/runslist runs — see the filters below
GET/durable/api/runs/valuesdistinct values of one filter axis, with counts — what the pickers list
GET/durable/api/runs/:idrun detail: the run, its step timeline, and its children
GET/durable/api/runs/:id/streamServer-Sent Events: this run's lifecycle events, live

GET /api/runs accepts two spellings of the same filter, both served by the console's RunFilter class (a BaseModelFilter over a run-query draft):

  • the flat spelling the console has always sent: ?status=failed&tag=etl (repeat a key for a set: ?tag=etl&tag=nightly), attr as key:op:value repeats;
  • the structured envelope @adonis-agora/filter-client builds: filter[status]=failed, filter[tag][]=etl&filter[tag][]=nightly, filter[attr]=tier:eq:enterprise.
ParamMeaning
statusone of pending, running, suspended, blocked, completed, failed, cancelled, dead. An unrecognised value is ignored, not rejected. Repeat for a set.
workflowexact workflow-name match. Repeatable — repeats match ANY of them.
tagexact match against the run's tags. Repeatable — repeats match runs carrying ANY of them.
namespaceexact match against the run's worker-pool namespace — omit it and the listing spans every namespace. Repeatable — repeats match ANY of them.
originexact match against the run's package attribution. The "unknown" bucket (absent origin) can't be expressed as an exact match — the console filters that one client-side.
createdAfter / createdBeforetime-range bounds on the run's creation, as epoch ms or an ISO date string — pushed down server-side.
attra search-attribute filter, key:op:value. Repeatable, and repeats are ANDed — opaque in the envelope too (filter[attr]), since attribute keys are dynamic; filter[attr.<key>][<op>]=value works as well.
page1-based page number, default 1
sizepage size, default 50, capped at 200

The attr ops are eq, ne, gt, gte, lt, lte, in — opaque through both spellings (filter[attr.<key>] takes the canonical equals/notEquals/in/isAnyOf aliases). Values are coerced: true/false become booleans, anything numeric becomes a number, everything else stays a string. Only the first two colons delimit, so a value may contain colons of its own. in takes a |-separated set (tier:in:pro|enterprise) and matches it as OR inside the one predicate — two eq predicates on one key are ANDed like every other pair, which no run can satisfy.

GET /durable/api/runs?namespace=eu-west&attr=amount:gte:5000&attr=tier:eq:enterprise&status=failed

A refused structured filter — unknown field, unsupported operator, a group the draft cannot express — answers 400 instead of silently widening. Unknown flat params stay ignored, so old callers (and endpoint mechanics like page) keep working. (static throwOnInvalid on the filter class is what makes the envelope strict.)

Paging is the same offset shape every Agora library takes (@adonis-agora/filter's page/size): a 1-based page number and a page size, ?page=2&size=100. The 0-based offset it resolves to is the store's business, never the wire's.

The response is { runs, meta: { page, size, count }, statuses }. The envelope key is meta, the same name AdonisJS/Lucid's own .paginate() uses — every @adonis-agora/* listing spells it that way. meta.count is how many rows this page returned, not a total — count === size is the "there may be more" signal.

Breaking in 0.39.0. This endpoint took ?limit=&offset= (0-based) before, and answered under a page envelope key. limit/offset are now ignored like any other unknown flat param — a stale client gets an unpaged first page rather than a 400, so translate as page = offset / limit + 1, size = limit; and read the window off body.meta instead of body.page. Same page/size rename on RunQuery (engine.listRuns({ page, size })) and on the cross-pod listRuns gateway request.

The picker enumeration

GET /api/runs/values answers what the console's tag, tenant and attribute pickers list: the distinct values one axis takes across the runs matching every OTHER active predicate, with counts — most common first, engine-minted tags (singleton:<key>) last. Served through the filter lib's groupByCountFromRequest with the console's adapter. The axis rides groupByCount[field] (workflow, status, namespace, tag, attr for the keys in use, attr.<key> for the values under one key — top-level field works too); the scope rides the same params as GET /api/runs (minus status, which is deliberately dropped so the offers don't collapse to the status being viewed). limit bounds the rows (default 100, capped at 200), offset pages them, search narrows them server-side — this endpoint keeps limit/offset on purpose: it is filter's own group-by-count aggregation (groupByCount[limit]/groupByCount[offset]), not a run listing, so limit/offset IS the aligned spelling here — a rare value outside the first page stays reachable by typing. The response is a bare [{ value, count }] array.

There is no separate checkpoints route: a run's step timeline comes back as the timeline array of GET /api/runs/:id, each entry carrying seq, name, kind, status, attempts, workerGroup, input/output/error, events, parallelGroup, and the derived durationMs / queueMs.

Live-tailing one run

GET /api/runs/:id/stream is an SSE endpoint. Each frame is a default message event whose data is one serialized engine lifecycle event for that run:

const source = new EventSource(`/durable/api/runs/${runId}/stream`)

source.onmessage = (msg) => {
  const event = JSON.parse(msg.data)
  // event.type is one of run.started / run.completed / run.failed / run.suspended /
  // step.started / step.completed / step.failed / capability.unavailable / protocol.incompatible
  console.log(event.type, event.name, event.durationMs)
}

The stream sends no keepalive comments and has no server-side idle timeout — it ends when the client disconnects. Behind a proxy that closes idle connections, reconnect from the client (EventSource does this on its own by default). The response carries x-accel-buffering: no so nginx does not buffer it.

Acting on runs

MethodRoutePurpose
POST/durable/api/runs/:id/retryre-enqueue the run (→ pending, a worker resumes it)
POST/durable/api/runs/:id/retry-with-inputfix and replay: start a new run from a corrected input
POST/durable/api/runs/:id/continueresume a run parked at a ctx.breakpoint()
POST/durable/api/runs/:id/cancelcancel the run — ask for the saga undo with ?compensate=true or a { compensate: true } body
POST/durable/api/runs/:id/redispatchre-enqueue every remote step still pending on this run
POST/durable/api/runs/:id/signaldeliver a signal payload on a token the run is waiting on
POST/durable/api/runs/:id/update/:namedeliver a validated update to a ctx.onUpdate point
POST/durable/api/runs/:id/tasks/:name/completecomplete an external ctx.task
POST/durable/api/runs/:id/tasks/:name/failfail an external ctx.task
POST/durable/api/bulk/:actionapply retry or cancel to everything a filter matches

Every one of these is non-blocking: they enqueue work and return, never replaying a workflow inline in the HTTP request.

Fix and replay

A run that failed on a bad input cannot be "retried" into success — a retry replays the same input. retry-with-input instead starts a brand-new run with a corrected input:

const res = await fetch(`/durable/api/runs/${runId}/retry-with-input`, {
  method: 'POST',
  headers: { 'content-type': 'application/json', ...headers },
  body: JSON.stringify({ input: { orderId: 'ord_42', amount: 1999 } }),
})

const { result } = await res.json()
result.runId // 'run-abc~retry~1f4c9e02'

The new run inherits the original's workflow, tags and namespace, and starts with a clean history — no checkpoints are replayed. The original run is left completely intact, keeping its status, its error and its full timeline as the record of what went wrong.

The link between the two is the run id itself: the new id is <originalRunId>~retry~<short-uuid>, and the console parses that suffix to draw the lineage between them. There is no separate parent column.

Continuing a breakpoint

ctx.breakpoint() parks a run indefinitely at a known point, with a visible breakpoint checkpoint in its timeline and no wake timer — a deliberate manual gate rather than a failure. POST /api/runs/:id/continue releases it, which is the console's counterpart to calling engine.continue(runId) from code.

It answers 404 with run <id> is not paused at a breakpoint when the run has no pending breakpoint checkpoint, so the call is safe to fire blindly.

Human-in-the-loop: signals, updates and tasks

The runs list already names what a suspended run is parked on (the waiting column); these three verbs let the operator act on it from the console instead of dropping into a REPL:

  • POST /api/runs/:id/signal delivers { token, payload? } to a ctx.waitForSignal rendezvous. It is guarded: unless you pass force: true, the token must be one the run is currently waiting on — a typo'd token would otherwise buffer a stray payload silently instead of resuming anything. A mismatch answers 409 carrying the waitingOn token list, so a client can offer the real choices; force: true buffers the payload anyway (the reliable-delivery path).
  • POST /api/runs/:id/update/:name delivers { arg? } to a ctx.onUpdate(name) point. The workflow's registered validator arbitrates server-side: a rejection answers 422 with the reason, and nothing is delivered.
  • POST /api/runs/:id/tasks/:name/complete (body { result? }) and .../fail (body { error }) settle an external ctx.task — the console equivalents of engine.completeTask / engine.failTask. The response is honest about delivery: { result, delivered: true } when a live waiter consumed it, { result: null, delivered: false } when no waiter was live yet and the completion was buffered for the run to consume when it reaches the task's wait — buffered is success, not a 404.

On a store-less tenant pod these verbs answer 404 (like retry-with-input and continue) — they need the engine's signal surface.

Bulk actions

POST /api/bulk/:action applies one action to everything a filter matches, where :action is retry or cancel (anything else is a 400). The selection comes from the query string, using the same status / workflow / tag / namespace / attr filters as GET /api/runs — there is no id list to assemble:

POST /durable/api/bulk/retry?status=failed&workflow=charge-order&namespace=eu-west
POST /durable/api/bulk/cancel?status=suspended&tag=stuck&compensate=true

?compensate=true runs the saga compensations and is honoured for cancel only.

Asking for the saga undo

Both cancel routes read compensate from either the query string or the JSON body, so ?compensate=true and { "compensate": true } are equivalent. A value spelled true / 1 / yes / on (or the bare ?compensate) means yes; false / 0 / no / off means no; absent means no.

Anything else is a 400. That is deliberate: the two ways to guess at an unreadable value are "skip an undo the operator asked for" and "run an undo they did not", and both would be silent. An explicit rejection is the only answer that cannot mislead.

A compensating cancel answers with the run's pre-cancel status, not cancelled. The undo replays the workflow's saga in the background so the HTTP request never blocks on it, and the run reaches cancelled once the compensations finish — watch the run's stream or poll it rather than reading the response body as the final state. A plain cancel is terminal in the response itself.

The response is { matched, applied }. matched is how many runs the filter returned; applied is how many the action actually changed. They differ when a run has already moved on — a run that throws or answers "nothing to do" is skipped rather than aborting the batch.

Bulk acts on at most 500 runs per call

The filter is evaluated with a hard cap of 500 and no paging. A filter matching more than that silently acts on the first 500 — narrow the filter (a tighter namespace, a time-bounded attr) and repeat, rather than assuming one call drained the backlog. Compare matched against 500 to tell.

Schedules

MethodRoutePurpose
GET/durable/api/schedulesthe ticked schedules with their live control state and fire windows
POST/durable/api/schedules/:key/pausepause the schedule at runtime, fleet-wide
POST/durable/api/schedules/:key/resumeresume it
POST/durable/api/schedules/:key/triggerfire the current window now (idempotent)

These are the console's Schedules tab, riding engine.listSchedules / setSchedulePaused / triggerSchedule. The listing returns { schedules }, each entry carrying the cadence (cron/everyMs/timezone), the effective paused state with a pausedAtRuntime flag telling you whether a console override (which wins over config until the next deploy) is in force, the current window's lastFireAt/nextFireAt/currentWindowRunId, and that run's lastRunStatus when it exists. trigger is idempotent by the window's deterministic run id — triggering a window that already fired returns the existing run rather than forking a duplicate. An unknown :key answers 404; an action other than pause/resume/trigger is a 400.

Fleet and topology

MethodRoutePurpose
GET/durable/api/workersper-worker heartbeats, grouped by routing token
GET/durable/api/healththe compact shape: queue depth and a stalled flag per group
GET/durable/api/topologythis pod's role, plus its tenant when it is a store-less pod
GET/durable/api/compatprotocol/capability negotiation across the fleet, plus blocked runs
GET/durable/api/openapi.jsonthe machine-readable API contract (OpenAPI 3.1)

GET /api/workers returns the raw GroupHealth[] — for each routing token, its queue depth and the individual live workers with their instanceId and lastBeatAt. That is what tells "one worker is wedged" apart from "the whole group is down".

On the BullMQ transport, each worker's heartbeat also carries a live status telemetry snapshot (WorkerStatus) that the console renders on its worker cards: the concurrency mode and limit (fixed or adaptive, with min/max), inFlight, RSS (rssBytes, plus rssLimitBytes/rssPct when a ceiling is known), cpuPct, throughputPerMin, p95Ms, and the last adaptive concurrency adjustment with its reason. Workers on other transports simply omit status — the cards degrade to liveness only.

GET /api/health is the older, compact view of the same data: it collapses liveWorkers to a count and adds stalled: depth > 0 && liveWorkers === 0. It stays for hand-written clients and probes; prefer /api/workers for anything new.

GET /api/compat powers the compatibility panel described below: per group, each pod's protocol range and capabilities with a compatible / degraded / incompatible outcome and the exact reason, alongside every blocked run and why it parked.

The routes are all named (durable_dashboard.index, durable_dashboard.runs.show, durable_dashboard.runs.retry_with_input, durable_dashboard.bulk, …), so you can reference them from your own code or wrap them in middleware.

What it shows

  • Runs — every run with its status, filterable by status, workflow, tag, namespace, origin, creation time and search attribute. The tag, tenant and attribute filters are value pickers: they list what the runs actually contain (counted server-side over the rest of the active filter, searchable, paged), take several values at once, and still accept a typed value. pending is a run created and enqueued but not yet picked up; a dead run carries a distinct badge so a poison pill is easy to spot.
  • What a suspended run is waiting on — a suspended run parked on a signal, a webhook callback, a child run or a breakpoint is surfaced as waiting, with what it is blocked on right in the row (signal approve, child ord-42.child.0, breakpoint). See below.
  • Step timeline — the selected run's steps in order, tagged by kind (local / remote / sleep / signal), with worker group, queue wait and duration. In-flight steps appear while they run, not only on completion — a remote step in flight reads as pending, and a local step's executing body as running (the trackStepStart option).
  • Live tail — the open run streams its events over SSE, so a long run updates without a reload.
  • Children and lineage — a run's child workflows are linked from its detail, as is the original of a fix-and-replay run.
  • Actions — retry, cancel, cancel + undo (saga compensation), fix-and-replay, redispatch, continue, the human-in-the-loop verbs (deliver a signal, a validated update, or a task completion), and the bulk equivalents of retry and cancel.
  • Schedules — the ticked scheduled workflows with their fire windows and effective pause state, plus pause / resume / run-now controls that apply fleet-wide at runtime.
  • Worker cards — on the BullMQ transport, each worker's live telemetry: concurrency, in-flight, RSS/CPU, throughput and p95.

The waiting state

waiting is not a ninth run status — it is a derived view of a suspended run. The engine records one generic suspended status; the listing joins it against the signal-waiter table (listSignalWaiters) and reports what the run is parked on:

waiting.onThe run is parked on
signala ctx.waitForSignal or ctx.waitForEvent; name is the signal or event name
webhooka ctx.webhook().wait() callback
childa child run; name is the child's run id
breakpointa ctx.breakpoint() — a human has to release it

A suspended run with no waiter gets no waiting value, and that absence is itself informative: a bare ctx.sleep, an in-flight remote step and a run whose wake was lost all fall into that bucket. Use durable:runs --stale to separate the last one from the first two.

waiting appears on the rows of GET /api/runs only, and only when the store exposes listSignalWaiters — a store-less tenant pod skips it silently.

Fleet health

When you run a split or polyglot cluster, the console grows a health and compatibility panel fed by the handshake descriptors and diagnostics events:

  • Per pod — its protocol version, negotiated level (compatible / degraded / incompatible), and a red flag with the exact reason on a mismatch (e.g. "no common protocol major: local speaks [1, 1], remote speaks [2, 2]").
  • Blocked runs — runs parked because no live worker advertises a required capability, so a stuck fleet is diagnosable at a glance rather than a silent hang.

A store-less tenant api/dashboard pod serves this same view — its reads proxy to the control plane over the wire, so the panel looks identical whether or not the pod owns a store. Some verbs have no wire equivalent yet and always answer 404 on such a pod: retry-with-input, continue, the human-in-the-loop verbs (signal, update, tasks/*), and the schedule controls.

Calling the API directly

The JSON API is a small, framework-light surface — listRuns, getRun, runValues, retryRun, retryWithInputRun, continueRun, redispatchPendingRun, cancelRun, bulkAction, health, workers, topology and compat, all exported from @adonis-agora/durable/dashboard — so you can build your own UI or operational scripts on the same routes. The full wire contract, human-in-the-loop and schedule verbs included, is published as the OpenAPI document:

// Retry every run that failed on a transient charge error, in one call.
const res = await fetch('/durable/api/bulk/retry?status=failed&workflow=charge-order', {
  method: 'POST',
  headers,
})

const { matched, applied } = await res.json()
if (matched === 500) {
  // The cap was hit — narrow the filter and repeat.
}

The OpenAPI contract

GET /durable/api/openapi.json serves a hand-written OpenAPI 3.1 document describing every route above — paths, request bodies, response shapes, the run and checkpoint schemas. It is the single machine-readable statement of the wire contract: generate a typed client from it, diff it in CI to catch a response reshaping, or import it into an API console. It is kept in lockstep with the handlers by a spec that asserts the document matches the registered route table, so it cannot silently drift.

For deep operational health views (success rate, throughput, top failures) inside your existing observability UI, see the Telescope integration.

On this page