Control plane

@dudousxd/nestjs-durable-dashboard mounts a control plane at /durable: a bundled React SPA plus its JSON API. It reads straight from the state store, so it works without an OpenTelemetry collector.

import { DurableDashboardModule } from '@dudousxd/nestjs-durable-dashboard';

@Module({
  imports: [DurableModule.forRoot({ store, transport }), DurableDashboardModule.forRoot()],
})
export class AppModule {}

DurableModule is global, so the dashboard resolves the engine and store automatically. Front the base route with your own guard to protect it.

Where it mounts

forRoot() defaults to /durable (SPA at /durable, JSON API at /durable/api). Pass basePath to move it — e.g. under your app's /api prefix so the same auth/proxy rules cover the dashboard API:

DurableDashboardModule.forRoot({ basePath: '/api/durable' });
// SPA at /api/durable, API at /api/durable/api

The SPA's asset URLs and its API base are derived from basePath at serve time, so the bundle works at any mount point. If your app sets a global prefix (app.setGlobalPrefix('api')), exclude the base route so the path isn't prefixed twice — or fold it in by setting basePath to include the prefix.

Serving it from an API pod (without processing there)

When you split your app into API and driving pods, you usually want the dashboard served from the API (it handles HTTP) while only the driving pods process runs. Set drive: false on the API instance's DurableModule:

DurableModule.forRoot({
  store,
  transport,
  drive: process.env.APP_TYPE === 'API' ? false : true, // driving pods process; the API only serves the dashboard
});

With drive: false the engine is still fully available — the dashboard can dispatch retries and cancels, and reads work — but the instance does not drive: it won't register @Step handlers (so it never consumes the task queue), won't recover incomplete runs on boot, and won't poll due timers. Leave that to the driving pods. So a retry clicked in the UI on an API pod still works: the API dispatches the step, a driving pod picks it up and runs it. Only the processing is gated, not the control plane. (Default is drive: true, so single-process apps need nothing.)

Console auth

Three ways to gate the console (the SPA at basePath and its JSON API at apiBasePath), from least to most work — pick one, or compose two:

guards — front it with your own auth

guards (+ imports for its dependencies) stamps a guard class onto BOTH dashboard controllers — the SPA at basePath and its JSON API at apiBasePath:

DurableDashboardModule.forRoot({
  guards: [ConsoleAuthGuard],
  imports: [AuthModule], // resolves ConsoleAuthGuard's own dependencies
});

A guard here must handle cookies, not just headers: the JSON API is fetched by the SPA's own fetch/XHR (a header-based guard works fine there), but the SPA shell itself is loaded by a full browser navigation — the browser sends whatever ambient cookie it has, never a custom Authorization header. A guard that only checks a bearer token will 403 the page shell for an already-logged-in admin whose next fetch to the API would have passed. A cookie-session guard that optionally redirects unauthenticated page navigations to your own sign-in screen:

import { type CanActivate, type ExecutionContext, Inject, Injectable } from '@nestjs/common';
import { SessionService } from './session.service';

@Injectable()
export class ConsoleAuthGuard implements CanActivate {
  constructor(@Inject(SessionService) private readonly sessions: SessionService) {}

  async canActivate(context: ExecutionContext): Promise<boolean> {
    const request = context.switchToHttp().getRequest();
    const response = context.switchToHttp().getResponse();
    const user = await this.sessions.verifyFromCookie(request);
    if (user?.isAdmin) return true;

    // A real browser navigation (loading the shell) vs the SPA's own fetch calls.
    if (request.headers?.['sec-fetch-mode'] === 'navigate') {
      response.redirect(302, '/signin?next=' + encodeURIComponent(request.originalUrl));
    } else {
      response.status(401).send({ message: 'Unauthorized' });
    }
    return false;
  }
}

dashboardAuth — the built-in session cookie

dashboardAuth gates the console behind the dashboard's own stateless, signed session cookie — no infra, and no changes to the bundled React SPA (unlike @dudousxd/nestjs-telescope, whose dashboardAuth renders its login screen inside its own SPA bundle, the durable dashboard's auth-adjacent pages are plain HTML served directly by their own controller, so gating the SPA shell never touches the Vite build). There are two ways to mint that cookie — Mode A (session) and Mode B (login) — used alone or together; at least one is required, or forRoot/forRootAsync throws at boot (an un-mintable gate is a boot error, not a silently-open or silently-stuck console).

Mode A — session, ride your host's own auth

Your host app already has its own auth (SSO/OIDC/whatever). The host frontend, carrying that auth, POSTs to <basePath>/session; your session hook validates the raw request and returns the session user (or null to deny), and the library mints its cookie from that. No credential this library understands ever exists — your identity provider stays the source of truth:

DurableDashboardModule.forRoot({
  dashboardAuth: {
    secret: process.env.DURABLE_AUTH_SECRET, // REQUIRED HMAC-SHA256 key, 32+ bytes recommended
    ttl: '8h', // optional, default 8h
    session: (request) => {
      const user = requireAuthenticatedUser(request); // however your host already authenticates
      return user?.isAdmin ? { id: user.id, name: user.name, roles: ['admin'] } : null;
    },
  },
});

Your own console launcher (a page in your app, gated by your own auth) POSTs <basePath>/session before sending the visitor to /durable — a successful response mints the durable_dashboard_session cookie (HttpOnly, SameSite=Lax, Secure over https) exactly like Mode B's login does. With only Mode A configured there is no login page to redirect to, so a page-level request (a full-page navigation) with a missing/invalid/expired cookie instead renders a small, static instruction page telling the visitor to sign in through the host app; an API request (apiBasePath, fetched by the SPA's own JS) still gets a plain 401, same as Mode B.

Opening the console from your app

Your launcher has to mint the Mode A cookie before navigating (see Mode A). The package ships that at three levels — take the one that fits and drop a level whenever it stops fitting:

// 1. Drop-in. Unstyled; forwards className/style/every button prop.
import { OpenDurableConsoleButton } from '@dudousxd/nestjs-durable-dashboard/react';
<OpenDurableConsoleButton className="btn" headers={() => authHeaders()} />;

// 2. Hook. You own the markup, it owns the states.
import { useOpenDurableConsole } from '@dudousxd/nestjs-durable-dashboard/react';
const { open, isPending, error } = useOpenDurableConsole({ headers });

// 3. Headless. No React at all.
import { openDurableConsole } from '@dudousxd/nestjs-durable-dashboard/client';
await openDurableConsole({ headers });

Already on TanStack Query? openDurableConsoleMutationOptions() returns the object useMutation takes — this package does not depend on TanStack, so you get the integration and a host that doesn't use Query pays nothing.

React and react-dom are optional peer dependencies and the React code lives behind the ./react subpath, so mounting only the NestJS module pulls in neither.

unauthenticatedPage — render that page yourself

The built-in instruction page cannot know who hosts the console, so it can only say "open this console from your application" in the abstract — it can't name your launcher, link to it, or look like the rest of your product. Pass unauthenticatedPage and the whole response is yours:

DurableDashboardModule.forRoot({
  dashboardAuth: {
    secret: process.env.DURABLE_AUTH_SECRET,
    session: (request) => resolveAdmin(request),
    unauthenticatedPage: ({ request, response, basePath }) => {
      // `request`/`response` are your platform's own objects (Express here). Render however you
      // already render — a template engine, @dudousxd/nestjs-inertia, a plain string.
      (response as Response).status(401).render('console-locked', { returnTo: basePath });
    },
  },
});

The hook owns the response: it must write AND end it. Serving happens at the console's own URL, so /durable stays /durable (no redirect, no second route for you to own).

It cannot open the console — it only ever runs on a request that has already been denied. If it throws, or returns without writing anything, the library logs a warning once and falls back to the built-in page, so a broken page can't hang the request or turn a denial into a 500.

This is not a replacement for Mode B's login form. To ship your own login UI, combine Mode A with this hook and POST <basePath>/session from your page — the mint endpoint is the supported primitive for exactly that.

Mode B — login, the built-in login page

No host frontend/IdP to lean on. The dashboard serves its own small, dependency-free, server-rendered login page and your login hook validates submitted username/password:

DurableDashboardModule.forRoot({
  dashboardAuth: {
    secret: process.env.DURABLE_AUTH_SECRET, // REQUIRED HMAC-SHA256 key, 32+ bytes recommended
    ttl: '8h', // optional, default 8h
    login: (username, password) =>
      username === process.env.DURABLE_USER && password === process.env.DURABLE_PASS
        ? { id: 'ops' }
        : null, // return null to deny — a uniform 401, no user-enumeration
  },
});

Open /durable, and an unauthenticated visit redirects to GET /durable/login?returnTo=/durable — a plain username/password form. The password field is never required: the page passes it through to your login hook verbatim (an empty string when left blank), so a host that gates on username alone (e.g. any active admin's email, password ignored) works with this same built-in page. On success it mints the same durable_dashboard_session cookie as Mode A and sends the browser back to returnTo. GET /durable/logout clears the cookie and returns to the login page when Mode B is configured, or to basePath (re-triggering whichever page-level gate applies) when it isn't.

Use forRootAsync when a hook needs injected services (e.g. an EntityManager to look up a real admin user instead of an env var pair) — the same factory shape covers session, login, or both:

DurableDashboardModule.forRootAsync({
  imports: [UsersModule],
  inject: [UsersService],
  useDashboardAuth: (users: UsersService) => ({
    secret: process.env.DURABLE_AUTH_SECRET,
    login: async (username, password) => {
      const user = await users.verifyPassword(username, password);
      return user?.isAdmin ? { id: user.id, name: user.name, roles: ['admin'] } : null;
    },
  }),
});

Returning undefined from useDashboardAuth leaves the console open, exactly like omitting dashboardAuth from forRoot.

Sliding renewal and revalidate

Sliding renewal keeps an active session alive: a cookie past 50% of its TTL is transparently re-issued on the response, so a working operator isn't forced to re-authenticate mid-session. Add revalidate to have that renewal re-check the user instead of blindly extending them:

DurableDashboardModule.forRoot({
  dashboardAuth: {
    secret: process.env.DURABLE_AUTH_SECRET,
    session: verifyHostSession,
    revalidate: async (session) => {
      const user = await users.findById(session.id);
      return user?.isActive === true; // false/throw revokes — the cookie is cleared, request denied
    },
  },
});

revalidate receives the already-minted session (not a fresh request — a console XHR carries no host auth to re-check, unlike Mode A's session hook), and runs at most once per ttl/2 per session, so a DB round-trip here is cheap — this holds even under real console traffic: a page load firing several parallel API calls with the same past-half-life cookie triggers exactly one revalidate call, not one per call, in-flight renewals of the same session share the result. Returning false, or throwing, clears the cookie and denies the request in place, the same treatment as an absent cookie. This closes the gap a sliding cookie otherwise leaves open — but it isn't instant: revalidate only runs on the renewal path, so a deactivated or demoted operator keeps console access for up to ttl/2 (4 hours at the default 8h TTL) after their last renewal, not the moment they're deactivated. revalidate alone can't mint a session, so it doesn't count toward the session/login "at least one" requirement — pair it with one of those.

A revalidate denial (or any other mid-session 401 from the JSON API) sends the console SPA to the matching auth surface automatically — the login page under Mode B, the session-required page under Mode A — instead of surfacing a raw 401 as a query error.

Composes with guards: when both are set, a request must pass the built-in session check AND every guard in guards — the built-in guard runs first, so an invalid session never even reaches your own guard.

DurableDashboardModule.forRoot({
  dashboardAuth: { secret: process.env.DURABLE_AUTH_SECRET, login: verifyOpsCredentials },
  guards: [IpAllowlistGuard], // an extra restriction on top of the built-in login
  imports: [NetworkModule],
});

Auth here is deliberately mount-level and role-agnostic: neither guards nor dashboardAuth has any notion of the durable dashboard's control-plane vs tenant topology (the GET topology route behind the header badge, described below) — logging in only answers "is this a legitimate operator", nothing about what they can see changes based on who they are.

What it shows

• Runs — every run with its stored status (pending / running / suspended / cancelling / completed / failed / cancelled / dead), filterable. pending is a run that's been created and enqueued but not yet picked up by a worker — you'll see it briefly on every run, and persistently on an API pod (drive: false) whose runs execute on separate driving pods. cancelling is a transient, durable status for a cancel(runId, { compensate: true }) whose saga undo is still running in the background — it survives a crash mid-compensation and finalizes to cancelled. The status filter is a row of chips, including a dead chip for the dead-letter state; a dead run carries a distinct badge so a poison pill is easy to spot. The generic suspended status covers very different situations — sleeping on a timer, waiting on a signal, blocked with no worker, queued behind a singleton leader — so the list and detail views refine it into a more legible display state; see Waiting states below. Alongside status the list narrows by tag, search attribute, tenant and the library that declared the workflow — the last two answer "whose run is this" and "whose code is this", and both have a default worth understanding; see Tenant and origin filters. The tag, tenant and attribute controls are pickers over what the runs actually contain, and each takes several values — see Filter pickers.
• Timeline as a graph — the selected run rendered with react-flow: a node pipeline from start through each step to the terminal state, tagged by kind (local / remote / sleep / signal), with worker group and duration. This is the "see the whole flow" view — a workflow whose steps span apps shown as one rail. In-flight steps show up here while they run, not only on completion: a remote step in flight reads as pending and a local step's executing body as running (both rendered as an in-flight node). Local in-flight visibility is on by default — it's the engine's trackStepStart option — so a long ctx.step is visible the moment it begins.
• Step timing — click a step to inspect it. A remote step shows its queued time (how long it sat in the queue before a worker picked it up) next to its processing duration, so you can tell a slow handler apart from a backed-up worker pool. Queue-wait is reported by every transport and every language SDK (including the Python worker).
• Live-tail — the run view streams new lifecycle events as they happen (see below) instead of polling, so a running workflow updates in place.
• Actions — retry a failed/suspended/dead run, cancel a running one, cancel + undo (cancel with saga compensation), and continue a run paused at a breakpoint. These are non-blocking: retry re-enqueues the run (engine.requeue(runId) → pending + dispatch) and cancel + undo runs the saga compensation in the background — neither replays the workflow inline in the HTTP request, so the dashboard action returns immediately and a worker does the work. (Previously an inline replay could hang the request.)

Filter pickers

The tag, tenant and search-attribute filters list the values the runs actually carry, with a count each, and take several of them.

They used to be free-text boxes, and that made them usable only by an operator who already knew a tenant name or a tag: a typo returned an empty list, which reads exactly like "no runs match". Search attributes were worst — both the key and the value are data nobody memorises.

#  [ etl ×] [ nightly ×]        ← several values, ORed
@  [ acme ×]                    ← tenant
⛃ [ tier is any of pro, enterprise ×]  [+ attribute]

Three things make them worth the machinery:

• The options are counted over the runs the OTHER filters already select. Pick a tenant, and the tag picker offers that tenant's tags. A picker never offers a value whose result set is empty.
• Several values on one axis are ORed, and the axes are ANDed — "these two tenants, tagged either of these two ways". On search attributes that needs a set operator (is any of), because two equality predicates on one key are ANDed like every other pair and no run has one attribute with two values.
• Typed text still works. The offered list is a bounded top-N (tag cardinality grows with the data — a singleton:<key> tag is minted per key), so a rare value can be real and absent; Enter takes it as typed.

A tag or tenant chip on a run row adds to the selection rather than replacing it, which is how a "these two kinds of run" view gets built by clicking.

The values come from GET runs/values, backed by StateStore.runValueFacets. A store that does not implement it answers [], and the pickers degrade to free-text entry — no error, no empty console.

Tenant and origin filters

Two of the run-list filters are about provenance rather than state, and they are wired differently on purpose — one pushes down to the store, the other cannot.

Tenant is a sidebar picker matched exactly against the run's namespace — the same field the engine partitions worker pools by, conventionally fed from DURABLE_TENANT (see Tenancy). It is sent to the store as the namespace param on GET runs and POST bulk/:action, so a narrowed list is narrowed in SQL. Its default is every namespace, and that is a decision rather than an omission: read paths (the dashboard, getRun) are deliberately not namespace-scoped, so defaulting the console to one tenant would have silently removed runs that every existing operator had been looking at. This is a filter an operator chooses, not a scope imposed on them — and it is not an authorization boundary either (Console auth is mount-level and role-agnostic). An empty selection is treated as absent, not as an exact match on a tenant named "", so clearing it returns the all-tenants view rather than an empty one. Selecting several compares them side by side (RunQuery.namespaces). The tenant chip on a run row adds itself to the selection, like the tag chips.

Origin — which package declared the run's workflow — is a row of facet chips above the list: all, one chip per package (labelled without the npm scope, full name in the tooltip), and unknown when any run in the list has none, each with a count. It is derived, never declared: @Workflow resolves the file it is applied in to the nearest enclosing package.json, so a library is attributed without opting into anything and nothing a caller passes to start can claim another library's name. The registrar warns once at boot naming whatever it could not attribute, because a wrong origin is worse than an absent one.

Which makes undefined a real value meaning UNKNOWN, not a blank to be tidied away. Runs created before the field existed have none; so do registerRemote, convention routing, and any workflow whose declaring package couldn't be resolved with confidence. RunQuery.origin is plain equality, so those runs match no origin value at all — picking a package hides them rather than folding them into a bucket. Hence the shape of the facet: all is the default and never goes away, the unknown count stays on screen while a package is selected, and an origin filter that comes back empty says which of the two things happened ("nothing matched" vs "these N runs cannot be matched by any package filter") and offers a jump into the unclassified ones. A run's detail header states origin unknown outright for the same reason, instead of leaving the field blank and letting it read as "the app".

Both filters push down to the store, including the unknown one. RunQuery.origin accepts null for exactly that bucket (origin IS NULL), which the console sends as ?unattributed=true — its own param rather than a reserved origin value, because any reserved string is a package name someone can legitimately publish. The chips' counts come from GET runs/facets, not from the rows on screen, so the unknown count stays visible and exact while a package is selected — the property the facet exists for. Retry all / cancel all send the same predicate, so a bulk action is scoped to precisely the set the list was showing, unattributed runs included.

At scale: paging, facets and what the numbers mean

A control plane accumulates runs for as long as its retention policy keeps them, so "list the runs" is not a bounded request. The console treats it as a page:

• GET runs takes limit / offset, and the SPA asks for 100 rows at a time, extending the page as the last rows scroll into view. Every store adapter implements both. The load starts a lookahead before the bottom, so the list reads as continuous rather than stalling at each boundary, and it is gated on the fetch in flight and on the count the page was last requested at — otherwise a control plane whose total briefly runs ahead of its own listing would walk the page size up on every render, which is the failure paging exists to prevent.
• The list is virtualised: only the rows in the viewport are mounted, so a deep page costs the same DOM as a shallow one.
• GET runs returns list rows, not whole runs — input, output and error are omitted, because no row renders them and on a busy deployment they are most of the bytes. GET runs/:id is unchanged and still carries all three; that is where they are displayed.

The numbers behind the design, measured against a deployment holding 9,533 runs: the unpaged listing was 12.24 MB every three seconds, of which 63% was error — a stack trace on every failed run, which the list never shows. Rendering it mounted 115,636 DOM nodes and blocked the tab's main thread for 26.9 out of every 30 seconds.

Counting without listing

Paging the list raises an obvious question: if the console holds 100 rows, where do the status and origin chips get their numbers? Not from the page — from GET runs/facets, one GROUP BY status, origin aggregate over the same tag / tenant / attribute predicates the listing uses:

const facets = await durableClient.facets(tag, attrs, namespace);
// → [{ status: 'failed', origin: '@acme/billing', count: 3658 }, { status: 'failed', origin: null, count: 12 }, …]

origin: null is the unattributed bucket — a real, countable cell rather than the absence a value match cannot express. status and origin are deliberately not accepted as params: they are the axes being counted, so narrowing by them would report the answer back to itself.

It is backed by StateStore.runFacets (optional on the port; a store without it falls back to counting a listing) and RunGateway.runFacets, which a tenant deployment round-trips to the control plane the same way it does listRuns.

Waiting states

The engine only ever persists one generic suspended status for a durably-parked run — that's what drives recovery, timers, and queries, and it never changes. But why a run is parked reads very differently depending on what it's waiting on, so the run list and the run detail header refine it into a more legible display state (both derive it the same way, so they always agree):

The awaiting label is a RunWaiting ({ on: 'signal' | 'webhook' | 'child' | 'breakpoint', name }), resolved from the run's pending signal waiters and classified by the waiter token's prefix — wh: → webhook, child: → child, bp: → breakpoint, anything else → a named signal. A ctx.breakpoint() pause is implemented as a signal wait under the hood, so it's classified and shown as breakpoint rather than a raw signal bp:<runId>:<seq> token.

no-worker is gated on a real backlog, not bare zero live workers. A worker only heartbeats for a group while it's actively serving it, so a group that's simply idle right now — a suspended run parked on its reconcile timer with nothing enqueued, or a scheduled workflow between cron runs — legitimately reports zero live workers without anything being blocked. Flagging that no-worker was a false positive, fixed in dashboard 0.29.6: the check is depth > 0 && liveWorkers.length === 0 (the same alert condition GroupHealth documents, see below) — a real backlog with no consumer. A parked/settled run with nothing enqueued now reads running (open, in flight) and only flips to no-worker once its resume actually enqueues behind a stalled queue. This also fixed the header banner, which used to count completed-work orphans as stalled.

Bulk-resolving waiting state — RunGateway.waitingFor: if you're building your own filtered/ paginated run list and need to know which of your runs are parked on a breakpoint (or any other event wait) without re-deriving the waiter scan yourself or querying checkpoints directly, call waitingFor on the injected RunGateway:

import { RunGateway } from '@dudousxd/nestjs-durable';

constructor(private readonly runGateway: RunGateway) {}

async whichAreStuck(runIds: string[]) {
  return this.runGateway.waitingFor(runIds);
  // { 'run-1': { on: 'breakpoint', name: 'breakpoint' }, 'run-3': { on: 'signal', name: 'approve' } }
}

It resolves in two bulk store scans (never one query per id), and on a tenant deployment it proxies over the control-plane transport in a single request for the whole id list, scoped to runs the requesting tenant actually owns.

Live-tail over SSE

The run view tails a run's lifecycle events over Server-Sent Events rather than re-fetching. The server exposes an @Sse route, GET runs/:id/stream, and the client subscribes with durableClient.streamRun(id, onEvent):

import { durableClient } from '@dudousxd/nestjs-durable-dashboard/client';

const close = durableClient.streamRun(runId, (event) => {
  // event: { type: 'step.completed' | 'run.failed' | …, runId, seq, name, … }
  console.log(event.type, event.name);
});
// later: close();

This is cross-pod when the engine has a control plane: an event produced by a worker pod is broadcast over the control plane, and a dashboard-only API pod re-delivers it to its SSE subscribers. So you can live-tail a run on the API pod even though the work runs elsewhere. Without a control plane, the stream still works for runs executing on the same instance (single-process apps).

Cancel + Undo (saga compensation)

The run actions include both a plain Cancel and a Cancel + Undo. Plain cancel is immediate — the run is marked cancelled and a late worker result is dropped. Cancel + Undo instead asks the engine to compensate the saga — in the background, not in the HTTP request: the run is resumed on a worker so its completed steps' compensate callbacks run in reverse order (each visible as a compensate:<step> step event in the timeline), then the run is marked cancelled. The HTTP action returns immediately. The UI sends this as cancel with ?compensate=true:

await durableClient.cancel(runId);                      // immediate, no undo
await durableClient.cancel(runId, { compensate: true }); // ?compensate=true — undo first

Use Cancel + Undo when the run has already done outside-world work (a charge, a reservation) that must be reversed; use plain Cancel to just stop it.

Dead-letter runs and their links

A run that crash-recovery gives up on — after exceeding maxRecoveryAttempts — is moved to the dead dead-letter state (a poison pill that would otherwise crash the process every boot). The dashboard surfaces these as a first-class status: a dead filter chip and a badge on the run. A dead run is terminal but inspectable, and still retriable from the UI once you've fixed the cause.

When you configure a deadLetterWorkflow, dead-lettering also starts a handler run (idempotent by a dlq:<runId> id) carrying { deadRunId, workflow, input, error }. The dashboard renders the relationship both ways: the dead run links forward to its dlq:<id> handler, and the handler links back to the dead run it was started for — so you can jump between the failure and whatever your DLQ workflow did about it (alert, compensate, queue for review).

Recovering a lost remote step dispatch

A remote step with no timeoutMs dispatches its task, persists a pending checkpoint, and suspends until the worker's result resumes it. If the worker crashed mid-step or the transport dropped the job (a Redis flush/eviction, or a stalled BullMQ job moved to failed and removed), the result never comes — and reconcile re-drives, by design, just re-suspend a still-pending step rather than re-dispatching it (so a merely-slow worker is never double-run). Left alone, the run hangs on pending forever.

The dashboard flags this instead of masquerading it as a healthy in-flight step: a remote step pending past 10 minutes (STALE_PENDING_MS) shows in the timeline as "awaiting worker result — dispatched Nm ago (possibly lost)". From there:

• Re-dispatch action — POST runs/:id/redispatch (durableClient.redispatch(id), or the dashboard's "Re-dispatch" button) re-enqueues the run's stuck pending remote steps, bumping their attempts so the idempotent step re-runs and its result resumes the run — the manual escape hatch, equivalent to WorkflowEngine.redispatchPending(runId).

• BullMQ self-heals for free. The BullMQ transport bridges a crashed/stalled task job's terminal failure into a synthetic failed StepResult (via the worker's 'failed' event), so the engine marks the checkpoint failed and its normal durable retry re-dispatches — no config needed, and the run never gets stuck on pending in the first place on this transport.

• Opt-in self-heal — remoteRedispatchMs (any transport). Set this engine option and a reconcile re-drive that finds a remote step still pending past the window automatically re-dispatches it, bounded by remoteRedispatchMax (default 10) so a step that never settles fails as a remote_step_lost error instead of looping forever:

  DurableModule.forRoot({
    store,
    transport,
    remoteRedispatchMs: 15 * 60_000, // self-heal a remote step still pending after 15 minutes
    remoteRedispatchMax: 10,
  });

  Off by default — re-dispatch can double-run a step whose original job is merely slow, so the window must exceed the longest such step's real duration and the step must be idempotent. Prefer a per-step timeoutMs where you can (tighter, heartbeat-aware — see Retries & backoff); remoteRedispatchMs is the store-driven net for the no-timeout steps that must still survive a lost dispatch.

Webhook callbacks

A ctx.webhook() hands a third party a callback URL (built by your webhookUrl option). When they POST it, the dashboard's POST webhooks/:token endpoint turns that body into the signal the waiting run resumes on. The token embeds runId:seq, so treat it as a secret — this endpoint is reachable by external systems; front it with signature verification in your own middleware.

Live queries and updates

Two more endpoints expose a run's query/update surface over HTTP:

• GET runs/:id/events/:key — read the latest value a run published with ctx.setEvent(key, …), a side-effect-free live query of an in-flight (or finished) run's state (progress, a partial result). Returns undefined if the run never published that key.
• POST runs/:id/updates/:name — deliver a validated update to a run waiting at ctx.onUpdate(name); the request body is the update argument. Any validator registered via engine.registerUpdateValidator runs first, so a rejected update never reaches the run.

Fleet health

When you run a split or polyglot cluster, the dashboard grows a health / compatibility panel fed by the handshake descriptors and diagnostics events:

• Per pod — its protocol version, negotiated level (compatible / degraded / incompatible), and a red flag with the exact reason on a mismatch (e.g. "worker speaks protocol 2, control-plane speaks 1 → stopped").
• Blocked runs — runs parked because no live worker advertises a required capability ("requires capability X, no worker"), so a stuck fleet is diagnosable at a glance rather than a silent hang.

A store-less tenant api/dashboard pod serves this same view — its reads proxy to the control plane over the wire, so the panel looks identical whether or not the pod owns a store.

Engine health & admin surface

The dashboard's Workers panel and its retry/cancel actions are a thin layer over methods WorkflowEngine exposes directly — useful when you're scripting an operational task instead of clicking through the SPA.

Worker liveness and adaptive concurrency — engine.workerHealth(extra?: string[]) returns one GroupHealth per worker group (the same call the dashboard's /durable/api/workers route and the Telescope worker panels use):

interface GroupHealth {
  group: string;
  /** Outstanding jobs in the group's task queue (waiting + active + delayed + prioritized). */
  depth: number;
  /** Workers with a non-expired heartbeat for this group. */
  liveWorkers: WorkerHeartbeat[];
}

interface WorkerHeartbeat {
  group: string;
  /** Stable per-process id (host + pid) — N replicas of a group each show as a distinct worker. */
  instanceId: string;
  lastBeatAt: number;
  /** Present on newer SDKs: concurrency mode/limit, in-flight, RAM %/CPU %, throughput, p95, last adjust. */
  status?: WorkerStatus;
}

depth > 0 && liveWorkers.length === 0 is the actionable alert — work queued with nothing consuming it. Pass extra to include local-step groups the engine can't derive from its registrations (they have no heartbeat to discover otherwise). Only a groupHealth-capable transport (BullMQ) returns non-empty results.

Bulk operations:

• engine.cancelWhere(filter, opts?) — cancel every run matching a RunQuery (workflow / status / tag / search-attribute predicates) — the same matching the dashboard's run list uses. Each match runs through the normal cancel path (child cascade, optional saga compensate, cancel listeners, control-plane broadcast), so it's the scriptable form of selecting rows and hitting Cancel. Returns one RunResult per matched run.
• engine.deleteRun(runId) — hard-delete a run and its entire subtree (checkpoints, signal waiters, search-attribute rows), cascading depth-first via getRunChildren. Unlike cancel, the run vanishes from getRun/listRuns entirely. Returns the number of runs deleted. Prefer cancel for a still-live run — deleting one mid-flight orphans its worker.
• engine.getRunChildren(parentRunId) — the ids a run spawned via ctx.child (awaited) or ctx.startChild (fire-and-forget), the same parent→children edge cancelWhere's cascade and the dashboard's run-tree both walk.

Graceful shutdown — engine.drain(timeoutMs = 10_000) stops the engine picking up new runs (recovery and the timer poller become no-ops) and waits for in-flight executions to settle, up to timeoutMs. Call it from your app's shutdown hook so a deploy hands off cleanly instead of leaving runs to time out their lease.

Cross-instance hooks — onEnqueued/onCancel let a worker react to activity from other instances over the control plane, without polling:

// Wire a worker to pick up a run the instant another pod enqueues it.
const stopEnqueued = engine.onEnqueued((runId) => engine.runOne(runId));

// Cooperative cancellation: abort in-flight work instead of finishing it just to
// have the result discarded.
const stopCancel = engine.onCancel((runId) => myWorkerBridge.abort(runId));

Both return an unsubscribe function. onEnqueued is only useful on instances that should execute runs (workers); onCancel is for a worker bridge doing cooperative cancellation of long in-flight work (a subprocess, a remote call) that a plain status flip can't interrupt.

API

The SPA is backed by a small JSON API you can also call directly. A codegen-emitted typed client ships at the ./client subpath as durableClient, so an external dashboard or script can call the same routes with full types instead of hand-rolling fetch:

import { durableClient } from '@dudousxd/nestjs-durable-dashboard/client';

const runs = await durableClient.runs('failed', undefined, undefined, { limit: 100 });
const detail = await durableClient.run(runs[0].id);
await durableClient.retry(detail.run.id);