Agora
Observability

Metrics

Dependency-free run/step counters and duration percentiles. collectMetrics serves a Prometheus scrape, attachDurableMetrics a JSON snapshot with p50/p95/max, both built from the engine's own lifecycle events.

@adonis-agora/durable ships two metrics collectors. Neither needs an OpenTelemetry SDK or a Prometheus client library. Both subscribe to the same engine lifecycle events the dashboard, the OTel bridge and the Telescope view use, and turn them into counters you read on demand:

CollectorImport fromShapeReach for it when
collectMetrics(engine)@adonis-agora/durablePrometheus text exposition + raw snapshotyou already scrape Prometheus
attachDurableMetrics(engine)@adonis-agora/durable/otelJSON snapshot with duration percentilesyou want p50/p95/max out of the box, or JSON for a custom dashboard

collectMetrics: Prometheus text exposition

collectMetrics lives in the main entry point and is shaped for a real Prometheus scrape:

import { collectMetrics } from '@adonis-agora/durable'

const metrics = collectMetrics(engine)

Its handle:

export interface MetricsCollector {
  snapshot(): MetricsSnapshot
  /** Prometheus text exposition of the counters — serve this from a `/metrics` endpoint. */
  prometheus(): string
  /** Unsubscribe from the engine. */
  stop(): void
}

snapshot() returns a deep copy of the counters, including a per-workflow breakdown and a step-started counter:

export interface MetricsSnapshot {
  /** `routed`: failures handed to an onError handler; `unrouted`: routes the engine could not apply. */
  runs: {
    started: number
    completed: number
    failed: number
    suspended: number
    routed: number
    unrouted: number
  }
  /** Routed failures by error code (`*` for a failure without one). */
  routedByCode: Record<string, number>
  steps: { started: number; completed: number; failed: number }
  /** Per-workflow run counters, keyed by workflow name. */
  byWorkflow: Record<string, { started: number; completed: number; failed: number }>
  /** Sum + count of recorded step durations (ms), for an average. */
  stepDuration: { sumMs: number; count: number }
}

Counting across a fleet

Both collectors count only the events their own instance produced. With a control plane, every instance also hears every other instance's events (that is how a dashboard pod live-tails a run on a worker), but those are counted where they happened. Scrape every pod and sum the series: each run and each step is counted once. (Before 0.51, every pod counted what it heard as well, so a sum over N pods read N times too high.) If you write your own collector on engine.subscribe, skip isRemoteEvent(event) the same way.

The dashboard's scrape endpoint

With the dashboard mounted, GET <path>/api/metrics (new in 0.51) already serves a Prometheus exposition (text/plain; version=0.0.4), behind the console's guard. It is the collectMetrics counters of the process that answers, followed by gauges read at scrape time:

GaugeLabelsValue
durable_pending_runsruns waiting to be picked up (the dispatch backlog)
durable_running_runsruns currently running
durable_dead_runsruns in the dead-letter queue
durable_group_queue_depthgroupqueued remote work per worker group
durable_group_live_workersgrouplive workers heartbeating for the group

durable_group_queue_depth > 0 with durable_group_live_workers == 0 is the alert: work is queued for a group nobody serves. The run-count gauges are store reads that every pod answers with the same number, so take them from one target (or use max), not a sum across pods. They are left out when the count fails (a control plane too old to answer it, read from a tenant pod) rather than reported as 0. The group gauges appear only when the transport reports worker health (BullMQ, queue and db today).

Serving a scrape endpoint yourself

To serve the counters from your own route instead, create the collector once, at module scope, and serve prometheus() from a route:

start/routes.ts
import router from '@adonisjs/core/services/router'
import engine from '@adonis-agora/durable/services/main'
import { collectMetrics } from '@adonis-agora/durable'

const durableMetrics = collectMetrics(engine)

router.get('/metrics', ({ response }) => {
  response.header('content-type', 'text/plain; version=0.0.4')
  return durableMetrics.prometheus()
})

Don't call collectMetrics(engine) inside the handler. Each call adds a new engine subscription that starts counting from zero, so a per-request collector would always report nothing and would keep adding listeners. Put the route behind your own middleware/guard (or on an internal-only port) if the numbers shouldn't be public.

A scrape returns:

# TYPE durable_runs_total counter
durable_runs_total{event="started"} 42
durable_runs_total{event="completed"} 38
durable_runs_total{event="failed"} 2
durable_runs_total{event="suspended"} 6
durable_runs_total{event="routed"} 1
durable_runs_total{event="unrouted"} 0
# TYPE durable_steps_total counter
durable_steps_total{event="started"} 120
durable_steps_total{event="completed"} 115
durable_steps_total{event="failed"} 3
# TYPE durable_runs_by_workflow_total counter
durable_runs_by_workflow_total{workflow="charge-order",event="started"} 42
durable_runs_by_workflow_total{workflow="charge-order",event="completed"} 38
durable_runs_by_workflow_total{workflow="charge-order",event="failed"} 2
# TYPE durable_runs_routed_total counter
durable_runs_routed_total{code="E_CARD_DECLINED"} 1
# TYPE durable_step_duration_ms_sum counter
durable_step_duration_ms_sum 184230
# TYPE durable_step_duration_count counter
durable_step_duration_count 115

The step-duration pair counts only completed steps that reported a durationMs. Divide the sum by the count for an average. A run event without a workflow name lands under workflow="unknown". Call stop() to detach the collector from the engine.

attachDurableMetrics: JSON snapshot with percentiles

attachDurableMetrics lives in the @adonis-agora/durable/otel subpath, next to the OTel bridge, but it has no OpenTelemetry dependency and works without @opentelemetry/api:

import { attachDurableMetrics } from '@adonis-agora/durable/otel'

const metrics = attachDurableMetrics(engine)

It returns a handle with three methods:

export interface DurableMetrics {
  /** A point-in-time copy of the counters and duration percentiles. */
  snapshot(): DurableMetricsSnapshot
  /** Zero everything (e.g. between scrape windows). */
  reset(): void
  /** Stop collecting. */
  unsubscribe(): void
}

snapshot() returns run/step counts by outcome plus p50/p95/max duration stats for both runs and steps:

export interface DurableMetricsSnapshot {
  runs: {
    started: number
    completed: number
    failed: number
    suspended: number
    /** Failures handed to an onError handler instead of failing the run. */
    routed: number
    /** Failures that had onError routes the engine could not apply (workers disagree). */
    unrouted: number
  }
  /** Routed failures by error code (`*` for a failure without one). */
  routedByCode: Record<string, number>
  steps: { completed: number; failed: number }
  /** Per-step wall-clock (from the `durationMs` on step events). */
  stepDurationMs: DurationStats
  /** Per-run wall-clock, measured `run.started` → terminal. */
  runDurationMs: DurationStats
}

export interface DurationStats {
  count: number
  /** Median, 95th percentile, and max — in milliseconds. */
  p50: number
  p95: number
  max: number
}

stepDurationMs includes both completed and failed step attempts. runDurationMs is measured from run.started to run.completed / run.failed as seen by this collector: a run that started before the collector was attached (or in another process) is counted in runs but contributes no duration.

Serving a JSON snapshot

snapshot() returns plain JSON, so it drops straight into a route:

start/routes.ts
import router from '@adonisjs/core/services/router'
import engine from '@adonis-agora/durable/services/main'
import { attachDurableMetrics } from '@adonis-agora/durable/otel'

const durableMetrics = attachDurableMetrics(engine)

router.get('/metrics/durable', () => durableMetrics.snapshot())

Call reset() after reading if you want each window's counters (and percentiles) independent instead of cumulative since boot. reset() also forgets the start time of in-flight runs, so a run spanning the reset doesn't contribute a duration. Call unsubscribe() to stop listening.

Percentiles are computed in memory

attachDurableMetrics keeps every recorded step and run duration until the next reset() and sorts them on each snapshot(). In a long-lived, high-throughput process, call reset() periodically (for example after each scrape) to keep that buffer bounded.

Which one to use

Both subscribe to the same event stream through engine.subscribe and are safe to run side by side. Each just adds its own listener. Use collectMetrics for Prometheus text and per-workflow counters. Use attachDurableMetrics for duration percentiles, a JSON snapshot, or a resettable window.

A failure handed to an onError handler is counted as routed, not failed, and is broken down by error code in routedByCode (Prometheus: durable_runs_routed_total{code}). An unrouted count above zero means a deploy left workers announcing different routes.

Counters are per process

Both collectors count the events this process's engine emits, and both reset when the process restarts. A multi-pod deployment needs one scrape target per instance (or an aggregating layer), the same as any Prometheus client library. In a split cluster the run lifecycle is emitted where the engine runs (the standalone / control-plane pods), not on store-less tenant pods, which have no engine to subscribe to.

Need labels these collectors don't have? Subscribe to the stream yourself with engine.subscribe and feed your own client, or bridge the events onto the diagnostics bus.

On this page

Partner

Your product as the official integration for Durable, right here.

Become a partner →