OTel export
Ship recorded Telescope entries — starting with every agora:<lib>:<event> diagnostics event — as OTLP spans and logs to a self-hosted OTel Collector, so the whole Agora ecosystem shows up in Grafana (Tempo/Loki) with zero per-lib code.
Telescope's generic diagnostics watcher already captures
every agora:<lib>:<event> any Agora library emits — no per-lib code, just one watcher reading
node:diagnostics_channel. OTel export takes that SAME already-captured stream and, when
turned on, converts it into OTLP spans (for timed operations) and logs (for point-in-time
events), batched and shipped to an OTel Collector — which then routes them into Tempo (traces),
Loki (logs), and Grafana visualizes both. No library in the ecosystem needs new code: Telescope
already sees everything through the diagnostics registry, so this is one new capability, not
N per-lib bridges.
This is a v1: traces and logs. Metrics are intentionally deferred — see Future work for why.
Install
OTel export lives inside @adonis-agora/telescope — no separate subpath/provider, since it
plugs into the SAME store chain redaction/sampling/streaming already use. Turning it on needs
two things: the config, and the OTel packages it lazily imports.
npm i @opentelemetry/api @opentelemetry/sdk-trace-base @opentelemetry/sdk-logs \
@opentelemetry/exporter-trace-otlp-http @opentelemetry/exporter-logs-otlp-http \
@opentelemetry/resourcesAll six are OPTIONAL peers. They are never imported unless otel.enabled is true in
config/telescope.ts — a host that never turns this on never needs them installed, and
@adonis-agora/telescope never pulls them in as a hard dependency. If you enable it without
installing them, boot logs an error and telescope disables OTel export for that run instead of
crashing.
Configuration
import { defineConfig } from '@adonis-agora/telescope'
export default defineConfig({
otel: {
enabled: true,
endpoint: 'http://localhost:4318', // your OTel Collector's OTLP/HTTP port
headers: {}, // e.g. { Authorization: 'Bearer <token>' } for a hosted ingress
serviceName: 'my-app',
entryTypes: ['diagnostic'], // which recorded entry types get exported
timeoutMs: 5000,
},
})| Key | Default | Description |
|---|---|---|
enabled | false | Master switch. Genuinely opt-in — no @opentelemetry/* import happens while off. |
endpoint | 'http://localhost:4318' | Base URL of the OTLP/HTTP receiver (the Collector's own default port). |
tracesPath | '/v1/traces' | Appended to endpoint for the traces signal. |
logsPath | '/v1/logs' | Appended to endpoint for the logs signal. |
headers | {} | Extra HTTP headers on every export request (auth for a hosted Collector ingress). |
serviceName | OTEL_SERVICE_NAME env, else 'adonis-app' | The service.name resource attribute stamped on every span/log. |
entryTypes | ['diagnostic'] | Which recorded entry TYPES get exported. Widen later (e.g. 'request') as the mapper grows. |
timeoutMs | 5000 | Per-export HTTP timeout. |
The mapping: diagnostics entry → span or log
This is the crux of the feature, and it is a PURE, independently unit-tested function
(packages/core/src/otel/mapper.ts, also importable standalone from
@adonis-agora/telescope/otel with zero OTel packages required) — the rest of this page is
really just documenting what that function does.
Every recorded entry already has a uniform shape (type, content, tags, durationMs,
traceId, createdAt). For a diagnostic entry specifically, content is { lib, event, ts, traceId, payload, durationMs } — exactly what @adonis-agora/diagnostics's wire envelope
carries.
1. Has a duration → SPAN. The mapper looks for a duration in two places:
- the entry's own top-level
durationMs— set whenever the emitting library calledemit(lib, event, payload, { durationMs }); - otherwise, a numeric
payload.durationMs— the convention a library uses when it stuffs a richer lifecycle-event object as the payload.@adonis-agora/durable's diagnostics bridge is exactly this shape:emit('durable', event.type, event), whereevent(astep.completed/run.failed/ …) already carries its owndurationMs.
The span's start time is reconstructed as end - durationMs — the event is ALREADY FINISHED
by the time Telescope observes it (there is no live span to attach to, only a historical
record). This is the same reconstruction technique @adonis-agora/durable's own bespoke OTel
bridge (attachDurableOtel) uses for its step spans.
2. No duration → LOG. A point-in-time occurrence (agora:cache:hit, a one-shot
notification, …) becomes an OTel log record instead of a zero-duration span — more idiomatic
(Loki, not Tempo, is where "something happened" belongs) and cheaper.
Both forms share:
- a name/body of
agora.<lib>.<event>— the SAME naming@adonis-agora/diagnostics's own OTel bridge uses for the identical channels, so a host running both sees consistent naming; - flattened scalar/array
payloadfields asagora.payload.<key>attributes; agora.lib,agora.event, andagora.entry_id(the Telescope entry id — click through from a Grafana span/log straight back to the exact entry in the Telescope dashboard);- error detection: the SAME
isErrorEntryheuristic the sampling store'skeepErrorsalready uses (tags includefailed,content.failed === true,content.statusCode >= 500, or a warn/error/fatalcontent.level), PLUS two diagnostics-specific additions it can't see on its own — a truthypayload.errorfield, or aneventname containingerror/fail('run.failed','step.failed', …). A span becomesERRORstatus; a log gets promoted toERRORseverity (unless an explicitpayload.levelsays otherwise).
Trace correlation
entry.traceId is whatever @adonis-agora/context resolved at record time — either a fresh
random 16-byte value, or (when the inbound HTTP request carried one) the exact W3C
traceparent trace-id, parsed byte-for-byte. When it is present and well-formed (32 lowercase
hex chars, not all-zero), the exporter parents the reconstructed span/log under a SYNTHESIZED
remote-parent SpanContext carrying that trace id — so Tempo/Loki group it into the SAME
trace an upstream, actually-instrumented service produced, even though Telescope's span was
built after the fact from a diagnostics event rather than from a live OTel context. The
synthetic parent span id never resolves to a real span in the backend — Tempo groups by trace
id, so this is enough for correlation without needing one.
agora.trace_id is ALSO kept as a plain attribute regardless of W3C validity, purely for
search/filter inside Grafana — it's whatever @adonis-agora/context produced, which may not
always be a strict W3C id (e.g. a host-supplied custom trace id hook).
Redaction, sampling, and the overload guard
The OTel exporter is wired in as a TelescopeStore decorator, positioned in the SAME chain
redaction/sampling/streaming already use:
- Redaction happens first. The decorator calls the inner store's
record()and exports from the RETURNED entry — never the raw input — so whatever the redaction pipeline masked (authorization,password,token,secret, …) is what leaves the process as OTel attributes/log bodies. Exporting to an external Collector is a strictly bigger exfiltration surface than the local Telescope store, so it must never see anything the local store itself wasn't already allowed to keep. - Sampling is SHARED, not independent. There is no separate OTel sampling rate. A sampled-away
entry never reaches the exporter — the SAME tail-sampling decision
config.samplingmakes for local storage applies to OTel export. This was a deliberate choice over giving OTel export its own rate: an entry dropped from Telescope storage almost always SHOULD also be dropped from export (it's noise either way), and a second independent knob would double the ways to misconfigure "why am I not seeing this event anywhere." - The overload guard is honoured. While the guard has paused ingestion (event-loop p99 lag over threshold), no NEW export work starts either — entries still record normally, but the OTLP HTTP call is skipped until load recovers. Telescope's own health can never be made worse by its own OTel export.
Receiving end: a minimal OTel Collector config
You're not setting up Grafana itself here — just the Collector that receives Telescope's OTLP
export and routes it to Tempo (traces) and Loki (logs). A minimal otelcol config:
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
processors:
batch: {}
exporters:
otlp/tempo:
endpoint: tempo:4317
tls:
insecure: true
loki:
endpoint: http://loki:3100/loki/api/v1/push
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp/tempo]
logs:
receivers: [otlp]
processors: [batch]
exporters: [loki]Point config/telescope.ts's otel.endpoint at this Collector's http://<host>:4318, add
Tempo + Loki as Grafana data sources, and traces/logs from every @adonis-agora/* library show
up correlated by trace id — no per-lib Grafana wiring.
Future work
- Metrics are deferred. Turning arbitrary, heterogeneous diagnostics payloads into
meaningful COUNTERS/HISTOGRAMS needs per-event semantic knowledge (is
payload.amounta gauge, a counter increment, or just an attribute?) that spans/logs don't need — a span or log is a reasonable default for ANY event, but a metric is not. A future iteration could add an opt-in convention (e.g. ametric: { name, kind, value }field libraries can stamp onto their payload) rather than guessing. - Real span reconstruction from
trace()'s span sub-channels.@adonis-agora/diagnosticsalso publishes proper start/end/error phase events on five sub-channels per(lib, event)pair (mirroring Node's owntracingChannel) for libraries that use itstrace()helper instead ofemit(). No library in the ecosystem uses it yet, and those sub-channel names aren't in the discovery registry the generic diagnostics watcher reads — bridging them would need its own subscription + redaction + sampling pass, independent of the entry-based mechanism this page documents. Deferred until a library actually adoptstrace(). @adonis-agora/durable's bespokeotel/durable-otel.tsbridge can likely be simplified (or removed) once this ships. Durable already bridges its own engine lifecycle events onto the diagnostics bus asagora:durable:<type>(seediagnostics-bridge.ts), withdurationMsnested in the payload — exactly the shape this mapper resolves as a fallback. The one thing durable's bespoke bridge gives that this generic one does not (yet) is proper span PARENTING (a real root-span-per-run with child-spans-per-step, reconstructed via an in-processMap<runId, Span>) — this generic bridge parents everything under the resolved trace id only, not under a sibling span. If that parenting fidelity matters less than "one fewer bespoke OTel integration to maintain," durable's bridge is a candidate for retirement. Out of scope here — flagged for a follow-up, not touched.
Notable exports
defineConfig,resolveConfig(main@adonis-agora/telescopeentry); typesOtelConfig,ResolvedOtelConfig.@adonis-agora/telescope/otel— the PURE mapping surface, importable without any OTel package installed:mapEntryToOtel,resolveDurationMs,isValidW3cTraceId; typesSpanExportInput,LogExportInput,MappedEntry,AttributeMap,AttributeValue.
@adonis-agora/telescope/cpu_profiling
On-demand V8 CPU profiling subpath of @adonis-agora/telescope — captures real Profiler.start/Profiler.stop samples via node:inspector around a request, aggregates them into a flamegraph tree, and renders it in the dashboard.
@adonis-agora/telescope/mcp
The Model Context Protocol subpath of @adonis-agora/telescope — a stateless JSON-RPC endpoint that lets a coding agent (Claude Code, Cursor, …) query the app's captured telemetry with six read tools (list entries, get entry, get trace, get waterfall, Pulse health, AI diagnose), behind the same auth guard as the dashboard.