@adonis-agora/telescope/alerts

Capturing exceptions is half the job — you also want to know when a new one appears. The @adonis-agora/telescope/alerts subpath polls the Telescope store for exception entries, evaluates alerting rules over them, and dispatches a rich payload to every configured channel: a new error family pages you, a spike pages you, and a re-occurrence after you thought it was fixed pages you again.

Install

Alerting ships inside the one @adonis-agora/telescope package. Enable it when configuring telescope:

npm i @adonis-agora/telescope
node ace configure @adonis-agora/telescope   # then pick "Alerts" at the prompt

Selecting Alerts registers @adonis-agora/telescope/alerts_provider and publishes config/telescope_alerts.ts.

How it hooks in

The alerter watches the same store everything else reads. Every every interval (default 30s) it picks up the exception entries recorded since the last look and evaluates your rules against them. Nothing is instrumented or intercepted — if an exception made it into Telescope, the alerter sees it.

Two consequences worth knowing:

• Alerts are as late as the interval. A 30s cadence means up to 30 seconds between the error and the page. Shorten every if that matters; the work per cycle is proportional to what was recorded, not to how often you ask.
• Only what was recorded is alertable. Exceptions arrive automatically once the core's request middleware is wired, and whenever you call recordException yourself. An exception that sampling dropped never reaches a rule.

Alerting starts from "now" at boot, so restarting your app never re-pages the whole backlog, and it never keeps the process alive on its own. A store failure during a cycle is logged and retried on the next one rather than thrown at your app.

When rules do and don't run

Two behaviours regularly surprise people:

Configuration

import env from '#start/env'
import { defineConfig } from '@adonis-agora/telescope/alerts'

export default defineConfig({
  channels: [
    { type: 'slack', url: env.get('TELESCOPE_SLACK_WEBHOOK') },
  ],
  rules: [{ type: 'new-exception', window: '1h' }],
  dashboardUrl: 'https://telescope.example.com/',
  every: '30s',
  cooldown: '15m',
})

Rules

type AlertRule =
  | { type: 'new-exception'; window: string }
  | { type: 'every-exception'; window?: string }   // window is OPTIONAL — display-only occurrence count
  | { type: 'exception-rate'; window: string; threshold: number }
  | {
      type: 'metric-threshold'
      metric: 'request-p95-ms' | 'request-p99-ms' | 'query-p95-ms' | 'query-p99-ms'
            | 'cache-hit-rate' | 'exception-count'
      window: string
      comparator: 'gte' | 'lte'
      threshold: number
      minSamples?: number   // guard a percentile page against a single slow request; default 1
    }

• new-exception — fires the first time an exception's familyHash is seen within window (a genuinely new error family), and again if the family re-appears after the window elapses, or after an explicit alerter.resolveFamily(hash) — the "resolved → re-occurred" signal.
• every-exception — fires for every exception (server exception and browser-reported client_exception), not just brand-new families — parity with a "notify on every error" setup. It is still rate-limited by cooldown per family, on an independent clock from new-exception, so a hot loop of the same error re-pages once per cooldown rather than on every single occurrence. Its window is optional and used only to count the occurrences shown on the alert (it does not gate firing); omit it and a 1h display window is assumed. Fired payloads carry the full exception context (the same shape new-exception gets), and a fresh vs repeat firing is badged New vs Recurring via the isNew flag (see the payload). Because it also fires on client_exception entries, this is the rule to reach for when you want front-end errors ingested via client-error reporting to page you the same way server errors do.
• exception-rate — fires when >= threshold exception entries land in the trailing window.
• metric-threshold — evaluates a computed metric over the window and fires when it crosses the threshold (comparator: 'gte' | 'lte'). The metrics are derived from the same windowed aggregation the Pulse / stats views use, so an alert means exactly what the dashboard shows — page on request-p99-ms >= 800 (a slow endpoint), cache-hit-rate <= 0.8, or an exception-count >= N spike. minSamples (default 1) guards a percentile or ratio rule against a single slow request on a quiet host — below that many samples in the window the rule holds its current state instead of deciding. It is deliberately ignored for exception-count, where a count of zero is a real, meaningful value that has to be able to auto-resolve a firing spike.

The exception rules are rate-limited by cooldown (per-rule for rate, per-family for new-exception and every-exception — each on its own independent per-family clock, so enabling both never double-suppresses).

For a "page me on every error" setup — including front-end client_exception reports — reach for every-exception. It's happy alongside a metric-threshold spike rule:

import env from '#start/env'
import { defineConfig } from '@adonis-agora/telescope/alerts'

export default defineConfig({
  channels: [{ type: 'slack', url: env.get('TELESCOPE_SLACK_WEBHOOK') }],
  rules: [
    // Fire on EVERY exception (server + browser), rate-limited per family by `cooldown`.
    // `window` is display-only here — it just labels the occurrence count on the alert.
    { type: 'every-exception', window: '15m' },
    // …and still page loudly on a broad exception spike.
    { type: 'metric-threshold', metric: 'exception-count', window: '5m', comparator: 'gte', threshold: 25 },
  ],
  cooldown: '10m',   // one page per error family per 10m, however hot the loop
})

Stateful raise / auto-resolve

metric-threshold rules are stateful, unlike the fire-and-cooldown exception rules. The alerter re-evaluates each rule on the poll interval and tracks whether it is currently firing:

• crossing the threshold when not already firing → raise (dispatch status: 'firing');
• no longer crossing while firing → auto-resolve (dispatch status: 'resolved');
• a just-resolved rule is held down for cooldown before it can re-raise (flap control).

So a Slack channel shows a spike and its recovery, and you are not re-paged every poll while an incident is ongoing. The exception rules carry no status (⇒ 'firing') — they have no resolve signal.

Deduplication

The new-exception rule's only state is a NewExceptionTracker — a bounded in-memory map (default cap 10,000 families, oldest-evicted) of family → last-seen. It's per-process by design: in a multi-replica deployment the same family is reported new once per replica, so budget your cooldown accordingly. The window check discards stale last-seen times, so it doubles as the re-occurrence detector.

Channels

Each fired alert fans out to every channel concurrently; one channel failing never blocks the others, and a per-channel failure is warn-logged (rate-limited by channel name) rather than thrown into the host.

HTTP channels use the global fetch, race every request against a 5s abort timeout, and treat a non-2xx response as a failure.

import env from '#start/env'
import { defineConfig, slackChannel, customChannel } from '@adonis-agora/telescope/alerts'

export default defineConfig({
  channels: [
    slackChannel(env.get('TELESCOPE_SLACK_WEBHOOK'), { username: 'Telescope', iconEmoji: ':rotating_light:' }),
    customChannel(async (alert) => { await pagerduty.trigger(alert) }, 'pagerduty'),
  ],
})

The alert payload

Every channel receives the same AlertPayload:

interface AlertPayload {
  rule: AlertRule
  value: number          // measured value that crossed the threshold (occurrences, count, or metric)
  threshold: number      // the rule threshold (1 for new-exception)
  firedAt: string        // ISO-8601
  instanceId: string
  status?: 'firing' | 'resolved'   // present on stateful metric-threshold alerts; absent ⇒ firing
  metric?: AlertMetric             // present on metric-threshold alerts
  exception?: {          // present for new-exception + every-exception; absent for rate/metric rules
    familyHash: string
    class: string
    message: string
    stack: string | null
    route: string | null           // route/uri/url of the request (or the page url for a client_exception)
    method: string | null          // always null for a client_exception
    statusCode: number | null
    userAgent: string | null       // reporting browser UA (client) or the sibling request's UA (server)
    referer: string | null         // `Referer` of the originating request; null for a client_exception
    componentStack: string | null  // React error-boundary stack — client_exception only
    extra: Record<string, unknown> | null  // host-defined debugging bag — client_exception only
    client: boolean                // true when the alert is a browser-reported client_exception
    clientIp: string | null        // originating IP; NEVER sourced from an untrusted body
    geo: AlertGeoLocation | null   // populated only when a geoLookup hook is configured (see below)
    durationMs: number | null      // request duration, when captured
    user: string | null            // from a `user:<id>` tag, when present
    occurrences: number            // times this family was seen in the window (>= 1)
    isNew: boolean                 // occurrences === 1 — a first occurrence vs a recurrence
    entryId: string                // for the dashboard deep link
  }
  diagnosis?: {          // AI probable-cause, on exception alerts when AI is configured
    cause: string
    fix: string
    confidence: string
    model: string
  }
  dashboardUrl?: string
}

The exception context is read generically from the exception entry (class/name, message, stack, route/uri/url, method, statusCode/status, userAgent, referer/referrer, durationMs, and a user:<id> tag), so it works with any source of exception entries. A browser-reported client_exception additionally carries componentStack (the React error-boundary stack), a host-defined extra bag, and the server-filled clientIp, with client: true and method/referer left null; a server exception carries method/referer and its recorded request IP, with componentStack/extra null and client: false.

New vs Recurring, and the fuller exception fields

Every fired exception alert carries isNew — true when this is the family's first occurrence in the window (occurrences === 1), false on a recurrence. Channels badge the two distinctly so on-call can triage urgency at a glance: the Slack channel appends · 🆕 New or · 🔁 Recurring to the end of the header line. This matters most with the every-exception rule, which pages on repeats as well as first sightings.

Beyond the core error fields, the Slack channel renders these enriched fields as context (each shown only when present, so a sparse alert stays compact):

AI probable-cause on exception alerts

When AI diagnosis is installed, every alert a new-exception or an every-exception rule fires is enriched with a diagnosis — the coordinator runs a time-bounded, fail-safe diagnosis of the offending exception and the alerter attaches a compact { cause, fix, confidence, model } summary to the payload. The rate rules carry no single offending entry, so they are never enriched. Slack renders it as a "Probable cause (AI)" section; the raw webhook carries it verbatim. The diagnose hook is guarded, so a diagnosis failure (or a slow model) never blocks or breaks the alert — the alert simply goes out without the section. Omit AI and alerts behave exactly as before.

Geo-enrichment

An exception alert can be enriched with a coarse geo location resolved from the originating client IP — so a Slack alert reads Location: 🇺🇸 San Francisco, California, United States next to the route and user. Geo is opt-in and dependency-light: the library ships no geo database and no HTTP client. You provide a geoLookup hook and own the lookup (and its caching / rate-limiting); telescope only calls it and renders the result.

import { defineConfig } from '@adonis-agora/telescope/alerts'
import { lookupCity } from '#services/geoip'   // your MaxMind / ip-api / Cloudflare wrapper

export default defineConfig({
  channels: [{ type: 'slack', url: env.get('TELESCOPE_SLACK_WEBHOOK') }],
  rules: [{ type: 'new-exception', window: '1h' }],
  // Called ONLY when an exception alert actually fires AND carries a clientIp.
  geoLookup: async (ip) => {
    const city = await lookupCity(ip)          // your provider
    if (!city) return null                     // null simply omits the Location field
    return {
      city: city.name,
      region: city.region,
      country: city.country,
      countryCode: city.isoCode,               // ISO 3166-1 alpha-2 → renders a flag emoji
    }
  },
})

The hook resolves an IP to an AlertGeoLocation — every field optional, so a partial result (country only) still renders:

interface AlertGeoLocation {
  city?: string
  region?: string
  country?: string
  countryCode?: string   // ISO 3166-1 alpha-2 (e.g. 'US'), used to render a flag emoji
}

// Sync or async; returning null (or throwing — swallowed) omits the Location field.
type GeoLookup = (ip: string) => AlertGeoLocation | null | Promise<AlertGeoLocation | null>

The result lands on the geo field of the alert's ExceptionAlertContext (and thus on the AlertPayload.exception), alongside the clientIp it was resolved from:

interface ExceptionAlertContext {
  // …familyHash, class, message, stack, route, method, statusCode, userAgent, referer,
  //   componentStack, extra, client, durationMs, user, occurrences, isNew, entryId…
  clientIp: string | null   // the originating IP; NEVER sourced from an untrusted body
  geo: AlertGeoLocation | null   // populated only when a geoLookup hook is configured AND clientIp is present
}

The Slack channel renders geo as a Location context field (formatSlackMessage composes 🇺🇸 City, Region, Country, skipping absent or duplicated parts); the raw webhook channel carries the geo object verbatim.

Notable exports

• Channels: slackChannel, webhookChannel, consoleChannel, customChannel; types AlertChannel, ChannelFetch, ConsoleSink.
• formatSlackMessage; types SlackMessage, SlackChannelOptions.
• Alerter, ExceptionPoller, NewExceptionTracker, DEFAULT_MAX_FAMILIES.
• AlerterService — the interval metric-threshold rule evaluator with raise/resolve; types AlerterServiceDeps, MetricSource.
• durationToMs; defineConfig, resolveConfig, DEFAULT_RULES.
• Types: AlertRule, AlertMetric, AlertPayload, AlertDiagnosis, ExceptionAlertContext, ResolvedAlerts, ChannelSpec, ChannelConfig, TelescopeAlertsConfig.
• Geo-enrichment types: GeoLookup (the IP→location hook) and AlertGeoLocation (its result), both re-exported from @adonis-agora/telescope/alerts.