Scaling storage

Every driver implements the same TelescopeStore contract, so the dashboard, the query API, alerts and the pruner behave the same on all of them. What changes as you climb is how much Telescope holds, for how long, and what it costs the rest of your app. Pick the lowest rung that fits; move up when a signal below says so.

Rungs are not a sequence you have to walk: an app that knows it writes a lot can start at 3. A columnar database (ClickHouse) is the natural rung after 3; it is not shipped yet, and a custom store can provide it today.

Signals that say "move up"Copy Anchor Link

• Your app's own queries slow down while Telescope writes. Rung 1 is sharing a database it shouldn't — go to 2.
• The telescope_entries table keeps growing although prune is configured. Each prune cycle is one DELETE of everything past prune.after; at a high write rate that delete gets long and the table bloats. Shorter retention or rung 3 fix it; rung 3 keeps the history.
• You want days of request / query history, not hours. A row store can hold it, but every prune cycle pays to delete it again.

Rung 1 → 2: a dedicated databaseCopy Anchor Link

Same driver, its own Lucid connection. Nothing else changes.

export default defineConfig({
  store: 'lucid',
  stores: { lucid: storage.lucid({ connection: 'telescope' }) },
  prune: { after: '24h' },
})

Rung 3: tiered — hot database, cold ParquetCopy Anchor Link

The hot tier is the store you already have, kept minutes deep. When the pruner runs, entries older than hotFor are written as Parquet files instead of being thrown away, and deleted from the hot tier only after the files are written. Reads span both tiers.

import { defineConfig, storage } from '@adonis-agora/telescope'

export default defineConfig({
  store: 'tiered',
  stores: {
    tiered: storage.tiered({
      hot: storage.lucid(),                                   // the app's own database
      cold: storage.parquet({ location: { dir: 'storage/telescope' } }),
      hotFor: '5m',                                           // how long entries stay hot
    }),
  },
  // Drives the move to cold. `after` is the COLD tier's retention.
  prune: { after: '7d', intervalMs: 60_000 },
})

Install the cold tier's packages (optional peers of the core, pure JavaScript):

npm install hyparquet hyparquet-writer hyparquet-compressors

What happens on each prune cycle:

1. Move. Hot entries older than now - hotFor (and newer than prune.after) are written to cold, then the hot tier deletes everything older than now - hotFor. If the write fails, the files it wrote are removed and the hot tier is left as it was; the next cycle retries.
2. Expire. Cold files whose whole hour is older than prune.after are deleted. Retention is per hourly file, so an hour stays visible until all of it has expired. prune.keepLast keeps the newest expired files until they hold at least that many entries.
3. Compact. Each closed hour's files (one per cycle) are merged into one file per 200k rows, written before the inputs are deleted.

Reads go to the hot tier first and to cold for whatever the answer still needs. list merges both newest-first and drops duplicate ids, so { page, size } pages straight across the tier boundary; when the hot tier fills a page with entries newer than anything in cold, cold is not read at all. get falls through to cold on a hot miss, and the traces screen and throughput chart combine both tiers. If the cold tier can't be read, hot results are still returned and a warning is logged.

LayoutCopy Anchor Link

type=<type>/dt=<yyyy-mm-dd>/hour=<hh>/<instanceId>-<uuid>.parquet

type comes first, so a query on one type lists only that type's files and a bucket lifecycle rule can give each type its own retention. Rows are sorted by created_at, id, row groups hold 10,000 rows with statistics on, and id / trace_id carry bloom filters, so a time-bounded read or a trace lookup skips the row groups that can't match. content and tags are JSON text, as in the Lucid table.

On S3Copy Anchor Link

Pass your own S3Client (@aws-sdk/client-s3). Telescope uses it for every byte, so your credentials, region, endpoint (GovCloud, a VPC endpoint, MinIO) and retry policy apply.

import { S3Client } from '@aws-sdk/client-s3'

const s3 = new S3Client({ region: 'us-east-1' })

storage.parquet({
  location: { s3: { client: s3, bucket: 'observability', prefix: 'telescope/v1' } },
})

Reads are ranged GetObject calls: a footer or a row group is fetched without the whole file.

Cold engine: pure JavaScript by defaultCopy Anchor Link

The cold tier is read with hyparquet: plain JavaScript, nothing native, nothing downloaded at runtime, and every byte of it visible to image scanners.

If your images may ship native code and you run wide scans over the cold tier (traces and tags across days, full-text search), you can opt into DuckDB instead, an optional peer:

npm install @duckdb/node-api

storage.parquet({
  location: { dir: 'storage/telescope' },
  engine: { duckdb: { memoryLimit: '512MB', threads: 2 } },   // or simply 'duckdb'
})

Selecting DuckDB without installing it fails at boot with an error naming the package; there is no silent fallback. The hyparquet packages are still needed with DuckDB (they write the files and read their footers).

Measured on one machine (300k entries in one directory, 30 row groups):

hyparquet wins the newest-first pages the dashboard opens with; DuckDB wins anything that has to look at every row.

OptionsCopy Anchor Link

Moving between rungsCopy Anchor Link

A new store starts empty and the old one's entries age out on their own retention — Telescope is observability, not a system of record, so there is nothing to migrate. Moving from rung 1 or 2 to 3 keeps the same hot table: only entries older than hotFor start moving to cold.