Retention & archival

By default the engine keeps every run forever. That is the right default for a workflow engine — a run's history is the audit trail — but on a busy deployment the terminal runs (completed, failed, cancelled, dead) accumulate without bound, and eventually the runs table is the biggest thing in your database. Retention bounds it: declare how long each terminal status is worth keeping, and the worker tick hard-deletes runs past their age — optionally archiving each one first.

The retention configCopy Anchor Link

Set a maximum age per terminal status in config/durable.ts — a duration string ('30d', '12h') or milliseconds. Age is measured from the run's last activity (updatedAt), not its creation:

export default defineConfig({
  // … plus your transport/store (see Getting Started)
  retention: {
    completed: '30d', // the happy path is cheap to keep and cheap to drop
    cancelled: '30d',
    failed: '90d',    // failures are evidence — keep them longer
    dead: '180d',     // a poison pill's post-mortem window
  },
})

Every field is optional and an omitted status is kept forever — so retention: { completed: '7d' } alone drops old successes and touches nothing else. Omit retention entirely (the default) and nothing is ever deleted.

Durations go through the same parser as every other duration option: a number is milliseconds, a string is <number><unit> with ms, s, m (minutes), h, d or w (long forms like '7 days' work too). A malformed string ('1 month', '7dayz') throws when the provider builds the engine, so a typo fails boot instead of quietly never pruning.

Scoped policies — one tenant, one kind of runCopy Anchor Link

The per-status ages apply to every run the worker sees. retention.policies adds scoped rules next to them: each one hard-deletes terminal runs in its statuses that match its scope once their last activity is older than maxAge, and/or once more than maxCount newer ones exist (see below). The scope is the identity axes of a run query — namespace/namespaces, workflow/workflows, tag/tags, and search-attribute attributes:

export default defineConfig({
  retention: {
    // The default for everything…
    completed: '30d',
    failed: '90d',
    policies: [
      // …but chat turns are throwaway after a day,
      { statuses: ['completed', 'cancelled'], maxAge: '1d', scope: { tags: ['chat'] } },
      // and one noisy tenant keeps its successes for a week.
      { statuses: ['completed'], maxAge: '7d', scope: { namespace: 'acme' } },
    ],
  },
})

Every policy needs statuses, scope, and at least one bound — maxAge, maxCount, or both. A policy with neither throws at boot (each retention policy must set maxAge and/or maxCount). Only the terminal statuses (completed, failed, cancelled, dead) count; any other status in statuses is ignored, because pruning a live run would race the engine. Each per-status age and each policy is its own pass over the store, so a run matched by several rules is deleted by whichever one is strictest.

A scope matches each run by its own row. Children inherit their parent's namespace but not its tags. The child subtree is deleted along with a matched parent anyway, but a child is only matched on its own through a scope that fits it, so use namespace or workflows (listing the child workflows too) when a whole run tree has to age out on one clock.

Keeping the newest N — maxCountCopy Anchor Link

A policy can bound by count instead of age, or as well:

retention: {
  policies: [
    // keep acme's 200 newest finished runs, delete the rest
    { statuses: ['completed', 'failed'], maxCount: 200, scope: { namespace: 'acme' } },
    // chat turns: at most a day old AND at most 1000 of them
    { statuses: ['completed'], maxAge: '1d', maxCount: 1000, scope: { tags: ['chat'] } },
  ],
},

• maxCount keeps the N newest runs by start time (createdAt) across the policy's whole statuses set, counted within its scope — "acme's newest 200" counts acme's runs only. Older ones are deleted.
• With both bounds the most restrictive wins: a run goes once it is past either.
• A run that started in the same millisecond as the last kept run is kept too, so a tie never deletes one of the newest N.
• maxCount: 0 deletes every matching run. It must be a non-negative integer, or boot fails.
• Deletion goes through the same sweep as everything else: onEvict archiving, the child subtree deleted with its parent, and the same per-pass batch and once-a-minute throttle, so a large overshoot drains over several passes.

How the sweep runsCopy Anchor Link

The sweep (engine.sweepRetention()) is a phase of the durable:work tick — the same loop that drives recovery, timers and schedules — self-throttled to one pass per minute per instance, so calling it every tick is a no-op most of the time. Each pass is bounded (100 candidates per status), so a large backlog drains across passes rather than stalling a tick. Deletion is a hard delete: the run, its checkpoints, its signal waiters and search-attribute rows, and — cascading down — its entire child subtree go with it. A deleted run's count includes that subtree. Each pass collects its runs and their subtrees (children first) and deletes them in one call to the store's optional deleteRuns, a few IN (...) statements in one transaction, instead of four statements per run; a store without deleteRuns gets one deleteRun per run. If the batch delete fails, the whole pass is kept and the next pass retries it. With eventBufferTtl set, the same pass also deletes buffered events older than the TTL, in batches of 500.

Retention only runs inside the worker loop: durable:work, or the embedded loop of worker: { embedded: true }. A web process without the embedded loop, a node ace one-off or anything else that boots the engine never prunes, and while the engine is draining for shutdown the sweep is skipped.

Every worker instance runs its own sweep (the one-a-minute throttle is per instance), so with several workers two of them can pick the same candidates in the same minute. Deleting an already-deleted run is harmless, but the same run can reach onEvict on two instances: make the archive write idempotent (key it by run id, as below).

Archiving before deletion — engine.onEvictCopy Anchor Link

Retention deletes; onEvict is how you keep a copy. Register a hook and it runs before each run's deletion, receiving the run and its full checkpoint timeline — write them to S3, a cold table, a file:

import engine from '@adonis-agora/durable/services/main'

engine.onEvict(async (run, checkpoints) => {
  await s3.putObject({
    Bucket: 'durable-archive',
    Key: `${run.workflow}/${run.id}.json`,
    Body: JSON.stringify({ run, checkpoints }),
  })
})

Hooks are awaited in registration order, and a hook that throws skips that run's deletion — the run stays in the store and the next sweep retries it. A broken archive can therefore never lose data; the failure surfaces as a warning log, not a silent gap in the archive. onEvict returns an unsubscribe function, like every other engine listener.

Purging on demand — engine.purgeRunsCopy Anchor Link

Retention ages history out. Sometimes you need everything for a scope gone now: a tenant deleted their account, or a feature's data is being wiped. engine.purgeRuns(scope, opts?) hard-deletes every run matching a scope (the same scope shape as a retention policy) in bounded batches, and engine.purgeNamespace(namespace, opts?) is the shortcut for one tenant:

import engine from '@adonis-agora/durable/services/main'

await engine.purgeNamespace('acme')                    // every run a tenant/partition owns
await engine.purgeRuns({ tag: `tenant:${tenantId}` }) // or any other scope

• Live runs in the scope are cancelled first (a plain cancel with no saga undo), then deleted. Pass cancelLive: false to leave live runs, and live descendants of finished ones, alone and delete terminal history only.
• Child runs are deleted with their root even when they don't match the scope themselves. Pass children: false to delete exactly the matching runs.
• batchSize (default 500) bounds each round, and rounds repeat until the scope is empty. Each round's runs are deleted with one deleteRuns call when the store has it.
• An empty scope is rejected, because it would delete every run in the store.
• onEvict hooks are not called. A purge is an explicit delete, not retention, so archive first if you need a copy.

It returns how many runs were deleted, children included.

Dashboard implicationsCopy Anchor Link

A deleted run's history is gone. You can't drill into a completed run from three weeks ago if completed is capped at 14 days, and there is no undo. That trade is deliberate: retention gives up long-tail auditability for a bounded runs table. Size each age around how long you actually need to debug or audit that outcome, not around "just in case", and use onEvict when you need both.

Operational guidanceCopy Anchor Link

Because ages are set per status, different outcomes can be kept for different lengths of time. A reasonable starting point:

• completed / cancelled are the "nothing to see here" outcomes. A couple of weeks is usually enough to answer "did this run, and when".
• failed / dead are the outcomes you actually investigate. Keep them longer ('90d' or more), since a failed or dead-lettered run is exactly the history you'll want when debugging weeks later.

Start with long ages and tighten them once you've confirmed nothing depends on the older history. Pruning can't be undone, and leaving retention off costs nothing until table growth becomes a real problem.