Run retention & pruning

Every run — completed, failed, cancelled, or dead — stays in the store forever by default. That's the right default for a system whose whole point is auditable history: you can always trace a run back through its checkpoints. But left alone indefinitely, durable_workflow_runs (and its child tables) only grows, and the timer poller's per-tick status scans get linearly slower as it does. The retention module option hard-prunes old terminal history on an interval so the tables stay bounded, while you keep control over exactly what "old" means, per status.

Declaring retention: the module's retention optionCopy Anchor Link

DurableModule.forRoot({
  store,
  transport,
  retention: {
    sweepInterval: '1m',
    batchSize: 1_000,
    policies: [
      { statuses: ['completed', 'cancelled'], maxAge: '14d', maxCount: 200 },
      { statuses: ['failed'], maxAge: '90d' }, // keep failures longer for debugging
    ],
  },
});

DurableRetentionOptions has:

• policies — the retention rules, one per (disjoint, per scope) status group. Required; omit retention entirely to keep everything forever (the default).
• sweepInterval — how often the prune sweep runs. A number is ms; a string is an ms-style duration ('1m', '5m' — note 'm' is minutes). 0 runs the sweep once, on boot only. Defaults to 60000 (1 minute).
• batchSize — max runs hard-deleted per batch, per policy, looped until a batch comes back short (so a big backlog drains over several batches instead of one giant delete). Defaults to 1000.

Retention only runs on a driving operator instance — a dashboard/ dispatch-only instance (drive: false) or a thin worker (no store) never prunes, so you don't need to worry about two instances racing each other's sweeps. It also requires a store adapter that implements pruneTerminalRuns — every bundled adapter does (in-memory, Drizzle SQLite/Postgres, TypeORM, Prisma, MikroORM); a custom adapter that omits it logs a warning and disables retention rather than failing boot.

RetentionPolicy: what gets pruned vs. keptCopy Anchor Link

Within a policy's statuses, a run is kept only if it satisfies every bound you set on that policy, and pruned the moment it violates any:

• maxAge — prune runs whose updatedAt (≈ when they reached their terminal status) is older than now - maxAge. A number is ms; a string is a duration ('7d', '2w', '90m' — again, 'm' is minutes; use '30d' for roughly a month).
• maxCount — keep only the maxCount most-recent (by updatedAt) runs in the status set; prune everything past that.

Set one or both — with both, the most restrictive wins: a run is pruned once it's either past maxCount or older than maxAge, whichever bites first. Boot-time validation rejects a misconfigured policy before it can silently do the wrong thing:

• every policy must set at least one of maxAge/maxCount;
• statuses may only name terminal statuses — completed, failed, cancelled, dead (TERMINAL_RUN_STATUSES); pruning a live status would race the engine and delete work in progress;
• a status may appear in only one policy per scope — two policies over the same runs must not both claim a status, so "most-recent N" is unambiguous (policies with different scopes may share statuses);
• a malformed duration string ('7days', '1 month') throws at boot instead of quietly never pruning.

Statuses you don't name in any policy are never pruned — e.g. the example above prunes completed/cancelled at 14 days (capped at 200 rows) and failed at 90 days, but never touches dead runs at all, so a poison-pill run stays inspectable indefinitely unless you add a policy for it.

Scoped policies: one tenant, one kind of runCopy Anchor Link

Add a scope to confine a policy to the runs matching it — the identity axes of a run query: namespace/namespaces, workflow/workflows, tag/tags (any of), and search-attribute attributes. A scoped policy only ever selects, and for maxCount only ever counts, runs in its scope, so "keep acme's newest 100" means 100 of acme's runs, not 100 overall.

retention: {
  policies: [
    // The default for everything…
    { statuses: ['completed', 'failed', 'cancelled'], maxAge: '30d' },
    // …but chat turns are throwaway after a day,
    { statuses: ['completed', 'cancelled'], maxAge: '1d', scope: { tags: ['chat'] } },
    // and one noisy tenant keeps only its newest 500 runs.
    { statuses: ['completed'], maxCount: 500, scope: { namespace: 'acme' } },
  ],
},

Policies are applied one after another, so a run matched by several scopes is pruned by the strictest one. A scope matches each run by its own row: children inherit their parent's namespace but not its tags, so scope by namespace or by workflows (listing the child workflows too) when a whole run tree has to age out together.

Purging on demandCopy Anchor Link

Retention ages history out; sometimes you need everything for a scope gone now — a tenant deleted their account, a feature's data is being wiped. WorkflowEngine.purgeRuns(scope, opts?) hard-deletes every run matching a scope (same RunScope shape as above), in bounded batches, through the store API:

await engine.purgeNamespace('acme');                 // every run a tenant/partition owns
await engine.purgeRuns({ tag: `tenant:${tenantId}` }); // or any other scope

• Live runs in the scope are cancelled first (plain cancel, no saga undo), then deleted. cancelLive: false leaves live runs (and live descendants of finished ones) alone and deletes terminal history only.
• Child runs are deleted with their root even when they don't match the scope themselves (children: false deletes exactly the matching runs).
• batchSize (default 500) bounds each round; each round is one bulk StateStore.deleteRuns (a custom store without it falls back to per-run deleteRun).
• An empty scope is rejected — it would delete every run in the store.

It returns how many runs were deleted, children included.

What pruning removesCopy Anchor Link

A prune sweep hard-deletes — not archives, not soft-deletes — the run row plus its checkpoints, signal waiters, and normalized search-attribute rows, in one transaction per batch, mirroring what deleteRun does for a single run. Once pruned, a run is gone from getRun/listRuns/the dashboard — there's no undo. Pruning is evaluated per run, independent of parent/child relationships: unlike engine.deleteRun (which recursively cascades to a run's children), pruneTerminalRuns only ever deletes runs matched by a policy's own statuses/maxAge/maxCount — a pruned parent's still-young child run (or vice versa) is left alone on its own terminal-status clock.

Dashboard implicationsCopy Anchor Link

A pruned run's history simply isn't there anymore — you can't drill into a completed run from three weeks ago if your policy caps completed at 14 days. This is deliberate: retention trades long-tail auditability for a bounded table (and a bounded poller scan), so size the policy around how long you actually need to debug or audit a given outcome, not around "just in case."

Operational guidanceCopy Anchor Link

Because maxAge/maxCount are set per status set, you don't have to apply the same retention to every outcome. A reasonable starting point:

• completed/cancelled — the "nothing to see here" outcomes. A couple of weeks ('14d') plus a row cap (maxCount) is usually enough to answer "did this run, and when" without the table growing forever.
• failed/dead — the outcomes you actually investigate. Keep these longer ('90d' or more, or a much higher maxCount) since a failure or dead-lettered run is exactly the history you'll want when debugging weeks later, and pruning it early loses the evidence.

Start conservative (longer maxAge, higher maxCount) and tighten once you've confirmed nothing depends on the older history — pruning is irreversible, and there's no cost to leaving retention off entirely until table growth actually becomes a problem.