Tenancy
The two isolation axes — namespace on the engine and partition on the transport — what each one actually partitions, and the boundary that keeps a store-less pod to its own runs.
Tenancy answers one question: whose runs are these, and which pool executes them. It is independent of how many processes you run — Topologies covers that. This page is what the isolation axes mean once you reach for them.
There are exactly two, they take the same kind of value, and they are not interchangeable:
| Axis | Set on | Partitions |
|---|---|---|
namespace | the engine, in config/durable.ts | the store: which runs this engine polls, recovers and resumes. Also prefixes the transport's keys. |
partition | the transport (and a tenant pod's config) | the queues: the <name>@<partition> routing token a dispatch targets and a worker serves. |
A useful shorthand: namespace is what a control plane polls; partition is what a worker
subscribes to.
namespace — the store poll scope
import { defineConfig, stores, transports } from '@adonis-agora/durable'
export default defineConfig({
namespace: 'blue', // this engine's pool
transport: 'bullmq',
transports: { bullmq: transports.bullmq({ connection: process.env.REDIS_URL }) },
store: 'lucid',
stores: { lucid: stores.lucid({ connection: 'pg' }) },
})It isolates two things at once:
- The store. Every run this engine creates is stamped with its namespace, and the poll paths —
driving pending runs, recovering incomplete ones, resuming due timers, waking blocked runs — only
act on runs carrying it.
resumeof a run from another namespace throwsNamespaceMismatchand releases the lock rather than replaying someone else's history. - The transport. A non-default namespace segments the key prefix:
durablebecomesdurable-<namespace>across every queue and key the pool uses.
Both follow one default-is-bare rule: 'default' (the default value) leaves every name
byte-identical to a single-pool deployment. Only a real, non-default namespace prefixes anything.
What an unset namespace means depends on the role. On a standalone engine (the default role)
it means 'default': the provider gives the engine namespace: 'default', so it drives, recovers and
resumes only the default pool's runs. That includes legacy rows written before runs carried a
namespace: the Lucid schema heal adds the column as NOT NULL DEFAULT 'default', and every shipped store
reads a missing (NULL) value back as 'default'. An empty string is the default pool too, as it is
for the transports: an engine configured with namespace: '' stamps and polls 'default', the stores
write '' as 'default', and a legacy row stored with '' reads back as 'default' and matches every
'default' filter. On a
control-plane it makes the engine the operator: it scopes nothing, so the store's poll filters
no-op, it drives runs of every namespace, and its transports stay on the bare prefix. "Seeing
everyone" is the absence of a namespace on a control plane, not a wildcard value. There is no standalone
operator: a process that must drive every pool is a control-plane.
Behaviour change. A standalone engine without a namespace used to be the operator, driving every
pool's runs on a shared store. It now stays in 'default'. If you relied on that, move the process to
role: 'control-plane'.
The engine's namespace and a run's namespace are not the same kind of value. A control plane's may be
undefined (the operator). A stored run's never is: every store normalizes it to 'default' on
write, matching the SQL column default. So "a control plane with no namespace" and "a run with no
namespace" mean different things, and only the first one means all.
partition — the queue suffix
Where namespace decides which runs a control plane owns, partition decides which pool of
workers executes them. It is set on the transport, and both sides must agree:
// Dispatch side — this engine's steps route to `<name>@acme`
transports.bullmq({ connection: process.env.REDIS_URL, partition: 'acme' })
// Worker side — a store-less pod declares the tenant it serves
export default defineConfig({
role: 'tenant',
transport: 'bullmq',
transports: { bullmq: transports.bullmq({ connection: process.env.REDIS_URL }) },
partition: 'acme',
})Every routing token becomes <name>@<partition>, so two pools serving the same step name never cross.
A worker on no partition serves the bare token — which is the same queue every un-partitioned process
in the deployment is on.
On the dispatch side the suffix comes from the run's namespace, not from the engine doing the
dispatching (a step's own partition still wins when it names one). That is what lets a single operator
serve many pools: it drives every namespace's runs, and each one's steps still land on that namespace's
queues rather than being pulled onto the operator's bare tokens.
Inside a thin worker's own workflow turn, ctx.step(name, input, { group }) overrides the token for
one step; omitted, it inherits the worker's partition, so a workflow and the steps it dispatches stay
in the same pool by default. The same suffix is what a
Python worker matches with run_redis_worker(worker, partition=...).
The wire between control plane and tenant
A store-less pod holds no database credentials. Everything it would otherwise read from or write to the store crosses the transport as plain JSON, in the same shapes the NestJS and Python SDKs use:
StartRunMessageon<prefix>-start-run:{ tenant, workflow, input, runId?, tags?, searchAttributes? }. The control plane turns it into a durable run.RunRequest/RunReplyon<prefix>-run-requestand<prefix>-run-reply, correlated byrequestId: every read and control verb (getRunDetail,listRuns,cancel,retry,continue,retryWithInput, …) proxied to the control plane and answered on the reply.TenantEvent: lifecycle events re-published per tenant, so a tenant's dashboard can tail its own runs live.
Which verbs each role exposes is covered in Reading and steering runs.
The isolation boundary
Every read, control action and run-start a store-less pod performs is a wire request answered by the
control plane's RunRequestResponder. That responder — not the client — is the boundary, and it
enforces:
listRuns,runFacetsandrunValueFacetsare namespace-forced to the tenant it derived. The client'snamespaceis discarded, never merely validated, so a tenant cannot list, count or read the tag vocabulary of another's runs.waitingForanswers only for the run ids the tenant owns.- Every runId-bearing verb (
getRun,getCheckpoints,getRunChildren,getRunDetail,getFanMembers,getSearchAttributes,signal,cancel,redispatch,retry,continue,retryWithInput) loads the run first and rejects withcross-tenantwhenrun.namespaceisn't the derived tenant — anti-IDOR, before anything is read or mutated. workerHealthis group-scoped to the tenant's own@<tenant>queues; the operator's bare groups and every other tenant's are dropped.startforces the run's namespace to the derived tenant, so a run a tenant starts is owned by, and later only reachable by, that tenant.- Unknown verbs are rejected rather than silently ignored.
The guarantee therefore holds even against a misbehaving client.
It is only as strong as the claim it reads. The tenant on a wire request is asserted by the caller.
Pair this with
layered tenant authentication:
network/prefix segmentation, plus a signed token the control plane derives the tenant from instead of
trusting the body.
Read scoping: scopeReads
The responder fences what a store-less pod can reach. A process that does hold the store can be
fenced too: with scopeReads: true and a namespace in effect (an explicit namespace, or the implicit
'default' of a standalone engine), the provider hands
the engine a view of the store whose run reads (getRun, listRuns, the counts and facets, and the
recovery, dispatch and timer scans) only see that namespace's runs. Writes are not fenced. It is off by
default, because a control plane's own operator screens should see everything, and it has no effect on
a store without scoping support (the Lucid and in-memory stores have it). See
Scoped reads.
Which namespace a new run gets
- A top-level
startis stamped with the starting engine's namespace, unless the call passesopts.namespaceexplicitly. - A run started by a store-less pod is stamped with the tenant the responder verified — the pod cannot choose.
- A retry (
retryWithInput) re-stamps the new run with the original run's namespace, so a fix-and-replay of a tenant's run stays that tenant's instead of falling back to the engine's own. - A child run inherits the parent run's namespace, not the namespace of whichever engine happens to execute that parent. This matters precisely because an operator legitimately executes every namespace's parents: without the rule, a recovery-resumed parent's child would fall back to the operator's own pool instead of the tenant's.
Sharing one store and one broker
Because the two axes are independent, one Redis/Postgres pair can host several non-interchangeable pools:
- Per-tenant worker pools, one control plane. Keep every run in the control plane's namespace and
route execution with
partition. The control plane polls and recovers everything; each tenant's workers only ever see<name>@<their tenant>. This is the shape the cluster docs describe. - Fully separate pools. Give each pool its own
namespace. Runs, queues and keys are all prefixed, so a dev cluster and a developer's laptop can share infrastructure without either driving the other's runs. Each namespaced engine drives only its own; an operator (acontrol-planewith nonamespace) alongside them drives all of them, which is what lets one deployed control plane orchestrate work that executes on many pools.
The two combine, and they answer different questions. Reach for partition when the runs are the same
system's but the execution must be isolated. Reach for namespace when the runs themselves belong
to different deployments.
Pitfalls
- A worker on no partition claims the shared pool's work. An un-partitioned worker is on the bare
<name>tokens — the exact queues every un-partitioned process in the deployment is on. A local process left running against a shared broker is not "another consumer"; it competes for, and steals, the deployed fleet's tasks. Give it a real partition. - A run stamped into a namespace nothing polls sits pending forever. Passing
opts.namespace(or starting from a tenant pod) routes the run to a pool that must actually have someone polling it — either an engine on that namespace, or an operator control plane. Between namespaced engines there is no fallback owner, so a namespace with no operator above it and no engine of its own is a dead end. - A stale local worker with an old build can misroute a run even when partitioned. A process left
running after a step or workflow rename can pick up a run with code that no longer matches its
history — a
NonDeterminismErroror a hung run. Kill stale processes; a distinct partition is not protection against a stale build. listRuns's namespace is enforced server-side. A tenant cannot widen its query by passing a differentnamespace— the responder overwrites it unconditionally. Don't build a client-side filter and call it isolation; there isn't one to bypass in the first place.
Next steps
- Topologies — the deployment shapes these axes ride on.
- Roles & config — the role-discriminated config, the
runGatewaysurface, and layered tenant authentication. - Thin workers — the store-less runtime that serves a partition.
Sleep & signals
Pause a workflow durably — ctx.sleep for time-based waits (minutes to months, no compute), ctx.waitForSignal for human approvals and webhooks, and ctx.waitForEvent for name-based pub/sub with reliable (buffered) delivery, all surviving restarts.
Authoring
Everything you build a real workflow from. The app/workflows convention, child workflows, durable entities, queries and updates, events, webhooks and external tasks, versioning, and both config-driven and runtime-persisted scheduling.