Singleton workflows

Some work must never overlap for a given key, even though many different keys should run freely in parallel — one inventory sync per store, one statement generation per account, one deploy per environment. singleton is a durable, FIFO mutex over a workflow's runs: declare a key function once, and the engine admits at most limit runs sharing that key at a time, across every instance that shares the store.

As a live system: three keys, one mutex slot each. Burst store:A and only its lane backs up, while the waiters sit suspended with zero compute:

Declaring a singletonCopy Anchor Link

Put singleton in the class's static workflow config:

import { BaseWorkflow } from '@adonis-agora/durable'
import type { WorkflowCtx } from '@adonis-agora/durable'

export default class SyncInventoryWorkflow extends BaseWorkflow {
  static workflow = {
    name: 'sync-inventory',
    version: '1',
    singleton: {
      key: (input: unknown) => `store:${(input as { storeId: string }).storeId}`,
    },
  }

  async run(ctx: WorkflowCtx, input: { storeId: string }) {
    const items = await ctx.step('inventory:fetch', input)
    await ctx.step('inventory:reconcile', items)
  }
}

The same singleton option is accepted by engine.register(name, version, fn, { singleton }) and engine.registerRemote(...).

With the key derived from storeId, two sync-inventory runs for storeId: 'A' never overlap, while a run for storeId: 'B' proceeds independently — its own key, its own slot. SingletonConfig has three fields:

The key is scoped to the workflow: admission only looks at runs of the same workflow sharing the key. (A concurrency quota key, by contrast, is shared across workflows.)

What happens when a second start arrivesCopy Anchor Link

A singleton run is tagged singleton:<key> at start time. Start never blocks and, without maxQueueDepth, never fails: the run is created as usual. The gate runs when the run is executed — first marked running, then checked before its body executes. The check is one atomic call to the store, tryAdmitSingleton:

1. Inside a single transaction that serializes competing claims, the store reads this workflow's runs carrying the same singleton:<key> tag in pending, running, suspended or blocked.
2. Runs that already hold a slot keep it. A holder is marked durably on the run itself, with the reserved tag durable:singleton:admitted.
3. The free slots (limit minus the holders) go to the oldest waiters by (createdAt, id). If this run is one of them, the store adds the admission tag in the same transaction and the body runs.
4. Otherwise the store parks the run suspended with a retry wakeAt (~1s, jittered ±250ms so a queue of waiters doesn't wake in lockstep), still inside that transaction, and the engine releases the lease. The timer poller wakes it and the claim runs again.

A slot is held from admission until the run reaches a terminal status (completed, failed, cancelled, dead), and nothing else releases it. A holder that sleeps, waits on a signal, is parked blocked, is recovered by another instance, or whose execution lease expires keeps its slot. A later start can't take a slot away from a holder, including a start in the same millisecond whose id happens to sort first. Each admission check after the first is a cheap confirmation.

Completion is push, not just poll: when a singleton run reaches a terminal state — including a cancel or an execution-timeout sweep — the engine immediately makes the oldest limit gated waiters sharing its key due and dispatches them, instead of making them wait out their retry timer. Sleeping holders are left alone. Each woken waiter claims through the store, so only as many as there are free slots actually run. The durable retry timer still exists as the crash/cross-instance fallback — if the notify is missed (e.g. the notifying instance dies first), the next timer tick catches up. It is also how waiters learn about a holder that was dead-lettered: the dead write drops the holder's admission tag, and the waiters claim the slot on their next retry.

// A: holds the slot for store A
await engine.start('sync-inventory', { storeId: 'A' }, 'run-1')
// B: same key, key is taken → suspended, gated behind A
await engine.start('sync-inventory', { storeId: 'A' }, 'run-2')
// C: different key → its own slot, runs immediately
await engine.start('sync-inventory', { storeId: 'B' }, 'run-3')

Guarantees across instancesCopy Anchor Link

The claim is atomic, so several instances sharing one store can never admit more than limit runs for a key. Two consequences of the store doing the claim:

• No over-admission. Two replicas claiming at the same moment are serialized by the store: the second sees the first's holder and waits. An older run that a worker picks up after a younger one is never admitted alongside it. Both are in the FIFO queue from the moment they are created (pending counts), so whichever is picked up first, only the oldest waiter takes a free slot.
• A cancel is never undone by the gate. A run cancelled while its admission check is in flight stays cancelled: the store only parks a run that is still active, under the same lock as the read.

A holder whose process crashes keeps its slot while it is running with an expired lease, until recovery re-drives it (or a cancel, timeout or dead-letter settles it). That is deliberate: the run's earlier steps already ran under the slot.

Admission is global, whatever the engine's scopeReads setting. scopeReads fences an engine's run reads to its namespace, but the slot claim (tryAdmitSingleton) always counts every active run of the workflow that shares the key, in every namespace. That is the point of a singleton key: an engine scoped to acme and another scoped to globex starting sync-inventory with the same key still get one run at a time. To get one slot per tenant, put the tenant in the key the key function returns, e.g. `${tenant}:store:${storeId}`.

Store supportCopy Anchor Link

Admission needs StateStore.tryAdmitSingleton. The bundled stores implement it:

• Lucid store: a transaction that locks the key's active rows with SELECT … FOR UPDATE (ordered by id) on Postgres and MySQL. On SQLite, which has no row locks, the transaction takes the database write lock first with a no-op UPDATE, then reads. No migration is needed: the slot is a tag on the existing tags column.
• In-memory store: the claim runs in one synchronous turn.
• CodecStateStore (a payload codec — encryption, compression — around another store) forwards the call to the store it wraps (when that store implements it), so wrapping a store keeps admission atomic.

A custom StateStore without tryAdmitSingleton keeps working after an upgrade, singleton workflows included. On such a store the engine falls back to the previous admission: a scan of the key's running/suspended runs, admitting the run if it is among the limit oldest by (createdAt, id). That fallback is a read followed by a decision, not a claim. Replicas can briefly admit more than limit runs, an older run picked up after a younger one is admitted alongside it, and a same-millisecond start can displace a holder. The engine prints one warning per engine (console.warn, naming StateStore.tryAdmitSingleton) the first time it admits this way. To get atomic admission, implement tryAdmitSingleton. The helpers singletonAdmissionTags and singletonAdmissionDenialPatch (exported from @adonis-agora/durable) implement the decision, including the count of holders admitted by pre-0.51 replicas (see the upgrade note below). A store wraps them in its own transaction and must drop the durable:singleton:admitted and durable:singleton:tracked tags when updateRun/updateRunIf writes a terminal status (withoutSingletonMarkers does it). The state-store conformance suite covers the claim.

Bounding the queue: maxQueueDepth and SingletonQueueFullErrorCopy Anchor Link

Left unset, the same-key backlog can grow without bound — every start for a busy key is accepted and simply queues behind the ones ahead of it, and the failure surfaces late, as a store full of runs that will not finish for hours. Set maxQueueDepth to cap it: once the runs sharing the key reach limit + maxQueueDepth, a further start is rejected outright with SingletonQueueFullError, and no run is created for it:

static workflow = {
  name: 'sync-inventory',
  version: '1',
  singleton: {
    key: (input: unknown) => `store:${(input as { storeId: string }).storeId}`,
    limit: 1,
    maxQueueDepth: 2, // at most 1 running + 2 waiting per store; a 4th start is rejected
  },
}

import { SingletonQueueFullError } from '@adonis-agora/durable'

try {
  await SyncInventoryWorkflow.dispatch({ storeId: 'A' })
} catch (error) {
  if (error instanceof SingletonQueueFullError) {
    // error.workflow, error.key, error.maxQueueDepth — retry later or shed the request
    return response.tooManyRequests({ retryAfter: 60 })
  }
  throw error
}

The capacity check counts the pending/running/suspended/blocked runs of the workflow sharing the key in one scan and compares against (limit ?? 1) + maxQueueDepth. Unlike the slot claim, that scan is an ordinary run read, so with scopeReads on it only counts the engine's own namespace. It runs before the run row is written, so a rejected start leaves no trace in the store. Once a slot frees (notify-on-release promotes the oldest waiter, which later settles), the backlog count drops below the cap and the next start is accepted again.

The check applies wherever the workflow is started — engine.start, Wf.dispatch / Wf.start, child workflows (a rejected child start is delivered to the parent as a failed child), event triggers and schedules. An idempotent re-start of an existing runId returns that run before the check and is never rejected.

Singleton schedulesCopy Anchor Link

A schedule fires a new run for every window, whether or not the previous one is still going. Making the scheduled workflow a singleton (with a constant key) serializes those runs instead of letting them overlap; the schedule's own overlap: 'skip' is the alternative when a late window should be dropped rather than queued.

Interaction with signalWithStartCopy Anchor Link

signalWithStart is just start (idempotent by runId) followed by a signal — it doesn't bypass the singleton gate. If the workflow it starts is a singleton and the derived key is already busy, the new run is created (or, with maxQueueDepth set, rejected with SingletonQueueFullError exactly as any other start) and gates like any other same-key run. The signal is buffered, but the body that waits for it doesn't execute until the run is admitted. Don't rely on signalWithStart to get a gated singleton entity moving immediately; it still queues behind whatever else shares its key.

Singleton key vs. a deterministic runIdCopy Anchor Link

Both look like "dedupe", but they solve different problems:

• Deterministic runId dedupes the start itself. start is idempotent by runId: if a run with that id already exists, the call is a no-op that returns the existing run's state — no new run, nothing queues. Use this when a redelivered trigger (an at-least-once queue, a retried request) must not create a second run of the same logical job.
• singleton key serializes distinct runs. Two different runIds that share a singleton key are two real runs — each gets its own history, its own dashboard entry — but the engine admits only limit of them at a time and gates the rest. Use this when overlapping runs for the same key are the problem even though each one is a legitimately distinct execution (e.g. two independently-triggered inventory syncs for the same store shouldn't race each other, but both should eventually run and both should be individually inspectable).

In short: a deterministic runId says "this is the same job, don't start it twice"; singleton says "these may be different jobs, but only limit of them may run at a time." They compose — a singleton workflow can still be started with a deterministic runId to also dedupe exact-duplicate triggers on top of the per-key serialization.