Singleton workflows

Some work must never overlap for a given key, even though many different keys should run freely in parallel — one inventory sync per store, one statement generation per account, one deploy per environment. singleton is a durable, FIFO mutex over a workflow's runs: declare a key function once, and the engine guarantees at most limit runs sharing that key are ever in flight at the same time, across every instance.

Watch it work: two starts arrive for the same key — the first is admitted, the second queues (suspended, zero compute) until the first settles and the gate wakes it; a start for a different key would run immediately on its own slot.

1// a durable mutex per store2@Workflow({3  name: 'sync-inventory',4  version: '1',5  singleton: { key: (input) => `store:${(input as SyncInput).storeId}` },6})7export class SyncInventoryWorkflow {8  constructor(private readonly inventory: InventoryService) {}9 10  async run(ctx: WorkflowCtx, input: SyncInput) {11    const stock = await ctx.step(this.inventory.pull, input);12    await ctx.step(this.inventory.reconcile, stock);13  }14}

And as a live system — three keys, one mutex slot each. Burst store:A and only its lane backs up:

Declaring a singleton: @Workflow({ singleton })Copy Anchor Link

@Workflow({
  name: 'sync-inventory',
  singleton: { key: (input) => `store:${(input as { storeId: string }).storeId}` },
})
export class SyncInventoryWorkflow {
  async run(ctx: WorkflowCtx, input: { storeId: string }) {
    const items = await ctx.step(this.inventory.fetch, input);
    await ctx.step(this.inventory.reconcile, items);
  }
}

With the key derived from storeId, two sync-inventory runs for storeId: 'A' never overlap, while a run for storeId: 'B' proceeds independently — its own key, its own slot. SingletonConfig has three fields:

• key — derives the serialization key from the workflow's input. Any two runs whose key(input) returns the same string compete for the same slot(s); everything else is unaffected.
• limit — max concurrent runs sharing the key. Defaults to 1 (a true mutex, one at a time).
• maxQueueDepth — caps the same-key backlog (see below). Omit for the old unbounded-queue behavior.

What happens when a second start arrivesCopy Anchor Link

A singleton run is tagged singleton:<key> at start time. Before its body executes, the store atomically checks the workflow's active runs sharing that key and claims a slot. An admitted run persists the reserved durable:singleton:admitted tag. Existing holders keep their slots; the remaining slots go to waiters ordered by (createdAt, id). A later start with the same timestamp and an earlier sorting ID cannot displace a holder.

A holder retains its slot while running, suspended (including sleep and signal waits), blocked, or cancelling. Execution lease expiry and engine restarts do not release admission. Terminal settlement releases it and removes the marker, so retrying a settled run must claim a slot again. Pending runs participate in waiter ordering but do not occupy a slot until admitted.

A run that cannot claim a slot suspends with a retry wakeAt (~1s, jittered ±250ms) and checks again when the timer poller wakes it. Each check performs an atomic store operation, so multiple instances cannot claim the same capacity from a shared stale snapshot.

Completion is push, not just poll: when a singleton run settles, the engine immediately notifies the oldest gated waiter(s) sharing its key instead of making them wait out their retry timer. The durable retry timer still exists as the crash/cross-instance fallback — if the notify is missed (e.g. the notifying instance dies first), the next timer tick catches up.

// A: singleton run, holds the slot on a long wait
await startRun(engine, 'sync-inventory', { storeId: 'A' }, 'run-1');
// B: same key, key is taken → suspended, gated behind A
await startRun(engine, 'sync-inventory', { storeId: 'A' }, 'run-2');
// C: different key → its own slot, runs immediately
await startRun(engine, 'sync-inventory', { storeId: 'B' }, 'run-3');

A cancelling run skips new admission while it tears down. If it was already admitted, it continues holding its slot until cancellation settles.

Store support and upgradesCopy Anchor Link

All bundled stores implement StateStore.tryAdmitSingleton(runId, tag, workflow, limit, retryWakeAt?), and CodecStateStore forwards it to the underlying store. No schema migration is needed: admission uses the existing tags column. The SQL adapters lock active rows for the workflow before reading and persisting the claim in one transaction; this briefly serializes admission checks for that workflow, including different keys. SQLite acquires its write lock before reading. Transaction errors fail admission without running the workflow body; normal run recovery can retry the claim.

Custom stores must implement this optional StateStore method to use singleton workflows. Starting a singleton with an unsupported store rejects before creating the run. Implementations must serialize competing claims before reading, preserve admitted holders, select remaining waiters in (createdAt, id) order, and persist the marker before returning true. When retryWakeAt is supplied, atomically park a denied active waiter as suspended with that wake time inside the same transaction. Never replace a terminal or cancelling status on denial; the exported singletonAdmissionDenialPatch helper selects this patch. The exported singletonAdmissionTags helper handles selection inside that atomic transaction. Clear the marker on terminal status updates and preserve it across active recovery. Follow the store's existing read scope: an unscoped store shares the key across namespaces, while a scoped store sees only its own namespace. Treat the admission marker as reserved metadata when changing tags.

Before upgrading from scan-based singleton admission, stop new singleton starts and let existing singleton runs settle, then update all workers and resume starts. Older active rows lack admission markers, so holders and gated waiters cannot be distinguished safely. Mixing old and new workers while singleton runs remain active does not provide the new admission guarantee.

Bounding the queue: maxQueueDepth and SingletonQueueFullErrorCopy Anchor Link

Left unset, the same-key backlog can grow without bound — every start for a busy key is accepted and simply queues behind the ones ahead of it. Set maxQueueDepth to cap it: once in-flight + gated runs sharing the key reach limit + maxQueueDepth, a further start is rejected outright with SingletonQueueFullError, and no run is created for it — back-pressure against an unbounded backlog instead of silently piling up work nobody will get to for hours:

@Workflow({
  name: 'sync-inventory',
  singleton: {
    key: (input) => `store:${(input as { storeId: string }).storeId}`,
    limit: 1,
    maxQueueDepth: 2, // at most 1 running + 2 waiting per store; a 4th start is rejected
  },
})
export class SyncInventoryWorkflow { /* ... */ }

try {
  await engine.start('sync-inventory', { storeId: 'A' }, requestId);
} catch (err) {
  if (err instanceof SingletonQueueFullError) {
    // err.workflow, err.key, err.maxQueueDepth are all populated — retry later or shed the request.
  }
  throw err;
}

The capacity check counts pending/running/ suspended/blocked/cancelling runs sharing the key in one scan and compares against (cfg.limit ?? 1) + cfg.maxQueueDepth. It runs before the run row is written, so a rejected start leaves no trace in the store. Once a slot frees (notify-on-release promotes the oldest waiter), the backlog count drops below the cap and the next start is admitted again.

Interaction with signalWithStartCopy Anchor Link

signalWithStart is just start (idempotent by runId) followed by a signal — it doesn't bypass the singleton gate. If the workflow it starts is a singleton and the derived key is already busy, the new run is created (or, with maxQueueDepth set, rejected with SingletonQueueFullError exactly as any other start) and gates like any other same-key run before its body — and therefore the signal it's waiting for — ever executes. Don't rely on signalWithStart to get a gated singleton entity moving immediately; it still queues behind whatever else shares its key.

Singleton key vs. a deterministic runIdCopy Anchor Link

Both look like "dedupe," but they solve different problems:

• Deterministic runId dedupes the start itself. start is idempotent by runId : if a run with that id already exists, the call is a no-op that returns the existing run's state — no new run, nothing queues. Use this when a redelivered trigger (an at-least-once queue, a retried request) must not create a second run of the same logical job.
• singleton key serializes distinct runs. Two different runIds that share a singleton key are two real runs — each gets its own history, its own dashboard entry — but the engine admits only limit of them at a time and gates the rest. Use this when overlapping runs for the same key are the problem even though each one is a legitimately distinct execution (e.g. two independently-triggered inventory syncs for the same store shouldn't race each other, but both should eventually run and both should be individually inspectable).

In short: a deterministic runId says "this is the same job, don't start it twice"; singleton says "these may be different jobs, but only one of them may run at a time." They compose — a singleton workflow can still be started with a deterministic runId to also dedupe exact-duplicate triggers on top of the per-key serialization.