Duolith docs

0167. A workflow is declared on the domain, checkpointed in the store, and always reaches an end

  • Status: Accepted (Mik, 18 Nov 2026)
  • Date: 2026-11-18
  • Decided by: Mik asked on 18 Nov 2026 for "the ability to restart processes so say a action has a workflow it can track and always finalize that as well", and to consider "something we can build in house". The design is the integrator's, approved by Mik on 18 Nov 2026 with the answers recorded at the end.
  • Planning item: none (after version 1, with decisions 0163 to 0166)

Context

Some actions are not finished when the request is answered. Generating a cover letter spends credits, calls a model, stores a file and tells the user. A process can die between any two of those. What matters is not that the process restarts (decision 0166 does that) but that the work restarts where it stopped, and that it ends: finished, or undone, or in front of a person. Never half done and forgotten.

IDX-Studios/work-stable-api already needs this, and writes it by hand each time:

  • src/services/db/cover-letter-generation-recovery.ts keeps a lease on each generation, sweeps expired ones on an hourly trigger and refunds the reserved credits.
  • src/app/tokens/token-commands.ts publishes its events "best-effort" after the database commit, with a comment at each of three places explaining that a lost event is recoverable and a double debit is not.
  • Stripe webhooks carry their own idempotency claim, written for that one case.

Each is correct and each is a one-off. This is the wiring Mik asked decisions 0164 and 0165 to generate.

What exists (read 18 Nov 2026)

EngineHow it worksWhat it costs to runWhat it asks of the author
TemporalA separate service and a fleet of workers. Replays the event history of each runIts cloud starts near $100 a month, or a cluster to operate. 51,200 events per runWorkflow code must be deterministic. Changes need version flags in the code
DBOSA library over Postgres. One row per run, one row per finished step"One database write per step … plus two additional database writes per workflow", about 1,000 steps a second per Postgres core. Recovery across servers needs its Conductor service, and suspension, sharding and upgrades are its paid editionDeterministic workflow code; steps "should be idempotent"
Vercel Workflow"use workflow" and "use step" compiled into routes, an event log, and a storage seam it calls a World$0.02 per 1,000 events on Vercel. Runs stay on the deployment that started them. Its Postgres World does not pin yet. In betaDeterministic workflow code, in a sandbox
Cloudflare Workflowsstep.do, step.sleep, step.waitForEvent on Workers500,000 steps a month included, then $0.80 per 100,000. Sleeping costs storage only. Up to 25,000 steps and a year of sleepRuns only on Cloudflare
AWS Lambda durable functionsCheckpoint and replay inside LambdaWaits are not billed for compute. Runs up to a yearDeterministic code outside steps. Runs only on Lambda
Inngest, Upstash Workflow, Trigger.devA hosted service calls back into your routes step by step$99 a month per million runs (Inngest), $1 per 100,000 steps (Upstash)A third party holds the state of every run

Two things are common to all of them. The flow is ordinary code, so the engine must replay it, and replay only works if the code is deterministic: no clock, no random number, no outside call except inside a step. Every one of them documents that rule and none can check it fully. And changing a flow while runs are in flight is left to the author (version flags) or to the host (pinned deployments).

Decision

Workflows are part of the domains extension (decision 0164), built in house on the store seam of decision 0165. There is no outside engine and no extra service. The flow is declared in the domain file, and each step's body is a method the author writes, exactly as an operation is.

1. A workflow is declared where the domain is

# domains/coverLetters.domain.yaml
commands:
  generateCoverLetter:
    input: { userId: id, jobId: id }
    returns: generation.Run            # answered at once; the work goes on
    allow: self(userId)
    starts: generation

workflows:
  generation:
    input: generateCoverLetter.input
    one per: [userId, jobId]           # a second start joins the run that exists
    deadline: 10m
    at once: 4 per userId              # fairness between users
    steps:
      reserve:  { does: tokens.spendTokens, undo: tokens.refundTokens }
      draft:    { retries: 3, timeout: 90s, needs: [ai] }
      store:    { changes: CoverLetter }
      tell:     { publishes: coverLetterReady }
  • A step is a method, named in the file and written by the author, with typed input (the workflow's input and the results of earlier steps) and a typed result. A missing step, or a wrong type, fails the build.
  • The shapes of a flow are a short fixed list: steps in order; together: (several at once); each: over a list, with a limit on how many run at once; wait: 2d; waits for: <event>, up to: 7d; when: a named outcome of an earlier step; does: a command of this or another domain; undo: the step that reverses this one.
  • Anything cleverer goes inside a step. A step that has to choose returns a named outcome (one of [approved, rejected]) and the flow branches on it.
  • Every name is a reference the build resolves (decision 0164, section 1a): steps, events, commands, models, services. A run is a model, coverLetters.generation.Run, with a derived key.
  • A schedule is a workflow with every: 1h or a calendar time. It replaces a scheduler entry that calls a route.
  • A job is a workflow of one step. There is no separate job mechanism (decision 0166 points here), so one pattern covers all work that outlives a request.

2. Why the flow is declared and not replayed

Because the flow is data, the engine is a state machine that reads the run's row and does the next thing. It never runs the author's code twice to find its place. That removes the hardest part of every engine above:

Replayed code (Temporal, DBOS, Vercel, Lambda)Declared flow (this decision)
Rules on the authorWorkflow code must be deterministic, and nothing can fully check itNone. There is no workflow code, only steps
Changing a flow with runs in flightVersion flags in the code, or runs pinned to old deploymentsThe build compares the old and new flow and says which runs can continue (section 5)
What the build can checkLittle: the flow is only known by running itEverything: every step, its types, its undo, what it changes and calls
What the tools can showA log of eventsThe flow as a picture, with each run's place in it
The priceNoneA flow can only take the declared shapes

The price is real, and it is the one Vercel's engineers argue against ("the best workflow engine is a programming language"). We take the other side because of what Mik asked for throughout: the domain file says what the system is, and the build proves it. Mik was content with either form (18 Nov 2026) and asked that everything be "strict where possible", with more allowed later through extensions. So the declared form is the only form at first. A flow written as code is a later extension that compiles to the same state machine and must pass the same crash test.

3. A run always reaches an end

Every run ends in exactly one of these, and the end is recorded:

EndMeaning
doneEvery step finished
undoneA step failed for good, and the undo of every finished step ran, last first
stoppedThe caller or an operator stopped it, and its undos ran

A run that cannot end by itself (an undo that keeps failing, or the deadline passed) becomes **stuck**. That is not an end. It raises an alert, shows in duolith workflows list --stuck, and waits for a person to retry or stop it. The promise is exact: no run is lost, none runs for ever, and none is left half done without a person being told. No engine can promise that an outside service will succeed.

How it holds:

  1. Starting is atomic with the command. On Postgres the command's own writes and the run's row commit in one transaction, through the outbox of decision 0165. Either both happened or neither did. On other stores the run's row is written before the command answers.
  2. Events are never best-effort. A publishes: on a command or a step goes through the same outbox, so the event is sent if and only if the change was committed. This replaces the three hand-written "best-effort publish" blocks in work-stable-api.
  3. One write per finished step. The result is stored before the next step starts. A finished step never runs again.
  4. A runner holds a lease on a run, with a number that goes up each time the lease changes hands. (A runner is the part of a server that advances runs.) A runner that stalls and wakes later is refused by that number, so two runners never both advance a run.
  5. A watchdog finds runs whose lease has run out and hands them to a live runner, which continues after the last finished step. This is the generated form of work-stable-api's recovery sweep.
  6. The step that was interrupted runs again. That is the only place where work can happen twice, so the build makes every step say how that is safe, and refuses the build otherwise: | The step… | Why a repeat is safe | | --- | --- | | changes: a model in the same Postgres as the run | Its write and its checkpoint commit together. It happens exactly once | | needs: a service that duolith.yaml lists as accepting a once-key | The generated client sends a key derived from the run and the step (Stripe's Idempotency-Key, for example), so the service ignores the repeat | | is marked repeatable | The author states that doing it twice is the same as once | | publishes: an event | Events carry an identity, and the generated subscriber drops one it has seen |
  1. Retries back off, and failures are named. A step's fails: are typed, as an operation's are. A named failure is final and starts the undos. Anything else is retried up to retries:.

4. Where it runs, and what wakes it

The engine needs two things from a target: a store with an atomic compare-and-set, and something that calls it at a given time. Both are seams, with the same conformance tests on each.

TargetRun stateWhat wakes a run
Bun, Node, Axum (a container that stays up)Postgres, claimed with SKIP LOCKEDThe runners in the same server, woken by LISTEN/NOTIFY and by the next wake time
A container that scales to zero (Cloud Run and others)Postgres or FirestoreThe platform's task queue calls an internal route that only the server hosting the runners mounts (decision 0169) and only the platform's identity may call
Cloudflare WorkersOne Durable Object per runIts alarm
VercelPostgres or RedisVercel Queues, with a cron as the watchdog
duolith dev and testsMemoryA timer, with a clock the test can move

A run that sleeps or waits is a row with a wake time. It holds no process, no connection and no memory, so waiting a week costs the same as waiting a second. The memory store is refused for workflows outside development.

A later adapter can hand a workflow to the platform's own engine (Cloudflare Workflows, Vercel Workflow, Lambda durable functions) where an app prefers that. The declared flow maps onto each, since each is a superset of it.

5. Changing a workflow while runs are in flight

Each run records the version of the flow it started on. duolith domains diff (decision 0164) compares the flow with the main branch:

  • Safe, and runs continue: a step's body changes, a step is added after every place a run can be, a retry or a timeout changes.
  • Not safe: a step is removed, reordered or retyped where a run may already have passed or be waiting. The build then requires the workflow to say what happens to runs in flight, and fails without it: ``yaml in flight: finish on the old steps # the old steps stay in the build until none use them # or: restart | undo

duolith workflows drain reports how many runs are still on an old version, and the build warns when old steps can be deleted.

6. Seeing and steering runs

  • A page can show a run. state.load(coverLetters.generation.run({ userId, jobId })) gives its step, its progress and its end, and follows it live when the model is live (decision 0166). No polling route is written.
  • Operators have duolith workflows list | show | retry | stop | drain, and the same as generated operations guarded by allow:, for an admin page.
  • Every step is a span in the trace of the command that started the run, and emits the telemetry event of decision 0165 with the run, the step, the attempt and the time taken.

7. Security

  • Stored inputs and results follow the field marks of decision 0165. A secret is never written to a run: the step receives a reference and reads the secret when it runs. personal fields are encrypted in the run and removed when it ends. Finished runs are kept for a set time (keep:), then reduced to their summary.
  • Starting is authorised once; resuming is authorised again. allow: is checked when the command starts the run. An event that resumes a waiting run is an operation with its own allow:. One that arrives from outside (a webhook) carries a signed, single-use token bound to that run and step, and the signature is checked before anything else.
  • A step never holds the caller's token. It runs as the system, on behalf of the recorded caller, with only the services its step names in needs: (decision 0168, section 4).
  • Limits against abuse and runaway cost: runs per caller and per one per key, steps per run, the size of a step's result, the total time a run may take. Each has a default and each refusal is a typed failure.
  • The wake route is never public. It is mounted only on the server that hosts the runners, as work-stable-api's /events routes are only on its events role.

8. Proved by killing it

Every build that has workflows generates a crash test and CI runs it. Using the mock services of decision 0164 and a clock it controls, it runs each workflow and kills the runner:

  • after every checkpoint, and in the middle of every step;
  • during every undo;
  • while two runners race for the same run.

It passes only if every run reaches a declared end, every outside effect happened once, and every undo that should have run did. A replaced template (decision 0165, section 4) must pass the same test.

Stages

  1. Run and step rows, leases and the watchdog on Postgres and memory; steps in order with retries, timeouts and undo; the outbox for starting and for events; the crash test.
  2. wait, waits for, together, each, when; schedules; at once and fairness; the run as a live model.
  3. Wake seams for scale-to-zero containers, Cloudflare Durable Objects and Vercel; the in flight: rules in domains diff; duolith workflows commands.
  4. Later extensions: adapters that hand a workflow to a platform's own engine, and flows written as code and compiled to the same state machine.

Alternatives considered

OptionWhy not
Adopt TemporalA second system to run or about $100 a month before the first user, a worker fleet, and determinism rules the build cannot check. The strongest choice for very large, very long workflows, and more than an early-stage business needs
Adopt DBOS as a libraryThe closest in spirit, and its cost model (one write per step) is the one taken here. But recovery across servers, suspension and upgrades are its paid edition, it is Postgres only, and its workflow code must be deterministic
Use each platform's engine (Cloudflare Workflows, Vercel Workflow, Lambda durable functions)Three behaviours, three sets of limits, and an app that cannot move between targets. Kept as a later adapter
A hosted service that calls back (Inngest, Upstash, Trigger.dev)A third party holds every run's state, and a per-step fee that grows with success
Flows as code, replayedDeterminism rules that cannot be fully checked, and version flags. See section 2
Only jobs with retries (as Oban gives)Retries one piece of work. Does not undo the steps before it or carry results between steps

Consequences

  • New context in tools/boundaries.yaml: the workflow engine inside packages/domains and its Rust crate, depending only on the store seam.
  • The domain schema gains workflows:, starts:, needs: on steps, and repeatable. duolith.yaml says which services accept once-keys. The glossary gains workflow, step, run, undo, lease, stuck.
  • Decision 0166 no longer has a job mechanism of its own and points here.
  • Run state is a new kind in the store rules of decision 0165, runs. Like the other kinds that must not be lost, it refuses when its store is down and may not resolve to memory outside development.
  • A step runs through the compiled pipeline of decision 0165 without the request-only steps: it keeps the deadline, the trace, validation of its input and result, the error step and the record.
  • Postgres is the store this is built and measured on first. One write per step is the cost to beat, and the pipeline benchmark of decision 0165 gains a workflow case.
  • work-stable-api's recovery sweep, its best-effort publishes and its webhook idempotency become declarations when it moves over. That move stays Mik's call.

Sources read on 18 Nov 2026

  • DBOS: architecture (docs.dbos.dev/architecture), "Why workflows should be Postgres rows", "Why durable execution should be lightweight".
  • Vercel: Workflow concepts (docs, updated 10 Sep 2026) and "The best workflow engine is a programming language" (27 Aug 2026).
  • Cloudflare: Workflows limits (10 Nov 2026) and pricing (4 Nov 2026).
  • AWS: Lambda durable functions, basic concepts, and the comparison with Step Functions.
  • Upstash, "Durable workflow engines compared" (20 Jul 2026, written by a vendor) and Alex Cloudstar, "Temporal vs Inngest vs Vercel Workflow" (prices rechecked 22 Oct 2026). The two disagree on Temporal's price, so only "near $100 a month" is claimed.
  • Not read today, described from general knowledge: Restate, Hatchet, Oban.

Mik's answers (18 Nov 2026)

  • Built in house.
  • Content with either form of flow, so the declared form is first and flows written as code are a later extension.
  • Strict where possible, with more allowed later through extensions.
  • "Build out everything" (18 Nov 2026). The proof is the example app of decision 0170, not WorkStable.