workflow-engine.md

Workflow engine

Maintainer notes on src/workflow/. The workflow file format and the driver contract are in the Workflows and Advanced workflows guide pages.

Each rule below spans more than one file. The rule names the code that owns it, so you can read the details there.

Rules that span files

• A run ends only in settleRun. The shared finishRun path calls it when settling reaches the top workflow's result, before it hands any lease back or out, and on a call that takes no lease too. reportAttempt first reconciles earlier reports, so an end cancels a later report before it can advance the revision. It records an accepted outcome and never ends the run. After a failed report, the next next skips the dependents and ends the run. Respond, cancel and retry in run-controls.ts, and next in engine.ts, share that path. Whole-run cancel settles directly as cancelled, without reconciling another round. To run code when a run ends, hook into finishRun. Do network I/O only after withRun releases the run lock. That path queues the end record, a local write, under the lock, and saves whether it did in the run's endRecordQueued field. When writeRun then fails, the queue is one save ahead of the run file, and the next next settles the run again and overwrites the queued <run>.end.json. A later next reports a record state only for a run whose field is true (runRecordState in records.ts).

• Reconcile the latest report before applying another or handing out a lease. reconcileRun, shared by leaseNext and reportAttempt, settles every driverless transition to a fixpoint at the run's current revision, through settleScope and settleItems, including workflow starts, fan-out expansion, render failures and bound exhaustion. reportAttempt advances the revision only after it, so only the latest revision can be unreconciled. A reached end applies before a loop decision. Only after reconciliation does leaseNext hand back an omitted open lease or issue the first ready leaf in declaration order. A running workflow occupies its parent's position in that order. canTakeLease: false still reconciles driverless transitions. Every changed run is saved, including a wait.

• A report is checked against the revision of its own lease. reportAttempt and retryStep increment revision; skips, if, end, and the creation of child steps inside next never do. A lease keeps the run revision it was handed out at, and reportAttempt compares a report with that, so the report of a parallel lease does not make it stale. An attempt that is not an open lease keeps the older answers: stale_revision when its revision is not the run's, otherwise wrong_attempt.

• An end or break stops what is still open under its workflow. stopWhere cancels every leased step under the workflow that reached the end, its sub-workflows included, and keeps the attempt on the step and in cancelledAttempts. It cancels running workflow steps there and skips pending and waiting ones. The walk takes a path predicate and failure. End and break construct their own end_reached failure; manual cancel and retry supply theirs. Step descendants use the same isUnder predicate as archiveUnder in loop.ts. Cancel records the ids this walk changed before reconciliation can settle or skip other steps. reportAttempt answers a cancelled attempt lease_cancelled before it checks the driver; cancelledAttempts is never trimmed. Every next answer lists in cancelled the attempts the driver says it runs that are no longer its open leases, so the driver learns of a cancellation from any answer, done included.

• The local driver runs commands in parallel and a prompt alone. runLocalDriver runs up to workflows.max_parallel commands (local config, default 1) and sends them as running. It holds a leased prompt until no command runs or is still exiting, because the prompt's read-only check compares the whole worktree before and after it. On cancelled, it drops a held lease, and it stops a running command with SIGTERM, then SIGKILL after 5 s while any captured descendant or process in a detached group remains, even after its command exited. Attached commands snapshot their descendant tree and start times before stopping, then probe and send SIGKILL only to descendants whose start times still match. If the identity snapshot fails, only the shell is stopped; detached commands use their process group. It does not send a stopping command as running again, so next does not cancel it twice. The signal policy follows max_parallel. At 1, a command runs in teamai's process group, as env exec runs one (src/env-exec.ts): the terminal's Ctrl-C reaches it, and it can read /dev/tty. Above 1, each command runs detached in a process group of its own, with no terminal, and SIGINT, SIGQUIT, SIGTERM or SIGHUP makes the driver stop every group, wait for them, and exit with that signal. Windows uses taskkill /T /F in place of the group kill.

• The end reached first wins. A reported step's settledAt is the revision its report made. Every terminal transition, including a skip or cancellation, carries settledAt. A step the engine settles takes the run's current revision, which is that of the report that caused it, because every earlier revision was reconciled before that report was accepted. An open lease it fails takes the revision it was leased at, and a cancellation the winning end's revision. reportAttempt accepts a report only after reconcileRun has applied every end reached before it, so an end a later report reaches never competes with an earlier one. Every end settleScope has yet to apply was reached at the run's current revision, and it applies the one declared first. The winner therefore does not depend on when next runs between reports.

• A break is a nameless succeeded end. A succeeded step whose break holds competes with the steps that reached an end by the same rule. It ends its workflow, or that workflow's loop, as success. Its value cannot change after its step succeeds, because pull rejects break inside a step loop's range, where a previous-pass read could change.

• A workflow loop is decided when a round has nothing left. Once every step of a looping workflow has settled and no end or break was reached, decideRound in loop.ts evaluates the workflow's loop.while. When it is false, the workflow ends with the round's outcome and outputs. Otherwise every step of the workflow repends for the next round, unless the workflow has run max_iterations rounds: then it fails loop_exhausted (WorkflowResult.failure, and run.failure, shown in done, at the top). A failed step alone does not stop the loop. The round is kept on the step that runs the workflow (Scope.runner), so repend resets it with that step, or on the run at the top. Pull rejects a step loop in a looping workflow.

• A step waits for every step it reads. readsOf collects the steps that the step's expressions (stepExpressions), templates, and with read. For a step that runs an inline workflow, it also collects the outer steps that the inline steps read. skipUnreachable uses that set to decide readiness. If a new field reads steps, add it to stepExpressions or templatedTexts. Otherwise the step can read a value before it exists, or an inline workflow can wait forever. A step outside a loop that needs or reads a step inside it also waits until the loop owner is settled (waitsFor in loop.ts), and is neither skipped nor ready before.

• A read of a later step in the same loop is a previous-pass read, and never waits. previous in loop.ts classifies it: the target is in the reader's loop range and needs the reader, directly or not. For an inline step, the reader is the step that runs its workflow. Pull allows the read, the engine leaves it out of the reads a step waits for, and expressionSteps gives the reader the target's latest archived pass (previousPass), which holds no pass fields on pass 1. Classify a new kind of read through previous too, or pull and the engine disagree on it.

• A step loop is decided once per run of its owner. After checking for an end, settleScope calls decideStepLoop for each succeeded owner without loopFinal. A false while sets loopFinal. Otherwise the range runs again through repend, unless a while owner has run max_iterations times, which fails it loop_exhausted. A loop without while is bounded where the owner's action would start, after the skip rule and if, for a leaf and a workflow step alike (exhaustedAtStart). The owner counts its runs in iteration. Pull rejects ranges that share a step and loops that wait for each other (loopProblems), so the engine does not check them again.

• A pass id is never reused. repend moves a step's current pass into its passes and bumps pass. archiveUnder moves what a repended workflow or for_each step ran into run.archive, and firstPass starts a new instance after every archived pass of its path. The full record of a run is run.steps, with each step's passes, plus run.archive; a run record, scorer, or budget reads both. steps.<id>.passes is a step's passes plus its current pass once it settled (passesOf). The expression projection keeps results for a for_each pass, including in previousPass. Pull checks the same outputs-or-results shape in step reads and workflow output aliases.

• An expression reads its absolute paths, wherever they are. expressionPaths returns every inputs and steps path, including those inside function arguments and predicates, and never a relative path such as .size, which reads the current element of a predicate. readsOf, skipUnreachable and the pull read checks use it, so a new kind of expression must return its paths there.

• Expression types are checked in two places, through typeProblems. parseExpression rejects what an expression shows by itself, such as x < 'a' or any(...) < 1. readDiagnostics checks paths against the types the workflow declares. It checks each path only in the workflow that owns it, so a predicate an inline step runs over an enclosing step is reported once.

• Pull and the engine share one list of templated fields, one list of step expressions, and one list of actions. The templated fields are in templatedTexts and the expressions, such as if and break, in stepExpressions, both in template.ts. HITL declarations normalize in parse-hitl.ts. The actions are in STEP_ACTIONS in types.ts. Add a new field or action there, not to a copy in parse.ts or engine.ts. Pull, the cache's prompt_file check, the lease render and readsOf all split a template with parseTemplate, which decides which {{ … }} is a template, and the render evaluates it with evaluateExpression, as for an if. parseReads in engine.ts and the step parse in parse.ts key a table by each expression, so a new one does not compile until both say how to read it.

• A new step status must fit stopped(), settled() and stopWhere(). The first two are in step-state.ts, and stopWhere() is in engine.ts. stopped() decides which dependents are skipped, which steps can recover, and the run outcome. settled() decides readiness. stopWhere() decides what an end or a break does to each step under it. Each switches over every status, so a status added to StepStateSchema does not compile until all three place it.

• Only the transitions in step-state.ts write step fields. waitForAnswer, answerHitl, resetPending, resetIteration, skip, lease, release, start, succeed, fail, cancel, repend, nextRound, endLoop and exhaust change a step in place and drop the fields its new status does not have. readRun parses every step through StepStateSchema, which gives a run file from an older teamai its defaults: settledAt 0, outcome from status, and no fields its status does not have.

• run.steps is replaced, never changed in place. Its type is readonly, so addItems, insertChildren and archiveUnder add or remove instances by assigning a new array. An instance's id is readonly too, so an instance cannot move to another scope's group. statesIn in scope.ts keeps the instances of every scope per array, in a WeakMap, so settling a scope reads its own instances without scanning every step. A new array starts a new index, except that insertChildren, which adds the instances of a workflow a step starts, carries the old index over.

• A pull check that changes what a file means carries a code. dropsKeptVersion in cache.ts removes the cached copy when a diagnostic has a code. An older definition then cannot keep running with the new meaning. A diagnostic without a code keeps the last valid version, but only after checkReferences checks that version again against what the workflows it runs settle to in the same sync.

• The workflow cache holds only what one sync settled. checkReferences checks each file's version against the versions the other files settle to, and dropUnsettled removes every other cache entry. startRun relies on this: it reads the workflows a run uses from the cache and does not check them against each other again. A new way for a file to keep or lose its version goes through checkReferences.

• The items of a for_each step settle like a workflow. When settling reaches a for_each step, advanceInstance evaluates its list and addItems puts an instance per item right after the step; the step is running until every item settled. settleRunning settles a running for_each step through settleItems, the way it settles a workflow step through settleScope. An item waits for the items its after names and is skipped when one of them stopped (afterStanding). It starts only while fewer than max_parallel items are leased or running (startedItems, counted once per scan). A failed item cancels nothing; the step fails once every item settled (itemsSettled). The rules that decide the items are in fan-out.ts, the settling in engine.ts. Pull rejects for_each with loop, end or break (loopProblems).

• item is bound per instance, in the engine and in pull alike. contextOf gives an item its own value and any other instance its workflow's (Scope.item), which childScope passes into the inline workflow an item runs. A for_each step's own if and for_each therefore read the enclosing item, never their own. Pull checks the same binding: ReadSite.item marks the templates of a for_each step, and ReadScope.item a workflow that runs per item. A template reads item only where it is bound: parseTemplate takes the binding, which pull decides from the step and the engine from whether the context holds an item, so elsewhere {{ item }} stays text in both. A new place that reads item must decide its binding in both.

• Step ids are paths. A step inside a sub-workflow has an id such as ship.check.classify, an item of a for_each step one such as build[2], and a step inside the workflow an item runs one such as build[2].verify. parseStepPath in step-path.ts reads them all; don't split ids by hand. A child step exists in the run only after the engine reaches it. To find a step's definition, use stepDefinitions in scope.ts, which reads the definitions alone, so it also finds the step of an archived pass. locateStep in engine.ts also gives the step's scope, and needs the instances along the path. Don't look a step up by top-level id.

• A HITL request waits before any action or end. settleScope checks loop exhaustion first, renders a HITL request message, and moves an unanswered step to waiting. Waiting is neither stopped nor settled. An answer moves it to pending without a revision bump. Approve no skips, choose succeeds, approve yes continues. An end-only step with HITL reaches its end only after yes. The shared templatedTexts makes HITL reads participate in readiness. Only driver.can.hitl declares HITL. parseNextPayload reads it into NextOptions.hitl; stored DriverIdentity and leases contain only kind, machine and workspace. Busy polls never answer waiting HITL requests through the no-HITL rules. Only a driver declaring HITL receives a waiting request; a busy poll without HITL receives wait. Both new leases and hand-backs enforce HITL for interactive steps. A refused hand-back fails at its lease revision and invalidates the attempt. The process-only autoApprove option is never read by the JSON payload parser. respond --json uses the same identity schema as next, reports unread fields through ignored, and records a driver UI's declared person answer with channel driver and agent false. Answer driver identity stays local; records publish only choice, channel and agent.

• Retry validates every container before resetting any. retryContainers in run-controls.ts climbs from the target through scopes and fan-out items until a running runner or the top. Each dependency cone includes needs, reads and loop waits, or item after dependencies. An independent standing end or break rejects the retry. The apply pass archives replaced passes and descendants, reopens settled runners with start to keep inputs and round, then bumps revision and reconciles. Only the targeted loop owner's iteration resets; the top-level round does not. An answered refused approve or succeeded choose is retryable too, so a changed answer resets its dependent branches. A HITL request in the cone that is waiting, or answered and not yet run, starts a new pass too, so its answer is asked again in the new context and an answer to the old pass is stale. Unanswered skips in every affected container reset so skips caused by cancellation can run again. Retry's prelude reconciles without finishing a provisional outcome. Reopening clears the run's end, failure and endRecordQueued flag and removes its queued end file through records.ts. These three pieces stay consistent: run state, the pending end record, and published stats. The next end upserts the same record path; stats replace the origin end's outcome and duration contribution with the new end's, without adding another run. Replaying the same endedAt contributes nothing.

• The local driver keeps tools and questions interruptible. A single question joins command and prompt exits in the driver's race. An interactive tool owns the terminal alone: no question opens while it runs, and starting one aborts and closes an open question. A poll closes the question when its own step is leased or a different step or pass waits; an unrelated lease leaves it open. It polls next every second while asking, every two seconds while tools run, and aborts a replaced HITL request or cancelled prompt. process-stop.ts shares process stopping between commands and adapters. Headless tools detach and stop with SIGTERM; interactive tools stay attached and stop with their captured descendant tree. Foreground interactive signals reuse env-exec's terminal detection and signal policy: when the terminal reaches the child, teamai ignores SIGINT and SIGQUIT, while SIGTERM and SIGHUP are forwarded and stop the driver. Interactive adapters inherit stdio and accept an optional result file. Repository edits are kept for interactive steps; only headless prompts run the read-only check. A nonzero interactive exit without a result reports its exit status. leaseNext records the identity of the latest driver that called next in RunState.driver, without capabilities. Resume uses this machine's open local lease workspace, then run.driver.workspace for a local driver on this machine, then cwd. Controls pass run.driver to finishRun so end records retain the driver kind.

• One local driver owns a run on this machine. runLocalDriver takes driverLockPath once with acquireLock and releases it in finally. A live holder returns driver_running, including its pid; the existing lock machinery reclaims dead owners. The driver lock spans tool execution and questions, while withRun holds a different run lock only during one engine operation. Neither lock's scope is extended by the other. Every HITL decision without a HITL driver and interactive hitl_required failure logs once when the engine changes the step. HITL replies carry one literal respond command template, which drivers display unchanged.

Tests

Test parsing through parseWorkflow and syncWorkflowDefinitions. Test the engine through startRun, leaseNext, and reportAttempt. Every driver must pass the conformance suite in src/workflow/conformance.ts. If a driver can see a new behavior, add a scenario there.