[feat] Run benchmark suites as workflows #1180

Parent: #1137

Goal

Labels from sessions and workflow runs show how agents did on whatever work came along. They cannot say whether a different tool or model would do better on the same work, because the work differs from run to run. A benchmark answers that: the same tasks, from the same starting commit, judged by the same check, run once per arm and repeated.

This issue adds suites with arms that change the tool or the model. Arms that change the team's skills, rules or instructions are #1181.

Files

In the team repo, next to workflows/:

# benchmarks/auth-fixes/suite.yaml
name: auth-fixes
description: Bugs from the auth module, fixed by the implementer role
repo: https://github.com/acme/api          # the project the tasks start from
defaults: { tool: claude, model: strong }  # model is optional; use an alias when arms change the tool
arms:
  - name: baseline
  - name: codex
    tool: codex
  - name: fast
    model: fast
repetitions: 3                             # 1 to 10, default 3
timeout: 6h                                # required, the run's timeout (#1082)
setup: npm ci                              # optional, runs in each trial workspace before the agent (#1082)
scorers: [diff-quality]                    # optional, scorers run on every trial (#1085)

# benchmarks/auth-fixes/tasks/expired-token.yaml
title: Reject expired refresh tokens
prompt: |
  Refresh tokens past their expiry are still accepted. Fix it.
startingCommit: 3f2c9a7d1e0b4c6a8f5d2e1b0c9a8f7e6d5c4b3a   # full 40-character commit of repo
check:
  run: npx vitest run test/auth/refresh.test.ts
  protect: ['test/**', 'vitest.config.ts', 'package.json', 'package-lock.json']
source: { platform: claude-code, sessionId: 4b1e... }   # optional, the session the task came from

A task has exactly one of check and successCriteria. successCriteria is text a correctness judge reads, for work no command can decide.

Validation, at pull, reported per file like workflows:

• 1 to 6 arms with unique names. Each arm changes at most one of the defaults keys, and at most one arm changes none.
• A model name in defaults.model while arms change the tool is a warning: an alias resolves per tool, a name does not.
• startingCommit is a full commit. check.run is non-empty, and check.protect is a non-empty list of paths relative to the repo. A task without check has successCriteria.
• timeout is set. When it passes, the next call to the run settles the trials no driver took as not_run ([feat] Enforce workflow requirements and execution limits #1082). Cancelling them, once [feat] Add manual run controls and human gates #1081 adds cancel, settles them sooner.
• A suite with no valid task is not cached. The last valid version is kept, as for workflows.
• teamai review --branch reports the same diagnostics as fail findings, as it does for workflows ([feat] Validate workflows, prompts and scorers in review --branch #1248). Today it does not read benchmarks/ from the branch, so this adds it.

Running

teamai workflow start <suite> --benchmark, an option on the existing command, with the usual --driver. It creates an ordinary run, worked through next and report like a workflow. --driver local runs it in process. The skill driver (#1086), the Paseo adapter (#1087) or a custom driver leases its trials, checks and scorer work once it follows the trial rules below. Nothing below is specific to the local driver unless it says so.

The plan is frozen at start. Before any trial is leased, the run stores in its state, the way startRun (engine.ts) already stores the definition and sourceHash: the sourceHash of the suite file and of each task file, each task's starting commit, the ordered trial list, each arm's intended tool and resolved model and effort, the baseline team revision, the role and projects of the config the run starts in, and the rubric.sourceHash of every scorer used. A later pull does not change a running benchmark. The baseline team revision is the one #1176 reads for the checkout the run starts in. When #1176 would leave team out or report team.localChanges, or that commit is not on the team repo's remote, the run fails at start, before any trial: Benchmarks start from the team revision this checkout last pulled, and it has none that another machine can pull. Push any team changes, run teamai pull, and start again. An arm whose model alias gives no model for its tool, under #1177's rules, fails the run at start, before any trial: Arm "codex" uses model alias "strong", which has no model for codex in models/aliases.yaml. Add one and start again.

Trials. One per task × arm × repetition, ordered by repetition, then task in file name order, then arm. In repetition r, counting from 0, the arms are rotated by r (A B C, then B C A, then C A B), so no arm always runs first. The order is not a dependency: a failed or unscored trial never skips another.

Each trial:

1. A new workspace at startingCommit ([feat] Add workflow workspaces, branches, and session continuation #1080). For a benchmark it is a fresh clone of the project with its own git data and no remotes, not a worktree that shares the member's .git. Its agent configuration is the starting commit's, as [feat] Add workflow workspaces, branches, and session continuation #1080 makes it for every step. The driver then sets teamai up in it with teamai init <team repo> --scope workspace --agent <tool>, passing in --role and --project the role and projects the lease carries from the frozen plan. It then runs teamai pull --team-ref <teamRevision> ([feat] Pin a workspace to a team ref with teamai pull --team-ref #1183), with the frozen baseline team revision the lease carries in teamRevision. Every trial thus gets the same team assets on every driver, even when the team pushes during the run. When init or the pull exits non-zero, the trial ends as error. When the trial's steps are done, the driver removes its workspace and the teamai data init created for it. A trial pushes no branch. Its check continues its copy ([feat] Add workflow workspaces, branches, and session continuation #1080 continue). Its scorers are scorer work outside run.steps ([feat] Add workflow scorers and recorded quality labels #1085), leased to the driver that holds the copy, and they do not continue it. The check and the scorers therefore run on the driver that ran the prompt, and the copy stays until their evidence is prepared.
2. setup runs in it ([feat] Enforce workflow requirements and execution limits #1082). A failed setup ends the trial as error.
3. The prompt step runs the task's prompt with the arm's tool and model. The lease names the tool in tool. It carries in model and effort the model name and effort the frozen plan resolved for the arm, not an alias, so every driver runs the same model even if aliases change during the run. The driver passes them to the tool without alias lookup, as it passes a model name in a step model ([feat] Let a workflow choose the model of its prompt steps #1177). The local driver launches the lease's tool instead of its --tools mapping. A driver that runs another tool anyway shows up as a deviation. These generated step fields and their lease handling belong to this issue.
4. For a task with check, the check is an ordinary command step. Its generated command uses git, with hooks turned off, to restore the check.protect paths to their content at startingCommit and to delete files the agent added under them, commits included. It then runs check.run. Every driver that runs command steps applies it the same way. The command comes from the task file. Exit code 0 passes.

Each check is tried first. Before a task's first trial, setup and then the check run once in a clean workspace at startingCommit. If the check passes there, the task is reported invalid: check passes without changes and its trials do not run. If setup fails, or the check cannot start, times out or is killed by a signal, the task is reported check could not run and its trials are not_run: that is not the failure the task expects, and not proof the check works. The author lists in protect everything the check reads and the agent must not supply: tests, test config, scripts, helpers. A check whose inputs cannot be listed belongs in successCriteria.

Correctness judge. A task with successCriteria is judged by a correctness scorer generated from it: labels correct (score 1) and incorrect (score 0), passingScore 1. It runs through #1085 on the trial's prompt step, reading the diff from startingCommit, commits included, and the step's session. Sampling is 100 for every scorer on a trial.

Isolation. A trial must not be able to publish anything: no push to a real remote, no forge login, no git credential helper, no SSH agent, no authenticated MCP server or plugin of the member's. The driver that runs the trial provides the clone without remotes and removes GITHUB_TOKEN, GH_TOKEN, GITLAB_TOKEN, GL_TOKEN, CI_JOB_TOKEN, SSH_AUTH_SOCK, GIT_ASKPASS and SSH_ASKPASS from the tool's environment. The local driver does both. The rest, logins stored in the member's home and their MCP servers, only the environment can guarantee. So a trial requires isolation: sandbox, the level proposed in #1082 for containment beyond a worktree. The local driver declares it only when the member's config says the machine is one (workflows.sandbox: true, or the key #1082 added for it), such as a CI runner or a container without personal logins. If #1082 shipped without a sandbox level, this issue adds the level and that key. This strict rule applies to benchmark trials only and does not change how ordinary workflow steps treat isolation.

Trials that cannot run. A trial is leased only to a driver that declares sandbox and lists the trial's tool in can.trials. Listing can.trials commits the driver to the trial rules above: the lease's tool, model and effort used as given, the lease's team revision pulled with --team-ref, a fresh clone without remotes, the variables above removed from the tool's environment, and no branch pushed. A driver written before this issue lists no can.trials, so it never gets a trial. No driver is handed a trial it would have to release, since next hands a released step straight back to it. The local driver lists the tools its adapter runs. A driver lists can.trials only on a machine where the team does not also reach a trial through user scope, since #1183's pinned pull fails there. The local driver checks this at start and settles the trials as not_run with #1183's user-scope message. A trial no driver takes stays pending. The local driver is the run's only driver, so it settles the trials it cannot take as not_run at once and the run completes. Without a sandbox it prints This machine is not declared a sandbox, so benchmark trials did not run. Run benchmarks in CI or a container, and declare it in the config. For a tool it cannot run it prints Arm "<arm>" uses <tool>, which the local driver cannot run, so its trials did not run. With other drivers, the trials still pending settle as not_run when the suite's timeout passes, when they are cancelled once #1081 adds cancel, or when the run hits its limits. Finishing the run waits until every trial's scorer entries have settled, labelled or not (#1085). A trial without check whose correctness judge gave no label is unscored. Ordinary workflow runs settle as before.

The member's own checkout is never changed.

Results

Each trial ends as one of:

• The record. The end record gains benchmark: { suite: { sourceHash }, repetitions, baselineTeamRevision, scorers: [{ name, rubric: { sourceHash } }], tasks: [{ id, startingCommit, sourceHash }], arms: [{ name, tool, model?, effort? }], trials: [{ task, arm, repetition, order, step, result, durationMs? }] }. baselineTeamRevision is the full commit. It and the scorer hashes come from the frozen plan, even when no trial ran or no scorer gave a label. No prompts, commands, URLs with credentials or local paths. Each trial's step entry carries its agent ([feat] Record the agent each workflow step ran as #1176). A reader rebuilds the comparison from the record alone, not from the current suite files.
• durationMs is measured around the prompt step, after setup and before any judge.
• A deviation is a trial whose recorded agent contradicts its arm on a value that can be compared: the tool name, the team revision, or a concrete model id requested and a different one reported. A missing value is not verified, not a deviation. A tool's own model selector such as opus is not compared with a provider's model id.
• The table. When the run ends, the local driver prints a table per arm, and every done envelope that next returns for the run carries it, so the skill driver or any other driver can show it. A run ends only inside next, never in report (docs/dev/workflow-engine.md). Per arm: intended tool and model, planned trials, passed, failed, error, unscored, not run, deviations, not verified, pass rate, median tokens and median duration. The pass rate is passed / planned trials. With no planned trials there is no rate. Medians leave out unknown values and show how many they used. Token medians name the components they add up.
• Under the table: These results hold for these tasks, this run and this machine.

Acceptance criteria

1. Pull reports a suite with 7 arms, a suite without timeout, an arm that changes both tool and model, a task with both check and successCriteria, a task with an empty protect, and a startingCommit that is not a full commit, each with its file.
2. In a declared sandbox, a fixture suite with 2 tasks, 2 arms and 2 repetitions runs 8 trials. The record lists all 8 with task, arm, repetition and order, and the arm order of repetition 2 is the reverse of repetition 1. When the first trial fails, the other 7 still run.
3. Every trial's agent.project.commit is the task's startingCommit, its workspace has no remotes and its own git data, and git status of the member's checkout is the same before and after the run. Every trial's agent.team.revision is the frozen baseline team revision, also when a team commit is pushed during the run. A checkout with edited team assets, or one whose team revision is not on the remote, fails the run at start with the message above.
4. Inside a trial, env shows none of the variables listed above.
5. A trial that edits vitest.config.ts to skip the test, or adds a helper under test/, is checked against the restored files and fails.
6. A task whose check passes on startingCommit is reported invalid and none of its trials run. A baseline whose setup fails, or whose check cannot start, times out or is killed, is reported check could not run, and its trials are not_run. A fixture shows the failing baseline passes with a known correct patch. A check that times out after a successful trial prompt gives error, not failed.
7. In a model arm, each trial's agent.model.requested is the model the frozen plan resolved for the arm, on every driver, also when models/aliases.yaml changes during the run. In a tool arm, each trial's agent.tool.name is the arm's tool. An arm whose alias has no model for its tool fails the run at start with the message above, and no trial runs.
8. A trial whose reported model contradicts a concrete requested id counts as a deviation and stays in the denominator. A trial with no reported model is not verified.
9. A successCriteria task has a correctness label on each trial. A trial whose judge fails is unscored, not passed.
10. On a machine not declared a sandbox, every trial is not_run, the sandbox message above is printed, and the run completes. In a declared sandbox, an arm whose tool the local driver cannot run has its trials not_run with the Arm "<arm>" uses <tool> message, and the other arms' trials run. One passed trial out of ten planned shows a pass rate of 10%.
11. Editing the suite and pulling during a run does not change that run's plan. After removing the suite from the cache, the printed numbers can still be computed from the published record with the jq command the docs give. The record keeps the baseline team revision and every scorer's rubric hash even when observations are missing, a scorer failed, or every trial is not_run.
12. A fake external driver that calls teamai workflow next and report, declares sandbox and lists only claude in can.trials runs a one-task suite with repetitions: 1, a Claude arm and a Codex arm. It runs the Claude trial and its check, is never leased the Codex trial, and the suite's timeout settles the Codex trial as not_run. The done envelope that next then returns carries the table. A driver that does not declare sandbox, or lists no can.trials, gets no trial lease.
13. A branch holding the 7-arm suite from criterion 1 fails teamai review --branch with exit 1 and a fail finding for that file.

Test plan

• Parser tests for criterion 1.
• Engine tests for the frozen plan, the trial list, the rotation, independent trials, the invalid-task paths, not_run settlement and the results table, with a fake tool.
• E2E with a fixture project repo and a fake tool for criteria 2 to 6 and 10 to 12, with the sandbox declared in the test config except for the first case of criterion 10. In criterion 12 the fake driver declares it in its next payload.
• One real-CLI run for the PR body: a one-task suite with two arms on Claude in a container or CI job, plus Codex if available.

Watch out for

• [feat] Add workflow workspaces, branches, and session continuation #1080 runs each tool "with the agent configuration from the run's base". A trial adds on top of it the team assets its pinned pull delivers ([feat] Pin a workspace to a team ref with teamai pull --team-ref #1183), and nothing resets them before the prompt step. Make that the only exception.
• --benchmark is new on workflow start. Regenerate skill-data/core/references/commands.md with npx vitest run commands-reference -u.
• New guide pages: docs/guide/benchmarks.md and docs/guide/zh-CN/benchmarks.md. They must say what protect has to cover, why trials need a sandbox, and to keep the suite's timeout below the host's job limit, since a timeout takes effect only on the next call to the run.
• can.trials is a new driver capability, checked before leasing like the ones [feat] Enforce workflow requirements and execution limits #1082 checks. The skill driver ([feat] Add the session skill driver and invocable workflows #1086) and the Paseo adapter ([feat] Add a Paseo driver adapter #1087) send it only once they follow the trial rules.
• The guide pages carry a CI job to copy, for GitHub Actions and for GitLab CI: install teamai and Claude Code, log the tool in from a CI secret, declare the sandbox in the job's config, and run teamai workflow start <suite> --benchmark --driver local. The PR's real-CLI run uses that job.
• A suite's defaults sets its baseline arm. It is not a workflow's defaults ([feat] Let a workflow choose the model of its prompt steps #1177), and the guide pages do not present them as one key.

Out of scope

• Arms that change team assets ([feat] Compare team asset sets in benchmarks #1181).
• Turning a past session into a task. source is recorded, but the task file is written by hand from the session's gitAtStart.
• A results page ([feat] Show scorer results in the dashboard #1182 covers labels), uncertainty intervals and a recommendation.

Dependencies

• Blocked by: [feat] Add workflow workspaces, branches, and session continuation #1080 (a workspace per trial), [feat] Enforce workflow requirements and execution limits #1082 (isolation levels, setup, timeouts), [feat] Add workflow scorers and recorded quality labels #1085 (scorers on trial steps), [feat] Record the agent each workflow step ran as #1176 (what each trial ran as), [feat] Let a workflow choose the model of its prompt steps #1177 (the model of an arm), [feat] Pin a workspace to a team ref with teamai pull --team-ref #1183 (the team revision of a trial), [feat] Close the #1080 gaps that #1175 leaves open #1246 (a trial's own base and optional ref, F2 and F6)
• Unblocks: [feat] Compare team asset sets in benchmarks #1181, [feat] Score what a workflow run delivered #1247