Skip to content

Commit d3b6ff4

Browse files
feat(scenarios): run-scenarios --driver codex and --driver external:COMMAND, with the driver on each gallery app page (#31265)
<!-- claude-projects-attribution: {"github_login":"os-project-manager"} --> _Requested by **Leehom** · [project thread](https://claude.ai/code/project/chan_01TenzsRopZj4jMbvjX8uz1X?thread=cmsg_01TenzsRopZj4jMbvjX8uz1XSm2B8fDTCJGABZvwCSJJvu)_ Fixes #31159 Clause-②: yes Metadata-protocol: none ## 维护者速读 - 改了什么:场景执行器新增两种驱动。`--driver codex` 用本机 Codex CLI(`codex exec --json`)跑卡片,只认 ChatGPT 套餐登录;`--driver external:命令` 让任何能按约定读写 JSON 的命令接进来(每次调用一个进程,stdin 进 JSON、stdout 出 JSON,约定写在包 README)。画廊每个应用页多一行 `- Driver:`,写明驱动名和版本;回放记录写明原录制的驱动。 - 为什么改:维护者原话「前面的建议都同意,立卡」(epic cloud#3600):同一张卡要能横向比较不同 agent 工具。WorkBuddy 只做核查:没找到无头模式加结构化输出的入口,所以不接,也不做 UI 桥。 - 风险与回滚:只动 `packages/qa/scenarios`,默认驱动 `claude` 的行为不变(只是把进程启动函数挪到 `drivers/process.ts` 共用)。最大的风险是人物 API key 泄露:key 只写进 0600 文件(外部命令拿到的是文件路径;Codex 是 `$CODEX_HOME` 下的临时 profile),用完即删,从不进 argv、环境变量或 stdin;端到端测试把整张 crm 卡经外部命令跑完,再在运行记录、录制、评分卡和画廊导出里逐文件搜 key。回滚即 revert 本 PR。 - 先看:`packages/qa/scenarios/src/runner/drivers/external.ts`(外部约定与 key 文件)、`packages/qa/scenarios/src/runner/drivers/codex.ts`(登录守卫、profile 文件、JSONL 解析)、`packages/qa/scenarios/test/runner/external-driver.test.ts`(端到端 key 搜索)。 - 两分钟验证:`pnpm --filter '@objectstack/scenarios^...' build && pnpm --filter @objectstack/scenarios exec vitest run test/runner/external-driver.test.ts test/runner/codex-driver.test.ts`,预期 2 个文件 20/20 通过。 - 需要你做的:无,席位落地。 ## What changes - **`--driver external:COMMAND`** (`src/runner/drivers/external.ts`, `ExternalDriver`): the `AgentDriver` seam as a process contract, `objectstack-scenario-driver/1`. - One process per call. The call is one JSON object on stdin, the answer one JSON object on stdout, logs on stderr (the record keeps the last 2,000 characters). Three calls: `info` once at start (the command names itself and its version: the record's driver is `external:NAME`, its version the one it gives; a command that cannot answer is refused before any card runs), `author` (`step`, `prompt`, `cwd`, `tier`, `limits`; runs in `cwd`) and `ask` (`step`, `prompt`, `mcp: { url, configFile }`, `limits`; runs in the config file's directory). - The answer is strict (`ExternalTurnSchema`: `replyText`, `toolCalls[{name, input, output, isError}]`, optional `model` and `numTurns`; an unknown key is refused). Exit 0 with that shape is the turn. A non-zero exit, a signal or the runner's timeout is a failed turn `no-result(exit N)`; stdout that is not one JSON object of that shape is `invalid-output(…)` with `isError`. A failed authoring turn leaves its Build half not measured (`authorNotMeasured`), a failed persona turn is graded on its empty reply. Never an empty pass. - ⛔ The key: `ask` writes `{"mcpServers":{"app":{"type":"http","url":…,"headers":{"Authorization":"Bearer KEY"}}}}` (the Claude driver's `--mcp-config` shape) to a 0600 file in a fresh temp directory, passes its PATH in `mcp.configFile`, and deletes the directory when the call returns. The key is never on argv, in the environment or on stdin. - `limits` carries `timeoutMs` (`--author-timeout` for `author`, 5 min for `ask`; the runner kills the call then) and, when given, `maxTurns` and `model`. The command may ignore them; the record states them (`DriverInfo`, `meta.process.maxTurns` / `requestedModel`). - The command is split on whitespace (no shell); a word starting `./` or `../`, and a first word holding `/`, resolves against the directory the runner starts in. The runner removes `ANTHROPIC_API_KEY`, `ANTHROPIC_AUTH_TOKEN`, `OPENAI_API_KEY` and `CODEX_API_KEY` from the command's environment; it cannot see the command's login, so `authMethod` reads `not verified (an external command: …)`. - **`--driver codex`** (`src/runner/drivers/codex.ts`, `CodexDriver`), built from the installed CLI (measurements below): - Login: refuses to start while `OPENAI_API_KEY` or `CODEX_API_KEY` is set, or when `codex login status` says anything but `Logged in using ChatGPT` (`assertCodexSubscriptionLogin`, pure, the counterpart of `assertSubscriptionLogin`); every turn also passes `-c forced_login_method="chatgpt"`, and both variables are removed from every child. - A turn: `codex exec --json --ephemeral --skip-git-repo-check -c forced_login_method="chatgpt" -c approval_policy="never" -p PROFILE -s … -C DIR [-m MODEL] -`, the prompt on stdin. Authoring: `-s workspace-write -c sandbox_workspace_write.network_access=true` in the project (the Claude driver's Bash has the network too). Persona turns: `-s read-only` in an empty temp directory. - MCP and the key: a profile layer `$CODEX_HOME/scenario-runner-HEX16.config.toml`, mode 0600, deleted in `finally`, holding `[mcp_servers."app"]` with `url` and `http_headers = { Authorization = "Bearer …" }`, plus `enabled = false` for every server of the user's own config (`codex mcp list --json`, read once at start), the counterpart of `--strict-mcp-config`. argv carries the profile's NAME only. A user server already named `app` is refused at start. - `parseCodexJsonl` reads the events into the same `ParsedTurn`: `agent_message` texts joined into `replyText`; `mcp_tool_call` → the app's own tool name (`rawName` `mcp__SERVER__TOOL`), output the result's text (or `structured_content`, or the error message), `isError` on `failed`; `command_execution`, `file_change`, `web_search` kept as calls by their item type; a call started and never completed keeps `output: undefined`; `turn.completed` → `success`; `turn.failed` → `turn.failed: MESSAGE` with `isError`; a top-level `error` event is a reconnect notice, not the end. - Flags: `--model`, `--author-model`, `--ask-model` → `-m` per turn kind; `--author-timeout` is the runner's own clock. `codex exec` has no turn cap, so `--author-max-turns` and `--ask-max-turns` are refused at `makeDriver` (as the recorded driver refuses every live flag). - **The driver on record** (Ask 4): `DriverInfo` gains `replayOf?: { driver, version? }`, which `RecordedDriver.info()` fills from the recording (a recording of a replay keeps the original). The gallery exporter writes `- Driver: NAME VERSION` on every app page (`driverText`, exported from `./export`); a replay reads `RECORDED-DRIVER VERSION (re-graded from a recording of that run)`, never the recording's path; an old replay record with no `replayOf` says it does not name its driver. The record schema reads `env.driver.version` / `replayOf` (loose, both optional). - **Shared process runner**: `runCli` and `noResult` moved from `driver.ts` to `src/runner/drivers/process.ts` (`runTurnProcess`, now taking the child environment), unchanged otherwise; the tool-result text reader moved there too (`contentText`) and `transcript.ts` uses it. `process.ts` is not re-exported from `./runner`. - `makeDriver`: `codex` and `external:COMMAND` (an empty command refused); the unknown-driver message lists `claude, codex, external:COMMAND, recorded:FILE`. `ApiBillingRefusedError` takes an optional `{ tool, login }` (default Claude Code's, message unchanged). - README: the Codex section and the external contract (stdin shapes for `info`/`author`/`ask`, the stdout shape, exit codes, timeout, limits, the key file), and the WorkBuddy finding. Changeset `patch` with both declaration lines. ### Public surface (Clause-②: yes, additive) Nothing removed or renamed. Added to `@objectstack/scenarios/runner`: `CodexDriver`, `CodexDriverOptions`, `CODEX_API_BILLING_ENV`, `CODEX_TURN_CONFIG`, `CODEX_APP_SERVER`, `codexSubscriptionOnlyEnv`, `assertCodexSubscriptionLogin`, `codexProfileToml`, `parseCodexJsonl`, `ExternalDriver`, `ExternalDriverOptions`, `EXTERNAL_DRIVER_CONTRACT`, `EXTERNAL_STRIPPED_ENV`, `ExternalLimits`, `ExternalInfoInput`, `ExternalAuthorInput`, `ExternalAskInput`, `ExternalInfoSchema`, `ExternalToolCallSchema`, `ExternalTurnSchema`, `ExternalTurn`, `parseExternalCommand`; the optional member `DriverInfo.replayOf`; an optional second parameter on `ApiBillingRefusedError`. Added to `./export`: `driverText`. CLI: `--driver codex`, `--driver external:COMMAND`. ### Ask 2: what was measured on the Codex CLI Installed `@openai/codex@0.167.0` (npm, `codex-cli 0.167.0`, linux-x64) into a scratch prefix in this container. No model turn ran: no login was used, and the container's proxy refuses OpenAI's hosts (`HTTP CONNECT failed with status 403`), so every `exec` below stopped at the model call. <details><summary>Readings, each with its source</summary> - `codex --help`, `codex exec --help` (installed binary): `exec` "Run Codex non-interactively"; `[PROMPT]` "If not provided as an argument (or if `-` is used), instructions are read from stdin"; `--json` "Print events to stdout as JSONL"; `-m, --model`; `-s, --sandbox read-only|workspace-write|danger-full-access`; `-C, --cd`; `--skip-git-repo-check`; `--ephemeral` "Run without persisting session files to disk"; `-p, --profile` "Layer $CODEX_HOME/NAME.config.toml on top of the base user config"; `--ignore-user-config`; `-c key=value`; `--strict-config`. No turn-cap flag; `max_turns` and its spellings appear nowhere in the binary's strings. - JSONL event shapes: `codex-rs/exec/src/exec_events.rs` at tag `rust-v0.167.0` (raw.githubusercontent.com): `thread.started {thread_id}`, `turn.started`, `turn.completed {usage}`, `turn.failed {error{message}}`, `item.started|updated|completed {item{id,type,…}}`, `error {message}`; item types `agent_message {text}`, `mcp_tool_call {server, tool, arguments, result{content, structured_content}, error{message}, status}`, `command_execution {command, aggregated_output, exit_code, status}`, `file_change`, `web_search`, `todo_list`, `reasoning`, `error`. No event carries the served model. Seen live: `thread.started`, `turn.started`, then `error` events `Reconnecting... 2/5 (…)` while the turn continued, so a top-level `error` is read as a notice. The docs the repository ships (`docs/exec.md`, `docs/config.md`, `docs/authentication.md` at that tag) only point to developers.openai.com, which this container cannot reach (connect refused): the config reference was not read; every config key below was measured instead. - MCP from a profile: `$CODEX_HOME/scen.config.toml` with `[mcp_servers.app] url = "http://127.0.0.1:39123/mcp"`, `http_headers = { Authorization = "Bearer FAKEKEY123" }`: `codex -p scen mcp list --json` shows it `streamable_http` with that header; `codex exec --json -p scen …` POSTed `initialize` to a local listener with `authorization: Bearer FAKEKEY123` (`clientInfo.name` `codex-mcp-client`). A project `.codex/config.toml` in the working directory was not read by `mcp list` (untrusted directory), so the driver does not use one. - The user's servers: with a base `config.toml` server `other` (port 39124) and the profile's `app` (39123), plain `exec -p scen` contacted both; `--ignore-user-config -p scen` contacted neither (it drops the profile too); `[mcp_servers.other] enabled = false` in the profile → `mcp list` shows it `enabled: false`, and `exec` contacted `app` only. The key appeared in no file under `$CODEX_HOME` but the profile. - Login: `codex login --with-api-key` (a fake key) → `codex login status` prints on stderr `API key configured - sk-fake-***-0000`, exit 0; an empty `CODEX_HOME` → `Not logged in`, exit 1, also with `CODEX_API_KEY` or `OPENAI_API_KEY` set; the binary's strings hold `Logged in using ChatGPT` (and `… using access token`, `… personal access token`, `… workload identity`, `… Amazon Bedrock …`). With `-c forced_login_method="chatgpt"`, the API-key login reads `Not logged in`, exit 1. - Config keys under `exec --strict-config -c …`: `forced_login_method="chatgpt"`, `approval_policy="never"` and `sandbox_workspace_write.network_access=true` are accepted; `forced_login_method="bogus"` → "expected `chatgpt` or `api`", `approval_policy="bogus"` → "expected one of `untrusted`, `on-failure`, `on-request`, `granular`, `never`", `sandbox_workspace_write.bogus_key` and `bogus_top` → "unknown configuration field". - Not measured: a real `codex exec` turn end to end (no ChatGPT login here, and live model runs are paused by the maintainer, cloud#3600), so a `turn.completed` stream was not seen live; its shape is the source's. The stand-in in `codex-driver.test.ts` replays those shapes. </details> ### Ask 3: WorkBuddy Checked on 2026-11-07; no wiring, no UI bridge. - npm search `workbuddy`: the one package calling itself a WorkBuddy command line, `@workbuddy/cli-vnext@1.0.153` (maintainer `workbuddy_developer`, `developer@workbuddy.com`), read from its tarball without running it: a REST client for a WorkBuddy tenant API (`commander`; commands `auth`, `profiles`, `use`, `whoami`; options `--base-url`, `--client-id`, `--output json|table|raw`), with no agent, no prompt mode and no MCP client. Its only URL is `https://yourcompany.workbuddy.com`; it is not Tencent's WorkBuddy. - Tencent's WorkBuddy (CN and the international WorkBuddy AI) is a desktop app: the community `dsh-workbuddy-provider` README (npm) describes reusing "WorkBuddy 桌面 App" credentials on macOS, and the app's model catalogue is fetched with "内置 CLI 版本号" (a built-in CLI version), but no published, documented headless mode with structured output and an MCP client was found. - So the PR says no, per the card. If one appears, it is a wrapper on `--driver external:` (README), not runner code. ## Tests - **`test/runner/external-driver.test.ts`** (new, 8 cases), against `test/runner/fixtures/fake-external-driver.mjs`, a stand-in command that speaks the contract and reports what it saw of the key (a SHA-256, never the key): - `info` names the driver and version; a command exiting 4, printing prose, or missing is refused at construction with the reason. - `author`: the exact stdin object, the call running in the project, the answer read into the turn (MCP prefix dropped, `isError` kept, `model`, `numTurns`, `success`), the project files snapshotted. - `ask`: the config file is 0600, holds the URL and the key, the call runs in its directory, the key is on neither argv, environment nor stdin, and the file and its directory are gone afterwards. - Failed calls: exit 3 → `no-result(exit 3)` and not measured; prose → `invalid-output(stdout is not one JSON object)`; another shape → `invalid-output(…)` (is_error); a hang → killed at the timeout, `author timed out after 0.03 min`. - Command parsing, `external:` with no command refused, the unknown-driver list; every flag in `limits` and on the record. - **End to end, the key pin**: the real `crm_account_with_contacts` card through `runCard` (both modes), each scripted turn sent through the stand-in command (the exporter fixture's in-memory app; `runCrm` gains `keyPrefix` and `wrap`), every persona and the admin issued a key `sk-scenario-e2e-key-9f27c1d4` + the persona id. Every ask read one of those keys from a 0600 file and from nowhere else, and each file is gone. Then the run record (where the CLI writes it), the recording, the scorecard (`renderScorecard`) and the full gallery export (`exportGallery`, which exported the card) are written, and every written file is searched for the key prefix: none holds it. The record names `external:fake-agent 1.2.3`, the app page reads `- Driver: external:fake-agent 1.2.3`; a replay of that recording names `replayOf` and its page reads `(re-graded from a recording of that run)` without the recording's path. - **`test/runner/codex-driver.test.ts`** (new, 12 cases), against a stand-in `codex` script (it answers `--version`, `login status` on stderr, `mcp list --json`, and for `exec` reads the profile it is pointed at): - `parseCodexJsonl` on the 0.167.0 shapes (pairing, failed MCP call, command execution, a call cut short, torn last line, notice not an end, `turn.failed`). - The cost guard: `OPENAI_API_KEY` / `CODEX_API_KEY` set, an API-key login, no login, an empty status, a non-plan login (its masked credential not echoed) refused; the billing variables stripped. - `codexProfileToml` exact text (TOML-escaped key and names). - The driver: version and login on record; refused at construction for an API-key login, no login and `OPENAI_API_KEY`; an ask turn's argv (`exec --json --ephemeral --skip-git-repo-check`, the two `-c` overrides, `-s read-only`, `-`), a 190 KB prompt on stdin, the profile 0600 with the key, the key on neither argv nor environment, the profile file gone and `$CODEX_HOME` empty afterwards, no billing variable in the child; an author turn's `workspace-write`, `-C`, the network override and the user's server turned off; a user server named `app` refused; timeout, exit-without-result and `turn.failed` each not measured; `--model` / `--author-model` / `--ask-model` → `-m`; both turn-cap flags refused. <details><summary>Ablations (each restored: blob == HEAD, git diff HEAD empty)</summary> Both through `scripts/ablation-replace.mjs` (anchor x1 → x0, replacement x0 → x1) on commit `fb1d76f7`. The suites import `src/` directly, so no dist leg. - **Ablation 1, the key written into the record** (`external.ts` `ask`: `return await this.call(input, dir);` → the turn's `meta.model` set to `task.mcp.apiKey`): the end-to-end case → `1 failed | 7 skipped`, `expected [ …(7) ] to deeply equal []`, naming the run record, `recording.json`, `scorecard.md`, and the gallery's `README.md`, `run-record.json`, `scorecard.json`, `scorecard.md`. - **Ablation 2, an API-key Codex login accepted** (`codex.ts`: the ChatGPT test widened to `/Logged in using ChatGPT|API key configured/`): `codex-driver.test.ts -t "cost guard|refuses an API-key login"` → `2 failed | 2 passed`, both "expected function to throw an error, but it didn't". - Restore leg: both files → `Test Files 2 passed (2)`, `Tests 20 passed (20)`. </details> ## Acceptance notes - **The external driver's cost claim.** `DriverInfo.apiBilled` is typed `false`, and the runner cannot see an external command's login, so the record says `authMethod: not verified (…)` and the README states the cost ruling applies to the command; the four billing variables are stripped from its environment. Whether an external command must also be refused when one of them is set in the caller's environment (as the two built-in drivers are) was not in the rulings; this PR strips and does not refuse. - **No served model for Codex.** The JSONL names none, so a Codex turn's `model` is absent and the scorecard reads `model not reported (asked for …)` rather than guessing; `modelMismatch` is never set for it. - **Codex writes into `$CODEX_HOME`.** The profile layer is the only key-in-a-file route the CLI was measured to read (`-p`); the file is 0600, uniquely named, and removed in `finally`. A run killed by SIGKILL could leave one behind; it holds a persona key of a local dev app that is torn down with the run. - **The external stand-in's e2e uses the fixture's in-memory app.** The command answers the turn the scripted model gave; the double applies the turn's writes to the in-memory app in-process. Every turn's bytes still cross the process boundary, and the key reaches the command only through the file. - `codex mcp list --json` prints the user's other servers' headers to the runner's memory once at start; only `name` is read and nothing is logged. - Out of scope, untouched: any live model run (paused, cloud#3600), a WorkBuddy or other UI bridge, `orchestrate.ts`, the gallery `index.json` schema (the driver is on the app page, as Ask 4 says; `run-record.json` already carries `env.driver`). ## Verification All on head `fb1d76f7` (base `origin/main` `169ff7e5`, which carries #31221 as `fba4c00a` and #31204 as `0d0c377b`), worktree `objectstack-issue-31159`. - `pnpm --filter '@objectstack/scenarios^...' build` (under the verify lock) → `VERDICT command-exit 0`. - `pnpm --filter @objectstack/scenarios build && pnpm --filter @objectstack/scenarios test` → `Test Files 30 passed (30)`, `Tests 573 passed (573)`, `VERDICT command-exit 0`. - `pnpm --filter @objectstack/scenarios typecheck` (`tsc --noEmit`, tests included) → exit 0. - `node --stack-size=4000 node_modules/eslint/bin/eslint.js packages/qa/scenarios --no-inline-config` → exit 0 (the package has no `lint` script). - The built entries load: `require('dist/runner.cjs')` (141 keys; `ExternalDriver`, `CodexDriver` functions), `dist/export.cjs` (`driverText`), `dist/index.cjs`, `dist/cli.cjs`, and `import('dist/runner.js')` (`EXTERNAL_DRIVER_CONTRACT` = `objectstack-scenario-driver/1`). - `GITHUB_EVENT_NAME=pull_request GITHUB_EVENT_PATH=EVENT_WITH_THIS_BODY node scripts/check-changeset-no-major.mjs --base origin/main` → exit 0 (`Clause-②: yes`, `Metadata-protocol: none`, `patch`). <details><summary>Derived gates: 60 families, 58 run green, 2 NOT MEASURED, then 1 re-measured</summary> `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` (tree at `fb1d76f7`) → 60 commands, each run with its exit captured before any pipe; `--ran` → "60 derived famil(ies) accounted for — 58 run, 2 NOT-MEASURED". - `node scripts/check-plugin-teardown-shape.mjs --self-test` → exit 3 (shallow clone, control commit missing). After `git fetch --depth=1 origin 621a487…`, rerun → exit 0, "48 cases pass". - `pnpm check:dual-build-cjs-loads` → exit 3, PREREQUISITE NOT MET (needs every package's `dist/`; this worktree built the scenarios closure only). NOT MEASURED; the package's own CJS and ESM entries were loaded directly instead (above). CI runs it. - Beyond the derivation: `pnpm --filter @objectstack/spec run check:error-code-provenance` → exit 0 (this diff adds no error-code literal). - Every other family exit 0, among them `check:cross-package-test-inputs`, `check:test-source-alias`, `check:dts-closure`, `check:published-files`, `check:undeclared-dep-imports`, `check:type-check-debt`, `check:type-check-coverage`, `check:nul-bytes`, `check-changeset-no-major`, `check-empty-changeset`, `check-adr-0087-registration`. - The full record: `pm-orders/devx/31159-gates-ran.txt` (project files). </details> - `test:repo`, the dogfood battery and the full CI farm are CI's (no `!` in the title, no `(narrowing)` arm). --- _Generated by [Claude Code](https://claude.ai/code/session_01FEkQ5mx64pxVGCbKfuxEwi)_ Co-authored-by: Claude <noreply@anthropic.com>
1 parent ea62933 commit d3b6ff4

15 files changed

Lines changed: 1496 additions & 112 deletions

File tree

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
---
2+
'@objectstack/scenarios': patch
3+
---
4+
5+
feat(scenarios): the scenario runner drives the Codex CLI (`--driver codex`) and any command that speaks a stdin/stdout driver contract (`--driver external:COMMAND`), and each gallery app page names the driver that ran it
6+
7+
Clause-②: yes
8+
Metadata-protocol: none
9+
10+
The same scenario card can now measure another agent tool. Additive: nothing is removed or renamed.
11+
12+
- **`--driver codex`** (`CodexDriver`): `codex exec --json` on your own ChatGPT plan login. It refuses to start while `OPENAI_API_KEY` or `CODEX_API_KEY` is set or `codex login status` reports anything but a ChatGPT login, and passes `-c forced_login_method="chatgpt"` on every turn. The persona's API key reaches the CLI only in a 0600 profile file under `$CODEX_HOME`, deleted when the turn returns. `--model`, `--author-model`, `--ask-model` and `--author-timeout` apply; `codex exec` has no turn cap, so `--author-max-turns` and `--ask-max-turns` are refused with it.
13+
- **`--driver external:COMMAND`** (`ExternalDriver`, contract `objectstack-scenario-driver/1`): one process per call, the call as JSON on stdin (`info`, `author`, `ask`), the turn as JSON on stdout. A non-zero exit or an answer of another shape is a failed turn with a named reason (`no-result(exit N)`, `invalid-output(…)`). The persona's key is only in a 0600 file whose path the `ask` call names, never on argv, in the environment or on stdin. The package README states the contract.
14+
- **The driver on record**: `DriverInfo` gains `replayOf`, so a replay (`--driver recorded:F`) names the driver and version of the recording it replays. Each exported gallery app page carries a `- Driver:` line with the driver's name and version (`driverText`).
15+
- `ApiBillingRefusedError` takes an optional second argument naming the agent tool and its login command; its default message is unchanged.

‎packages/qa/scenarios/README.md‎

Lines changed: 71 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -138,6 +138,8 @@ pnpm --filter @objectstack/scenarios run-scenarios # every
138138
pnpm --filter @objectstack/scenarios run-scenarios --only service_desk --modes scratch
139139
pnpm --filter @objectstack/scenarios run-scenarios --record /tmp/rec.json # also save every turn
140140
pnpm --filter @objectstack/scenarios run-scenarios --driver recorded:/tmp/rec.json # replay, no model
141+
pnpm --filter @objectstack/scenarios run-scenarios --driver codex # the Codex CLI on your ChatGPT plan
142+
pnpm --filter @objectstack/scenarios run-scenarios --driver external:./my-agent.mjs # any command on the driver contract
141143
pnpm --filter @objectstack/scenarios run-scenarios --author-timeout 90 # minutes per authoring turn (default 60)
142144
pnpm --filter @objectstack/scenarios run-scenarios --author-max-turns 400 # turns per authoring turn, every card (default by tier)
143145
pnpm --filter @objectstack/scenarios run-scenarios --ask-max-turns 20 # turns per question or operation turn (default 12)
@@ -196,7 +198,9 @@ npx @objectstack/scenarios help # the comma
196198
is logged in with an API key (the cost ruling on cloud#3600), and the record states the driver, the login, the
197199
authoring timeout and the turn caps. Every turn records its process: elapsed time, the turn cap it ran under
198200
(`--max-turns`), and for a turn that did not finish its exit code or signal, whether the runner's timeout killed it,
199-
and the end of its stderr. `RecordedDriver` replays a recording, and refuses the live-driver limits and models below.
201+
and the end of its stderr. `RecordedDriver` replays a recording, and refuses the live-driver limits and models below;
202+
its record keeps the driver of the recording (`replayOf`). For Codex and any other agent tool, see
203+
[Other agent tools](#other-agent-tools--codex-and-the-external-driver-contract).
200204
- **Model** (`driver.ts`): `--model M` passes `claude --model M` on every turn, an alias (`sonnet`) or a name
201205
(`claude-sonnet-4-5`); `--author-model` sets the authoring turns (build and iterate) and `--ask-model` the question
202206
and operation turns apart, each in place of `--model`. With none, a turn takes the login's default model. Every turn
@@ -255,3 +259,69 @@ npx @objectstack/scenarios help # the comma
255259
states its change's rows the same way: the new objects (with the card's field minimum), lookups, option fields,
256260
dashboards and nav targets. `test/runner/prompt-trace.test.ts` reads the checks off the scorer itself and holds every
257261
card to it.
262+
263+
## Other agent tools — Codex and the external driver contract
264+
265+
The same card can measure another agent tool. Every record (`env.driver`) and every gallery app page (`- Driver:`)
266+
names the driver and its version; a replay (`--driver recorded:F`) names the driver of the recording it replays
267+
(`replayOf`), never the recording's path.
268+
269+
### `--driver codex`
270+
271+
`drivers/codex.ts` runs the OpenAI Codex CLI in non-interactive mode, on your own ChatGPT plan. What it reads of the
272+
CLI was measured on `codex-cli 0.167.0`:
273+
274+
- **Login**: the ChatGPT plan only. It refuses to start while `OPENAI_API_KEY` or `CODEX_API_KEY` is set, or when
275+
`codex login status` reports anything but `Logged in using ChatGPT` (an API key, no login, an access token); every
276+
turn passes `-c forced_login_method="chatgpt"`, and both variables are removed from its environment.
277+
- **A turn**: `codex exec --json --ephemeral --skip-git-repo-check -c approval_policy="never" -p PROFILE … -`, the
278+
prompt on stdin. An authoring turn runs `-s workspace-write -c sandbox_workspace_write.network_access=true -C
279+
<project>`; a question or operation runs `-s read-only` in an empty temp directory.
280+
- **MCP**: the app is served under the name `app` from a profile layer, `$CODEX_HOME/scenario-runner-<random>.config.toml`
281+
(0600, deleted when the turn returns), that holds the persona's key as an `http_headers` `Authorization` header and
282+
turns off every MCP server of your own config (`enabled = false`). A server of your own named `app` is refused.
283+
- **Flags**: `--model`, `--author-model` and `--ask-model` pass `-m`; `--author-timeout` is the runner's own clock.
284+
`codex exec` has no cap on its agentic turns, so `--author-max-turns` and `--ask-max-turns` are refused. Its JSONL
285+
names no model, so the scorecard shows `model not reported (asked for …)`.
286+
287+
### `--driver external:COMMAND` — the contract
288+
289+
`drivers/external.ts` runs any command that speaks the driver seam as a process contract. `COMMAND` is split on
290+
whitespace (no shell, no quoting); a word that starts `./` or `../`, and a first word holding a `/`, is resolved against
291+
the directory you run from. For anything more, write a wrapper script.
292+
293+
- **One process per call.** The call is one JSON object on stdin; the answer is one JSON object on stdout; logs go to
294+
stderr (the record keeps its last 2,000 characters). Every input starts with `"contract":
295+
"objectstack-scenario-driver/1"`; a command should refuse a contract it does not know.
296+
- **`info`**, once, before any card runs: `{"contract":…,"kind":"info"}` → `{"name":"my-agent","version":"1.4.2"}`.
297+
The record's driver is `external:my-agent`, its version `1.4.2`. A command that cannot answer it is refused.
298+
- **`author`**: build or iterate the app in `cwd` with shell and file tools; the call runs in `cwd`.
299+
`{"contract":…,"kind":"author","step":{"card","mode","phase","step"},"prompt":"…","cwd":"/abs/project","tier":"starter","limits":{…}}`
300+
- **`ask`**: answer one data question or write operation as a persona, over the app's MCP endpoint; the call runs in
301+
the directory of `mcp.configFile`.
302+
`{"contract":…,"kind":"ask","step":{"card","mode","phase","step","persona"},"prompt":"…","mcp":{"url":"…","configFile":"/tmp/…/mcp.json"},"limits":{…}}`
303+
- **The answer** to `author` and `ask`, strict (an unknown key is refused):
304+
`{"replyText":"every text the agent said, joined","toolCalls":[{"name":"query_records","input":{…},"output":"…","isError":false}],"model":"…","numTurns":3}`.
305+
`model` and `numTurns` are optional; an MCP tool may be named bare or `mcp__<server>__<tool>`; a call that never
306+
answered leaves `output` out.
307+
- **Exit codes**: 0 with an answer of that shape is the turn. Any other end is a failed turn with a named reason,
308+
never an empty pass: a non-zero exit or a signal is `no-result(exit N)`, stdout that is not one JSON object of that
309+
shape is `invalid-output(…)`. A failed authoring turn leaves its Build half not measured; a failed persona turn is
310+
graded as the empty answer it is.
311+
- **Timeout**: `limits.timeoutMs` — `--author-timeout` for `author` (default 60 min), 5 min for `ask`. The runner kills
312+
a call still running then (SIGTERM).
313+
- **Limits**: `limits` carries `timeoutMs`, and `maxTurns` (`--author-max-turns` / `--ask-max-turns`) and `model`
314+
(`--model`, `--author-model`, `--ask-model`) when given. The command may ignore them; the record states them.
315+
- ⛔ **The key**: the persona's API key is only in `mcp.configFile`, a 0600 file in the shape of Claude Code's
316+
`--mcp-config` (`{"mcpServers":{"app":{"type":"http","url":…,"headers":{"Authorization":"Bearer …"}}}}`), deleted
317+
when the call returns. It is never on argv, in the environment or on stdin, and so never in a run record, a
318+
scorecard or a gallery export. A command must not print it.
319+
- **Cost**: the runner cannot see an external command's login, so the record's login reads `not verified`; the cost
320+
ruling still applies, and the runner removes `ANTHROPIC_API_KEY`, `ANTHROPIC_AUTH_TOKEN`, `OPENAI_API_KEY` and
321+
`CODEX_API_KEY` from the command's environment.
322+
323+
`test/runner/fixtures/fake-external-driver.mjs` is a complete command in about a hundred lines.
324+
325+
**WorkBuddy** (checked 2026-11-07): Tencent's WorkBuddy is a desktop app; no headless mode with structured output
326+
and an MCP client was found for it, so it is not wired here. A tool that has both rides `--driver external:` with a
327+
wrapper, with no change to the runner.

‎packages/qa/scenarios/src/export/exporter.ts‎

Lines changed: 18 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@
2020
* and then swapped in, so a re-export first deletes the old directory and
2121
* nothing stale survives (see {@link APP_LAYOUT}):
2222
*
23-
* apps/<id>/README.md what the app is, its scorecard, the files
23+
* apps/<id>/README.md what the app is, the driver that ran it ({@link driverText}), its scorecard, the files
2424
* apps/<id>/source/… the authored metadata source (`source.ts`)
2525
* apps/<id>/run-record.json the run record, the work directories redacted (`workDirsOf`)
2626
* apps/<id>/scorecard.json|.md the per-half scorecard
@@ -54,7 +54,7 @@ import { renderScorecard, type CardScorecard } from '../runner/scorecard.js';
5454
import { GALLERY_GUARD, type GalleryGuard } from './generic.js';
5555
import { parseIndex, serializeIndex, type GalleryApp, type GalleryIndex } from './manifest.js';
5656
import { judgeRecord, type PassJudgement } from './pass.js';
57-
import { readRunRecords, redactWorkDir, unpublishedRunRefusal, type RecordFile } from './record.js';
57+
import { readRunRecords, redactWorkDir, unpublishedRunRefusal, type RecordFile, type ScenarioRunRecord } from './record.js';
5858
import type { CaptureScreenshots, PlannedShot, SkippedShot } from './screenshots.js';
5959
import { projectFrameworkVersion, readAuthoredSource } from './source.js';
6060
import { buildTranscript, mergeRecordings, renderTranscript } from './transcript.js';
@@ -124,7 +124,20 @@ class Refused extends Error {
124124

125125
const json = (v: unknown) => `${JSON.stringify(v, null, 2)}\n`;
126126

127-
function appReadme(scenario: LoadedScenario, entry: GalleryApp, judgement: PassJudgement, scorecard: CardScorecard, shots: PlannedShot[], skipped: SkippedShot[]): string {
127+
/**
128+
* The page's driver line: the agent tool whose turns the app was built and
129+
* graded on, with its version. A replay names the driver of the recording it
130+
* replayed, never the recording's path; a replay whose record does not name
131+
* that driver says so.
132+
*/
133+
export function driverText(driver: ScenarioRunRecord['env']['driver']): string {
134+
const named = (d: { driver: string; version?: string | undefined }) => `${d.driver}${d.version ? ` ${d.version}` : ''}`;
135+
if (driver.replayOf) return `${named(driver.replayOf)} (re-graded from a recording of that run)`;
136+
if (driver.driver === 'recorded') return 'a recording (the record does not name the driver that made it)';
137+
return named(driver);
138+
}
139+
140+
function appReadme(scenario: LoadedScenario, record: ScenarioRunRecord, entry: GalleryApp, judgement: PassJudgement, scorecard: CardScorecard, shots: PlannedShot[], skipped: SkippedShot[]): string {
128141
const c = scenario.card;
129142
const lines = [
130143
`# ${c.title.en}`,
@@ -135,6 +148,7 @@ function appReadme(scenario: LoadedScenario, entry: GalleryApp, judgement: PassJ
135148
'',
136149
`- Scenario: \`${c.id}\` · business functions ${c.businessFunctions.map((f) => `\`${f}\``).join(', ')} · namespace \`${entry.namespace}\``,
137150
`- Framework: @objectstack/spec ${entry.framework.version} (runner checkout ${entry.framework.runner}) · run ${entry.run} on ${entry.runDate}`,
151+
`- Driver: ${driverText(record.env.driver)}`,
138152
`- 问数: ${entry.qa.questions} questions, ${entry.qa.turns} persona turns · ${entry.operations} write operations`,
139153
'',
140154
'## The requirement',
@@ -324,7 +338,7 @@ async function exportOne(
324338
[APP_LAYOUT.scorecardMd, `${renderScorecard(scorecard)}\n`],
325339
[APP_LAYOUT.transcriptJson, json(transcript)],
326340
[APP_LAYOUT.transcriptMd, renderTranscript(transcript, card.title.en)],
327-
[APP_LAYOUT.readme, redact(appReadme(scenario, entry, judgement, scorecard, shots, skipped))],
341+
[APP_LAYOUT.readme, redact(appReadme(scenario, record, entry, judgement, scorecard, shots, skipped))],
328342
]);
329343
return { entry, shots, skipped };
330344
}

‎packages/qa/scenarios/src/export/record.ts‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,12 @@ export const ScenarioRunRecordSchema = z.looseObject({
5656
framework: z.string().min(1),
5757
/** `runner/packages.ts` `PackageSource`; judged by {@link unpublishedRunRefusal}, so a record without it still reads. */
5858
packages: z.unknown().optional(),
59-
driver: z.looseObject({ driver: z.string().min(1) }),
59+
/** `runner/driver.ts` `DriverInfo`: what ran the turns; a replay names the driver of its recording (`replayOf`). */
60+
driver: z.looseObject({
61+
driver: z.string().min(1),
62+
version: z.string().optional(),
63+
replayOf: z.looseObject({ driver: z.string().min(1), version: z.string().optional() }).optional(),
64+
}),
6065
}),
6166
evidenceDir: z.string().min(1),
6267
results: z.array(z.looseObject({ id: z.string().min(1), verdict: VerdictSchema })),

‎packages/qa/scenarios/src/runner/cli.ts‎

Lines changed: 32 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,13 @@
1111
* --only ID[,ID] run only these cards
1212
* --modes M[,M] scratch, template (default: both)
1313
* --driver claude the local Claude Code CLI on your own login (default)
14+
* --driver codex the local OpenAI Codex CLI on your own ChatGPT plan
15+
* login (`drivers/codex.ts`); it has no turn cap, so the
16+
* two --*-max-turns flags are refused with it
17+
* --driver external:C any command C that speaks the driver contract (one
18+
* process per call, JSON in on stdin, JSON out on stdout;
19+
* `drivers/external.ts` and the package README); every
20+
* flag below reaches it in the call's `limits`
1421
* --driver recorded:F replay the recording F (no model, no network to a model)
1522
* --record F also save every live turn to F, for later replay
1623
* --author-timeout MIN the live driver's budget per authoring turn, build and
@@ -56,6 +63,8 @@ import { fileURLToPath } from 'node:url';
5663
import { formatRefusal, libraryOf, loadScenarios, readCapabilityAreas } from '../loader.js';
5764
import { HttpApp } from './app.js';
5865
import { ClaudeCodeDriver, RecordedDriver, RecordingDriver, type AgentDriver } from './driver.js';
66+
import { CodexDriver } from './drivers/codex.js';
67+
import { ExternalDriver } from './drivers/external.js';
5968
import { CAPABILITY_AREAS_DIR, locateLibraryRoot } from './library.js';
6069
import { minutesText, runCard, type ProjectOps } from './orchestrate.js';
6170
import { installedSpecVersion, PACKAGES_MODES, prepareWorkspacePackages, settleWorkspaceProject, workspaceScaffolder, type PackageSource, type PackagesMode } from './packages.js';
@@ -189,7 +198,29 @@ export function makeDriver(spec: string, opts: DriverLimits & DriverModels = {})
189198
if (given.length) throw new Error(`${given.join(', ')} ${given.length === 1 ? 'applies' : 'apply'} to the live driver only: a replay re-reads each turn as it was recorded`);
190199
return RecordedDriver.fromFile(spec.slice('recorded:'.length));
191200
}
192-
throw new Error(`unknown driver \`${spec}\` (claude, recorded:FILE)`);
201+
if (spec === 'codex') {
202+
// Measured on codex-cli 0.167.0: `codex exec` takes no cap on its agentic turns (`drivers/codex.ts`).
203+
const capped = (['authorMaxTurns', 'askMaxTurns'] as const).filter((k) => opts[k] !== undefined).map((k) => LIVE_FLAGS.find(([key]) => key === k)![1]);
204+
if (capped.length) throw new Error(`${capped.join(', ')} ${capped.length === 1 ? 'has' : 'have'} no equivalent in the Codex CLI: \`codex exec\` takes no turn cap`);
205+
return new CodexDriver({
206+
...(opts.authorTimeoutMin !== undefined ? { authorTimeoutMs: opts.authorTimeoutMin * 60_000 } : {}),
207+
...(opts.model !== undefined ? { model: opts.model } : {}),
208+
...(opts.authorModel !== undefined ? { authorModel: opts.authorModel } : {}),
209+
...(opts.askModel !== undefined ? { askModel: opts.askModel } : {}),
210+
});
211+
}
212+
if (spec.startsWith('external:')) {
213+
return new ExternalDriver({
214+
command: spec.slice('external:'.length),
215+
...(opts.authorTimeoutMin !== undefined ? { authorTimeoutMs: opts.authorTimeoutMin * 60_000 } : {}),
216+
...(opts.authorMaxTurns !== undefined ? { authorMaxTurns: opts.authorMaxTurns } : {}),
217+
...(opts.askMaxTurns !== undefined ? { askMaxTurns: opts.askMaxTurns } : {}),
218+
...(opts.model !== undefined ? { model: opts.model } : {}),
219+
...(opts.authorModel !== undefined ? { authorModel: opts.authorModel } : {}),
220+
...(opts.askModel !== undefined ? { askModel: opts.askModel } : {}),
221+
});
222+
}
223+
throw new Error(`unknown driver \`${spec}\` (claude, codex, external:COMMAND, recorded:FILE)`);
193224
}
194225

195226
export async function main(argv: readonly string[]): Promise<number> {

0 commit comments

Comments
 (0)