Skip to content

Commit 9c1edbc

Browse files
fix(scenarios): seed a reference cycle in two passes; an author turn that does not finish leaves its build not measured (#31096)
Fixes #30982 Clause-②: no Metadata-protocol: none ## 维护者速读 - 改了什么:场景跑批器(`@objectstack/scenarios` 的 runner)改为两遍加载种子数据,能装进 department ↔ employee 这类互相引用的环;构建提示词写明「环里至少一侧必须可选」,两侧都必填的构建记为 Build finding;构建回合超时或失败时 Build 半场记为「未测量」,不再打分、不再跑种子、Run 和 iterate;提示词改经 stdin 传给 `claude`。 - 为什么改:8 卡实跑里每张中档卡的种子只装进 3–6 行,Run 半场全部测不了;中档构建在第 15 分钟被 runner 自己杀掉却被当成 AI 质量分;travel_expense 和 service_desk 的构建回合根本没启动——实测是提示词超过 Linux 单个参数 128 KiB 的上限,`spawn` 直接报 E2BIG,记录里什么都没留下。 - 风险与回滚:只动私有、不发布的跑批器包,平台运行时零改动;回滚就是 revert 本 PR。两处没能在真机上实测:`claude -p` 从 stdin 读提示词、第二遍用 `PATCH /api/v1/data/OBJECT/ID` 回填引用;若有一处不成立,记录会如实写出退出码和 stderr,不会静默丢失。 - 先看:`packages/qa/scenarios/src/runner/seed.ts`(`seedPlan`:强连通分量里挑可选的一侧推迟)、`src/runner/driver.ts` 的 `runCli`(stdin 传输、超时与进程记录)、`src/runner/orchestrate.ts` 的 `seedApp` 第二遍和 `scratchMode` 的未测量提前返回。 - 两分钟验证:`pnpm --filter @objectstack/scenarios exec vitest run test/runner/units.test.ts test/runner/orchestrate.test.ts`,预期 78 个测试全绿,其中 requisition_to_receipt 的种子 637/637 全部装入。 - 需要你做的:无,席位落地。 ## What changes **Defect 1 — the seed loads a reference cycle in two passes** (`src/runner/seed.ts`, new; `orchestrate.ts` `seedApp`). - The references come from the card's own declared lookups (`expect.lookups` between two seeded objects), never a hand list. Objects go in dependencies first, one strongly connected set at a time (Tarjan; on an acyclic card this is exactly the depth-first order the runner already used). - Inside a cycle, members go in by Kahn's algorithm; when every remaining member still waits on another, one whose waiting lookups are all **optional in the built schema** is let in early. Then every lookup value naming a row not yet in at its row's insert (a lookup to an object loaded later, or a self-lookup naming a later row of its own object) is left out of the insert and set by one `PATCH` per row in a second pass (`AppHarness.updateRow`, `HttpApp`: `PATCH /api/v1/data/OBJECT/ID`). - The built schema's `required` lookup fields are read through the scorer's one artifact reader (`artifactToEvalGraph`), scratch mode from the built artifact, template mode from the installed copy. - A cycle whose every lookup the build made `required` cannot be loaded. The build prompt's data contract now names the card's reference cycles and says "At least one side of a reference cycle must be optional" (`prompts.ts` `dataContract`, only on a card that has one), and the Build half records `requiredCycles`, which the scorecard lists as a Build finding naming the cycle. The seed is still sent, so the record holds the app's own refusals rather than the runner's prediction. - The tally still counts each row once: a row whose second-pass update is refused moves from `inserted` to `refused` with the `fields` it could not set, and `seedDependence` takes what hangs off it out as before. `seed.secondPass` records the lookup values set per `OBJECT.FIELD`. **Defect 2 — a build author turn that fails is recorded, and its Build half is not measured.** - Measured cause: the build prompt carries the whole seed and was passed on argv. On Linux one argv string over 128 KiB (`MAX_ARG_STRLEN`, 131,072 bytes) makes `spawn` throw `E2BIG` before the CLI starts; `runCli` rejected, the author turn threw, and `RecordingDriver` recorded nothing. The two cards with no build author turn are exactly the two over the limit (readings below). The prompt now goes to the CLI's stdin (`claude -p`, no prompt argument), for question turns too. - `runCli` never rejects: a process that could not start, was killed, or exited without a result is answered with what was seen — `meta.process`: elapsed ms, exit code or signal, whether the runner's timeout killed it, the stderr tail (2,000 chars), and a spawn or stdin error. It travels in the turn's `meta`, so a recording carries it and a replay re-reads a failed turn as the run saw it. - `authorNotMeasured`: a turn without a clean `success` result line — no result line, an error subtype, or `is_error: true` (a usage limit answers `success` with it) — leaves its Build half `notMeasured: "author turn failed: …"`; a turn the driver threw on is the same, with its error. Nothing after it runs: no `os validate`/`build`/`verify`, no score, no seed, no Run, no iterate. **Defect 3 — the authoring timeout is a runner option, and a killed turn is never a degraded score.** - `--author-timeout MIN` (default `DEFAULT_AUTHOR_TIMEOUT_MIN = 60`, a plain constant: `card.tier` is not read) reaches `authorTimeoutMs`, which both the build and the iterate author turns use; the run record's `env.driver.authorTimeoutMs` and the console's driver line state it. A replay refuses the flag (it re-reads what was recorded). - A turn the timeout killed makes its Build half `notMeasured: "author timed out after N min"`. An iterate author turn that does not finish leaves the first halves standing and the iterate build not measured, with no iterate Run half and no change size; template mode the same for its iterate. - The scorecard prints `not measured: …` in the Build cell (0 of 0, never `degraded`), adds `Build · author` and `Iterate · author` rows with every authoring turn's `resultSubtype` and elapsed time, and the run record gives every half that did not run because of it the reason in `notes` (`not run: the build was not measured (…)`), verdict `not-run`. Landing: all in `packages/qa/scenarios/src/runner/**`, its tests and README, plus `test/export/fixture.ts` (outside the claim's file surface, only to give its `AppHarness` double the new `updateRow`). Not `src/schema.ts`, `src/loader.ts` or the card files. The gallery exporter's screenshot boot (`src/export/screenshots.ts`, unchanged) seeds through `seedApp` and inherits the two-pass load. ## Tests - `test/runner/units.test.ts`, new top-level describes: - the seed's load plan on `requisition_to_receipt`: the cycle found from `expect.lookups`; departments first with five managers deferred; the side the build left optional is the one that waits (18 `employee.department` values when `department.manager` is required); both sides required names the cycle; `order_to_cash`'s `e09.manager → e10` self-reference waits; on every card in the library the first pass never names a row not in yet; the contract's cycle sentence. - the live `ClaudeCodeDriver` against a stand-in `claude` script (never the real CLI): a 200,000-byte prompt reaches it whole on stdin with argv under 1,000 bytes; a turn the timeout kills records `exit 143`, `timedOut`, elapsed and stderr tail and is `author timed out after …`; a crash and a vanished binary are failed turns with what was seen; `--author-timeout` parsing, its 60-min default and the replay refusal. - `test/runner/app.test.ts`, new top-level describe: `HttpApp.updateRow` sends `PATCH /api/v1/data/OBJECT/ID` with the bare field patch to a loopback door, and reads a refusal's code and message. - `test/runner/orchestrate.test.ts`, new top-level describes: a killed / failed / usage-limited / thrown build author turn, a killed iterate turn, and a failed template iterate turn, each not measured with nothing after it run; `seedApp` loading all 637 `requisition_to_receipt` rows into a double that checks reference integrity and `required` on insert and update; a refused second-pass update; `runCard` reporting a both-sides-required cycle as a Build finding. - `pnpm --filter @objectstack/scenarios typecheck` and `test`: green at the final head (readings below). Reverse verification, each through `node scripts/ablation-replace.mjs` in wrap mode (anchor hit 1 → 0, blob changed, restore proven `blob == HEAD`, `git diff HEAD` empty). The suites import `src/` directly, so no build sits between the mutation and the run. | Mutation | Expected | Observed | |:---|:---|:---| | `seed.ts`: no value is ever deferred | the cycle tests red | 9 failed / 69 passed | | `driver.ts`: the build prompt back on argv | the 200 KB test red (`spawn E2BIG`, empty reply) | 1 failed / 50 passed, `Received: ""` | | `orchestrate.ts`: delete the not-measured early return | the killed / failed build tests red | 2 failed / 25 passed | ## Acceptance notes - `error_max_turns` counts as `author turn failed`: it is the runner's own turn budget (80), the same class as the timeout. A 60-minute mid-tier build that then runs out of turns will read not measured, not a cut-off score. No `--author-max-turns` option is added. - The suggested route skipped the seed for a both-sides-required build. Here it is still sent: the finding names the cycle either way, and the record then shows whether the platform really refuses a missing required lookup on insert instead of assuming it. - `buildPassed` is unchanged: a both-sides-required cycle is a finding, not a scored check (scoring stays `@objectstack/scenarios/score`), so such a build can still be the template mode's verified build; its template seed is refused the same way and recorded. - Not measured live: `claude -p` reading its prompt from stdin (the CLI's help: `-p/--print … useful for pipes`, `--input-format text (default)`) and the `PATCH` data door against a real server (its shape is read from `packages/rest/src/crud-routes.ts`, and `app.ts` says so; the request the runner sends is pinned against a loopback door). The card's verification re-run settles both; if either fails, the turn or the second-pass refusal is in the record with its exit code, stderr or answer. - Measured in the same class and covered without extra code: `order_to_cash`'s `e09.manager → e10` was refused by the object-by-object load, and `service_desk` has an `agent_group.lead ↔ agent.group` cycle. - Changeset: none. `@objectstack/scenarios` is `private: true` and publishes nothing, so `skip-changeset`. ## Verification Readings at head `401e567b`, `objectstack-ai/objectstack` (the gate run and the package suite both after the final commit). <details><summary>Defect 2: build prompt bytes per card vs MAX_ARG_STRLEN (131,072)</summary> ```text asset_lifecycle build 109066 iterate 763 crm_account_with_contacts build 3563 iterate 724 invoice_with_line_items build 4250 iterate 751 leave_balances build 120716 iterate 883 order_to_cash build 123612 iterate 803 project_with_tasks build 4059 iterate 742 requisition_to_receipt build 128145 iterate 824 service_desk build 141922 iterate 814 OVER 128KiB (MAX_ARG_STRLEN) travel_expense build 162201 iterate 807 OVER 128KiB (MAX_ARG_STRLEN) node spawn('/bin/true', ['-p', 'x'.repeat(n)]): 128145 spawned 131071 spawned 131072 REJECTED E2BIG spawn E2BIG 141922 REJECTED E2BIG spawn E2BIG 162201 REJECTED E2BIG spawn E2BIG ``` </details> <details><summary>Package typecheck and tests</summary> ```text os-verify-lock: ACQUIRED after 0s — running: pnpm --filter @objectstack/scenarios build && pnpm --filter @objectstack/scenarios typecheck && pnpm --filter @objectstack/scenarios test check-dts-emitted: @objectstack/scenarios - 10/10 declared declaration file(s) present. > tsc --noEmit Test Files 23 passed (23) Tests 364 passed (364) os-verify-lock: VERDICT command-exit 0 · held the lock 42s · waited 0s · ⚠ SHARED-BOX SECONDS — this lock excluded other LOCKED runs, NOT unlocked sibling work (`check:*` gates and the rest), so these are not idle-box figures ``` </details> An earlier gate run on this branch read one red and two prerequisites, none left: `check:cross-package-test-inputs` flagged a new test's `readdirSync` of `docs/qa/scenarios/`, which the declared `*.json` glob does not hold (the test now reads the library through `loadScenarios`, as `test/library.test.ts` does); `check-plugin-teardown-shape --self-test` needed its pinned fixture commit fetched into this shallow clone; `check:dual-build-cjs-loads` needed built output. <details><summary>Derived gates (dispatch-gates --commands, reconciled with --ran)</summary> ```text dispatch-gates: gate list derived from the tree of 'objectstack-ai/objectstack' at commit 401e567 (/home/user/objectstack-issue-30982). Run reconciliation — 50 derived, 50 run, 0 NOT-MEASURED, 0 UNRUN. ✓ dispatch-gates --ran: 50 derived famil(ies) accounted for — 50 run, 0 NOT-MEASURED (a DERIVED zero — all 50 recorded an exit code and none of them is 3). node scripts/check-closing-keyword-parity.mjs :: exit 0 node scripts/check-closing-keyword-parity.mjs --self-test :: exit 0 node scripts/check-comment-mask-adoption.mjs :: exit 0 node scripts/check-comment-mask-adoption.mjs --self-test :: exit 0 node scripts/check-comment-mask-corpus.mjs :: exit 0 node scripts/check-dts-emitted.mjs --self-test :: exit 0 node scripts/check-issue-citations.mjs :: exit 0 node scripts/check-plugin-teardown-shape.mjs :: exit 0 node scripts/check-plugin-teardown-shape.mjs --self-test :: exit 0 node scripts/ci/assert-turbo-remote-cache.mjs --self-test :: exit 0 node scripts/ci/plan-test-core-shards.mjs --self-test :: exit 0 node scripts/ci/prune-turbo-cache.mjs --self-test :: exit 0 node scripts/docs-audit/check-affected-docs.mjs :: exit 0 node scripts/docs-audit/check-drift-comment.mjs :: exit 0 pnpm --filter @objectstack/spec run check:duration-unit-keys :: exit 0 pnpm check:ci-filter-parity :: exit 0 pnpm check:closed-ceremony-text :: exit 0 pnpm check:cross-package-test-inputs :: exit 0 pnpm check:dispatcher-error-vocabulary :: exit 0 pnpm check:doc-authoring :: exit 0 pnpm check:driver-memory-census :: exit 0 pnpm check:dts-closure :: exit 0 pnpm check:dual-build-cjs-loads :: exit 0 pnpm check:engine-double-contract :: exit 0 pnpm check:error-status-conformance :: exit 0 pnpm check:gitlink-declared :: exit 0 pnpm check:issue-citations :: exit 0 pnpm check:keyed-text-bounds :: exit 0 pnpm check:lean-entry-closure :: exit 0 pnpm check:logger-receiver-detach :: exit 0 pnpm check:nul-bytes :: exit 0 pnpm check:objectql-double-limit :: exit 0 pnpm check:org-identifier :: exit 0 pnpm check:page-declaration-shape :: exit 0 pnpm check:platform-object-tenancy-census :: exit 0 pnpm check:published-files :: exit 0 pnpm check:query-options-erasure :: exit 0 pnpm check:refd-timer-probe :: exit 0 pnpm check:registry-log-declared :: exit 0 pnpm check:rest-log-spy-declared :: exit 0 pnpm check:slot-lookup :: exit 0 pnpm check:sourcemap-no-sources-content :: exit 0 pnpm check:system-context-census :: exit 0 pnpm check:test-source-alias :: exit 0 pnpm check:tier-file-adoption :: exit 0 pnpm check:type-check-coverage :: exit 0 pnpm check:type-check-debt :: exit 0 pnpm check:undeclared-dep-imports :: exit 0 pnpm check:watch-hint-literal :: exit 0 pnpm check:where-matcher :: exit 0 ``` </details> --- _Generated by [Claude Code](https://claude.ai/code/session_012MmDsRzowAnXEmogPGSEKA)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent b631573 commit 9c1edbc

14 files changed

Lines changed: 1168 additions & 111 deletions

File tree

‎packages/qa/scenarios/README.md‎

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,7 @@ pnpm --filter @objectstack/scenarios run-scenarios # every
8989
pnpm --filter @objectstack/scenarios run-scenarios --only service_desk --modes scratch
9090
pnpm --filter @objectstack/scenarios run-scenarios --record /tmp/rec.json # also save every turn
9191
pnpm --filter @objectstack/scenarios run-scenarios --driver recorded:/tmp/rec.json # replay, no model
92+
pnpm --filter @objectstack/scenarios run-scenarios --author-timeout 90 # minutes per authoring turn (default 60)
9293
```
9394

9495
Per card and mode the runner writes `docs/qa/platform-checklist/runs/<date>-scenario-<card>.json`
@@ -99,9 +100,24 @@ Per card and mode the runner writes `docs/qa/platform-checklist/runs/<date>-scen
99100
| `scratch` | `create-objectstack` → the agent builds from the card prompt and the runner's data contract (`prompts.ts`) → `objectstack validate` / `build` / `verify` (proofs chosen from the tags) → score → `objectstack dev --fresh` → seed, one user + permission-set grant + API key per persona → every question as every persona it names and every operation as its persona, over the app's MCP → the iterate change → rebuild, re-score, re-run |
100101
| `template` | a blank host project (`host<code>`) → boot → install the card's verified build **unmanaged** (`/api/v1/marketplace/install-local`, `installMode: 'unmanaged'`) → the agent applies the iterate change to the installed copy → score → seed → Run. A refused install is a finding in the record |
101102

102-
- **Driver** (`driver.ts`): the local `claude` CLI on your own login (`claude -p --output-format stream-json`). It
103-
refuses to start when `ANTHROPIC_API_KEY` / `ANTHROPIC_AUTH_TOKEN` is set or the CLI is logged in with an API key
104-
(the cost ruling on cloud#3600), and the record states the driver and login. `RecordedDriver` replays a recording.
103+
- **Driver** (`driver.ts`): the local `claude` CLI on your own login (`claude -p --output-format stream-json`, the
104+
prompt written to its stdin: a build prompt carries the whole seed, and one argv string over 128 KiB makes `spawn`
105+
fail with `E2BIG` on Linux). It refuses to start when `ANTHROPIC_API_KEY` / `ANTHROPIC_AUTH_TOKEN` is set or the CLI
106+
is logged in with an API key (the cost ruling on cloud#3600), and the record states the driver, the login and the
107+
authoring timeout. Every turn records its process: elapsed time, and for a turn that did not finish its exit code or
108+
signal, whether the runner's timeout killed it, and the end of its stderr. `RecordedDriver` replays a recording.
109+
- **A build that did not finish is not measured** (`orchestrate.ts`): an authoring turn the timeout killed
110+
(`--author-timeout`, build and iterate alike) leaves its Build half **not measured: author timed out after N min**;
111+
one that ended with no result line or an error result (`error_max_turns`, `is_error`), or never ran, leaves it **not
112+
measured: author turn failed: …**. Never a degraded score, and nothing runs on it: no score, seed, Run or iterate. The
113+
scorecard's author rows show every authoring turn's `resultSubtype` and elapsed time.
114+
- **Seed** (`seed.ts`): rows go in as the admin, a lookup's target before its referrer. Where the card's declared
115+
lookups lead back to where they start — `department.manager → employee` with `employee.department → department`, or a
116+
self-lookup naming a later row — the seed loads in two passes: every row first with each lookup naming a row not in
117+
yet left out, then one update per row sets those (`seed.secondPass` in the record). The side that waits is one the
118+
build left optional; a build that makes every side of a cycle `required` cannot be seeded, which the data contract
119+
states and the scorecard reports as a Build finding naming the cycle (the seed is still sent, so the record holds
120+
the app's own refusals).
105121
- **Observation** (`observe.ts`): the write grader is fed what the runner reads itself: the agent's write calls and
106122
the door's answer (a coded refusal), the object's rows before and after the turn, and the audit trail's actor. A turn
107123
that wrote nothing because the object refused the read it needed first (a persona that may not read it) grades as

‎packages/qa/scenarios/src/runner/app.ts‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,10 @@
1818
* set reads `PERMISSION_DENIED` on every app object
1919
* a seed row POST /api/v1/data/:object with the card's own `id` (accepted
2020
* verbatim, so a lookup value in the seed names the real row)
21+
* a waiting lookup PATCH /api/v1/data/:object/:id with the bare field patch (the
22+
* seed's second pass, `seedApp` in `orchestrate.ts`; read from the
23+
* route's source, `packages/rest/src/crud-routes.ts`, not yet
24+
* measured live)
2125
* rows GET /api/v1/data/:object?limit=…
2226
* the audit actor GET /api/v1/data/sys_audit_log?record_id=… (`user_id`)
2327
* unmanaged install POST /api/v1/marketplace/install-local { packageId, manifest,
@@ -119,6 +123,8 @@ export interface AppHarness {
119123
/** An API key for the admin, for the questions no persona is named on. */
120124
adminApiKey(): Promise<string>;
121125
insertRow(object: string, row: Record<string, unknown>): Promise<HttpAnswer>;
126+
/** Set fields on a row already in (the seed's second pass: the lookups that close a reference cycle). */
127+
updateRow(object: string, id: string, patch: Record<string, unknown>): Promise<HttpAnswer>;
122128
listRows(object: string): Promise<HttpAnswer & { rows: Array<Record<string, unknown>> }>;
123129
/** The `user_id` the newest audit row for this record names; `null` when none names one. */
124130
auditActorUserId(object: string, recordId: string): Promise<string | null>;
@@ -262,6 +268,10 @@ export class HttpApp implements AppHarness {
262268
return this.call('POST', `/api/v1/data/${encodeURIComponent(object)}`, row, { session: this.admin });
263269
}
264270

271+
updateRow(object: string, id: string, patch: Record<string, unknown>): Promise<HttpAnswer> {
272+
return this.call('PATCH', `/api/v1/data/${encodeURIComponent(object)}/${encodeURIComponent(id)}`, patch, { session: this.admin });
273+
}
274+
265275
async listRows(object: string): Promise<HttpAnswer & { rows: Array<Record<string, unknown>> }> {
266276
const r = await this.call('GET', `/api/v1/data/${encodeURIComponent(object)}?limit=1000`, undefined, { session: this.admin });
267277
const records = (r.body as { records?: unknown } | null)?.records;

‎packages/qa/scenarios/src/runner/cli.ts‎

Lines changed: 23 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,9 @@
1010
* --driver claude the local Claude Code CLI on your own login (default)
1111
* --driver recorded:F replay the recording F (no model, no network to a model)
1212
* --record F also save every live turn to F, for later replay
13+
* --author-timeout MIN the live driver's budget per authoring turn, build and
14+
* iterate alike (default 60); a turn it kills leaves its
15+
* Build half not measured
1316
* --template F a verified build (dist/objectstack.json) for template mode
1417
* --no-iterate-run skip the Run half after the iterate (scratch mode)
1518
* --out DIR where records go (default docs/qa/platform-checklist/runs, git-ignored)
@@ -29,7 +32,7 @@ import { fileURLToPath } from 'node:url';
2932
import { formatRefusal, loadScenarios, readCapabilityAreas } from '../loader.js';
3033
import { HttpApp } from './app.js';
3134
import { ClaudeCodeDriver, RecordedDriver, RecordingDriver, type AgentDriver } from './driver.js';
32-
import { runCard, type ProjectOps } from './orchestrate.js';
35+
import { minutesText, runCard, type ProjectOps } from './orchestrate.js';
3336
import { bootDev, os, readArtifact, readNamespace, scaffold } from './project.js';
3437
import { cardScorecard, renderScorecard, runRecord } from './scorecard.js';
3538

@@ -54,6 +57,8 @@ export interface CliArgs {
5457
record?: string;
5558
template?: string;
5659
iterateRun: boolean;
60+
/** Minutes per authoring turn (`--author-timeout`); the driver's default when absent. */
61+
authorTimeoutMin?: number;
5762
out?: string;
5863
work?: string;
5964
createVersion?: string;
@@ -78,6 +83,12 @@ export function parseArgs(argv: readonly string[]): CliArgs {
7883
else if (flag === '--record') a.record = val();
7984
else if (flag === '--template') a.template = val();
8085
else if (flag === '--no-iterate-run') a.iterateRun = false;
86+
else if (flag === '--author-timeout') {
87+
const raw = val();
88+
const min = Number(raw);
89+
if (!Number.isFinite(min) || min <= 0) throw new Error(`--author-timeout takes a number of minutes above 0, not \`${raw}\``);
90+
a.authorTimeoutMin = min;
91+
}
8192
else if (flag === '--out') a.out = val();
8293
else if (flag === '--work') a.work = val();
8394
else if (flag === '--create-version') a.createVersion = val();
@@ -86,9 +97,12 @@ export function parseArgs(argv: readonly string[]): CliArgs {
8697
return a;
8798
}
8899

89-
export function makeDriver(spec: string): AgentDriver {
90-
if (spec === 'claude') return new ClaudeCodeDriver();
91-
if (spec.startsWith('recorded:')) return RecordedDriver.fromFile(spec.slice('recorded:'.length));
100+
export function makeDriver(spec: string, opts: { authorTimeoutMin?: number } = {}): AgentDriver {
101+
if (spec === 'claude') return new ClaudeCodeDriver(opts.authorTimeoutMin !== undefined ? { authorTimeoutMs: opts.authorTimeoutMin * 60_000 } : {});
102+
if (spec.startsWith('recorded:')) {
103+
if (opts.authorTimeoutMin !== undefined) throw new Error('--author-timeout applies to the live driver only: a replay re-reads each turn as it was recorded');
104+
return RecordedDriver.fromFile(spec.slice('recorded:'.length));
105+
}
92106
throw new Error(`unknown driver \`${spec}\` (claude, recorded:FILE)`);
93107
}
94108

@@ -110,10 +124,13 @@ export async function main(argv: readonly string[]): Promise<number> {
110124
return 2;
111125
}
112126

113-
const live = makeDriver(args.driver);
127+
const live = makeDriver(args.driver, args.authorTimeoutMin !== undefined ? { authorTimeoutMin: args.authorTimeoutMin } : {});
114128
const driver = args.record ? new RecordingDriver(live) : live;
115129
const info = driver.info();
116-
console.log(`driver: ${info.driver} ${info.version ?? ''} · login: ${info.authMethod ?? 'n/a'} · API-billed: no`);
130+
console.log(
131+
`driver: ${info.driver} ${info.version ?? ''} · login: ${info.authMethod ?? 'n/a'} · API-billed: no` +
132+
(info.authorTimeoutMs !== undefined ? ` · author timeout: ${minutesText(info.authorTimeoutMs)}` : ''),
133+
);
117134

118135
const ops: ProjectOps = {
119136
scaffold: (name, under) => {

‎packages/qa/scenarios/src/runner/driver.ts‎

Lines changed: 95 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -9,10 +9,11 @@
99
* exactly two drivers, and neither calls a model API itself:
1010
*
1111
* - {@link ClaudeCodeDriver} spawns the `claude` CLI in headless mode
12-
* (`claude -p … --output-format stream-json`). It REFUSES to start when the
13-
* process would bill an API key — `ANTHROPIC_API_KEY` / `ANTHROPIC_AUTH_TOKEN`
14-
* in the environment, or a CLI whose `auth status` reports an API-key login —
15-
* and strips both variables from every child it spawns.
12+
* (`claude -p --output-format stream-json`, the prompt on stdin). It REFUSES
13+
* to start when the process would bill an API key — `ANTHROPIC_API_KEY` /
14+
* `ANTHROPIC_AUTH_TOKEN` in the environment, or a CLI whose `auth status`
15+
* reports an API-key login — and strips both variables from every child it
16+
* spawns.
1617
* - {@link RecordedDriver} replays turns recorded earlier, so the runner's
1718
* own tests (and a re-grade of an old run) need no model and no network.
1819
*
@@ -25,7 +26,7 @@ import { mkdtempSync, mkdirSync, readdirSync, readFileSync, rmSync, writeFileSyn
2526
import { tmpdir } from 'node:os';
2627
import { dirname, join, relative, resolve, sep } from 'node:path';
2728

28-
import { parseStreamJson, type ParsedTurn } from './transcript.js';
29+
import { parseStreamJson, type ParsedTurn, type TurnProcess } from './transcript.js';
2930

3031
/** Which run step a turn belongs to; the replay key is built from it. */
3132
export interface StepRef {
@@ -68,6 +69,8 @@ export interface DriverInfo {
6869
authMethod?: string;
6970
/** Always `false`: no driver here bills an API key; the record states it. */
7071
apiBilled: false;
72+
/** The live driver's wall-clock budget per authoring turn (build and iterate), ms. */
73+
authorTimeoutMs?: number;
7174
}
7275

7376
export interface AgentDriver {
@@ -111,41 +114,104 @@ export function assertSubscriptionLogin(env: NodeJS.ProcessEnv, authStatus: unkn
111114
return method;
112115
}
113116

117+
/**
118+
* The authoring budget a run gets unless `--author-timeout` says otherwise:
119+
* sized for a mid-tier (`starter`) card, whose 7–8-object builds ran well past
120+
* a quarter of an hour.
121+
*/
122+
export const DEFAULT_AUTHOR_TIMEOUT_MIN = 60;
123+
114124
export interface ClaudeCodeDriverOptions {
115125
/** The CLI binary; default `claude`. */
116126
bin?: string;
117127
/** Agentic turns allowed for one question or operation; default 12. */
118128
maxTurnsAsk?: number;
119129
/** Agentic turns allowed for one authoring task; default 80. */
120130
maxTurnsAuthor?: number;
121-
/** Wall-clock budget per call, ms; default 15 min for authoring, 5 min for a question. */
131+
/** Wall-clock budget per authoring turn (build and iterate), ms; default {@link DEFAULT_AUTHOR_TIMEOUT_MIN}. */
122132
authorTimeoutMs?: number;
133+
/** Wall-clock budget per question or operation, ms; default 5 min. */
123134
askTimeoutMs?: number;
124135
/** Optional model alias passed as `--model` (the login's default otherwise). */
125136
model?: string;
126137
}
127138

128139
interface CliRun {
129140
stdout: string;
130-
stderr: string;
131-
exit: number | null;
141+
process: TurnProcess;
132142
}
133143

134-
function runCli(bin: string, args: string[], cwd: string, timeoutMs: number): Promise<CliRun> {
144+
const STDERR_TAIL = 2000;
145+
146+
/**
147+
* One CLI process, the prompt written to its stdin. The prompt never travels
148+
* on argv: a build prompt carries the card's whole seed, and on Linux one
149+
* argument longer than 128 KiB (`MAX_ARG_STRLEN`) makes `spawn` throw `E2BIG`
150+
* before the CLI starts — the two largest cards' build prompts are past it.
151+
*
152+
* Never rejects: a process that could not start, was killed by the timeout or
153+
* exited without a result is answered with what was seen of it, so the turn
154+
* it belongs to is recorded rather than lost.
155+
*/
156+
function runCli(bin: string, args: string[], cwd: string, input: string, timeoutMs: number): Promise<CliRun> {
135157
return new Promise((resolvePromise) => {
136-
const child = spawn(bin, args, { cwd, env: subscriptionOnlyEnv(), stdio: ['ignore', 'pipe', 'pipe'] });
158+
const started = Date.now();
137159
let stdout = '';
138160
let stderr = '';
139-
child.stdout.on('data', (d: Buffer) => (stdout += d.toString()));
140-
child.stderr.on('data', (d: Buffer) => (stderr += d.toString()));
141-
const timer = setTimeout(() => child.kill('SIGTERM'), timeoutMs);
142-
child.on('close', (exit) => {
143-
clearTimeout(timer);
144-
resolvePromise({ stdout, stderr, exit });
161+
let timedOut = false;
162+
let error: string | undefined;
163+
let settled = false;
164+
let timer: NodeJS.Timeout | undefined;
165+
const settle = (exit: number | null, signal: NodeJS.Signals | null) => {
166+
if (settled) return;
167+
settled = true;
168+
if (timer) clearTimeout(timer);
169+
const tail = stderr.trim().slice(-STDERR_TAIL);
170+
resolvePromise({
171+
stdout,
172+
process: {
173+
elapsedMs: Date.now() - started,
174+
exit,
175+
...(signal ? { signal } : {}),
176+
timedOut,
177+
timeoutMs,
178+
...(tail ? { stderrTail: tail } : {}),
179+
...(error ? { error } : {}),
180+
},
181+
});
182+
};
183+
let child: ReturnType<typeof spawn>;
184+
try {
185+
child = spawn(bin, args, { cwd, env: subscriptionOnlyEnv(), stdio: ['pipe', 'pipe', 'pipe'] });
186+
} catch (e) {
187+
error = e instanceof Error ? e.message : String(e); // `spawn E2BIG` and its kin throw here
188+
settle(null, null);
189+
return;
190+
}
191+
child.stdout!.on('data', (d: Buffer) => (stdout += d.toString()));
192+
child.stderr!.on('data', (d: Buffer) => (stderr += d.toString()));
193+
child.on('error', (e) => {
194+
error ??= e.message; // `spawn ENOENT`: no `close` may follow
195+
if (child.pid === undefined) settle(null, null);
145196
});
197+
child.stdin!.on('error', (e) => (error ??= e.message)); // EPIPE: the CLI exited before reading its prompt
198+
child.stdin!.end(input);
199+
timer = setTimeout(() => {
200+
timedOut = true;
201+
child.kill('SIGTERM');
202+
}, timeoutMs);
203+
child.on('close', (exit, signal) => settle(exit, signal));
146204
});
147205
}
148206

207+
/**
208+
* `no-result(exit 143)`, `no-result(signal SIGTERM)`, `no-result(spawn E2BIG)`:
209+
* a turn that printed no result line, by how its process ended — or, when no
210+
* process ever ran, why.
211+
*/
212+
const noResult = (p: TurnProcess): string =>
213+
`no-result(${p.exit !== null ? `exit ${p.exit}` : p.signal ? `signal ${p.signal}` : (p.error ?? 'no exit code, no signal')})`;
214+
149215
/** Every file under `dir` (skipping dependency and build trees) → content, for the author recording. */
150216
export function snapshotSources(dir: string): Record<string, string> {
151217
const out: Record<string, string> = {};
@@ -196,8 +262,12 @@ export class ClaudeCodeDriver implements AgentDriver {
196262
this.authMethod = assertSubscriptionLogin(process.env, parsed);
197263
}
198264

265+
private get authorTimeoutMs(): number {
266+
return this.opts.authorTimeoutMs ?? DEFAULT_AUTHOR_TIMEOUT_MIN * 60_000;
267+
}
268+
199269
info(): DriverInfo {
200-
return { driver: 'claude-code-cli', version: this.version, authMethod: this.authMethod, apiBilled: false };
270+
return { driver: 'claude-code-cli', version: this.version, authMethod: this.authMethod, apiBilled: false, authorTimeoutMs: this.authorTimeoutMs };
201271
}
202272

203273
private base(maxTurns: number): string[] {
@@ -211,15 +281,16 @@ export class ClaudeCodeDriver implements AgentDriver {
211281
writeFileSync(empty, JSON.stringify({ mcpServers: {} }));
212282
try {
213283
const args = [
214-
'-p', task.prompt,
284+
'-p',
215285
...this.base(this.opts.maxTurnsAuthor ?? 80),
216286
'--mcp-config', empty, '--strict-mcp-config',
217287
'--permission-mode', 'acceptEdits',
218288
'--allowedTools', 'Bash', 'Read', 'Edit', 'Write', 'Glob', 'Grep', 'Skill',
219289
];
220-
const run = await runCli(this.bin, args, task.cwd, this.opts.authorTimeoutMs ?? 15 * 60_000);
290+
const run = await runCli(this.bin, args, task.cwd, task.prompt, this.authorTimeoutMs);
221291
const turn = parseStreamJson(run.stdout);
222-
if (turn.meta.resultSubtype === undefined) turn.meta.resultSubtype = `no-result(exit ${run.exit})`;
292+
turn.meta.process = run.process;
293+
if (turn.meta.resultSubtype === undefined) turn.meta.resultSubtype = noResult(run.process);
223294
return { turn, files: snapshotSources(task.cwd) };
224295
} finally {
225296
rmSync(dirname(empty), { recursive: true, force: true });
@@ -237,10 +308,11 @@ export class ClaudeCodeDriver implements AgentDriver {
237308
{ mode: 0o600 },
238309
);
239310
try {
240-
const args = ['-p', task.prompt, ...this.base(this.opts.maxTurnsAsk ?? 12), '--mcp-config', cfg, '--strict-mcp-config', '--allowedTools', 'mcp__app'];
241-
const run = await runCli(this.bin, args, dir, this.opts.askTimeoutMs ?? 5 * 60_000);
311+
const args = ['-p', ...this.base(this.opts.maxTurnsAsk ?? 12), '--mcp-config', cfg, '--strict-mcp-config', '--allowedTools', 'mcp__app'];
312+
const run = await runCli(this.bin, args, dir, task.prompt, this.opts.askTimeoutMs ?? 5 * 60_000);
242313
const turn = parseStreamJson(run.stdout);
243-
if (turn.meta.resultSubtype === undefined) turn.meta.resultSubtype = `no-result(exit ${run.exit})`;
314+
turn.meta.process = run.process;
315+
if (turn.meta.resultSubtype === undefined) turn.meta.resultSubtype = noResult(run.process);
244316
return turn;
245317
} finally {
246318
rmSync(dir, { recursive: true, force: true });

‎packages/qa/scenarios/src/runner/index.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,4 +15,5 @@ export * from './orchestrate.js';
1515
export * from './project.js';
1616
export * from './prompts.js';
1717
export * from './scorecard.js';
18+
export * from './seed.js';
1819
export * from './transcript.js';

0 commit comments

Comments
 (0)