Skip to content

Commit 5f00374

Browse files
feat(scenarios): an authoring turn's turn cap is sized by the card's tier; --author-max-turns and --ask-max-turns reach the live driver (#31111)
Fixes #31106 Clause-②: no Metadata-protocol: none ## 维护者速读 - 改了什么:场景跑批器(私有包 `@objectstack/scenarios` 的 runner)构建回合的轮数上限改为按卡片 `tier` 取默认值(control 80、starter 300、delivery 600),新增 `--author-max-turns N` / `--ask-max-turns N` 两个参数覆盖它;每个回合记录它实际用的上限,记分卡 author 行显示「已用轮数/上限」,例如 `error_max_turns · 80/80 turns`。 - 为什么改:上限原来写死 80 且 CLI 不能改,实测 3 张 starter 卡的构建回合全部在 80 轮被截停(`error_max_turns`,15–24 分钟),没有一张中档卡能测完;control 卡只要 17–21 轮,维持 80 不变。 - 风险与回滚:只动不发布的跑批器包,平台运行时零改动;回滚就是 revert 本 PR。没有用真实 `claude` 跑过(成本裁决只许本地登录,本容器没有);starter 的 300 轮和 60 分钟超时大致同时触顶,构建仍可能以超时结束,两种情况都如实记为「未测量」。 - 先看:`packages/qa/scenarios/src/runner/driver.ts`(`DEFAULT_AUTHOR_MAX_TURNS` 表与 `author()` 按 `task.tier` 取上限)、`src/runner/cli.ts`(`makeDriver` 传参与回放拒收)、`src/runner/orchestrate.ts`(三处 author 调用带上 `tier: card.tier`)。 - 两分钟验证:`pnpm --filter @objectstack/scenarios exec vitest run test/runner/units.test.ts test/runner/orchestrate.test.ts`,预期 84 个测试全绿。 - 需要你做的:无,席位落地。 ## What changes **The cap is resolved per authoring turn, by the card's tier** (`src/runner/driver.ts`). - `DEFAULT_AUTHOR_MAX_TURNS`, a record keyed by `ScenarioTier`, = `{ control: 80, starter: 300, delivery: 600 }`, beside `DEFAULT_AUTHOR_TIMEOUT_MIN`, and `DEFAULT_ASK_MAX_TURNS = 12`. - The driver is built once per CLI run (`makeDriver`) and knows no card, so `AuthorTask` gains a required `tier: ScenarioTier`. The live driver computes `--max-turns` as `--author-max-turns`, else `DEFAULT_AUTHOR_MAX_TURNS[task.tier]`. The tier is not part of the replay key (`stepKey` is unchanged), so existing recordings still replay. - `orchestrate.ts` passes `tier: card.tier` on all three authoring calls: the scratch build, the scratch iterate, and the template iterate. The card asks for "build and iterate turns", so one cap covers both. - Ask turns (questions and operations) take `--ask-max-turns`, else 12, on every tier. - `ClaudeCodeDriverOptions.maxTurnsAuthor` / `maxTurnsAsk` are renamed `authorMaxTurns` / `askMaxTurns`, matching `authorTimeoutMs` / `askTimeoutMs` and the flags. Nothing set them (`git grep` at `a61f61a4`: only their own declarations and the two `??` reads). **The CLI** (`src/runner/cli.ts`). - `--author-max-turns N` and `--ask-max-turns N` take a whole number above 0, and any other value is refused with the flag and the value named. `makeDriver(spec, limits)` threads them to `ClaudeCodeDriver` the same way `--author-timeout` is threaded. - A replay (`--driver recorded:F`) refuses all three live-driver limits, naming every one it was given. This matches the existing `--author-timeout` refusal: a replay re-reads each turn as it was recorded. - The driver line now ends `· author max turns: control 80, starter 300, delivery 600 · ask max turns: 12`. **The cap used is recorded.** - On the turn: `TurnProcess.maxTurns` (`meta.process`, next to `timeoutMs`) is the `--max-turns` the CLI process was given, on authoring and ask turns alike, so a recording carries it. - On the run record: `TurnSummary.maxTurns` (beside `numTurns`) in `cardRun`, and `scorecard.modes[].first.build.author` / `iterate.build.author` carry `numTurns` and `maxTurns`. `env.driver` (`DriverInfo`) carries `authorMaxTurns` (the per-tier table this run applies) and `askMaxTurns`. - On the scorecard: the `Build · author` and `Iterate · author` rows show the turns taken of the cap, for example `error_max_turns · 80/80 turns · 23m 34s`. A turn killed before any result line shows `?/80 turns`. A turn recorded without a cap shows the count alone, and a turn with neither shows neither (the existing rows are byte-identical). Unchanged, by ruling: a turn ending `error_max_turns` is still `not measured: author turn failed: error_max_turns`, and nothing runs after it. Landing: `packages/qa/scenarios/src/runner/{driver,cli,orchestrate,scorecard,transcript}.ts`, `test/runner/{units,orchestrate}.test.ts` and the package README, all inside the claim's file surface. `src/schema.ts` is read (`ScenarioTier`, `SCENARIO_TIERS`), not changed. ## Tests - `test/runner/units.test.ts`, inside the existing describe of the live driver against a stand-in `claude` script (never the real CLI). The stand-in gains an `ARGV` prompt that answers its own argument list, so the tests read the `--max-turns` value the CLI process was really started with: - **Tier default, no flag:** a control, starter and delivery author turn start the CLI with `--max-turns` 80, 300 and 600. An ask turn starts it with 12. Each turn's `meta.process.maxTurns` matches, and `info()` states the table. The literal table is pinned. - **The flags reach the CLI's argument list:** `parseArgs(['--author-max-turns', '450', '--ask-max-turns', '20'])` goes through `makeDriver('claude', …)`, the binary resolves on `PATH` like the `--author-timeout` test, and the CLI gets `--max-turns 450` on every tier and `--max-turns 20` on an ask turn. `0`, `-3`, `1.5` and `many` are each refused, naming the flag and the value. A replay refuses each flag and all three together, naming them. - The two replay-driver author tasks now name their tier. The required field caught both at typecheck. - `test/runner/orchestrate.test.ts`: - An `error_max_turns` build author turn is not measured as before. `build.author` carries `numTurns`/`maxTurns`, the author row reads `error_max_turns · 80/80 turns · 23m 34s`, and the JSON-serialised run record carries the cap in `cardRun` and `scorecard`. - An iterate turn killed before any result line reads `?/80 turns`. A turn with neither a count nor a cap keeps its old row. - A new top-level describe checks that every authoring turn of the control card (scratch build, scratch iterate, template iterate) carries `control` to the driver. In the reference-cycle describe, the starter card `requisition_to_receipt` carries `starter` on its build and iterate turns. Reverse verification: one mutation per behaviour, each through `node scripts/ablation-replace.mjs` in wrap mode. In every case the anchor went from 1 hit to 0, the blob changed, the restored blob equalled HEAD, and `git diff HEAD` was empty afterwards. The suites import `src/` by relative path, so no build sits between a mutation and its run. | Mutation | Expected | Observed | |:---|:---|:---| | `driver.ts`: the author cap ignores the task's tier (`'control'`) | tier-default test red | 1 failed / 83 passed | | `cli.ts`: `makeDriver` drops `authorMaxTurns` | flag test red | 1 failed / 83 passed | | `orchestrate.ts`: the scratch build passes `tier: 'control'` | starter-card test red | 1 failed / 83 passed | | `scorecard.ts`: the author row drops the turns text | both author-row tests red | 2 failed / 82 passed | | `cli.ts`: the replay forgets `--author-max-turns` | flag test red (the replay opened the file instead) | 1 failed / 83 passed | | `orchestrate.ts`: `summarize` drops `maxTurns` | both author-row tests red | 2 failed / 82 passed | ## Acceptance notes - **Defaults: the card's example is kept, and the measurements support it.** From the card's three `starter` readings (80 turns in 23m 34s, 15m 05s and 14m 48s), a turn takes 11–18 s, so the 60-minute `--author-timeout` default allows about 200–325 turns. A cap of 300 makes the turn cap and the wall clock bind at about the same point. The cap stops a turn that loops, and the timeout stops a slow one. `control` stays at 80: its builds finished in 17–21 turns. - **How the two caps relate on `delivery`:** no delivery card exists in the library at `a61f61a4` (9 cards: 6 `starter`, 3 `control`). At the starter pace, 600 turns would take about 110–180 minutes, past the 60-minute default, so under the default the clock ends a delivery turn first. The README and the constant's doc say a delivery run should raise `--author-timeout` too. The timeout itself is not tier-sized: the card does not ask for it, and the last PR on this package made it a flat constant on purpose. - **Ask turns: 12 on every tier, as before.** Neither the card nor its thread has an ask-turn `error_max_turns` reading, so there is no measured reason for a per-tier table. It is a single constant. - `--author-max-turns` sets one cap for every card in the run. A per-tier override was not asked for, so it is not added. - Not measured live: no real `claude` run. The cost ruling allows only a local Claude Code login, and this container has none. The flag-to-argv path is pinned against the stand-in CLI. The epic seat's re-run of the six `starter` cards is the card's own verification. - `test:repo` was not run locally. The one repo-level pin that reads this package, `packages/core/src/security/second-object-read-exposure.pin.test.ts`, lists `referenceTargetOf` callers under `src/score/`. This diff adds no such call and touches no file there. CI's `Test Core` runs it. - Changeset: none. `@objectstack/scenarios` is `private: true` and publishes nothing, so the PR carries `skip-changeset`. ## Verification Readings at head `fdac4b9b`, `objectstack-ai/objectstack`. The gate sweep, the package build and the package suite all ran after the final commit. <details><summary>Package: typecheck, build, suite</summary> ```text pnpm --filter @objectstack/scenarios typecheck -> exit 0 (tsc --noEmit; tsconfig includes test/**) pnpm --filter @objectstack/scenarios build -> exit 0; check-dts-emitted: 10/10 declared declaration file(s) present pnpm --filter @objectstack/scenarios test -> Test Files 23 passed (23) · Tests 431 passed (431) vitest run test/runner/units.test.ts test/runner/orchestrate.test.ts -> Tests 84 passed (84) built entries: require() of dist/{index,score,score-full,runner,export}.cjs ok; import() of dist/runner.js ok; runner.cjs DEFAULT_AUTHOR_MAX_TURNS = {"control":80,"starter":300,"delivery":600} · DEFAULT_ASK_MAX_TURNS = 12 ``` </details> <details><summary>Gates: dispatch-gates --commands (49) and --ran</summary> ```text dispatch-gates: gate list derived from the tree of 'objectstack-ai/objectstack' at commit fdac4b9 change set: 8 path(s) vs merge base a61f61a · changed lines 288 (+256 / -32) vs the human-merge threshold 3000: under 49 command(s) run, each exit code recorded before any pipe: 48 x exit 0, check:dual-build-cjs-loads exit 3 (PREREQUISITE NOT MET: it reads every package's dist/; it exited 0 when re-run after the sweep's own lanes had built them: 116 published require entry point(s) across 71 package(s) load) check:dts-closure / check:sourcemap-no-sources-content re-run after this package was built: exit 0 / exit 0 Run reconciliation — 49 derived, 49 run, 0 NOT-MEASURED, 0 UNRUN. ``` </details> --- _Generated by [Claude Code](https://claude.ai/code/session_012MmDsRzowAnXEmogPGSEKA)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent aa3e21c commit 5f00374

8 files changed

Lines changed: 256 additions & 32 deletions

File tree

‎packages/qa/scenarios/README.md‎

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,8 @@ pnpm --filter @objectstack/scenarios run-scenarios --only service_desk --modes s
9797
pnpm --filter @objectstack/scenarios run-scenarios --record /tmp/rec.json # also save every turn
9898
pnpm --filter @objectstack/scenarios run-scenarios --driver recorded:/tmp/rec.json # replay, no model
9999
pnpm --filter @objectstack/scenarios run-scenarios --author-timeout 90 # minutes per authoring turn (default 60)
100+
pnpm --filter @objectstack/scenarios run-scenarios --author-max-turns 400 # turns per authoring turn, every card (default by tier)
101+
pnpm --filter @objectstack/scenarios run-scenarios --ask-max-turns 20 # turns per question or operation turn (default 12)
100102
```
101103

102104
Per card and mode the runner writes `docs/qa/platform-checklist/runs/<date>-scenario-<card>.json`
@@ -110,14 +112,22 @@ Per card and mode the runner writes `docs/qa/platform-checklist/runs/<date>-scen
110112
- **Driver** (`driver.ts`): the local `claude` CLI on your own login (`claude -p --output-format stream-json`, the
111113
prompt written to its stdin: a build prompt carries the whole seed, and one argv string over 128 KiB makes `spawn`
112114
fail with `E2BIG` on Linux). It refuses to start when `ANTHROPIC_API_KEY` / `ANTHROPIC_AUTH_TOKEN` is set or the CLI
113-
is logged in with an API key (the cost ruling on cloud#3600), and the record states the driver, the login and the
114-
authoring timeout. Every turn records its process: elapsed time, and for a turn that did not finish its exit code or
115-
signal, whether the runner's timeout killed it, and the end of its stderr. `RecordedDriver` replays a recording.
115+
is logged in with an API key (the cost ruling on cloud#3600), and the record states the driver, the login, the
116+
authoring timeout and the turn caps. Every turn records its process: elapsed time, the turn cap it ran under
117+
(`--max-turns`), and for a turn that did not finish its exit code or signal, whether the runner's timeout killed it,
118+
and the end of its stderr. `RecordedDriver` replays a recording, and refuses the three live-driver limits below.
119+
- **Turn caps** (`driver.ts`): an authoring turn, build and iterate alike, may take `DEFAULT_AUTHOR_MAX_TURNS` by the
120+
card's `tier` — `control` 80, `starter` 300, `delivery` 600 — unless `--author-max-turns N` sets one cap for every
121+
card; a question or operation turn takes 12 unless `--ask-max-turns N` says otherwise. The cap guards a looping turn
122+
and `--author-timeout` a slow one, whichever comes first: the `starter` builds measured ran 11–18 s a turn, so the
123+
60-minute default holds 200–325 of them and both limits bind at about the same point. A `delivery` card's 600 turns
124+
outlast that default; raise `--author-timeout` with it.
116125
- **A build that did not finish is not measured** (`orchestrate.ts`): an authoring turn the timeout killed
117126
(`--author-timeout`, build and iterate alike) leaves its Build half **not measured: author timed out after N min**;
118127
one that ended with no result line or an error result (`error_max_turns`, `is_error`), or never ran, leaves it **not
119128
measured: author turn failed: …**. Never a degraded score, and nothing runs on it: no score, seed, Run or iterate. The
120-
scorecard's author rows show every authoring turn's `resultSubtype` and elapsed time.
129+
scorecard's author rows show every authoring turn's `resultSubtype`, the turns it took of its cap
130+
(`error_max_turns · 300/300 turns`) and its elapsed time.
121131
- **Seed** (`seed.ts`): rows go in as the admin, a lookup's target before its referrer. Where the card's declared
122132
lookups lead back to where they start — `department.manager → employee` with `employee.department → department`, or a
123133
self-lookup naming a later row — the seed loads in two passes: every row first with each lookup naming a row not in

‎packages/qa/scenarios/src/runner/cli.ts‎

Lines changed: 42 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,13 @@
1313
* --author-timeout MIN the live driver's budget per authoring turn, build and
1414
* iterate alike (default 60); a turn it kills leaves its
1515
* Build half not measured
16+
* --author-max-turns N the live driver's turn cap per authoring turn, build
17+
* and iterate alike, on every card (default by the card's
18+
* tier: control 80, starter 300, delivery 600); a turn
19+
* that reaches it (`error_max_turns`) leaves its Build
20+
* half not measured
21+
* --ask-max-turns N the live driver's turn cap per question or operation
22+
* turn (default 12)
1623
* --template F a verified build (dist/objectstack.json) for template mode
1724
* --no-iterate-run skip the Run half after the iterate (scratch mode)
1825
* --out DIR where records go (default docs/qa/platform-checklist/runs, git-ignored)
@@ -59,6 +66,10 @@ export interface CliArgs {
5966
iterateRun: boolean;
6067
/** Minutes per authoring turn (`--author-timeout`); the driver's default when absent. */
6168
authorTimeoutMin?: number;
69+
/** Turns per authoring turn on every card (`--author-max-turns`); the card tier's default when absent. */
70+
authorMaxTurns?: number;
71+
/** Turns per question or operation turn (`--ask-max-turns`); the driver's default when absent. */
72+
askMaxTurns?: number;
6273
out?: string;
6374
work?: string;
6475
createVersion?: string;
@@ -73,6 +84,12 @@ export function parseArgs(argv: readonly string[]): CliArgs {
7384
if (v === undefined) throw new Error(`${flag} needs a value`);
7485
return v;
7586
};
87+
const turns = () => {
88+
const raw = val();
89+
const n = Number(raw);
90+
if (!Number.isInteger(n) || n <= 0) throw new Error(`${flag} takes a whole number of turns above 0, not \`${raw}\``);
91+
return n;
92+
};
7693
if (flag === '--cards') a.cards = val();
7794
else if (flag === '--only') a.only = val().split(',');
7895
else if (flag === '--modes') {
@@ -89,6 +106,8 @@ export function parseArgs(argv: readonly string[]): CliArgs {
89106
if (!Number.isFinite(min) || min <= 0) throw new Error(`--author-timeout takes a number of minutes above 0, not \`${raw}\``);
90107
a.authorTimeoutMin = min;
91108
}
109+
else if (flag === '--author-max-turns') a.authorMaxTurns = turns();
110+
else if (flag === '--ask-max-turns') a.askMaxTurns = turns();
92111
else if (flag === '--out') a.out = val();
93112
else if (flag === '--work') a.work = val();
94113
else if (flag === '--create-version') a.createVersion = val();
@@ -97,10 +116,26 @@ export function parseArgs(argv: readonly string[]): CliArgs {
97116
return a;
98117
}
99118

100-
export function makeDriver(spec: string, opts: { authorTimeoutMin?: number } = {}): AgentDriver {
101-
if (spec === 'claude') return new ClaudeCodeDriver(opts.authorTimeoutMin !== undefined ? { authorTimeoutMs: opts.authorTimeoutMin * 60_000 } : {});
119+
/** The live driver's limits a run may set: `--author-timeout`, `--author-max-turns`, `--ask-max-turns`. */
120+
export type DriverLimits = Pick<CliArgs, 'authorTimeoutMin' | 'authorMaxTurns' | 'askMaxTurns'>;
121+
122+
const LIMIT_FLAGS: ReadonlyArray<readonly [keyof DriverLimits, string]> = [
123+
['authorTimeoutMin', '--author-timeout'],
124+
['authorMaxTurns', '--author-max-turns'],
125+
['askMaxTurns', '--ask-max-turns'],
126+
];
127+
128+
export function makeDriver(spec: string, opts: DriverLimits = {}): AgentDriver {
129+
if (spec === 'claude') {
130+
return new ClaudeCodeDriver({
131+
...(opts.authorTimeoutMin !== undefined ? { authorTimeoutMs: opts.authorTimeoutMin * 60_000 } : {}),
132+
...(opts.authorMaxTurns !== undefined ? { authorMaxTurns: opts.authorMaxTurns } : {}),
133+
...(opts.askMaxTurns !== undefined ? { askMaxTurns: opts.askMaxTurns } : {}),
134+
});
135+
}
102136
if (spec.startsWith('recorded:')) {
103-
if (opts.authorTimeoutMin !== undefined) throw new Error('--author-timeout applies to the live driver only: a replay re-reads each turn as it was recorded');
137+
const given = LIMIT_FLAGS.filter(([key]) => opts[key] !== undefined).map(([, flag]) => flag);
138+
if (given.length) throw new Error(`${given.join(', ')} ${given.length === 1 ? 'applies' : 'apply'} to the live driver only: a replay re-reads each turn as it was recorded`);
104139
return RecordedDriver.fromFile(spec.slice('recorded:'.length));
105140
}
106141
throw new Error(`unknown driver \`${spec}\` (claude, recorded:FILE)`);
@@ -124,12 +159,14 @@ export async function main(argv: readonly string[]): Promise<number> {
124159
return 2;
125160
}
126161

127-
const live = makeDriver(args.driver, args.authorTimeoutMin !== undefined ? { authorTimeoutMin: args.authorTimeoutMin } : {});
162+
const live = makeDriver(args.driver, args);
128163
const driver = args.record ? new RecordingDriver(live) : live;
129164
const info = driver.info();
130165
console.log(
131166
`driver: ${info.driver} ${info.version ?? ''} · login: ${info.authMethod ?? 'n/a'} · API-billed: no` +
132-
(info.authorTimeoutMs !== undefined ? ` · author timeout: ${minutesText(info.authorTimeoutMs)}` : ''),
167+
(info.authorTimeoutMs !== undefined ? ` · author timeout: ${minutesText(info.authorTimeoutMs)}` : '') +
168+
(info.authorMaxTurns ? ` · author max turns: ${Object.entries(info.authorMaxTurns).map(([tier, n]) => `${tier} ${n}`).join(', ')}` : '') +
169+
(info.askMaxTurns !== undefined ? ` · ask max turns: ${info.askMaxTurns}` : ''),
133170
);
134171

135172
const ops: ProjectOps = {

‎packages/qa/scenarios/src/runner/driver.ts‎

Lines changed: 50 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ import { mkdtempSync, mkdirSync, readdirSync, readFileSync, rmSync, writeFileSyn
2626
import { tmpdir } from 'node:os';
2727
import { dirname, join, relative, resolve, sep } from 'node:path';
2828

29+
import { SCENARIO_TIERS, type ScenarioTier } from '../schema.js';
2930
import { parseStreamJson, type ParsedTurn, type TurnProcess } from './transcript.js';
3031

3132
/** Which run step a turn belongs to; the replay key is built from it. */
@@ -46,6 +47,8 @@ export interface AuthorTask extends StepRef {
4647
/** The project directory the agent edits. */
4748
cwd: string;
4849
prompt: string;
50+
/** The card's tier: it sizes the live driver's turn cap ({@link DEFAULT_AUTHOR_MAX_TURNS}). Not part of the replay key. */
51+
tier: ScenarioTier;
4952
}
5053

5154
export interface AskTask extends StepRef {
@@ -71,6 +74,10 @@ export interface DriverInfo {
7174
apiBilled: false;
7275
/** The live driver's wall-clock budget per authoring turn (build and iterate), ms. */
7376
authorTimeoutMs?: number;
77+
/** The live driver's turn cap per authoring turn, by card tier (`--author-max-turns` sets every tier alike). */
78+
authorMaxTurns?: Record<ScenarioTier, number>;
79+
/** The live driver's turn cap per question or operation turn. */
80+
askMaxTurns?: number;
7481
}
7582

7683
export interface AgentDriver {
@@ -121,13 +128,29 @@ export function assertSubscriptionLogin(env: NodeJS.ProcessEnv, authStatus: unkn
121128
*/
122129
export const DEFAULT_AUTHOR_TIMEOUT_MIN = 60;
123130

131+
/**
132+
* The agentic turns one authoring turn (build or iterate) may take unless
133+
* `--author-max-turns` says otherwise, by the card's tier. A `control` card's
134+
* 2–3-object build finished in 17–21 turns under the old flat cap of 80, which
135+
* stays. Every `starter` build measured hit that 80 (`error_max_turns`) after
136+
* 15–24 minutes, 11–18 s a turn: the 60-minute author timeout holds 200–325
137+
* such turns, so 300 puts the turn cap and the wall clock at about the same
138+
* point. A `delivery` card (none in the library yet) gets twice that; at the
139+
* same pace 600 turns outlast the default timeout, so its run raises
140+
* `--author-timeout` too, or the clock ends the turn first.
141+
*/
142+
export const DEFAULT_AUTHOR_MAX_TURNS: Readonly<Record<ScenarioTier, number>> = { control: 80, starter: 300, delivery: 600 };
143+
144+
/** The agentic turns one question or operation turn may take unless `--ask-max-turns` says otherwise, every tier alike. */
145+
export const DEFAULT_ASK_MAX_TURNS = 12;
146+
124147
export interface ClaudeCodeDriverOptions {
125148
/** The CLI binary; default `claude`. */
126149
bin?: string;
127-
/** Agentic turns allowed for one question or operation; default 12. */
128-
maxTurnsAsk?: number;
129-
/** Agentic turns allowed for one authoring task; default 80. */
130-
maxTurnsAuthor?: number;
150+
/** Agentic turns allowed for one question or operation; default {@link DEFAULT_ASK_MAX_TURNS}. */
151+
askMaxTurns?: number;
152+
/** Agentic turns allowed for one authoring task, every tier alike; default by the card's tier, {@link DEFAULT_AUTHOR_MAX_TURNS}. */
153+
authorMaxTurns?: number;
131154
/** Wall-clock budget per authoring turn (build and iterate), ms; default {@link DEFAULT_AUTHOR_TIMEOUT_MIN}. */
132155
authorTimeoutMs?: number;
133156
/** Wall-clock budget per question or operation, ms; default 5 min. */
@@ -266,8 +289,24 @@ export class ClaudeCodeDriver implements AgentDriver {
266289
return this.opts.authorTimeoutMs ?? DEFAULT_AUTHOR_TIMEOUT_MIN * 60_000;
267290
}
268291

292+
private authorMaxTurns(tier: ScenarioTier): number {
293+
return this.opts.authorMaxTurns ?? DEFAULT_AUTHOR_MAX_TURNS[tier];
294+
}
295+
296+
private get askMaxTurns(): number {
297+
return this.opts.askMaxTurns ?? DEFAULT_ASK_MAX_TURNS;
298+
}
299+
269300
info(): DriverInfo {
270-
return { driver: 'claude-code-cli', version: this.version, authMethod: this.authMethod, apiBilled: false, authorTimeoutMs: this.authorTimeoutMs };
301+
return {
302+
driver: 'claude-code-cli',
303+
version: this.version,
304+
authMethod: this.authMethod,
305+
apiBilled: false,
306+
authorTimeoutMs: this.authorTimeoutMs,
307+
authorMaxTurns: Object.fromEntries(SCENARIO_TIERS.map((t) => [t, this.authorMaxTurns(t)])) as Record<ScenarioTier, number>,
308+
askMaxTurns: this.askMaxTurns,
309+
};
271310
}
272311

273312
private base(maxTurns: number): string[] {
@@ -280,16 +319,17 @@ export class ClaudeCodeDriver implements AgentDriver {
280319
const empty = join(mkdtempSync(join(tmpdir(), 'scenario-mcp-')), 'none.json');
281320
writeFileSync(empty, JSON.stringify({ mcpServers: {} }));
282321
try {
322+
const maxTurns = this.authorMaxTurns(task.tier);
283323
const args = [
284324
'-p',
285-
...this.base(this.opts.maxTurnsAuthor ?? 80),
325+
...this.base(maxTurns),
286326
'--mcp-config', empty, '--strict-mcp-config',
287327
'--permission-mode', 'acceptEdits',
288328
'--allowedTools', 'Bash', 'Read', 'Edit', 'Write', 'Glob', 'Grep', 'Skill',
289329
];
290330
const run = await runCli(this.bin, args, task.cwd, task.prompt, this.authorTimeoutMs);
291331
const turn = parseStreamJson(run.stdout);
292-
turn.meta.process = run.process;
332+
turn.meta.process = { ...run.process, maxTurns };
293333
if (turn.meta.resultSubtype === undefined) turn.meta.resultSubtype = noResult(run.process);
294334
return { turn, files: snapshotSources(task.cwd) };
295335
} finally {
@@ -308,10 +348,11 @@ export class ClaudeCodeDriver implements AgentDriver {
308348
{ mode: 0o600 },
309349
);
310350
try {
311-
const args = ['-p', ...this.base(this.opts.maxTurnsAsk ?? 12), '--mcp-config', cfg, '--strict-mcp-config', '--allowedTools', 'mcp__app'];
351+
const maxTurns = this.askMaxTurns;
352+
const args = ['-p', ...this.base(maxTurns), '--mcp-config', cfg, '--strict-mcp-config', '--allowedTools', 'mcp__app'];
312353
const run = await runCli(this.bin, args, dir, task.prompt, this.opts.askTimeoutMs ?? 5 * 60_000);
313354
const turn = parseStreamJson(run.stdout);
314-
turn.meta.process = run.process;
355+
turn.meta.process = { ...run.process, maxTurns };
315356
if (turn.meta.resultSubtype === undefined) turn.meta.resultSubtype = noResult(run.process);
316357
return turn;
317358
} finally {

‎packages/qa/scenarios/src/runner/orchestrate.ts‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,8 @@ export interface TurnSummary {
9191
/** The result line said `is_error: true` (a usage limit answers `success` with it). */
9292
isError?: true;
9393
numTurns?: number;
94+
/** The turn cap the CLI process ran under (`meta.process`): `numTurns` of `maxTurns`. */
95+
maxTurns?: number;
9496
toolCalls: number;
9597
model?: string;
9698
/** Wall-clock time of the CLI process, when the driver ran one (`meta.process`). */
@@ -198,6 +200,7 @@ const summarize = (t: ParsedTurn): TurnSummary => {
198200
...(t.meta.resultSubtype ? { resultSubtype: t.meta.resultSubtype } : {}),
199201
...(t.meta.isError === true ? { isError: true as const } : {}),
200202
...(t.meta.numTurns !== undefined ? { numTurns: t.meta.numTurns } : {}),
203+
...(p?.maxTurns !== undefined ? { maxTurns: p.maxTurns } : {}),
201204
...(t.meta.model ? { model: t.meta.model } : {}),
202205
toolCalls: t.toolCalls.length,
203206
...(p ? { elapsedMs: p.elapsedMs } : {}),
@@ -705,14 +708,14 @@ async function scratchMode(original: LoadedScenario, driver: AgentDriver, ops: P
705708
const labels = citationLabels(original);
706709
const card = scenario.card;
707710
const ref = { card: card.id, mode: 'scratch' as const };
708-
mode.build = await buildHalf(scenario, dir, ops, async () => (await driver.author({ ...ref, phase: 'build', step: 'author', cwd: dir, prompt: buildPrompt(card) })).turn, scenario);
711+
mode.build = await buildHalf(scenario, dir, ops, async () => (await driver.author({ ...ref, phase: 'build', step: 'author', cwd: dir, prompt: buildPrompt(card), tier: card.tier })).turn, scenario);
709712
// A build whose authoring turn did not finish is not measured: no seed, no Run, no iterate on it.
710713
if (mode.build.notMeasured) return { mode };
711714
const builtArtifact = ops.readArtifact(dir);
712715
if (builtArtifact !== undefined) mode.run = await withApp(ops, dir, mode.findings, (app) => runHalf(scenario, app, driver, { ...ref, phase: 'run' }, labels, builtArtifact));
713716

714717
const iterated = iteratedScenario(scenario);
715-
const build = await buildHalf(iterated, dir, ops, async () => (await driver.author({ ...ref, phase: 'iterate', step: 'author', cwd: dir, prompt: iteratePrompt(card) })).turn, iterated);
718+
const build = await buildHalf(iterated, dir, ops, async () => (await driver.author({ ...ref, phase: 'iterate', step: 'author', cwd: dir, prompt: iteratePrompt(card), tier: card.tier })).turn, iterated);
716719
if (build.notMeasured) {
717720
mode.iterate = { build };
718721
return { mode, ...(buildPassed(mode.build) ? { verified: builtArtifact } : {}) };
@@ -767,7 +770,7 @@ async function templateMode(original: LoadedScenario, driver: AgentDriver, ops:
767770
}
768771
const before = await app.metadataAsArtifact(collections);
769772
const build: BuildHalf = { commands: [], score: null };
770-
await authorInto(build, async () => (await driver.author({ card: card.id, mode: 'template', phase: 'iterate', step: 'author', cwd: dir, prompt: templateIteratePrompt(card, { baseUrl: booted.url, adminEmail: opts.admin.email, adminPassword: opts.admin.password }) })).turn);
773+
await authorInto(build, async () => (await driver.author({ card: card.id, mode: 'template', phase: 'iterate', step: 'author', cwd: dir, prompt: templateIteratePrompt(card, { baseUrl: booted.url, adminEmail: opts.admin.email, adminPassword: opts.admin.password }), tier: card.tier })).turn);
771774
if (build.notMeasured) {
772775
mode.iterate = { build };
773776
return;

0 commit comments

Comments
 (0)