Skip to content

Commit 0d0c377

Browse files
feat(scenarios): run-scenarios --model / --author-model / --ask-model, with the served model recorded per turn (#31204)
<!-- claude-projects-attribution: {"github_login":"os-project-manager"} --> _Requested by **Leehom** · [project thread](https://claude.ai/code/project/chan_01TenzsRopZj4jMbvjX8uz1X?thread=cmsg_01TenzsRopZj4jMbvjX8uz1XJdp59LSRHc3ZMoadGaTvT9)_ Fixes #31160 Clause-②: no Metadata-protocol: none ## 维护者速读 - 改了什么:场景跑批器 `run-scenarios` 新增 `--model`(每一轮都用)、`--author-model`(建造/迭代轮)、`--ask-model`(人设问答与操作轮),原样作为 `claude --model` 传给本地 Claude Code。每一轮在运行记录里同时写明「要的模型」和「实际服务的模型」(读自流的 init 行),计分卡的作者行写出实际模型,新增 `Run · model` 行按实际模型统计人设轮次,不一致处标 `⚠`。 - 为什么改:2026-11-06/07 的所有轮次都跑在登录默认模型上,无法固定或对比模型;维护者同意此建议并立卡。 - 风险与回滚:只动私有包 `@objectstack/scenarios`,不发布任何东西;不传新参数时命令行与之前逐字节相同。回滚即还原本 PR。 - 先看:`packages/qa/scenarios/src/runner/driver.ts`(按轮次类型传 `--model`、`servedAsRequested` 判不一致)、`src/runner/scorecard.ts`(`Run · model` 行与作者行)、`src/runner/cli.ts`(三个参数、回放拒收)。 - 两分钟验证:`pnpm --filter @objectstack/scenarios exec vitest run --maxWorkers=2 test/runner/units.test.ts test/runner/orchestrate.test.ts -t model`,预期全部通过(含参数到达 CLI 参数表、记录带实际模型、不一致被标出)。 - 需要你做的:无,席位落地。 ## What changes - **Three flags on `run-scenarios`** (`runner/cli.ts`): `--model M` for every live turn, `--author-model M` for the authoring turns (build and iterate) and `--ask-model M` for the question and operation turns, each of the last two in place of `--model`. A value is an alias (`sonnet`) or a name (`claude-sonnet-4-5`), passed through as `claude --model` takes it; an empty value or one that starts with `-` (a forgotten value swallowing the next flag) is refused. With none, no turn passes `--model` and the login's default serves, as today. - **Threaded to the driver** (`runner/driver.ts`): `ClaudeCodeDriver` already took `model`; it now also takes `authorModel` / `askModel` and passes the one for the kind of turn. `DriverInfo` states `authorModel` / `askModel`, so the record's `env.driver` and the run's first console line say what was asked for. `RecordedDriver` refuses the three model flags as it refuses the live-driver limits (`makeDriver`): a replay re-reads each turn as it was recorded. - **The model asked for, beside the model served, per turn**: the live driver stamps each turn's `meta.process.requestedModel` (the `--model` it was given; `runner/transcript.ts`), next to the stream's own `meta.model` it already parsed. The turn summary in the record (`TurnSummary`, `runner/orchestrate.ts`) carries `model` (served), `requestedModel` and `modelMismatch: true` when the served name is not the requested one. A recording keeps both on every turn, so a replay re-reads them as the run saw them. - **Mismatch rule** (`servedAsRequested`, `runner/driver.ts`): the CLI takes an alias and the stream names the model it resolved to, so a turn is served as asked when the served name holds the requested one as a whole run of its dash-separated parts (`sonnet` and `claude-sonnet-4-5` in `claude-sonnet-4-5-20250929`; never `opus`, never a fragment like `son`); a context-size suffix (`[1m]`) is dropped. Any other answer, including an alias that names no model family (`default`, `opusplan`), is shown as a mismatch beside both names, for the reader to judge. - **The scorecard** (`runner/scorecard.ts`): the author rows end with the served model (`success · 17/80 turns · 3m 02s · claude-opus-5-5`, `⚠ asked for sonnet` on a mismatch, `model not reported (asked for sonnet)` when the stream named none); new rows `Run · model` and `Iterate · Run · model` count the half's question and operation turns per served model (`claude-sonnet-4-5-20250929 ×12, claude-opus-5-5 ×2 ⚠ 2 not sonnet`); `—` when no turn named a model and none was asked for. `HalfScores.models` (`ModelTally`) carries the same in the record's `scorecard`. - README: the two command lines and a **Model** paragraph. ## Why The card's three asks, verbatim: "`--model ALIAS` for every turn, plus optional `--author-model` / `--ask-model` to set the build and persona turns apart. Thread them to the driver."; "The run record and the scorecard state the model actually served per turn. The stream's `meta.model` is already parsed; show any mismatch with the requested model."; "Tests: the flag reaches the CLI argument list; the record carries the served model." Every recorded turn of the 2026-11-06/07 runs used the login's default model and nothing let a run pin or compare one. ## Tests - **The flag reaches the CLI's argument list** (`test/runner/units.test.ts`, the stand-in `claude` block, 2 new cases). The stand-in now serves a model the way the real CLI's init line names it (`--model X` → `claude-X-4-5-20250929`; `retired` falls back to `claude-opus-5-5`). Pins: `parseArgs(['--model', 'sonnet'])` → `makeDriver` → both an authoring and an ask turn's argv carry `--model sonnet`, each turn's `meta.process.requestedModel` is `sonnet` and `meta.model` is the served `claude-sonnet-4-5-20250929`; `--author-model opus --ask-model haiku` beside `--model sonnet` splits them; `--author-model` alone leaves the ask turn with no `--model` and nothing requested; no flag passes no `--model`; a fallback keeps both names. Negative pins: a missing, empty, blank or flag-shaped value is refused per flag; a replay refuses `--model`, and names `--author-model, --ask-model` together. - **The mismatch rule** (`units.test.ts`, new describe, a 12-row table): alias, name, dated name, case, `[1m]` suffix, other family, wrong version, a fragment of a part, `default`, empty. - **The record carries the served model** (`test/runner/orchestrate.test.ts`, new describe, 2 cases): the real crm card through `runCard`, its driver wrapped in a `RecordingDriver` and a stamp that asks for `opus` on authoring turns and `sonnet` on persona turns, with one question's turns served `claude-opus-5-5`. Pins: every turn summary carries `model` and `requestedModel`, exactly that question's turns carry `modelMismatch`; `first.models` counts the served models; the rendered table's author row names `claude-opus-4-1-20250805` and both `Run · model` rows show `⚠ N not sonnet`; the written record (`JSON` round trip) holds `env.driver.authorModel/askModel`, the mismatched turn and the tally; the saved recording replayed through `RecordedDriver.fromFile` (the `--driver recorded:F` path) answers that turn with both names. A run whose turns name no model and ask for none keeps `| Run · model | — |` and the author rows unchanged. <details><summary>Ablations (each restored: blob == HEAD, git diff HEAD empty)</summary> All four run through `scripts/ablation-replace.mjs` (anchor x1 → x0, replacement x0 → x1) on head `4798cbf1`; the suites import `src/` directly, so no dist leg. - Ablation 1, the mismatch marking disarmed (`&& !servedAsRequested(…)` in `orchestrate.ts` `summarize` replaced by an always-false `String("ABL1").length < 0`): `orchestrate.test.ts -t served` → `1 failed | 2 passed` (the served-model record case). - Ablation 2, the replay refusal of the model flags disarmed (`LIVE_FLAGS.filter` → `LIVE_FLAGS.slice(0, 3 + …).filter` in `cli.ts`): `units.test.ts -t model` → `1 failed | 6 passed` (the refusal case). - Ablation 3, the empty / flag-shaped value refusal disarmed (`raw.trim() === '' || raw.startsWith('-')` → `raw === String("ABL3")`): `units.test.ts -t model` → `1 failed | 6 passed` (the refusal case). - Ablation 4, the `--model` threading disarmed (`if (model) a.push` → `if (model && String("ABL4").length < 0) a.push` in `driver.ts` `base`): `units.test.ts -t model` → `1 failed | 6 passed` (the argument-list case). </details> ## Acceptance notes - **Served is read off the stream, never inferred from the request.** A turn whose stream named no model (no init line: a spawn that failed) shows `model not reported`, and is never counted as a mismatch: nothing was served to compare. - **No model allow-list.** The runner does not validate an alias against a list: `claude` answers an unknown model itself, and that turn is recorded as it failed. A list here would go stale with every model release. - **Recordings made before this change** carry no `requestedModel`; their turns show the served model and no mismatch. - **Serial with PR #31188** (`--packages`, same `cli.ts` / `scorecard.ts`): it was still open when this PR opened, so `origin/main` was merged once before it; the seat's order has this branch merge `origin/main` again after #31188 lands. #31159 (`--driver external`) follows on the same files; this change stays local to the flags, the driver options and the record. - `@objectstack/scenarios` is `private: true` and nothing else moved: `skip-changeset`. ## Verification **Current head `45ceeaff`** (second `origin/main` merge, at `065bba72`, carrying #31188; plain merge, no rebase). The only conflict was the README command block, and all three new lines were kept. Inside the merge, the new `orchestrate.test.ts` `runRecord` call passes `packages: PUBLISHED` because #31188 made `RunRecordMeta.packages` required. On this head: 25 files / 516 tests pass, typecheck and scoped eslint exit 0, and `dispatch-gates --ran` shows 49 derived, 49 run, 0 NOT MEASURED (`check:dual-build-cjs-loads` now measured). Ablations were not rerun because the merge touched none of the ablated lines. The rounds below are on the earlier head. All on head `0b43e745` (the final commit: the work at `4798cbf1` plus one merge of `origin/main` at `8acbfab9`), in worktree `objectstack-issue-31160`, base `620b6e6d`. `origin/main` has since moved (#31185 among others, which touches `packages/qa/scenarios` prompts and tests); `git merge-tree` against it is clean. - Package suite, under the verify lock after `pnpm --filter "@objectstack/scenarios..." build` on the merged head: `pnpm --filter @objectstack/scenarios test` → `Test Files 23 passed (23)`, `Tests 460 passed (460)`; `pnpm --filter @objectstack/scenarios typecheck` (`tsc --noEmit`, test files included: `--listFiles` lists `test/runner/units.test.ts`) exit 0; `VERDICT command-exit 0`. - Lint scoped to the package: `eslint packages/qa/scenarios --no-inline-config` exit 0. - Derived gates: `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` → 49 commands, all run on this head; 48 `exit 0`; `pnpm check:dual-build-cjs-loads` exit 3, PREREQUISITE NOT MET (it reads every package's `dist/`, and only the scenarios closure was built here) — NOT MEASURED, left to CI. `--ran` reconciliation: `49 derived famil(ies) accounted for — 48 run, 1 NOT-MEASURED`. - Changeset: none. The diff touches only the private `@objectstack/scenarios` (`"private": true`): `skip-changeset`. - Dogfood battery: n/a (no `!` in the title, no `(narrowing)` arm). - No live model run (paused by the maintainer): the flag-to-argv path is measured against a stand-in `claude`, and the replay path through `RecordedDriver.fromFile` on a recording the test writes. <details><summary>The 49 derived gates, as run</summary> ```text node scripts/check-closing-keyword-parity.mjs :: exit 0 node scripts/check-closing-keyword-parity.mjs --self-test :: exit 0 node scripts/check-comment-mask-adoption.mjs :: exit 0 node scripts/check-comment-mask-adoption.mjs --self-test :: exit 0 node scripts/check-comment-mask-corpus.mjs :: exit 0 node scripts/check-dts-emitted.mjs --self-test :: exit 0 node scripts/check-issue-citations.mjs :: exit 0 node scripts/check-plugin-teardown-shape.mjs :: exit 0 node scripts/check-plugin-teardown-shape.mjs --self-test :: exit 0 node scripts/ci/assert-turbo-remote-cache.mjs --self-test :: exit 0 node scripts/ci/plan-test-core-shards.mjs --self-test :: exit 0 node scripts/ci/prune-turbo-cache.mjs --self-test :: exit 0 node scripts/docs-audit/check-affected-docs.mjs :: exit 0 node scripts/docs-audit/check-drift-comment.mjs :: exit 0 pnpm --filter @objectstack/spec run check:duration-unit-keys :: exit 0 pnpm check:ci-filter-parity :: exit 0 pnpm check:closed-ceremony-text :: exit 0 pnpm check:cross-package-test-inputs :: exit 0 pnpm check:doc-authoring :: exit 0 pnpm check:driver-memory-census :: exit 0 pnpm check:dts-closure :: exit 0 pnpm check:dual-build-cjs-loads :: exit 3 pnpm check:engine-double-contract :: exit 0 pnpm check:gitlink-declared :: exit 0 pnpm check:issue-citations :: exit 0 pnpm check:keyed-text-bounds :: exit 0 pnpm check:lean-entry-closure :: exit 0 pnpm check:logger-receiver-detach :: exit 0 pnpm check:nul-bytes :: exit 0 pnpm check:objectql-double-limit :: exit 0 pnpm check:org-identifier :: exit 0 pnpm check:page-declaration-shape :: exit 0 pnpm check:platform-object-tenancy-census :: exit 0 pnpm check:published-files :: exit 0 pnpm check:query-options-erasure :: exit 0 pnpm check:refd-timer-probe :: exit 0 pnpm check:registry-log-declared :: exit 0 pnpm check:rest-log-spy-declared :: exit 0 pnpm check:slot-lookup :: exit 0 pnpm check:sourcemap-no-sources-content :: exit 0 pnpm check:system-context-census :: exit 0 pnpm check:test-source-alias :: exit 0 pnpm check:tier-file-adoption :: exit 0 pnpm check:type-check-coverage :: exit 0 pnpm check:type-check-debt :: exit 0 pnpm check:undeclared-dep-imports :: exit 0 pnpm check:vouch-for-reference-callers :: exit 0 pnpm check:watch-hint-literal :: exit 0 pnpm check:where-matcher :: exit 0 ``` </details> ## Human Minimum - [x] 1. Schemas start as Zod; `packages/spec` holds no runtime logic. (Not touched.) - [x] 2. Every name I added uses the casing for its kind. - [x] 3. No formatter ran; `pnpm lint` passes (scoped to the package). - [x] 4. One issue, one branch off an up-to-date `main`, one pull request. - [x] 5. A change that publishes carries a changeset, and both declaration lines are filled in here and in the changeset. (Publishes nothing: `skip-changeset`.) - [x] 6. `packages/spec` touched: its generated artifacts are regenerated and committed. (Not touched.) - [x] 7. No hand edits to `content/docs/references/`, `packages/*/CHANGELOG.md` or `content/docs/releases/`. - [x] 8. Title, body and comments are in English. - [x] The *Before Submitting* list in CONTRIBUTING.md is done. <!-- signed: domain:devx#3 · os-project-manager · session_01FEkQ5njkh3xbcDN6EtSfw7 --> Co-authored-by: Claude <noreply@anthropic.com>
1 parent b065fad commit 0d0c377

8 files changed

Lines changed: 353 additions & 20 deletions

File tree

‎packages/qa/scenarios/README.md‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -139,6 +139,8 @@ pnpm --filter @objectstack/scenarios run-scenarios --driver recorded:/tmp/rec.js
139139
pnpm --filter @objectstack/scenarios run-scenarios --author-timeout 90 # minutes per authoring turn (default 60)
140140
pnpm --filter @objectstack/scenarios run-scenarios --author-max-turns 400 # turns per authoring turn, every card (default by tier)
141141
pnpm --filter @objectstack/scenarios run-scenarios --ask-max-turns 20 # turns per question or operation turn (default 12)
142+
pnpm --filter @objectstack/scenarios run-scenarios --model sonnet # the model every turn asks for (default: the login's own)
143+
pnpm --filter @objectstack/scenarios run-scenarios --author-model opus --ask-model haiku # build and persona turns apart
142144
pnpm --filter @objectstack/scenarios run-scenarios --packages workspace # on this checkout's unreleased packages (default: published)
143145
```
144146

@@ -167,7 +169,15 @@ Per card and mode the runner writes `docs/qa/platform-checklist/runs/<date>-scen
167169
is logged in with an API key (the cost ruling on cloud#3600), and the record states the driver, the login, the
168170
authoring timeout and the turn caps. Every turn records its process: elapsed time, the turn cap it ran under
169171
(`--max-turns`), and for a turn that did not finish its exit code or signal, whether the runner's timeout killed it,
170-
and the end of its stderr. `RecordedDriver` replays a recording, and refuses the three live-driver limits below.
172+
and the end of its stderr. `RecordedDriver` replays a recording, and refuses the live-driver limits and models below.
173+
- **Model** (`driver.ts`): `--model M` passes `claude --model M` on every turn, an alias (`sonnet`) or a name
174+
(`claude-sonnet-4-5`); `--author-model` sets the authoring turns (build and iterate) and `--ask-model` the question
175+
and operation turns apart, each in place of `--model`. With none, a turn takes the login's default model. Every turn
176+
records the model it asked for (`requestedModel`) and the one it was served, read off the stream's init line
177+
(`model`); a served name that does not hold the requested one as whole dash-separated parts (`sonnet` is in
178+
`claude-sonnet-4-5-20250929`, `opus` is not) is a mismatch (`modelMismatch`). The scorecard's author rows name the
179+
served model (`⚠ asked for sonnet` on a mismatch), and its `Run · model` rows count the persona turns per served
180+
model (`claude-opus-5-5 ×9 ⚠ 9 not sonnet`).
171181
- **Turn caps** (`driver.ts`): an authoring turn, build and iterate alike, may take `DEFAULT_AUTHOR_MAX_TURNS` by the
172182
card's `tier` — `control` 80, `starter` 300, `delivery` 600 — unless `--author-max-turns N` sets one cap for every
173183
card; a question or operation turn takes 12 unless `--ask-max-turns N` says otherwise. The cap guards a looping turn

‎packages/qa/scenarios/src/runner/cli.ts‎

Lines changed: 37 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,14 @@
2222
* half not measured
2323
* --ask-max-turns N the live driver's turn cap per question or operation
2424
* turn (default 12)
25+
* --model M the model every live turn asks for, an alias (`sonnet`)
26+
* or a name (`claude-sonnet-4-5`), as `claude --model`
27+
* takes it (default: the login's own)
28+
* --author-model M the model for authoring turns, build and iterate, in
29+
* place of --model
30+
* --ask-model M the model for question and operation turns, in place
31+
* of --model; the record and the scorecard state the
32+
* model each turn was served, and mark a mismatch
2533
* --template F a verified build (dist/objectstack.json) for template mode
2634
* --no-iterate-run skip the Run half after the iterate (scratch mode)
2735
* --out DIR where records go (default docs/qa/platform-checklist/runs, git-ignored)
@@ -77,6 +85,12 @@ export interface CliArgs {
7785
authorMaxTurns?: number;
7886
/** Turns per question or operation turn (`--ask-max-turns`); the driver's default when absent. */
7987
askMaxTurns?: number;
88+
/** The model every live turn asks for (`--model`); the login's default when absent. */
89+
model?: string;
90+
/** The model an authoring turn asks for (`--author-model`), in place of `model`. */
91+
authorModel?: string;
92+
/** The model a question or operation turn asks for (`--ask-model`), in place of `model`. */
93+
askModel?: string;
8094
out?: string;
8195
work?: string;
8296
createVersion?: string;
@@ -99,6 +113,11 @@ export function parseArgs(argv: readonly string[]): CliArgs {
99113
if (!Number.isInteger(n) || n <= 0) throw new Error(`${flag} takes a whole number of turns above 0, not \`${raw}\``);
100114
return n;
101115
};
116+
const model = () => {
117+
const raw = val();
118+
if (raw.trim() === '' || raw.startsWith('-')) throw new Error(`${flag} takes a model alias or name (sonnet, opus, claude-sonnet-4-5), not \`${raw}\``);
119+
return raw;
120+
};
102121
if (flag === '--cards') a.cards = val();
103122
else if (flag === '--only') a.only = val().split(',');
104123
else if (flag === '--modes') {
@@ -117,6 +136,9 @@ export function parseArgs(argv: readonly string[]): CliArgs {
117136
}
118137
else if (flag === '--author-max-turns') a.authorMaxTurns = turns();
119138
else if (flag === '--ask-max-turns') a.askMaxTurns = turns();
139+
else if (flag === '--model') a.model = model();
140+
else if (flag === '--author-model') a.authorModel = model();
141+
else if (flag === '--ask-model') a.askModel = model();
120142
else if (flag === '--out') a.out = val();
121143
else if (flag === '--work') a.work = val();
122144
else if (flag === '--create-version') a.createVersion = val();
@@ -135,22 +157,31 @@ export function parseArgs(argv: readonly string[]): CliArgs {
135157
/** The live driver's limits a run may set: `--author-timeout`, `--author-max-turns`, `--ask-max-turns`. */
136158
export type DriverLimits = Pick<CliArgs, 'authorTimeoutMin' | 'authorMaxTurns' | 'askMaxTurns'>;
137159

138-
const LIMIT_FLAGS: ReadonlyArray<readonly [keyof DriverLimits, string]> = [
160+
/** The models a run may ask the live driver for: `--model`, `--author-model`, `--ask-model`. */
161+
export type DriverModels = Pick<CliArgs, 'model' | 'authorModel' | 'askModel'>;
162+
163+
const LIVE_FLAGS: ReadonlyArray<readonly [keyof (DriverLimits & DriverModels), string]> = [
139164
['authorTimeoutMin', '--author-timeout'],
140165
['authorMaxTurns', '--author-max-turns'],
141166
['askMaxTurns', '--ask-max-turns'],
167+
['model', '--model'],
168+
['authorModel', '--author-model'],
169+
['askModel', '--ask-model'],
142170
];
143171

144-
export function makeDriver(spec: string, opts: DriverLimits = {}): AgentDriver {
172+
export function makeDriver(spec: string, opts: DriverLimits & DriverModels = {}): AgentDriver {
145173
if (spec === 'claude') {
146174
return new ClaudeCodeDriver({
147175
...(opts.authorTimeoutMin !== undefined ? { authorTimeoutMs: opts.authorTimeoutMin * 60_000 } : {}),
148176
...(opts.authorMaxTurns !== undefined ? { authorMaxTurns: opts.authorMaxTurns } : {}),
149177
...(opts.askMaxTurns !== undefined ? { askMaxTurns: opts.askMaxTurns } : {}),
178+
...(opts.model !== undefined ? { model: opts.model } : {}),
179+
...(opts.authorModel !== undefined ? { authorModel: opts.authorModel } : {}),
180+
...(opts.askModel !== undefined ? { askModel: opts.askModel } : {}),
150181
});
151182
}
152183
if (spec.startsWith('recorded:')) {
153-
const given = LIMIT_FLAGS.filter(([key]) => opts[key] !== undefined).map(([, flag]) => flag);
184+
const given = LIVE_FLAGS.filter(([key]) => opts[key] !== undefined).map(([, flag]) => flag);
154185
if (given.length) throw new Error(`${given.join(', ')} ${given.length === 1 ? 'applies' : 'apply'} to the live driver only: a replay re-reads each turn as it was recorded`);
155186
return RecordedDriver.fromFile(spec.slice('recorded:'.length));
156187
}
@@ -185,7 +216,9 @@ export async function main(argv: readonly string[]): Promise<number> {
185216
`driver: ${info.driver} ${info.version ?? ''} · login: ${info.authMethod ?? 'n/a'} · API-billed: no` +
186217
(info.authorTimeoutMs !== undefined ? ` · author timeout: ${minutesText(info.authorTimeoutMs)}` : '') +
187218
(info.authorMaxTurns ? ` · author max turns: ${Object.entries(info.authorMaxTurns).map(([tier, n]) => `${tier} ${n}`).join(', ')}` : '') +
188-
(info.askMaxTurns !== undefined ? ` · ask max turns: ${info.askMaxTurns}` : ''),
219+
(info.askMaxTurns !== undefined ? ` · ask max turns: ${info.askMaxTurns}` : '') +
220+
(info.authorModel ? ` · author model: ${info.authorModel}` : '') +
221+
(info.askModel ? ` · ask model: ${info.askModel}` : ''),
189222
);
190223

191224
// Built and packed once, before the first card: every scaffold of this run installs the same tarballs.

‎packages/qa/scenarios/src/runner/driver.ts‎

Lines changed: 50 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,10 @@ export interface DriverInfo {
7878
authorMaxTurns?: Record<ScenarioTier, number>;
7979
/** The live driver's turn cap per question or operation turn. */
8080
askMaxTurns?: number;
81+
/** The model the live driver asks for on an authoring turn (`--model` / `--author-model`); absent ⇒ the login's default. */
82+
authorModel?: string;
83+
/** The model the live driver asks for on a question or operation turn (`--model` / `--ask-model`); absent ⇒ the login's default. */
84+
askModel?: string;
8185
}
8286

8387
export interface AgentDriver {
@@ -155,8 +159,35 @@ export interface ClaudeCodeDriverOptions {
155159
authorTimeoutMs?: number;
156160
/** Wall-clock budget per question or operation, ms; default 5 min. */
157161
askTimeoutMs?: number;
158-
/** Optional model alias passed as `--model` (the login's default otherwise). */
162+
/** The model passed as `--model` on every turn (an alias such as `sonnet`, or a full name); the login's default otherwise. */
159163
model?: string;
164+
/** The model for an authoring turn (build and iterate), in place of {@link model}. */
165+
authorModel?: string;
166+
/** The model for a question or operation turn, in place of {@link model}. */
167+
askModel?: string;
168+
}
169+
170+
/** `claude-sonnet-4-5-20250929[1m]` → `['claude', 'sonnet', '4', '5', '20250929']`: the name's parts, the context-size suffix dropped. */
171+
const modelParts = (name: string): string[] => name.toLowerCase().replace(/\[[^\]]*\]$/, '').split('-').filter(Boolean);
172+
173+
/**
174+
* Was the turn served the model it asked for? The CLI takes an alias
175+
* (`sonnet`) or a name (`claude-opus-4-1`) and the stream's init line names
176+
* the model it resolved to (`claude-sonnet-4-5-20250929`), so a turn is served
177+
* as asked when the served name holds the requested one as a whole run of its
178+
* dash-separated parts — `sonnet` and `claude-sonnet-4-5` in
179+
* `claude-sonnet-4-5-20250929`, never `opus` in it. Any other answer, an alias
180+
* that names no model family (`default`, `opusplan`) included, is a mismatch the
181+
* scorecard shows beside both names, for the reader to judge.
182+
*/
183+
export function servedAsRequested(requested: string, served: string): boolean {
184+
const want = modelParts(requested);
185+
const got = modelParts(served);
186+
if (want.length === 0) return false;
187+
for (let i = 0; i + want.length <= got.length; i++) {
188+
if (want.every((part, j) => got[i + j] === part)) return true;
189+
}
190+
return false;
160191
}
161192

162193
interface CliRun {
@@ -297,6 +328,14 @@ export class ClaudeCodeDriver implements AgentDriver {
297328
return this.opts.askMaxTurns ?? DEFAULT_ASK_MAX_TURNS;
298329
}
299330

331+
private get authorModel(): string | undefined {
332+
return this.opts.authorModel ?? this.opts.model;
333+
}
334+
335+
private get askModel(): string | undefined {
336+
return this.opts.askModel ?? this.opts.model;
337+
}
338+
300339
info(): DriverInfo {
301340
return {
302341
driver: 'claude-code-cli',
@@ -306,12 +345,14 @@ export class ClaudeCodeDriver implements AgentDriver {
306345
authorTimeoutMs: this.authorTimeoutMs,
307346
authorMaxTurns: Object.fromEntries(SCENARIO_TIERS.map((t) => [t, this.authorMaxTurns(t)])) as Record<ScenarioTier, number>,
308347
askMaxTurns: this.askMaxTurns,
348+
...(this.authorModel ? { authorModel: this.authorModel } : {}),
349+
...(this.askModel ? { askModel: this.askModel } : {}),
309350
};
310351
}
311352

312-
private base(maxTurns: number): string[] {
353+
private base(maxTurns: number, model: string | undefined): string[] {
313354
const a = ['--output-format', 'stream-json', '--verbose', '--max-turns', String(maxTurns)];
314-
if (this.opts.model) a.push('--model', this.opts.model);
355+
if (model) a.push('--model', model);
315356
return a;
316357
}
317358

@@ -320,16 +361,17 @@ export class ClaudeCodeDriver implements AgentDriver {
320361
writeFileSync(empty, JSON.stringify({ mcpServers: {} }));
321362
try {
322363
const maxTurns = this.authorMaxTurns(task.tier);
364+
const model = this.authorModel;
323365
const args = [
324366
'-p',
325-
...this.base(maxTurns),
367+
...this.base(maxTurns, model),
326368
'--mcp-config', empty, '--strict-mcp-config',
327369
'--permission-mode', 'acceptEdits',
328370
'--allowedTools', 'Bash', 'Read', 'Edit', 'Write', 'Glob', 'Grep', 'Skill',
329371
];
330372
const run = await runCli(this.bin, args, task.cwd, task.prompt, this.authorTimeoutMs);
331373
const turn = parseStreamJson(run.stdout);
332-
turn.meta.process = { ...run.process, maxTurns };
374+
turn.meta.process = { ...run.process, maxTurns, ...(model ? { requestedModel: model } : {}) };
333375
if (turn.meta.resultSubtype === undefined) turn.meta.resultSubtype = noResult(run.process);
334376
return { turn, files: snapshotSources(task.cwd) };
335377
} finally {
@@ -349,10 +391,11 @@ export class ClaudeCodeDriver implements AgentDriver {
349391
);
350392
try {
351393
const maxTurns = this.askMaxTurns;
352-
const args = ['-p', ...this.base(maxTurns), '--mcp-config', cfg, '--strict-mcp-config', '--allowedTools', 'mcp__app'];
394+
const model = this.askModel;
395+
const args = ['-p', ...this.base(maxTurns, model), '--mcp-config', cfg, '--strict-mcp-config', '--allowedTools', 'mcp__app'];
353396
const run = await runCli(this.bin, args, dir, task.prompt, this.opts.askTimeoutMs ?? 5 * 60_000);
354397
const turn = parseStreamJson(run.stdout);
355-
turn.meta.process = { ...run.process, maxTurns };
398+
turn.meta.process = { ...run.process, maxTurns, ...(model ? { requestedModel: model } : {}) };
356399
if (turn.meta.resultSubtype === undefined) turn.meta.resultSubtype = noResult(run.process);
357400
return turn;
358401
} finally {

‎packages/qa/scenarios/src/runner/orchestrate.ts‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ import {
4444
type ScenarioArtifactScore,
4545
} from '../score/index.js';
4646
import type { AppHarness, PersonaAccount } from './app.js';
47-
import type { AgentDriver, StepRef } from './driver.js';
47+
import { servedAsRequested, type AgentDriver, type StepRef } from './driver.js';
4848
import { citationLabels, namespaceOfArtifact, namespacedScenario, permissionSetName, templateHostName } from './namespace.js';
4949
import { agentSetOf, clockDates, observeOperation, observeProbe, probeCall, touchedRowId, type Row } from './observe.js';
5050
import type { CommandResult } from './project.js';
@@ -95,7 +95,12 @@ export interface TurnSummary {
9595
/** The turn cap the CLI process ran under (`meta.process`): `numTurns` of `maxTurns`. */
9696
maxTurns?: number;
9797
toolCalls: number;
98+
/** The model the turn was served (the stream's `meta.model`); absent when its stream named none. */
9899
model?: string;
100+
/** The model the driver asked for (`--model`, `meta.process`); absent ⇒ the login's default. */
101+
requestedModel?: string;
102+
/** The served model is not the requested one ({@link servedAsRequested}); only when both are known. */
103+
modelMismatch?: true;
99104
/** Wall-clock time of the CLI process, when the driver ran one (`meta.process`). */
100105
elapsedMs?: number;
101106
// What a turn that did not finish leaves to say why (`meta.process`):
@@ -204,6 +209,8 @@ const summarize = (t: ParsedTurn): TurnSummary => {
204209
...(t.meta.numTurns !== undefined ? { numTurns: t.meta.numTurns } : {}),
205210
...(p?.maxTurns !== undefined ? { maxTurns: p.maxTurns } : {}),
206211
...(t.meta.model ? { model: t.meta.model } : {}),
212+
...(p?.requestedModel ? { requestedModel: p.requestedModel } : {}),
213+
...(p?.requestedModel && t.meta.model && !servedAsRequested(p.requestedModel, t.meta.model) ? { modelMismatch: true as const } : {}),
207214
toolCalls: t.toolCalls.length,
208215
...(p ? { elapsedMs: p.elapsedMs } : {}),
209216
...(p && !finished(t)

0 commit comments

Comments
 (0)