Repository navigation
Commit 0d0c377
feat(scenarios): run-scenarios --model / --author-model / --ask-model, with the served model recorded per turn (#31204)
<!-- claude-projects-attribution: {"github_login":"os-project-manager"}
-->
_Requested by **Leehom** · [project
thread](https://claude.ai/code/project/chan_01TenzsRopZj4jMbvjX8uz1X?thread=cmsg_01TenzsRopZj4jMbvjX8uz1XJdp59LSRHc3ZMoadGaTvT9)_
Fixes #31160
Clause-②: no
Metadata-protocol: none
## 维护者速读
- 改了什么:场景跑批器 `run-scenarios` 新增
`--model`(每一轮都用)、`--author-model`(建造/迭代轮)、`--ask-model`(人设问答与操作轮),原样作为
`claude --model` 传给本地 Claude Code。每一轮在运行记录里同时写明「要的模型」和「实际服务的模型」(读自流的
init 行),计分卡的作者行写出实际模型,新增 `Run · model` 行按实际模型统计人设轮次,不一致处标 `⚠`。
- 为什么改:2026-11-06/07 的所有轮次都跑在登录默认模型上,无法固定或对比模型;维护者同意此建议并立卡。
- 风险与回滚:只动私有包 `@objectstack/scenarios`,不发布任何东西;不传新参数时命令行与之前逐字节相同。回滚即还原本
PR。
- 先看:`packages/qa/scenarios/src/runner/driver.ts`(按轮次类型传
`--model`、`servedAsRequested` 判不一致)、`src/runner/scorecard.ts`(`Run ·
model` 行与作者行)、`src/runner/cli.ts`(三个参数、回放拒收)。
- 两分钟验证:`pnpm --filter @objectstack/scenarios exec vitest run
--maxWorkers=2 test/runner/units.test.ts test/runner/orchestrate.test.ts
-t model`,预期全部通过(含参数到达 CLI 参数表、记录带实际模型、不一致被标出)。
- 需要你做的:无,席位落地。
## What changes
- **Three flags on `run-scenarios`** (`runner/cli.ts`): `--model M` for
every live turn, `--author-model M` for the authoring turns (build and
iterate) and `--ask-model M` for the question and operation turns, each
of the last two in place of `--model`. A value is an alias (`sonnet`) or
a name (`claude-sonnet-4-5`), passed through as `claude --model` takes
it; an empty value or one that starts with `-` (a forgotten value
swallowing the next flag) is refused. With none, no turn passes
`--model` and the login's default serves, as today.
- **Threaded to the driver** (`runner/driver.ts`): `ClaudeCodeDriver`
already took `model`; it now also takes `authorModel` / `askModel` and
passes the one for the kind of turn. `DriverInfo` states `authorModel` /
`askModel`, so the record's `env.driver` and the run's first console
line say what was asked for. `RecordedDriver` refuses the three model
flags as it refuses the live-driver limits (`makeDriver`): a replay
re-reads each turn as it was recorded.
- **The model asked for, beside the model served, per turn**: the live
driver stamps each turn's `meta.process.requestedModel` (the `--model`
it was given; `runner/transcript.ts`), next to the stream's own
`meta.model` it already parsed. The turn summary in the record
(`TurnSummary`, `runner/orchestrate.ts`) carries `model` (served),
`requestedModel` and `modelMismatch: true` when the served name is not
the requested one. A recording keeps both on every turn, so a replay
re-reads them as the run saw them.
- **Mismatch rule** (`servedAsRequested`, `runner/driver.ts`): the CLI
takes an alias and the stream names the model it resolved to, so a turn
is served as asked when the served name holds the requested one as a
whole run of its dash-separated parts (`sonnet` and `claude-sonnet-4-5`
in `claude-sonnet-4-5-20250929`; never `opus`, never a fragment like
`son`); a context-size suffix (`[1m]`) is dropped. Any other answer,
including an alias that names no model family (`default`, `opusplan`),
is shown as a mismatch beside both names, for the reader to judge.
- **The scorecard** (`runner/scorecard.ts`): the author rows end with
the served model (`success · 17/80 turns · 3m 02s · claude-opus-5-5`, `⚠
asked for sonnet` on a mismatch, `model not reported (asked for sonnet)`
when the stream named none); new rows `Run · model` and `Iterate · Run ·
model` count the half's question and operation turns per served model
(`claude-sonnet-4-5-20250929 ×12, claude-opus-5-5 ×2 ⚠ 2 not sonnet`);
`—` when no turn named a model and none was asked for.
`HalfScores.models` (`ModelTally`) carries the same in the record's
`scorecard`.
- README: the two command lines and a **Model** paragraph.
## Why
The card's three asks, verbatim: "`--model ALIAS` for every turn, plus
optional `--author-model` / `--ask-model` to set the build and persona
turns apart. Thread them to the driver."; "The run record and the
scorecard state the model actually served per turn. The stream's
`meta.model` is already parsed; show any mismatch with the requested
model."; "Tests: the flag reaches the CLI argument list; the record
carries the served model." Every recorded turn of the 2026-11-06/07 runs
used the login's default model and nothing let a run pin or compare one.
## Tests
- **The flag reaches the CLI's argument list**
(`test/runner/units.test.ts`, the stand-in `claude` block, 2 new cases).
The stand-in now serves a model the way the real CLI's init line names
it (`--model X` → `claude-X-4-5-20250929`; `retired` falls back to
`claude-opus-5-5`). Pins: `parseArgs(['--model', 'sonnet'])` →
`makeDriver` → both an authoring and an ask turn's argv carry `--model
sonnet`, each turn's `meta.process.requestedModel` is `sonnet` and
`meta.model` is the served `claude-sonnet-4-5-20250929`; `--author-model
opus --ask-model haiku` beside `--model sonnet` splits them;
`--author-model` alone leaves the ask turn with no `--model` and nothing
requested; no flag passes no `--model`; a fallback keeps both names.
Negative pins: a missing, empty, blank or flag-shaped value is refused
per flag; a replay refuses `--model`, and names `--author-model,
--ask-model` together.
- **The mismatch rule** (`units.test.ts`, new describe, a 12-row table):
alias, name, dated name, case, `[1m]` suffix, other family, wrong
version, a fragment of a part, `default`, empty.
- **The record carries the served model**
(`test/runner/orchestrate.test.ts`, new describe, 2 cases): the real crm
card through `runCard`, its driver wrapped in a `RecordingDriver` and a
stamp that asks for `opus` on authoring turns and `sonnet` on persona
turns, with one question's turns served `claude-opus-5-5`. Pins: every
turn summary carries `model` and `requestedModel`, exactly that
question's turns carry `modelMismatch`; `first.models` counts the served
models; the rendered table's author row names `claude-opus-4-1-20250805`
and both `Run · model` rows show `⚠ N not sonnet`; the written record
(`JSON` round trip) holds `env.driver.authorModel/askModel`, the
mismatched turn and the tally; the saved recording replayed through
`RecordedDriver.fromFile` (the `--driver recorded:F` path) answers that
turn with both names. A run whose turns name no model and ask for none
keeps `| Run · model | — |` and the author rows unchanged.
<details><summary>Ablations (each restored: blob == HEAD, git diff HEAD
empty)</summary>
All four run through `scripts/ablation-replace.mjs` (anchor x1 → x0,
replacement x0 → x1) on head `4798cbf1`; the suites import `src/`
directly, so no dist leg.
- Ablation 1, the mismatch marking disarmed (`&& !servedAsRequested(…)`
in `orchestrate.ts` `summarize` replaced by an always-false
`String("ABL1").length < 0`): `orchestrate.test.ts -t served` → `1
failed | 2 passed` (the served-model record case).
- Ablation 2, the replay refusal of the model flags disarmed
(`LIVE_FLAGS.filter` → `LIVE_FLAGS.slice(0, 3 + …).filter` in `cli.ts`):
`units.test.ts -t model` → `1 failed | 6 passed` (the refusal case).
- Ablation 3, the empty / flag-shaped value refusal disarmed
(`raw.trim() === '' || raw.startsWith('-')` → `raw === String("ABL3")`):
`units.test.ts -t model` → `1 failed | 6 passed` (the refusal case).
- Ablation 4, the `--model` threading disarmed (`if (model) a.push` →
`if (model && String("ABL4").length < 0) a.push` in `driver.ts` `base`):
`units.test.ts -t model` → `1 failed | 6 passed` (the argument-list
case).
</details>
## Acceptance notes
- **Served is read off the stream, never inferred from the request.** A
turn whose stream named no model (no init line: a spawn that failed)
shows `model not reported`, and is never counted as a mismatch: nothing
was served to compare.
- **No model allow-list.** The runner does not validate an alias against
a list: `claude` answers an unknown model itself, and that turn is
recorded as it failed. A list here would go stale with every model
release.
- **Recordings made before this change** carry no `requestedModel`;
their turns show the served model and no mismatch.
- **Serial with PR #31188** (`--packages`, same `cli.ts` /
`scorecard.ts`): it was still open when this PR opened, so `origin/main`
was merged once before it; the seat's order has this branch merge
`origin/main` again after #31188 lands. #31159 (`--driver external`)
follows on the same files; this change stays local to the flags, the
driver options and the record.
- `@objectstack/scenarios` is `private: true` and nothing else moved:
`skip-changeset`.
## Verification
**Current head `45ceeaff`** (second `origin/main` merge, at `065bba72`,
carrying #31188; plain merge, no rebase). The only conflict was the
README command block, and all three new lines were kept. Inside the
merge, the new `orchestrate.test.ts` `runRecord` call passes `packages:
PUBLISHED` because #31188 made `RunRecordMeta.packages` required. On
this head: 25 files / 516 tests pass, typecheck and scoped eslint exit
0, and `dispatch-gates --ran` shows 49 derived, 49 run, 0 NOT MEASURED
(`check:dual-build-cjs-loads` now measured). Ablations were not rerun
because the merge touched none of the ablated lines. The rounds below
are on the earlier head.
All on head `0b43e745` (the final commit: the work at `4798cbf1` plus
one merge of `origin/main` at `8acbfab9`), in worktree
`objectstack-issue-31160`, base `620b6e6d`. `origin/main` has since
moved (#31185 among others, which touches `packages/qa/scenarios`
prompts and tests); `git merge-tree` against it is clean.
- Package suite, under the verify lock after `pnpm --filter
"@objectstack/scenarios..." build` on the merged head: `pnpm --filter
@objectstack/scenarios test` → `Test Files 23 passed (23)`, `Tests 460
passed (460)`; `pnpm --filter @objectstack/scenarios typecheck` (`tsc
--noEmit`, test files included: `--listFiles` lists
`test/runner/units.test.ts`) exit 0; `VERDICT command-exit 0`.
- Lint scoped to the package: `eslint packages/qa/scenarios
--no-inline-config` exit 0.
- Derived gates: `node scripts/pm/dispatch-gates.mjs --commands --repo
objectstack-ai/objectstack` → 49 commands, all run on this head; 48
`exit 0`; `pnpm check:dual-build-cjs-loads` exit 3, PREREQUISITE NOT MET
(it reads every package's `dist/`, and only the scenarios closure was
built here) — NOT MEASURED, left to CI. `--ran` reconciliation: `49
derived famil(ies) accounted for — 48 run, 1 NOT-MEASURED`.
- Changeset: none. The diff touches only the private
`@objectstack/scenarios` (`"private": true`): `skip-changeset`.
- Dogfood battery: n/a (no `!` in the title, no `(narrowing)` arm).
- No live model run (paused by the maintainer): the flag-to-argv path is
measured against a stand-in `claude`, and the replay path through
`RecordedDriver.fromFile` on a recording the test writes.
<details><summary>The 49 derived gates, as run</summary>
```text
node scripts/check-closing-keyword-parity.mjs :: exit 0
node scripts/check-closing-keyword-parity.mjs --self-test :: exit 0
node scripts/check-comment-mask-adoption.mjs :: exit 0
node scripts/check-comment-mask-adoption.mjs --self-test :: exit 0
node scripts/check-comment-mask-corpus.mjs :: exit 0
node scripts/check-dts-emitted.mjs --self-test :: exit 0
node scripts/check-issue-citations.mjs :: exit 0
node scripts/check-plugin-teardown-shape.mjs :: exit 0
node scripts/check-plugin-teardown-shape.mjs --self-test :: exit 0
node scripts/ci/assert-turbo-remote-cache.mjs --self-test :: exit 0
node scripts/ci/plan-test-core-shards.mjs --self-test :: exit 0
node scripts/ci/prune-turbo-cache.mjs --self-test :: exit 0
node scripts/docs-audit/check-affected-docs.mjs :: exit 0
node scripts/docs-audit/check-drift-comment.mjs :: exit 0
pnpm --filter @objectstack/spec run check:duration-unit-keys :: exit 0
pnpm check:ci-filter-parity :: exit 0
pnpm check:closed-ceremony-text :: exit 0
pnpm check:cross-package-test-inputs :: exit 0
pnpm check:doc-authoring :: exit 0
pnpm check:driver-memory-census :: exit 0
pnpm check:dts-closure :: exit 0
pnpm check:dual-build-cjs-loads :: exit 3
pnpm check:engine-double-contract :: exit 0
pnpm check:gitlink-declared :: exit 0
pnpm check:issue-citations :: exit 0
pnpm check:keyed-text-bounds :: exit 0
pnpm check:lean-entry-closure :: exit 0
pnpm check:logger-receiver-detach :: exit 0
pnpm check:nul-bytes :: exit 0
pnpm check:objectql-double-limit :: exit 0
pnpm check:org-identifier :: exit 0
pnpm check:page-declaration-shape :: exit 0
pnpm check:platform-object-tenancy-census :: exit 0
pnpm check:published-files :: exit 0
pnpm check:query-options-erasure :: exit 0
pnpm check:refd-timer-probe :: exit 0
pnpm check:registry-log-declared :: exit 0
pnpm check:rest-log-spy-declared :: exit 0
pnpm check:slot-lookup :: exit 0
pnpm check:sourcemap-no-sources-content :: exit 0
pnpm check:system-context-census :: exit 0
pnpm check:test-source-alias :: exit 0
pnpm check:tier-file-adoption :: exit 0
pnpm check:type-check-coverage :: exit 0
pnpm check:type-check-debt :: exit 0
pnpm check:undeclared-dep-imports :: exit 0
pnpm check:vouch-for-reference-callers :: exit 0
pnpm check:watch-hint-literal :: exit 0
pnpm check:where-matcher :: exit 0
```
</details>
## Human Minimum
- [x] 1. Schemas start as Zod; `packages/spec` holds no runtime logic.
(Not touched.)
- [x] 2. Every name I added uses the casing for its kind.
- [x] 3. No formatter ran; `pnpm lint` passes (scoped to the package).
- [x] 4. One issue, one branch off an up-to-date `main`, one pull
request.
- [x] 5. A change that publishes carries a changeset, and both
declaration lines are filled in here and in the changeset. (Publishes
nothing: `skip-changeset`.)
- [x] 6. `packages/spec` touched: its generated artifacts are
regenerated and committed. (Not touched.)
- [x] 7. No hand edits to `content/docs/references/`,
`packages/*/CHANGELOG.md` or `content/docs/releases/`.
- [x] 8. Title, body and comments are in English.
- [x] The *Before Submitting* list in CONTRIBUTING.md is done.
<!-- signed: domain:devx#3 · os-project-manager ·
session_01FEkQ5njkh3xbcDN6EtSfw7 -->
Co-authored-by: Claude <noreply@anthropic.com>1 parent b065fad commit 0d0c377
8 files changed
Lines changed: 353 additions & 20 deletions
File tree
- packages/qa/scenarios
- src/runner
- test/runner
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
139 | 139 | | |
140 | 140 | | |
141 | 141 | | |
| 142 | + | |
| 143 | + | |
142 | 144 | | |
143 | 145 | | |
144 | 146 | | |
| |||
167 | 169 | | |
168 | 170 | | |
169 | 171 | | |
170 | | - | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
171 | 181 | | |
172 | 182 | | |
173 | 183 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
25 | 33 | | |
26 | 34 | | |
27 | 35 | | |
| |||
77 | 85 | | |
78 | 86 | | |
79 | 87 | | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
80 | 94 | | |
81 | 95 | | |
82 | 96 | | |
| |||
99 | 113 | | |
100 | 114 | | |
101 | 115 | | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
102 | 121 | | |
103 | 122 | | |
104 | 123 | | |
| |||
117 | 136 | | |
118 | 137 | | |
119 | 138 | | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
120 | 142 | | |
121 | 143 | | |
122 | 144 | | |
| |||
135 | 157 | | |
136 | 158 | | |
137 | 159 | | |
138 | | - | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
139 | 164 | | |
140 | 165 | | |
141 | 166 | | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
142 | 170 | | |
143 | 171 | | |
144 | | - | |
| 172 | + | |
145 | 173 | | |
146 | 174 | | |
147 | 175 | | |
148 | 176 | | |
149 | 177 | | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
150 | 181 | | |
151 | 182 | | |
152 | 183 | | |
153 | | - | |
| 184 | + | |
154 | 185 | | |
155 | 186 | | |
156 | 187 | | |
| |||
185 | 216 | | |
186 | 217 | | |
187 | 218 | | |
188 | | - | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
189 | 222 | | |
190 | 223 | | |
191 | 224 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
78 | 78 | | |
79 | 79 | | |
80 | 80 | | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
81 | 85 | | |
82 | 86 | | |
83 | 87 | | |
| |||
155 | 159 | | |
156 | 160 | | |
157 | 161 | | |
158 | | - | |
| 162 | + | |
159 | 163 | | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
160 | 191 | | |
161 | 192 | | |
162 | 193 | | |
| |||
297 | 328 | | |
298 | 329 | | |
299 | 330 | | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
300 | 339 | | |
301 | 340 | | |
302 | 341 | | |
| |||
306 | 345 | | |
307 | 346 | | |
308 | 347 | | |
| 348 | + | |
| 349 | + | |
309 | 350 | | |
310 | 351 | | |
311 | 352 | | |
312 | | - | |
| 353 | + | |
313 | 354 | | |
314 | | - | |
| 355 | + | |
315 | 356 | | |
316 | 357 | | |
317 | 358 | | |
| |||
320 | 361 | | |
321 | 362 | | |
322 | 363 | | |
| 364 | + | |
323 | 365 | | |
324 | 366 | | |
325 | | - | |
| 367 | + | |
326 | 368 | | |
327 | 369 | | |
328 | 370 | | |
329 | 371 | | |
330 | 372 | | |
331 | 373 | | |
332 | | - | |
| 374 | + | |
333 | 375 | | |
334 | 376 | | |
335 | 377 | | |
| |||
349 | 391 | | |
350 | 392 | | |
351 | 393 | | |
352 | | - | |
| 394 | + | |
| 395 | + | |
353 | 396 | | |
354 | 397 | | |
355 | | - | |
| 398 | + | |
356 | 399 | | |
357 | 400 | | |
358 | 401 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
47 | | - | |
| 47 | + | |
48 | 48 | | |
49 | 49 | | |
50 | 50 | | |
| |||
95 | 95 | | |
96 | 96 | | |
97 | 97 | | |
| 98 | + | |
98 | 99 | | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
99 | 104 | | |
100 | 105 | | |
101 | 106 | | |
| |||
204 | 209 | | |
205 | 210 | | |
206 | 211 | | |
| 212 | + | |
| 213 | + | |
207 | 214 | | |
208 | 215 | | |
209 | 216 | | |
| |||
0 commit comments