Repository navigation
Commit 5f00374
feat(scenarios): an authoring turn's turn cap is sized by the card's tier; --author-max-turns and --ask-max-turns reach the live driver (#31111)
Fixes #31106
Clause-②: no
Metadata-protocol: none
## 维护者速读
- 改了什么:场景跑批器(私有包 `@objectstack/scenarios` 的 runner)构建回合的轮数上限改为按卡片 `tier`
取默认值(control 80、starter 300、delivery 600),新增 `--author-max-turns N` /
`--ask-max-turns N` 两个参数覆盖它;每个回合记录它实际用的上限,记分卡 author 行显示「已用轮数/上限」,例如
`error_max_turns · 80/80 turns`。
- 为什么改:上限原来写死 80 且 CLI 不能改,实测 3 张 starter 卡的构建回合全部在 80
轮被截停(`error_max_turns`,15–24 分钟),没有一张中档卡能测完;control 卡只要 17–21 轮,维持 80
不变。
- 风险与回滚:只动不发布的跑批器包,平台运行时零改动;回滚就是 revert 本 PR。没有用真实 `claude`
跑过(成本裁决只许本地登录,本容器没有);starter 的 300 轮和 60
分钟超时大致同时触顶,构建仍可能以超时结束,两种情况都如实记为「未测量」。
-
先看:`packages/qa/scenarios/src/runner/driver.ts`(`DEFAULT_AUTHOR_MAX_TURNS`
表与 `author()` 按 `task.tier` 取上限)、`src/runner/cli.ts`(`makeDriver`
传参与回放拒收)、`src/runner/orchestrate.ts`(三处 author 调用带上 `tier: card.tier`)。
- 两分钟验证:`pnpm --filter @objectstack/scenarios exec vitest run
test/runner/units.test.ts test/runner/orchestrate.test.ts`,预期 84 个测试全绿。
- 需要你做的:无,席位落地。
## What changes
**The cap is resolved per authoring turn, by the card's tier**
(`src/runner/driver.ts`).
- `DEFAULT_AUTHOR_MAX_TURNS`, a record keyed by `ScenarioTier`, = `{
control: 80, starter: 300, delivery: 600 }`, beside
`DEFAULT_AUTHOR_TIMEOUT_MIN`, and `DEFAULT_ASK_MAX_TURNS = 12`.
- The driver is built once per CLI run (`makeDriver`) and knows no card,
so `AuthorTask` gains a required `tier: ScenarioTier`. The live driver
computes `--max-turns` as `--author-max-turns`, else
`DEFAULT_AUTHOR_MAX_TURNS[task.tier]`. The tier is not part of the
replay key (`stepKey` is unchanged), so existing recordings still
replay.
- `orchestrate.ts` passes `tier: card.tier` on all three authoring
calls: the scratch build, the scratch iterate, and the template iterate.
The card asks for "build and iterate turns", so one cap covers both.
- Ask turns (questions and operations) take `--ask-max-turns`, else 12,
on every tier.
- `ClaudeCodeDriverOptions.maxTurnsAuthor` / `maxTurnsAsk` are renamed
`authorMaxTurns` / `askMaxTurns`, matching `authorTimeoutMs` /
`askTimeoutMs` and the flags. Nothing set them (`git grep` at
`a61f61a4`: only their own declarations and the two `??` reads).
**The CLI** (`src/runner/cli.ts`).
- `--author-max-turns N` and `--ask-max-turns N` take a whole number
above 0, and any other value is refused with the flag and the value
named. `makeDriver(spec, limits)` threads them to `ClaudeCodeDriver` the
same way `--author-timeout` is threaded.
- A replay (`--driver recorded:F`) refuses all three live-driver limits,
naming every one it was given. This matches the existing
`--author-timeout` refusal: a replay re-reads each turn as it was
recorded.
- The driver line now ends `· author max turns: control 80, starter 300,
delivery 600 · ask max turns: 12`.
**The cap used is recorded.**
- On the turn: `TurnProcess.maxTurns` (`meta.process`, next to
`timeoutMs`) is the `--max-turns` the CLI process was given, on
authoring and ask turns alike, so a recording carries it.
- On the run record: `TurnSummary.maxTurns` (beside `numTurns`) in
`cardRun`, and `scorecard.modes[].first.build.author` /
`iterate.build.author` carry `numTurns` and `maxTurns`. `env.driver`
(`DriverInfo`) carries `authorMaxTurns` (the per-tier table this run
applies) and `askMaxTurns`.
- On the scorecard: the `Build · author` and `Iterate · author` rows
show the turns taken of the cap, for example `error_max_turns · 80/80
turns · 23m 34s`. A turn killed before any result line shows `?/80
turns`. A turn recorded without a cap shows the count alone, and a turn
with neither shows neither (the existing rows are byte-identical).
Unchanged, by ruling: a turn ending `error_max_turns` is still `not
measured: author turn failed: error_max_turns`, and nothing runs after
it.
Landing:
`packages/qa/scenarios/src/runner/{driver,cli,orchestrate,scorecard,transcript}.ts`,
`test/runner/{units,orchestrate}.test.ts` and the package README, all
inside the claim's file surface. `src/schema.ts` is read
(`ScenarioTier`, `SCENARIO_TIERS`), not changed.
## Tests
- `test/runner/units.test.ts`, inside the existing describe of the live
driver against a stand-in `claude` script (never the real CLI). The
stand-in gains an `ARGV` prompt that answers its own argument list, so
the tests read the `--max-turns` value the CLI process was really
started with:
- **Tier default, no flag:** a control, starter and delivery author turn
start the CLI with `--max-turns` 80, 300 and 600. An ask turn starts it
with 12. Each turn's `meta.process.maxTurns` matches, and `info()`
states the table. The literal table is pinned.
- **The flags reach the CLI's argument list:**
`parseArgs(['--author-max-turns', '450', '--ask-max-turns', '20'])` goes
through `makeDriver('claude', …)`, the binary resolves on `PATH` like
the `--author-timeout` test, and the CLI gets `--max-turns 450` on every
tier and `--max-turns 20` on an ask turn. `0`, `-3`, `1.5` and `many`
are each refused, naming the flag and the value. A replay refuses each
flag and all three together, naming them.
- The two replay-driver author tasks now name their tier. The required
field caught both at typecheck.
- `test/runner/orchestrate.test.ts`:
- An `error_max_turns` build author turn is not measured as before.
`build.author` carries `numTurns`/`maxTurns`, the author row reads
`error_max_turns · 80/80 turns · 23m 34s`, and the JSON-serialised run
record carries the cap in `cardRun` and `scorecard`.
- An iterate turn killed before any result line reads `?/80 turns`. A
turn with neither a count nor a cap keeps its old row.
- A new top-level describe checks that every authoring turn of the
control card (scratch build, scratch iterate, template iterate) carries
`control` to the driver. In the reference-cycle describe, the starter
card `requisition_to_receipt` carries `starter` on its build and iterate
turns.
Reverse verification: one mutation per behaviour, each through `node
scripts/ablation-replace.mjs` in wrap mode. In every case the anchor
went from 1 hit to 0, the blob changed, the restored blob equalled HEAD,
and `git diff HEAD` was empty afterwards. The suites import `src/` by
relative path, so no build sits between a mutation and its run.
| Mutation | Expected | Observed |
|:---|:---|:---|
| `driver.ts`: the author cap ignores the task's tier (`'control'`) |
tier-default test red | 1 failed / 83 passed |
| `cli.ts`: `makeDriver` drops `authorMaxTurns` | flag test red | 1
failed / 83 passed |
| `orchestrate.ts`: the scratch build passes `tier: 'control'` |
starter-card test red | 1 failed / 83 passed |
| `scorecard.ts`: the author row drops the turns text | both author-row
tests red | 2 failed / 82 passed |
| `cli.ts`: the replay forgets `--author-max-turns` | flag test red (the
replay opened the file instead) | 1 failed / 83 passed |
| `orchestrate.ts`: `summarize` drops `maxTurns` | both author-row tests
red | 2 failed / 82 passed |
## Acceptance notes
- **Defaults: the card's example is kept, and the measurements support
it.** From the card's three `starter` readings (80 turns in 23m 34s, 15m
05s and 14m 48s), a turn takes 11–18 s, so the 60-minute
`--author-timeout` default allows about 200–325 turns. A cap of 300
makes the turn cap and the wall clock bind at about the same point. The
cap stops a turn that loops, and the timeout stops a slow one. `control`
stays at 80: its builds finished in 17–21 turns.
- **How the two caps relate on `delivery`:** no delivery card exists in
the library at `a61f61a4` (9 cards: 6 `starter`, 3 `control`). At the
starter pace, 600 turns would take about 110–180 minutes, past the
60-minute default, so under the default the clock ends a delivery turn
first. The README and the constant's doc say a delivery run should raise
`--author-timeout` too. The timeout itself is not tier-sized: the card
does not ask for it, and the last PR on this package made it a flat
constant on purpose.
- **Ask turns: 12 on every tier, as before.** Neither the card nor its
thread has an ask-turn `error_max_turns` reading, so there is no
measured reason for a per-tier table. It is a single constant.
- `--author-max-turns` sets one cap for every card in the run. A
per-tier override was not asked for, so it is not added.
- Not measured live: no real `claude` run. The cost ruling allows only a
local Claude Code login, and this container has none. The flag-to-argv
path is pinned against the stand-in CLI. The epic seat's re-run of the
six `starter` cards is the card's own verification.
- `test:repo` was not run locally. The one repo-level pin that reads
this package,
`packages/core/src/security/second-object-read-exposure.pin.test.ts`,
lists `referenceTargetOf` callers under `src/score/`. This diff adds no
such call and touches no file there. CI's `Test Core` runs it.
- Changeset: none. `@objectstack/scenarios` is `private: true` and
publishes nothing, so the PR carries `skip-changeset`.
## Verification
Readings at head `fdac4b9b`, `objectstack-ai/objectstack`. The gate
sweep, the package build and the package suite all ran after the final
commit.
<details><summary>Package: typecheck, build, suite</summary>
```text
pnpm --filter @objectstack/scenarios typecheck -> exit 0 (tsc --noEmit; tsconfig includes test/**)
pnpm --filter @objectstack/scenarios build -> exit 0; check-dts-emitted: 10/10 declared declaration file(s) present
pnpm --filter @objectstack/scenarios test -> Test Files 23 passed (23) · Tests 431 passed (431)
vitest run test/runner/units.test.ts test/runner/orchestrate.test.ts -> Tests 84 passed (84)
built entries: require() of dist/{index,score,score-full,runner,export}.cjs ok; import() of dist/runner.js ok;
runner.cjs DEFAULT_AUTHOR_MAX_TURNS = {"control":80,"starter":300,"delivery":600} · DEFAULT_ASK_MAX_TURNS = 12
```
</details>
<details><summary>Gates: dispatch-gates --commands (49) and
--ran</summary>
```text
dispatch-gates: gate list derived from the tree of 'objectstack-ai/objectstack' at commit fdac4b9
change set: 8 path(s) vs merge base a61f61a · changed lines 288 (+256 / -32) vs the human-merge threshold 3000: under
49 command(s) run, each exit code recorded before any pipe: 48 x exit 0, check:dual-build-cjs-loads exit 3
(PREREQUISITE NOT MET: it reads every package's dist/; it exited 0 when re-run after the sweep's own lanes had built them:
116 published require entry point(s) across 71 package(s) load)
check:dts-closure / check:sourcemap-no-sources-content re-run after this package was built: exit 0 / exit 0
Run reconciliation — 49 derived, 49 run, 0 NOT-MEASURED, 0 UNRUN.
```
</details>
---
_Generated by [Claude
Code](https://claude.ai/code/session_012MmDsRzowAnXEmogPGSEKA)_
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent aa3e21c commit 5f00374
8 files changed
Lines changed: 256 additions & 32 deletions
File tree
- packages/qa/scenarios
- src/runner
- test/runner
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
97 | 97 | | |
98 | 98 | | |
99 | 99 | | |
| 100 | + | |
| 101 | + | |
100 | 102 | | |
101 | 103 | | |
102 | 104 | | |
| |||
110 | 112 | | |
111 | 113 | | |
112 | 114 | | |
113 | | - | |
114 | | - | |
115 | | - | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
116 | 125 | | |
117 | 126 | | |
118 | 127 | | |
119 | 128 | | |
120 | | - | |
| 129 | + | |
| 130 | + | |
121 | 131 | | |
122 | 132 | | |
123 | 133 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
16 | 23 | | |
17 | 24 | | |
18 | 25 | | |
| |||
59 | 66 | | |
60 | 67 | | |
61 | 68 | | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
62 | 73 | | |
63 | 74 | | |
64 | 75 | | |
| |||
73 | 84 | | |
74 | 85 | | |
75 | 86 | | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
76 | 93 | | |
77 | 94 | | |
78 | 95 | | |
| |||
89 | 106 | | |
90 | 107 | | |
91 | 108 | | |
| 109 | + | |
| 110 | + | |
92 | 111 | | |
93 | 112 | | |
94 | 113 | | |
| |||
97 | 116 | | |
98 | 117 | | |
99 | 118 | | |
100 | | - | |
101 | | - | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
102 | 136 | | |
103 | | - | |
| 137 | + | |
| 138 | + | |
104 | 139 | | |
105 | 140 | | |
106 | 141 | | |
| |||
124 | 159 | | |
125 | 160 | | |
126 | 161 | | |
127 | | - | |
| 162 | + | |
128 | 163 | | |
129 | 164 | | |
130 | 165 | | |
131 | 166 | | |
132 | | - | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
133 | 170 | | |
134 | 171 | | |
135 | 172 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
| 29 | + | |
29 | 30 | | |
30 | 31 | | |
31 | 32 | | |
| |||
46 | 47 | | |
47 | 48 | | |
48 | 49 | | |
| 50 | + | |
| 51 | + | |
49 | 52 | | |
50 | 53 | | |
51 | 54 | | |
| |||
71 | 74 | | |
72 | 75 | | |
73 | 76 | | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
74 | 81 | | |
75 | 82 | | |
76 | 83 | | |
| |||
121 | 128 | | |
122 | 129 | | |
123 | 130 | | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
124 | 147 | | |
125 | 148 | | |
126 | 149 | | |
127 | | - | |
128 | | - | |
129 | | - | |
130 | | - | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
131 | 154 | | |
132 | 155 | | |
133 | 156 | | |
| |||
266 | 289 | | |
267 | 290 | | |
268 | 291 | | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
269 | 300 | | |
270 | | - | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
271 | 310 | | |
272 | 311 | | |
273 | 312 | | |
| |||
280 | 319 | | |
281 | 320 | | |
282 | 321 | | |
| 322 | + | |
283 | 323 | | |
284 | 324 | | |
285 | | - | |
| 325 | + | |
286 | 326 | | |
287 | 327 | | |
288 | 328 | | |
289 | 329 | | |
290 | 330 | | |
291 | 331 | | |
292 | | - | |
| 332 | + | |
293 | 333 | | |
294 | 334 | | |
295 | 335 | | |
| |||
308 | 348 | | |
309 | 349 | | |
310 | 350 | | |
311 | | - | |
| 351 | + | |
| 352 | + | |
312 | 353 | | |
313 | 354 | | |
314 | | - | |
| 355 | + | |
315 | 356 | | |
316 | 357 | | |
317 | 358 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
91 | 91 | | |
92 | 92 | | |
93 | 93 | | |
| 94 | + | |
| 95 | + | |
94 | 96 | | |
95 | 97 | | |
96 | 98 | | |
| |||
198 | 200 | | |
199 | 201 | | |
200 | 202 | | |
| 203 | + | |
201 | 204 | | |
202 | 205 | | |
203 | 206 | | |
| |||
705 | 708 | | |
706 | 709 | | |
707 | 710 | | |
708 | | - | |
| 711 | + | |
709 | 712 | | |
710 | 713 | | |
711 | 714 | | |
712 | 715 | | |
713 | 716 | | |
714 | 717 | | |
715 | | - | |
| 718 | + | |
716 | 719 | | |
717 | 720 | | |
718 | 721 | | |
| |||
767 | 770 | | |
768 | 771 | | |
769 | 772 | | |
770 | | - | |
| 773 | + | |
771 | 774 | | |
772 | 775 | | |
773 | 776 | | |
| |||
0 commit comments