You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 9c1edbc
Browse filesBrowse the repository at this point in the historyBrowse files
fix(scenarios): seed a reference cycle in two passes; an author turn that does not finish leaves its build not measured (#31096)
Fixes#30982
Clause-②: no
Metadata-protocol: none
## 维护者速读
- 改了什么:场景跑批器(`@objectstack/scenarios` 的 runner)改为两遍加载种子数据,能装进 department
↔ employee 这类互相引用的环;构建提示词写明「环里至少一侧必须可选」,两侧都必填的构建记为 Build
finding;构建回合超时或失败时 Build 半场记为「未测量」,不再打分、不再跑种子、Run 和 iterate;提示词改经 stdin
传给 `claude`。
- 为什么改:8 卡实跑里每张中档卡的种子只装进 3–6 行,Run 半场全部测不了;中档构建在第 15 分钟被 runner 自己杀掉却被当成
AI 质量分;travel_expense 和 service_desk 的构建回合根本没启动——实测是提示词超过 Linux 单个参数 128
KiB 的上限,`spawn` 直接报 E2BIG,记录里什么都没留下。
- 风险与回滚:只动私有、不发布的跑批器包,平台运行时零改动;回滚就是 revert 本 PR。两处没能在真机上实测:`claude -p` 从
stdin 读提示词、第二遍用 `PATCH /api/v1/data/OBJECT/ID` 回填引用;若有一处不成立,记录会如实写出退出码和
stderr,不会静默丢失。
-
先看:`packages/qa/scenarios/src/runner/seed.ts`(`seedPlan`:强连通分量里挑可选的一侧推迟)、`src/runner/driver.ts`
的 `runCli`(stdin 传输、超时与进程记录)、`src/runner/orchestrate.ts` 的 `seedApp`
第二遍和 `scratchMode` 的未测量提前返回。
- 两分钟验证:`pnpm --filter @objectstack/scenarios exec vitest run
test/runner/units.test.ts test/runner/orchestrate.test.ts`,预期 78
个测试全绿,其中 requisition_to_receipt 的种子 637/637 全部装入。
- 需要你做的:无,席位落地。
## What changes
**Defect 1 — the seed loads a reference cycle in two passes**
(`src/runner/seed.ts`, new; `orchestrate.ts` `seedApp`).
- The references come from the card's own declared lookups
(`expect.lookups` between two seeded objects), never a hand list.
Objects go in dependencies first, one strongly connected set at a time
(Tarjan; on an acyclic card this is exactly the depth-first order the
runner already used).
- Inside a cycle, members go in by Kahn's algorithm; when every
remaining member still waits on another, one whose waiting lookups are
all **optional in the built schema** is let in early. Then every lookup
value naming a row not yet in at its row's insert (a lookup to an object
loaded later, or a self-lookup naming a later row of its own object) is
left out of the insert and set by one `PATCH` per row in a second pass
(`AppHarness.updateRow`, `HttpApp`: `PATCH /api/v1/data/OBJECT/ID`).
- The built schema's `required` lookup fields are read through the
scorer's one artifact reader (`artifactToEvalGraph`), scratch mode from
the built artifact, template mode from the installed copy.
- A cycle whose every lookup the build made `required` cannot be loaded.
The build prompt's data contract now names the card's reference cycles
and says "At least one side of a reference cycle must be optional"
(`prompts.ts` `dataContract`, only on a card that has one), and the
Build half records `requiredCycles`, which the scorecard lists as a
Build finding naming the cycle. The seed is still sent, so the record
holds the app's own refusals rather than the runner's prediction.
- The tally still counts each row once: a row whose second-pass update
is refused moves from `inserted` to `refused` with the `fields` it could
not set, and `seedDependence` takes what hangs off it out as before.
`seed.secondPass` records the lookup values set per `OBJECT.FIELD`.
**Defect 2 — a build author turn that fails is recorded, and its Build
half is not measured.**
- Measured cause: the build prompt carries the whole seed and was passed
on argv. On Linux one argv string over 128 KiB (`MAX_ARG_STRLEN`,
131,072 bytes) makes `spawn` throw `E2BIG` before the CLI starts;
`runCli` rejected, the author turn threw, and `RecordingDriver` recorded
nothing. The two cards with no build author turn are exactly the two
over the limit (readings below). The prompt now goes to the CLI's stdin
(`claude -p`, no prompt argument), for question turns too.
- `runCli` never rejects: a process that could not start, was killed, or
exited without a result is answered with what was seen — `meta.process`:
elapsed ms, exit code or signal, whether the runner's timeout killed it,
the stderr tail (2,000 chars), and a spawn or stdin error. It travels in
the turn's `meta`, so a recording carries it and a replay re-reads a
failed turn as the run saw it.
- `authorNotMeasured`: a turn without a clean `success` result line — no
result line, an error subtype, or `is_error: true` (a usage limit
answers `success` with it) — leaves its Build half `notMeasured: "author
turn failed: …"`; a turn the driver threw on is the same, with its
error. Nothing after it runs: no `os validate`/`build`/`verify`, no
score, no seed, no Run, no iterate.
**Defect 3 — the authoring timeout is a runner option, and a killed turn
is never a degraded score.**
- `--author-timeout MIN` (default `DEFAULT_AUTHOR_TIMEOUT_MIN = 60`, a
plain constant: `card.tier` is not read) reaches `authorTimeoutMs`,
which both the build and the iterate author turns use; the run record's
`env.driver.authorTimeoutMs` and the console's driver line state it. A
replay refuses the flag (it re-reads what was recorded).
- A turn the timeout killed makes its Build half `notMeasured: "author
timed out after N min"`. An iterate author turn that does not finish
leaves the first halves standing and the iterate build not measured,
with no iterate Run half and no change size; template mode the same for
its iterate.
- The scorecard prints `not measured: …` in the Build cell (0 of 0,
never `degraded`), adds `Build · author` and `Iterate · author` rows
with every authoring turn's `resultSubtype` and elapsed time, and the
run record gives every half that did not run because of it the reason in
`notes` (`not run: the build was not measured (…)`), verdict `not-run`.
Landing: all in `packages/qa/scenarios/src/runner/**`, its tests and
README, plus `test/export/fixture.ts` (outside the claim's file surface,
only to give its `AppHarness` double the new `updateRow`). Not
`src/schema.ts`, `src/loader.ts` or the card files. The gallery
exporter's screenshot boot (`src/export/screenshots.ts`, unchanged)
seeds through `seedApp` and inherits the two-pass load.
## Tests
- `test/runner/units.test.ts`, new top-level describes:
- the seed's load plan on `requisition_to_receipt`: the cycle found from
`expect.lookups`; departments first with five managers deferred; the
side the build left optional is the one that waits (18
`employee.department` values when `department.manager` is required);
both sides required names the cycle; `order_to_cash`'s `e09.manager →
e10` self-reference waits; on every card in the library the first pass
never names a row not in yet; the contract's cycle sentence.
- the live `ClaudeCodeDriver` against a stand-in `claude` script (never
the real CLI): a 200,000-byte prompt reaches it whole on stdin with argv
under 1,000 bytes; a turn the timeout kills records `exit 143`,
`timedOut`, elapsed and stderr tail and is `author timed out after …`; a
crash and a vanished binary are failed turns with what was seen;
`--author-timeout` parsing, its 60-min default and the replay refusal.
- `test/runner/app.test.ts`, new top-level describe: `HttpApp.updateRow`
sends `PATCH /api/v1/data/OBJECT/ID` with the bare field patch to a
loopback door, and reads a refusal's code and message.
- `test/runner/orchestrate.test.ts`, new top-level describes: a killed /
failed / usage-limited / thrown build author turn, a killed iterate
turn, and a failed template iterate turn, each not measured with nothing
after it run; `seedApp` loading all 637 `requisition_to_receipt` rows
into a double that checks reference integrity and `required` on insert
and update; a refused second-pass update; `runCard` reporting a
both-sides-required cycle as a Build finding.
- `pnpm --filter @objectstack/scenarios typecheck` and `test`: green at
the final head (readings below).
Reverse verification, each through `node scripts/ablation-replace.mjs`
in wrap mode (anchor hit 1 → 0, blob changed, restore proven `blob ==
HEAD`, `git diff HEAD` empty). The suites import `src/` directly, so no
build sits between the mutation and the run.
| Mutation | Expected | Observed |
|:---|:---|:---|
| `seed.ts`: no value is ever deferred | the cycle tests red | 9 failed
/ 69 passed |
| `driver.ts`: the build prompt back on argv | the 200 KB test red
(`spawn E2BIG`, empty reply) | 1 failed / 50 passed, `Received: ""` |
| `orchestrate.ts`: delete the not-measured early return | the killed /
failed build tests red | 2 failed / 25 passed |
## Acceptance notes
- `error_max_turns` counts as `author turn failed`: it is the runner's
own turn budget (80), the same class as the timeout. A 60-minute
mid-tier build that then runs out of turns will read not measured, not a
cut-off score. No `--author-max-turns` option is added.
- The suggested route skipped the seed for a both-sides-required build.
Here it is still sent: the finding names the cycle either way, and the
record then shows whether the platform really refuses a missing required
lookup on insert instead of assuming it.
- `buildPassed` is unchanged: a both-sides-required cycle is a finding,
not a scored check (scoring stays `@objectstack/scenarios/score`), so
such a build can still be the template mode's verified build; its
template seed is refused the same way and recorded.
- Not measured live: `claude -p` reading its prompt from stdin (the
CLI's help: `-p/--print … useful for pipes`, `--input-format text
(default)`) and the `PATCH` data door against a real server (its shape
is read from `packages/rest/src/crud-routes.ts`, and `app.ts` says so;
the request the runner sends is pinned against a loopback door). The
card's verification re-run settles both; if either fails, the turn or
the second-pass refusal is in the record with its exit code, stderr or
answer.
- Measured in the same class and covered without extra code:
`order_to_cash`'s `e09.manager → e10` was refused by the
object-by-object load, and `service_desk` has an `agent_group.lead ↔
agent.group` cycle.
- Changeset: none. `@objectstack/scenarios` is `private: true` and
publishes nothing, so `skip-changeset`.
## Verification
Readings at head `401e567b`, `objectstack-ai/objectstack` (the gate run
and the package suite both after the final commit).
<details><summary>Defect 2: build prompt bytes per card vs
MAX_ARG_STRLEN (131,072)</summary>
```text
asset_lifecycle build 109066 iterate 763
crm_account_with_contacts build 3563 iterate 724
invoice_with_line_items build 4250 iterate 751
leave_balances build 120716 iterate 883
order_to_cash build 123612 iterate 803
project_with_tasks build 4059 iterate 742
requisition_to_receipt build 128145 iterate 824
service_desk build 141922 iterate 814 OVER 128KiB (MAX_ARG_STRLEN)
travel_expense build 162201 iterate 807 OVER 128KiB (MAX_ARG_STRLEN)
node spawn('/bin/true', ['-p', 'x'.repeat(n)]):
128145 spawned
131071 spawned
131072 REJECTED E2BIG spawn E2BIG
141922 REJECTED E2BIG spawn E2BIG
162201 REJECTED E2BIG spawn E2BIG
```
</details>
<details><summary>Package typecheck and tests</summary>
```text
os-verify-lock: ACQUIRED after 0s — running: pnpm --filter @objectstack/scenarios build && pnpm --filter @objectstack/scenarios typecheck && pnpm --filter @objectstack/scenarios test
check-dts-emitted: @objectstack/scenarios - 10/10 declared declaration file(s) present.
> tsc --noEmit
Test Files 23 passed (23)
Tests 364 passed (364)
os-verify-lock: VERDICT command-exit 0 · held the lock 42s · waited 0s · ⚠ SHARED-BOX SECONDS — this lock excluded other LOCKED runs, NOT unlocked sibling work (`check:*` gates and the rest), so these are not idle-box figures
```
</details>
An earlier gate run on this branch read one red and two prerequisites,
none left: `check:cross-package-test-inputs` flagged a new test's
`readdirSync` of `docs/qa/scenarios/`, which the declared `*.json` glob
does not hold (the test now reads the library through `loadScenarios`,
as `test/library.test.ts` does); `check-plugin-teardown-shape
--self-test` needed its pinned fixture commit fetched into this shallow
clone; `check:dual-build-cjs-loads` needed built output.
<details><summary>Derived gates (dispatch-gates --commands, reconciled
with --ran)</summary>
```text
dispatch-gates: gate list derived from the tree of 'objectstack-ai/objectstack' at commit 401e567 (/home/user/objectstack-issue-30982).
Run reconciliation — 50 derived, 50 run, 0 NOT-MEASURED, 0 UNRUN.
✓ dispatch-gates --ran: 50 derived famil(ies) accounted for — 50 run, 0 NOT-MEASURED (a DERIVED zero — all 50 recorded an exit code and none of them is 3).
node scripts/check-closing-keyword-parity.mjs :: exit 0
node scripts/check-closing-keyword-parity.mjs --self-test :: exit 0
node scripts/check-comment-mask-adoption.mjs :: exit 0
node scripts/check-comment-mask-adoption.mjs --self-test :: exit 0
node scripts/check-comment-mask-corpus.mjs :: exit 0
node scripts/check-dts-emitted.mjs --self-test :: exit 0
node scripts/check-issue-citations.mjs :: exit 0
node scripts/check-plugin-teardown-shape.mjs :: exit 0
node scripts/check-plugin-teardown-shape.mjs --self-test :: exit 0
node scripts/ci/assert-turbo-remote-cache.mjs --self-test :: exit 0
node scripts/ci/plan-test-core-shards.mjs --self-test :: exit 0
node scripts/ci/prune-turbo-cache.mjs --self-test :: exit 0
node scripts/docs-audit/check-affected-docs.mjs :: exit 0
node scripts/docs-audit/check-drift-comment.mjs :: exit 0
pnpm --filter @objectstack/spec run check:duration-unit-keys :: exit 0
pnpm check:ci-filter-parity :: exit 0
pnpm check:closed-ceremony-text :: exit 0
pnpm check:cross-package-test-inputs :: exit 0
pnpm check:dispatcher-error-vocabulary :: exit 0
pnpm check:doc-authoring :: exit 0
pnpm check:driver-memory-census :: exit 0
pnpm check:dts-closure :: exit 0
pnpm check:dual-build-cjs-loads :: exit 0
pnpm check:engine-double-contract :: exit 0
pnpm check:error-status-conformance :: exit 0
pnpm check:gitlink-declared :: exit 0
pnpm check:issue-citations :: exit 0
pnpm check:keyed-text-bounds :: exit 0
pnpm check:lean-entry-closure :: exit 0
pnpm check:logger-receiver-detach :: exit 0
pnpm check:nul-bytes :: exit 0
pnpm check:objectql-double-limit :: exit 0
pnpm check:org-identifier :: exit 0
pnpm check:page-declaration-shape :: exit 0
pnpm check:platform-object-tenancy-census :: exit 0
pnpm check:published-files :: exit 0
pnpm check:query-options-erasure :: exit 0
pnpm check:refd-timer-probe :: exit 0
pnpm check:registry-log-declared :: exit 0
pnpm check:rest-log-spy-declared :: exit 0
pnpm check:slot-lookup :: exit 0
pnpm check:sourcemap-no-sources-content :: exit 0
pnpm check:system-context-census :: exit 0
pnpm check:test-source-alias :: exit 0
pnpm check:tier-file-adoption :: exit 0
pnpm check:type-check-coverage :: exit 0
pnpm check:type-check-debt :: exit 0
pnpm check:undeclared-dep-imports :: exit 0
pnpm check:watch-hint-literal :: exit 0
pnpm check:where-matcher :: exit 0
```
</details>
---
_Generated by [Claude
Code](https://claude.ai/code/session_012MmDsRzowAnXEmogPGSEKA)_
---------
Co-authored-by: Claude <noreply@anthropic.com>
Per card and mode the runner writes `docs/qa/platform-checklist/runs/<date>-scenario-<card>.json`
@@ -99,9 +100,24 @@ Per card and mode the runner writes `docs/qa/platform-checklist/runs/<date>-scen
99
100
|`scratch`|`create-objectstack` → the agent builds from the card prompt and the runner's data contract (`prompts.ts`) → `objectstack validate` / `build` / `verify` (proofs chosen from the tags) → score → `objectstack dev --fresh` → seed, one user + permission-set grant + API key per persona → every question as every persona it names and every operation as its persona, over the app's MCP → the iterate change → rebuild, re-score, re-run |
100
101
|`template`| a blank host project (`host<code>`) → boot → install the card's verified build **unmanaged** (`/api/v1/marketplace/install-local`, `installMode: 'unmanaged'`) → the agent applies the iterate change to the installed copy → score → seed → Run. A refused install is a finding in the record |
101
102
102
-
-**Driver** (`driver.ts`): the local `claude` CLI on your own login (`claude -p --output-format stream-json`). It
103
-
refuses to start when `ANTHROPIC_API_KEY` / `ANTHROPIC_AUTH_TOKEN` is set or the CLI is logged in with an API key
104
-
(the cost ruling on cloud#3600), and the record states the driver and login. `RecordedDriver` replays a recording.
103
+
-**Driver** (`driver.ts`): the local `claude` CLI on your own login (`claude -p --output-format stream-json`, the
104
+
prompt written to its stdin: a build prompt carries the whole seed, and one argv string over 128 KiB makes `spawn`
105
+
fail with `E2BIG` on Linux). It refuses to start when `ANTHROPIC_API_KEY` / `ANTHROPIC_AUTH_TOKEN` is set or the CLI
106
+
is logged in with an API key (the cost ruling on cloud#3600), and the record states the driver, the login and the
107
+
authoring timeout. Every turn records its process: elapsed time, and for a turn that did not finish its exit code or
108
+
signal, whether the runner's timeout killed it, and the end of its stderr. `RecordedDriver` replays a recording.
109
+
-**A build that did not finish is not measured** (`orchestrate.ts`): an authoring turn the timeout killed
110
+
(`--author-timeout`, build and iterate alike) leaves its Build half **not measured: author timed out after N min**;
111
+
one that ended with no result line or an error result (`error_max_turns`, `is_error`), or never ran, leaves it **not
112
+
measured: author turn failed: …**. Never a degraded score, and nothing runs on it: no score, seed, Run or iterate. The
113
+
scorecard's author rows show every authoring turn's `resultSubtype` and elapsed time.
114
+
-**Seed** (`seed.ts`): rows go in as the admin, a lookup's target before its referrer. Where the card's declared
115
+
lookups lead back to where they start — `department.manager → employee` with `employee.department → department`, or a
116
+
self-lookup naming a later row — the seed loads in two passes: every row first with each lookup naming a row not in
117
+
yet left out, then one update per row sets those (`seed.secondPass` in the record). The side that waits is one the
118
+
build left optional; a build that makes every side of a cycle `required` cannot be seeded, which the data contract
119
+
states and the scorecard reports as a Build finding naming the cycle (the seed is still sent, so the record holds
120
+
the app's own refusals).
105
121
-**Observation** (`observe.ts`): the write grader is fed what the runner reads itself: the agent's write calls and
106
122
the door's answer (a coded refusal), the object's rows before and after the turn, and the audit trail's actor. A turn
107
123
that wrote nothing because the object refused the read it needed first (a persona that may not read it) grades as
if(opts.authorTimeoutMin!==undefined)thrownewError('--author-timeout applies to the live driver only: a replay re-reads each turn as it was recorded');
0 commit comments