Skip to content

Commit 3f970b9

Browse files
test(cli): declare measured budgets on the generate-refuses and build spawn files (#23181 slice 11) (#26174)
<!-- claude-projects-attribution: {"github_login":"os-project-manager"} --> _Requested by **Leehom** · [project thread](https://claude.ai/code/project/chan_01TenzsRopZj4jMbvjX8uz1X?thread=cmsg_01TenzsRopZj4jMbvjX8uz1XUA7Q7GidaxhPzTGbwdi7pX)_ Part of #23181 (slice 11). The card is a series and stays open. Clause-②: no Metadata-protocol: none Test-only, comment-only diff: `skip-changeset` (nothing in any package's `files[]` changes; `packages/cli/test/**` is not published). ## Rows this PR addresses The ten `@objectstack/cli` integration files of the per-case CLI spawn family (`test/generate-refuses-*.test.ts` and the five named `test/build-*.test.ts`). Reading used: the per-file list of shard-timings refresh run `40275747937` (PR #25960, the newest refresh with a per-file list; it was not merged, and PR #26089 used the same list for the spec/lint rows). The card's own table (run `38340176409`) is older and lower. | file | spawns | slowest CI reading (run) | runs over 10 s / executed | local wall, 3 runs (s) | budget declared | |:--|--:|--:|--:|--:|--:| | `test/build-callable-names.test.ts` | 9 | 123.82 s (`40200279306`) | 5 / 5 | 84.5 / 89.7 / 86.5 | 160 | | `test/generate-refuses-unimportable-alias.test.ts` | 7 | 94.13 s (`40204927516`) | 3 / 3 | 65.7 / 68.4 / 64.7 | 120 | | `test/build-runtime-bundle-bodyless.test.ts` | 6 | 92.69 s (`40200279306`) | 3 / 3 | 57.8 / 57.8 / 57.8 | 120 | | `test/build-strict-body-count.test.ts` | 6 | 82.44 s (`40204927516`) | 3 / 3 | 55.1 / 58.4 / 59.0 | 110 | | `test/generate-refuses-namespace-prefix.test.ts` | 5 | 69.59 s (`40204927516`) | 3 / 3 | 49.0 / 50.9 / 49.5 | 90 | | `test/build-view-container-name.test.ts` | 5 | 66.67 s (`40204927516`) | 4 / 4 | 49.3 / 48.8 / 46.8 | 90 | | `test/generate-refuses-unparseable-name.test.ts` | 5 | 66.41 s (`40200279306`) | 5 / 5 | 47.9 / 48.5 / 45.5 | 90 | | `test/generate-refuses-name-outside-charset.test.ts` | 4 | 54.70 s (`40204927516`) | 4 / 4 | 37.9 / 36.7 / 37.5 | 70 | | `test/generate-refuses-retired-generator.test.ts` | 4 | 53.28 s (`40204927516`) | 4 / 4 | 36.2 / 35.2 / 37.8 | 70 | | `test/build-text-face-advisory-count.test.ts` | 3 | 41.41 s (`40200279306`) | 5 / 5 | 25.3 / 25.0 / 27.2 | 60 | Budget = slowest CI reading x 1.25, rounded up to the next 10 s (the grammar `readTestBudget` in `scripts/measure-test-shard-timings.mjs` enforces; all ten lines parse through it). Pin: the next refresh lists none of these ten in its over-budget table; each appears under "Held to the budget it declares" instead, and is listed again only if a run reads it over its own budget. ## Why a declared budget, not a cheaper test The order's hypothesis was a per-case boot that one `beforeAll` could share. Measured, it does not hold here: - **The cost is the spawn floor, not the cases.** One `tsx bin/run-dev.js` spawn costs 8.2 to 8.5 s locally, and `os --version` alone costs 8.4 to 8.8 s: the source entry has oclif import all 72 command modules through tsx on every start, because no `oclif.manifest.json` exists. Every file's local wall is its spawn count x 8.5 to 9.5 s (for example 7 spawns, 65 s; 3 spawns, 25 s). - **There is nothing to share.** The five `generate-refuses-*` files already run all their spawns in one `beforeAll`. Every spawn in all ten files is a different argv or a different fixture: each one is a single-cause case (a refusal, its dry-run form, the neighbouring layer's verdict, or a control). Cutting one removes a case. - So no file with even one spawn fits the 10 s line without changing the spawn entry, which the claim leaves out. These files are exactly the class ruling record 6181973363 letter B names ("spawns the real producer once per single-cause fixture"). Each now carries its one-line `// test-budget:` header, as eleven other cli spawn files already do. - Concurrency inside a file was looked at and not taken. Two parallel spawns cost the same 9.3 s as one, but four take 17.5 s and seven take 28.9 s on this 4-core box. That is packing, not less work, which is the reason `packages/cli/vitest.config.ts` records for rejecting `maxWorkers: 4`. ## The lever that would make these budgets fall (a decision, not taken here) Measured on this tree, with an `oclif.manifest.json` written into `packages/cli` for the probe and removed afterwards (`git status` clean): `tsx bin/run-dev.js generate object order_line --dry-run` drops from 8.2 to 8.5 s to **2.0 to 2.3 s**, with byte-identical output (1412 bytes) and the same exit code. `--version` drops to 0.4 s. A refusal case (`generate object class`) still exits 1 in 2.2 s. Applied to this family, local wall would fall from about 520 s to about 130 s. It is reported to the seat as an open decision because it changes the spawn shape these files document. ## Proof - Diff: 10 files, +10 lines, each a header comment at line 2. No case, assertion, timeout, tier or project changes. - Cases still isolated: shuffled run at `179c8078a`, `pnpm exec vitest run --project integration --maxWorkers=2 --sequence.shuffle --sequence.seed=23181` + the 10 files: Test Files 10 passed (10), Tests 91 passed (91), lock VERDICT command-exit 0. - Wall before: the table's three local runs, under `os-verify-lock`, `--project integration --maxWorkers=2`, the default reporter's per-file module line (the number CI's per-file timings read). The after wall is the same tree's behaviour (comment-only diff), so it was not re-run three times; the shuffled run above is the after reading. - `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` at `179c8078a`: 55 commands, all exit 0 (two first exited 3 on prerequisites: `check-plugin-teardown-shape --self-test` on a shallow clone, fixed by fetching `621a4876`; `check:dual-build-cjs-loads` on eight unbuilt packages, fixed by building them). `--ran`: "55 derived famil(ies) accounted for, 55 run, 0 NOT-MEASURED". - `pnpm --filter @objectstack/cli typecheck`: exit 0 (`check:test-typecheck: OK`). - `pnpm exec eslint --no-inline-config packages/cli`: exit 0. ## Acceptance notes - Three spawns repeat across files with the same argv in an empty project: `os g object class` (in `unimportable-alias`, `unparseable-name`, `name-outside-charset`), `os g object order-line` (in `unimportable-alias`, `name-outside-charset`) and `os g object order_line` (in `unparseable-name`, `name-outside-charset`). Each file asserts a different layer's wording on its copy, so they are not removed here. Folding them is about 4 spawns (about 35 s per run) and a cross-file reshuffle of the #16726, #16541 and #17410 layer pins. --- _Generated by [Claude Code](https://claude.ai/code/session_01Pq8DZ41armkBBdhQUufpjH)_ Co-authored-by: Claude <noreply@anthropic.com>
1 parent 51cc7f9 commit 3f970b9

10 files changed

Lines changed: 10 additions & 0 deletions

‎packages/cli/test/build-callable-names.test.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
// test-budget: 160 — spawns the real producer once per single-cause fixture: nine spawns of the source CLI, `os build` over one fixture per callable kind the census derives plus the control and the named-export stack, each artifact then loaded through loadArtifactBundle; each spawn's floor is the source entry loading all 72 commands (about 8.5 s, `os --version` included); slowest CI reading 123.82 s (run 40200279306) in the window of shard-timings refresh run 40275747937 (PR #25960), over 10 s in 5 of 5 runs that executed it, x 1.25, rounded up
23

34
/**
45
* `objectstack build` ships every authored `functions` name under its own name,

‎packages/cli/test/build-runtime-bundle-bodyless.test.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
// test-budget: 120 — spawns the real producer once per single-cause fixture: six spawns of the source CLI, `os build` over six stacks that differ only in which callables keep a body; each spawn's floor is the source entry loading all 72 commands (about 8.5 s, `os --version` included); slowest CI reading 92.69 s (run 40200279306) in the window of shard-timings refresh run 40275747937 (PR #25960), over 10 s in 3 of 3 runs that executed it, x 1.25, rounded up
23

34
/**
45
* `objectstack build` writes the runtime module (`objectstack-runtime.*.mjs`)

‎packages/cli/test/build-strict-body-count.test.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
// test-budget: 110 — spawns the real producer once per single-cause fixture: six spawns of the source CLI, `os build --strict-body` over the zero control, the one and four fixtures in the text and `--json` faces, and the `functions`-entry stack; each spawn's floor is the source entry loading all 72 commands (about 8.5 s, `os --version` included); slowest CI reading 82.44 s (run 40204927516) in the window of shard-timings refresh run 40275747937 (PR #25960), over 10 s in 3 of 3 runs that executed it, x 1.25, rounded up
23

34
/**
45
* [#22839] `objectstack build --strict-body` counts callables, and its

‎packages/cli/test/build-text-face-advisory-count.test.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
// test-budget: 60 — spawns the real producer once per single-cause fixture: three spawns of the source CLI, `os build` over the multi-package stack in the text and `--json` faces and over the single-package control; each spawn's floor is the source entry loading all 72 commands (about 8.5 s, `os --version` included); slowest CI reading 41.41 s (run 40200279306) in the window of shard-timings refresh run 40275747937 (PR #25960), over 10 s in 5 of 5 runs that executed it, x 1.25, rounded up
23

34
/**
45
* #18780 — `os build`'s TEXT face counted per-package advisories it never

‎packages/cli/test/build-view-container-name.test.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
// test-budget: 90 — spawns the real producer once per single-cause fixture: five spawns of the source CLI, `os build` over the divergent container in both faces, the `packages[]` stack and the two controls; each spawn's floor is the source entry loading all 72 commands (about 8.5 s, `os --version` included); slowest CI reading 66.67 s (run 40204927516) in the window of shard-timings refresh run 40275747937 (PR #25960), over 10 s in 4 of 4 runs that executed it, x 1.25, rounded up
23

34
/**
45
* [#20393] `os build` refuses a `views:` container whose own `name` disagrees

‎packages/cli/test/generate-refuses-name-outside-charset.test.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
// test-budget: 70 — spawns the real producer once per single-cause fixture: four spawns of the source CLI (`os g object order-line`, `os g flow lead-qual --dry-run`, `os g object class` and the `order_line` control), each in its own empty project; each spawn's floor is the source entry loading all 72 commands (about 8.5 s, `os --version` included); slowest CI reading 54.70 s (run 40204927516) in the window of shard-timings refresh run 40275747937 (PR #25960), over 10 s in 4 of 4 runs that executed it, x 1.25, rounded up
23

34
/**
45
* PIN (#16726) — `os generate <type> <name>` refuses a name outside the

‎packages/cli/test/generate-refuses-namespace-prefix.test.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
// test-budget: 90 — spawns the real producer once per single-cause fixture: five spawns of the source CLI (`os g object order__line`, its view dry run, `os g object customer` over a config that will not load, and one control in each project), each refusal read against its project before that control; each spawn's floor is the source entry loading all 72 commands (about 8.5 s, `os --version` included); slowest CI reading 69.59 s (run 40204927516) in the window of shard-timings refresh run 40275747937 (PR #25960), over 10 s in 3 of 3 runs that executed it, x 1.25, rounded up
23

34
/**
45
* PIN (#20197) — the two refusals the namespace prefix added to

‎packages/cli/test/generate-refuses-retired-generator.test.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
// test-budget: 70 — spawns the real producer once per single-cause fixture: four spawns of the source CLI (`os generate schema`, `os g schema -o`, `os g constructor thing` and the `object --dry-run` control) in one project, each listing read before the next; each spawn's floor is the source entry loading all 72 commands (about 8.5 s, `os --version` included); slowest CI reading 53.28 s (run 40204927516) in the window of shard-timings refresh run 40275747937 (PR #25960), over 10 s in 4 of 4 runs that executed it, x 1.25, rounded up
23

34
/**
45
* PIN (#19098) — the retired-generator door of `os generate`, in the run that

‎packages/cli/test/generate-refuses-unimportable-alias.test.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
// test-budget: 120 — spawns the real producer once per single-cause fixture: seven spawns of the source CLI (`os g dashboard class`, `os g skill let`, the dry run, `os g object class`, `os g object order-line`, and the `order_line` and `type` controls), each in its own empty project; each spawn's floor is the source entry loading all 72 commands (about 8.5 s, `os --version` included); slowest CI reading 94.13 s (run 40204927516) in the window of shard-timings refresh run 40275747937 (PR #25960), over 10 s in 3 of 3 runs that executed it, x 1.25, rounded up
23

34
/**
45
* PIN (#17410) — `os generate` must not EXIT 0 on a name whose barrel alias no

‎packages/cli/test/generate-refuses-unparseable-name.test.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
// test-budget: 90 — spawns the real producer once per single-cause fixture: five spawns of the source CLI (`os g object foo.bar`, `os g flow foo.bar`, the dry run, `os g object class` and the `order_line` control); each spawn's floor is the source entry loading all 72 commands (about 8.5 s, `os --version` included); slowest CI reading 66.41 s (run 40200279306) in the window of shard-timings refresh run 40275747937 (PR #25960), over 10 s in 5 of 5 runs that executed it, x 1.25, rounded up
23

34
/**
45
* PIN (#16541) — `os generate` must not EXIT 0 on a name whose emission does

0 commit comments

Comments
 (0)