Repository navigation
Commit fba4c00
feat(scenarios): publish @objectstack/scenarios with an objectstack-scenarios CLI (npx), carrying the open cards only (#31221)
<!-- claude-projects-attribution: {"github_login":"os-project-manager"}
-->
_Requested by **Leehom** · [project
thread](https://claude.ai/code/project/chan_01TenzsRopZj4jMbvjX8uz1X?thread=cmsg_01TenzsRopZj4jMbvjX8uz1XQogyQWa7pC8SuBM9gesGp2)_
Fixes #31172
Clause-②: yes
Metadata-protocol: none
## 维护者速读
- 改了什么:`@objectstack/scenarios` 去掉 `private`,随 `@objectstack/*`
发版列车一起发布,带一个命令 `objectstack-scenarios`。`npx @objectstack/scenarios run
…` 就是今天的
`run-scenarios`,参数、默认值和拒绝规则完全相同;包里只带开放卡库(`docs/qa/scenarios/*.json`
与能力区清单),构建时拷进 `dist/library/`。
- 为什么改:维护者原话「把执行器发布成一个独立的命令行工具,同意」。外部团队和其他 agent 工具不用克隆平台仓库就能跑卡片。不是 `os`
子命令,裁决 D 不变。
- 风险与回滚:新增一个公开包和一个命令,已有包的行为不变;最大的风险是私有卡片随包外泄,所以打包测试对 tarball 做白名单:除了
`package.json`、与 `docs/qa/scenarios/` 逐字节相同的卡片、与能力区逐字节相同的清单,任何 JSON
出现都红。回滚即 revert 本 PR(发版前没有任何东西发布出去)。
-
先看:`packages/qa/scenarios/test/build/packed-tarball.test.ts`(打包白名单与命令冒烟)、`packages/qa/scenarios/src/runner/library.ts`(装在
npm 里时去哪读卡库)、`packages/qa/scenarios/scripts/pack-library.mjs`(只拷两个目录)。
- 两分钟验证:`pnpm --filter @objectstack/scenarios build && pnpm --filter
@objectstack/scenarios exec vitest run
test/build/packed-tarball.test.ts`,预期 10/10 通过(打包、解包、用纯 Node 跑
`objectstack-scenarios`,读到包内 9 张卡,`--packages workspace` 与 API key
登录都被拒)。
- 需要你做的:无,席位落地。
## What changes
- **The package is published** (`packages/qa/scenarios/package.json`):
`private` removed, `publishConfig.access: public`, `files: ["dist",
"README.md", "CHANGELOG.md"]`, `engines.node >=22`, all copied from the
nearest published sibling (`@objectstack/verify`). It joins the
changesets `fixed` group (`.changeset/config.json`), so it versions with
the train; nothing is bumped by hand (it stays `0.1.0` in the tree until
the release's `changeset version`). The description now says what it is:
published, with the CLI and the open library.
- **The command line, a thin wrapper.** `bin.objectstack-scenarios` is
`bin/objectstack-scenarios.js` (the `create-objectstack` pattern): it
imports the BUILT `dist/bin.js` and nothing else. `src/bin.ts`
(`runCli`) dispatches `run` to the runner CLI's own `main` with every
following argument unchanged, plus `help` and `version`. The runner is
imported lazily, for `run` only, and the build keeps it out of
`dist/bin.js`: a tsup esbuild plugin (`binLoadsBuiltCli`) turns that
import into one of a separate built entry, `dist/cli.js` (`.cjs` for the
CJS build), so `help` and `version` load `dist/bin.js` alone (1.5 KB;
measured 30 ms, against about 1.2 s warm for the runner's graph).
`runner/cli.ts` is not restructured; it changes in two places: where it
finds the repository (below), and one refusal (next bullet).
- **The open library as package data.** The build's new step
`scripts/pack-library.mjs` copies `docs/qa/scenarios/*.json` and
`docs/qa/platform-checklist/areas/*.json` (one level, regular files,
nothing else) into `dist/library/`, keeping the repository's layout. The
package's `turbo.json` adds both directories to the build's inputs, so a
card edit invalidates the cached build the release would otherwise
replay (`$TURBO_EXTENDS$` keeps the root inputs; measured with `turbo
run build --dry=json`: 25 `docs/qa` inputs, 9 cards + 16 areas).
- **Where the runner reads the library** (`src/runner/library.ts`,
`locateLibraryRoot`): in a checkout it is the repository, as before;
installed from npm it is `dist/library/`. A checkout is recognised by
the package sitting at `packages/qa/scenarios` of a pnpm workspace
holding `docs/qa/scenarios` (compared by real path), so an install
inside some other workspace is never read as one. `cli.ts` uses it
instead of `findUp(pnpm-workspace.yaml)`, which from an npm install
would have found the user's workspace or thrown. From npm, `--out`
defaults to `./scenario-runs` (not into the package), and no framework
commit is stamped (the package may sit in someone else's git
repository). `libraryOf` is unchanged: the shipped copy is the `open`
library, any `--cards` directory the overlay.
- **`--packages workspace` is refused from npm** (`cli.ts`, right after
the library root is located): workspace mode (#31188) builds and packs a
checkout's packages, and an installed package has none, so the run stops
with that reason before it creates a work directory, builds or packs
anything. `published` (the default) is unchanged.
- **README**: "Running from npm" (prerequisites, the cards it carries,
where records go, the package source, not an `os` command); the intro
and build paragraphs no longer say "private, unpublished".
- **Changeset** `.changeset/31172-publish-scenarios-cli.md`, `patch`,
with both declaration lines. Corrected from `minor` in `9742af04`: with
`Metadata-protocol: none` the launch-window level rule (ruling A on
#25105) grades it `patch`, and `Check Changeset` refused `minor`.
- **`packages/spec/llms.txt`**: its package-ecosystem section counts the
published packages and `check:llms-txt` re-derives the count from the
workspace, so it moves 74 → 75. Not the expected surface; named here
because the gate makes it this PR's. The file ships in
`@objectstack/spec`, which is in the same `fixed` group, so it goes out
with the next release either way.
### Public surface (Clause-②: yes), measured on the built entry
declarations
Read with the TypeScript checker (`getExportsOfModule`) over
`dist/*.d.ts` after a full build of this head:
| Subpath | Exports | Values | Type-only |
|:--|--:|--:|--:|
| `@objectstack/scenarios` (`dist/index.d.ts`) | 79 | 55 | 24 |
| `@objectstack/scenarios/score` | 38 | 17 | 21 |
| `@objectstack/scenarios/score/full` | 56 | 25 | 31 |
| `@objectstack/scenarios/runner` | 165 | 118 | 47 |
| `@objectstack/scenarios/export` | 68 | 39 | 29 |
These five subpaths existed (private) and are unchanged by this PR
except that they are now on npm (`./runner` and `./export` count what
#31188 added to them on main). New: the `objectstack-scenarios` command
(`run`, `help`, `version`). `dist/bin.js` (`runCli`, `USAGE`, `CliIo`)
and `dist/cli.js` (the runner CLI) are not `exports` keys, so a consumer
cannot resolve them; `library.ts` is not re-exported from `./runner`.
## Tests
- **`test/build/packed-tarball.test.ts`** (new, 10 cases): `pnpm pack`
from the package directory (the tarball `pnpm publish` uploads), then
the tarball extracted into a project of its own with each runtime
dependency linked to the workspace build.
- The manifest is public, names the bin, and has no `workspace:` range
left.
- Every card of `docs/qa/scenarios/` and every area of
`docs/qa/platform-checklist/areas/` is in `dist/library/`, byte for
byte.
- **No other JSON anywhere in the tarball**: an allow-list of
`package.json`, those cards and those areas. A card from anywhere else,
wherever it lands, fails this case.
- Every shipped card is `origin: generic`.
- No TypeScript source (only `.d.ts`/`.d.cts`), and no `src/`, `test/`
or `scripts/`.
- `dist/bin.js` imports `fs`, `path` and `url` and nothing else
statically, and reaches the runner only through `import("./cli.js")`.
- The packed bin under plain Node: `--help` (exit 0), `version` (the
packed version), no command (exit 2); `run --only no_such_card` loads
the shipped library from `INSTALL/dist/library/docs/qa/scenarios` with
no refusal and exits 2 with exactly that one line; `run --packages
workspace` exits 1 on the refusal and creates no work directory; with a
stand-in `claude` on a subscription login and `ANTHROPIC_API_KEY` set,
`run --only service_desk` exits 1 on "`ANTHROPIC_API_KEY` is set" and
scaffolds nothing.
- **Budgets.** The three `run` cases each spawn Node over the runner's
whole graph once: about 1.1 to 1.5 s each here, 5.7 s and 6.5 s cold on
a CI Test Core runner (run 42400730182, where vitest's 5 s default timed
them out). Each gets an explicit 30 s (`RUN_SPAWN_BUDGET_MS`), as
`built-consumer.test.ts` budgets its own spawn of that graph. The
help/version case is three spawns of `dist/bin.js` alone (123 ms here)
and keeps the default. The whole file reads 5.1 s here.
- **`test/runner/library.test.ts`** (new, 6 cases): `locateLibraryRoot`
reads the checkout from the sources, the packed copy from an installed
package, never another workspace that holds a `docs/qa/scenarios` and a
`packages/qa/scenarios`, and refuses an installed package with no
library; `runCli` prints usage, refuses no/unknown command with exit 2,
and hands `run`'s arguments to the runner CLI unchanged (`--bogus`,
`--modes nightly` refused by `parseArgs`). The file imports
`src/runner/cli.ts` itself, so the lazy import's transform is paid in
the file's import phase, not inside a case.
<details><summary>Ablations (each restored: blob == HEAD, git diff HEAD
empty)</summary>
All through `scripts/ablation-replace.mjs` (anchor x1 → x0, replacement
x0 → x1); 1 and 2 on commit `2d69cfb4`, 3 on `2eb91a69`, 4 on
`149e988b`.
- **Ablation 1, the copy widened** (`pack-library.mjs` `DIRS` gains
`docs/qa/platform-checklist`): the inner script rebuilt the package
(`pack-library` ran, `dist/library/docs/qa/platform-checklist/` then
held `areas` and `coverage.json`) and ran `packed-tarball.test.ts` → `1
failed | 7 passed`, the failure being "carries no other JSON: a card
from anywhere but docs/qa/scenarios is refused". Restored, and the
package rebuilt afterwards (`dist/library/docs/qa/platform-checklist/`
holds `areas` only).
- **Ablation 2, the checkout test loosened** (`library.ts`: the
real-path comparison of `packages/qa/scenarios` with the package
replaced by an always-true `String("ABL2").length > 0`):
`library.test.ts` → `1 failed | 5 passed`, the failure being "never
reads another pnpm workspace holding a docs/qa/scenarios as the
checkout". The suite imports `src/`, so no dist leg.
- **Ablation 3, the runner bundled back into the bin**
(`tsup.config.ts`: `esbuildPlugins: [binLoadsBuiltCli].slice(0, 0)`):
rebuilt, `dist/bin.js` then carried 19 static `@objectstack/*` imports
(196 KB); `packed-tarball.test.ts` → `1 failed | 8 passed`, the failure
being "keeps the runner out of the bin". Restored, rebuilt.
- **Ablation 4, the workspace refusal disarmed** (`cli.ts`: its
condition ANDed with an always-false `String("ABL4").length < 0`):
rebuilt, `ablation-dist-preflight` found the marker in `dist/cli.js` and
`dist/cli.cjs`; `packed-tarball.test.ts -t 'refuses --packages
workspace'` → `1 failed`, the run reaching workspace mode and dying on
`fatal: not a git repository` instead of the reason. Restored, rebuilt,
preflight `--absent`: marker absent from all 81 built files, tree clean.
- The API-billed refusal the smoke test pins is the runner's existing
behaviour, which this diff does not change, so it is not ablated.
</details>
## Acceptance notes
- **The private overlay cannot reach the tarball by construction and is
refused if it does.** The overlay (objectstack-ai/cloud#3603) lives
outside this repository; `pack-library.mjs` copies two named directories
of this repository, one level deep; and the packaging test refuses any
JSON that is not byte-identical to an open card or area. It runs in Test
Core on every PR and in the merge queue, so a leak is red before it can
merge, rather than at publish. The triage audit's 1 parent 0d0c377 commit fba4c00
14 files changed
Lines changed: 619 additions & 25 deletions
File tree
- .changeset
- packages
- qa/scenarios
- bin
- scripts
- src
- runner
- test
- build
- runner
- spec
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
83 | 83 | | |
84 | 84 | | |
85 | 85 | | |
| 86 | + | |
86 | 87 | | |
87 | 88 | | |
88 | 89 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | | - | |
| 3 | + | |
| 4 | + | |
4 | 5 | | |
5 | 6 | | |
6 | 7 | | |
| |||
27 | 28 | | |
28 | 29 | | |
29 | 30 | | |
30 | | - | |
31 | | - | |
32 | | - | |
33 | | - | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
34 | 36 | | |
35 | 37 | | |
36 | 38 | | |
| |||
147 | 149 | | |
148 | 150 | | |
149 | 151 | | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
150 | 177 | | |
151 | 178 | | |
152 | 179 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
4 | | - | |
5 | 4 | | |
6 | | - | |
| 5 | + | |
7 | 6 | | |
8 | 7 | | |
9 | 8 | | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
10 | 12 | | |
11 | 13 | | |
12 | 14 | | |
| |||
60 | 62 | | |
61 | 63 | | |
62 | 64 | | |
63 | | - | |
| 65 | + | |
64 | 66 | | |
65 | 67 | | |
66 | 68 | | |
| |||
93 | 95 | | |
94 | 96 | | |
95 | 97 | | |
96 | | - | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
97 | 110 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | | - | |
| 7 | + | |
| 8 | + | |
8 | 9 | | |
9 | 10 | | |
10 | 11 | | |
| |||
32 | 33 | | |
33 | 34 | | |
34 | 35 | | |
35 | | - | |
| 36 | + | |
| 37 | + | |
36 | 38 | | |
37 | 39 | | |
38 | 40 | | |
39 | 41 | | |
40 | 42 | | |
41 | | - | |
| 43 | + | |
| 44 | + | |
42 | 45 | | |
43 | 46 | | |
44 | 47 | | |
45 | 48 | | |
46 | 49 | | |
47 | 50 | | |
48 | | - | |
| 51 | + | |
49 | 52 | | |
50 | 53 | | |
51 | 54 | | |
52 | 55 | | |
53 | 56 | | |
54 | 57 | | |
55 | 58 | | |
| 59 | + | |
56 | 60 | | |
57 | 61 | | |
58 | 62 | | |
| |||
191 | 195 | | |
192 | 196 | | |
193 | 197 | | |
194 | | - | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
195 | 203 | | |
196 | | - | |
| 204 | + | |
197 | 205 | | |
198 | 206 | | |
199 | 207 | | |
200 | 208 | | |
201 | 209 | | |
202 | | - | |
| 210 | + | |
203 | 211 | | |
204 | 212 | | |
205 | 213 | | |
| |||
250 | 258 | | |
251 | 259 | | |
252 | 260 | | |
253 | | - | |
| 261 | + | |
| 262 | + | |
254 | 263 | | |
255 | 264 | | |
256 | 265 | | |
| |||
0 commit comments