Skip to content

Commit fba4c00

Browse files
feat(scenarios): publish @objectstack/scenarios with an objectstack-scenarios CLI (npx), carrying the open cards only (#31221)
<!-- claude-projects-attribution: {"github_login":"os-project-manager"} --> _Requested by **Leehom** · [project thread](https://claude.ai/code/project/chan_01TenzsRopZj4jMbvjX8uz1X?thread=cmsg_01TenzsRopZj4jMbvjX8uz1XQogyQWa7pC8SuBM9gesGp2)_ Fixes #31172 Clause-②: yes Metadata-protocol: none ## 维护者速读 - 改了什么:`@objectstack/scenarios` 去掉 `private`,随 `@objectstack/*` 发版列车一起发布,带一个命令 `objectstack-scenarios`。`npx @objectstack/scenarios run …` 就是今天的 `run-scenarios`,参数、默认值和拒绝规则完全相同;包里只带开放卡库(`docs/qa/scenarios/*.json` 与能力区清单),构建时拷进 `dist/library/`。 - 为什么改:维护者原话「把执行器发布成一个独立的命令行工具,同意」。外部团队和其他 agent 工具不用克隆平台仓库就能跑卡片。不是 `os` 子命令,裁决 D 不变。 - 风险与回滚:新增一个公开包和一个命令,已有包的行为不变;最大的风险是私有卡片随包外泄,所以打包测试对 tarball 做白名单:除了 `package.json`、与 `docs/qa/scenarios/` 逐字节相同的卡片、与能力区逐字节相同的清单,任何 JSON 出现都红。回滚即 revert 本 PR(发版前没有任何东西发布出去)。 - 先看:`packages/qa/scenarios/test/build/packed-tarball.test.ts`(打包白名单与命令冒烟)、`packages/qa/scenarios/src/runner/library.ts`(装在 npm 里时去哪读卡库)、`packages/qa/scenarios/scripts/pack-library.mjs`(只拷两个目录)。 - 两分钟验证:`pnpm --filter @objectstack/scenarios build && pnpm --filter @objectstack/scenarios exec vitest run test/build/packed-tarball.test.ts`,预期 10/10 通过(打包、解包、用纯 Node 跑 `objectstack-scenarios`,读到包内 9 张卡,`--packages workspace` 与 API key 登录都被拒)。 - 需要你做的:无,席位落地。 ## What changes - **The package is published** (`packages/qa/scenarios/package.json`): `private` removed, `publishConfig.access: public`, `files: ["dist", "README.md", "CHANGELOG.md"]`, `engines.node >=22`, all copied from the nearest published sibling (`@objectstack/verify`). It joins the changesets `fixed` group (`.changeset/config.json`), so it versions with the train; nothing is bumped by hand (it stays `0.1.0` in the tree until the release's `changeset version`). The description now says what it is: published, with the CLI and the open library. - **The command line, a thin wrapper.** `bin.objectstack-scenarios` is `bin/objectstack-scenarios.js` (the `create-objectstack` pattern): it imports the BUILT `dist/bin.js` and nothing else. `src/bin.ts` (`runCli`) dispatches `run` to the runner CLI's own `main` with every following argument unchanged, plus `help` and `version`. The runner is imported lazily, for `run` only, and the build keeps it out of `dist/bin.js`: a tsup esbuild plugin (`binLoadsBuiltCli`) turns that import into one of a separate built entry, `dist/cli.js` (`.cjs` for the CJS build), so `help` and `version` load `dist/bin.js` alone (1.5 KB; measured 30 ms, against about 1.2 s warm for the runner's graph). `runner/cli.ts` is not restructured; it changes in two places: where it finds the repository (below), and one refusal (next bullet). - **The open library as package data.** The build's new step `scripts/pack-library.mjs` copies `docs/qa/scenarios/*.json` and `docs/qa/platform-checklist/areas/*.json` (one level, regular files, nothing else) into `dist/library/`, keeping the repository's layout. The package's `turbo.json` adds both directories to the build's inputs, so a card edit invalidates the cached build the release would otherwise replay (`$TURBO_EXTENDS$` keeps the root inputs; measured with `turbo run build --dry=json`: 25 `docs/qa` inputs, 9 cards + 16 areas). - **Where the runner reads the library** (`src/runner/library.ts`, `locateLibraryRoot`): in a checkout it is the repository, as before; installed from npm it is `dist/library/`. A checkout is recognised by the package sitting at `packages/qa/scenarios` of a pnpm workspace holding `docs/qa/scenarios` (compared by real path), so an install inside some other workspace is never read as one. `cli.ts` uses it instead of `findUp(pnpm-workspace.yaml)`, which from an npm install would have found the user's workspace or thrown. From npm, `--out` defaults to `./scenario-runs` (not into the package), and no framework commit is stamped (the package may sit in someone else's git repository). `libraryOf` is unchanged: the shipped copy is the `open` library, any `--cards` directory the overlay. - **`--packages workspace` is refused from npm** (`cli.ts`, right after the library root is located): workspace mode (#31188) builds and packs a checkout's packages, and an installed package has none, so the run stops with that reason before it creates a work directory, builds or packs anything. `published` (the default) is unchanged. - **README**: "Running from npm" (prerequisites, the cards it carries, where records go, the package source, not an `os` command); the intro and build paragraphs no longer say "private, unpublished". - **Changeset** `.changeset/31172-publish-scenarios-cli.md`, `patch`, with both declaration lines. Corrected from `minor` in `9742af04`: with `Metadata-protocol: none` the launch-window level rule (ruling A on #25105) grades it `patch`, and `Check Changeset` refused `minor`. - **`packages/spec/llms.txt`**: its package-ecosystem section counts the published packages and `check:llms-txt` re-derives the count from the workspace, so it moves 74 → 75. Not the expected surface; named here because the gate makes it this PR's. The file ships in `@objectstack/spec`, which is in the same `fixed` group, so it goes out with the next release either way. ### Public surface (Clause-②: yes), measured on the built entry declarations Read with the TypeScript checker (`getExportsOfModule`) over `dist/*.d.ts` after a full build of this head: | Subpath | Exports | Values | Type-only | |:--|--:|--:|--:| | `@objectstack/scenarios` (`dist/index.d.ts`) | 79 | 55 | 24 | | `@objectstack/scenarios/score` | 38 | 17 | 21 | | `@objectstack/scenarios/score/full` | 56 | 25 | 31 | | `@objectstack/scenarios/runner` | 165 | 118 | 47 | | `@objectstack/scenarios/export` | 68 | 39 | 29 | These five subpaths existed (private) and are unchanged by this PR except that they are now on npm (`./runner` and `./export` count what #31188 added to them on main). New: the `objectstack-scenarios` command (`run`, `help`, `version`). `dist/bin.js` (`runCli`, `USAGE`, `CliIo`) and `dist/cli.js` (the runner CLI) are not `exports` keys, so a consumer cannot resolve them; `library.ts` is not re-exported from `./runner`. ## Tests - **`test/build/packed-tarball.test.ts`** (new, 10 cases): `pnpm pack` from the package directory (the tarball `pnpm publish` uploads), then the tarball extracted into a project of its own with each runtime dependency linked to the workspace build. - The manifest is public, names the bin, and has no `workspace:` range left. - Every card of `docs/qa/scenarios/` and every area of `docs/qa/platform-checklist/areas/` is in `dist/library/`, byte for byte. - **No other JSON anywhere in the tarball**: an allow-list of `package.json`, those cards and those areas. A card from anywhere else, wherever it lands, fails this case. - Every shipped card is `origin: generic`. - No TypeScript source (only `.d.ts`/`.d.cts`), and no `src/`, `test/` or `scripts/`. - `dist/bin.js` imports `fs`, `path` and `url` and nothing else statically, and reaches the runner only through `import("./cli.js")`. - The packed bin under plain Node: `--help` (exit 0), `version` (the packed version), no command (exit 2); `run --only no_such_card` loads the shipped library from `INSTALL/dist/library/docs/qa/scenarios` with no refusal and exits 2 with exactly that one line; `run --packages workspace` exits 1 on the refusal and creates no work directory; with a stand-in `claude` on a subscription login and `ANTHROPIC_API_KEY` set, `run --only service_desk` exits 1 on "`ANTHROPIC_API_KEY` is set" and scaffolds nothing. - **Budgets.** The three `run` cases each spawn Node over the runner's whole graph once: about 1.1 to 1.5 s each here, 5.7 s and 6.5 s cold on a CI Test Core runner (run 42400730182, where vitest's 5 s default timed them out). Each gets an explicit 30 s (`RUN_SPAWN_BUDGET_MS`), as `built-consumer.test.ts` budgets its own spawn of that graph. The help/version case is three spawns of `dist/bin.js` alone (123 ms here) and keeps the default. The whole file reads 5.1 s here. - **`test/runner/library.test.ts`** (new, 6 cases): `locateLibraryRoot` reads the checkout from the sources, the packed copy from an installed package, never another workspace that holds a `docs/qa/scenarios` and a `packages/qa/scenarios`, and refuses an installed package with no library; `runCli` prints usage, refuses no/unknown command with exit 2, and hands `run`'s arguments to the runner CLI unchanged (`--bogus`, `--modes nightly` refused by `parseArgs`). The file imports `src/runner/cli.ts` itself, so the lazy import's transform is paid in the file's import phase, not inside a case. <details><summary>Ablations (each restored: blob == HEAD, git diff HEAD empty)</summary> All through `scripts/ablation-replace.mjs` (anchor x1 → x0, replacement x0 → x1); 1 and 2 on commit `2d69cfb4`, 3 on `2eb91a69`, 4 on `149e988b`. - **Ablation 1, the copy widened** (`pack-library.mjs` `DIRS` gains `docs/qa/platform-checklist`): the inner script rebuilt the package (`pack-library` ran, `dist/library/docs/qa/platform-checklist/` then held `areas` and `coverage.json`) and ran `packed-tarball.test.ts` → `1 failed | 7 passed`, the failure being "carries no other JSON: a card from anywhere but docs/qa/scenarios is refused". Restored, and the package rebuilt afterwards (`dist/library/docs/qa/platform-checklist/` holds `areas` only). - **Ablation 2, the checkout test loosened** (`library.ts`: the real-path comparison of `packages/qa/scenarios` with the package replaced by an always-true `String("ABL2").length > 0`): `library.test.ts` → `1 failed | 5 passed`, the failure being "never reads another pnpm workspace holding a docs/qa/scenarios as the checkout". The suite imports `src/`, so no dist leg. - **Ablation 3, the runner bundled back into the bin** (`tsup.config.ts`: `esbuildPlugins: [binLoadsBuiltCli].slice(0, 0)`): rebuilt, `dist/bin.js` then carried 19 static `@objectstack/*` imports (196 KB); `packed-tarball.test.ts` → `1 failed | 8 passed`, the failure being "keeps the runner out of the bin". Restored, rebuilt. - **Ablation 4, the workspace refusal disarmed** (`cli.ts`: its condition ANDed with an always-false `String("ABL4").length < 0`): rebuilt, `ablation-dist-preflight` found the marker in `dist/cli.js` and `dist/cli.cjs`; `packed-tarball.test.ts -t 'refuses --packages workspace'` → `1 failed`, the run reaching workspace mode and dying on `fatal: not a git repository` instead of the reason. Restored, rebuilt, preflight `--absent`: marker absent from all 81 built files, tree clean. - The API-billed refusal the smoke test pins is the runner's existing behaviour, which this diff does not change, so it is not ablated. </details> ## Acceptance notes - **The private overlay cannot reach the tarball by construction and is refused if it does.** The overlay (objectstack-ai/cloud#3603) lives outside this repository; `pack-library.mjs` copies two named directories of this repository, one level deep; and the packaging test refuses any JSON that is not byte-identical to an open card or area. It runs in Test Core on every PR and in the merge queue, so a leak is red before it can merge, rather than at publish. The triage audit's ⚠️ asked for a check in the publish job itself; this PR does not edit `release.yml` (the release pipeline is outside the claimed surface): the release publishes from a commit whose Test Core ran this test. - **The bin is a wrapper, not a second CLI.** `run` passes its arguments through; `help` names the flags and points at the README instead of copying the flag table, so #31160's `--model` flags (and any later flag) appear in `run` without touching `bin.ts`. - **Interaction with #31188 (`--packages`, merged as `86e9ef0a`).** Merged into this branch (no rebase), as the order's serial note asked. `published` (its default) works from npm unchanged; `workspace` is refused there (What changes). Its exporter rule (refusing `workspace` records) is in `./export` and needs nothing here. Its workspace mode packs every non-private `@objectstack/*` package, so it packs this one too (76). - `dist/bin.js` has no top-level effect (it only exports `runCli`); `bin/objectstack-scenarios.js` is the published entry. `dist/cli.js` keeps `cli.ts`'s own self-run guard, which compares `argv[1]` with its module URL, so it never fires under the bin. - `dist/bin.cjs` and `dist/cli.cjs` are emitted because the package's build emits both formats for every entry; nothing reads them. - The `framework` record field reads `unknown` for a run from npm; the packages a run used are stated by #31188's `packages` field. - Out of scope, untouched: customer-facing acceptance cards, an `os` subcommand (ruling D on #30291 stands), any release action. ## Verification All on head `149e988b` (after merging `origin/main` `065bba72`, which carries #31188 as `86e9ef0a`), worktree `objectstack-issue-31172`; `git merge-tree` against `origin/main` `733efbd4` is clean. - Full build of `./packages/**` and `./examples/**` under the verify lock (`turbo run build --concurrency=2`): `Tasks: 82 successful, 82 total`, `VERDICT command-exit 0`. - Package suite, after `pnpm --filter '@objectstack/scenarios...' build`: `pnpm --filter @objectstack/scenarios typecheck && pnpm --filter @objectstack/scenarios test` → `Test Files 27 passed (27)`, `Tests 527 passed (527)`, `VERDICT command-exit 0` (`tsc --noEmit`, whose `include` covers `test/**`). - Lint scoped to the package: `node --stack-size=4000 node_modules/eslint/bin/eslint.js packages/qa/scenarios --no-inline-config` → exit 0. - Derived gates: `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` → 77 commands, all 77 exit 0, including `check-changeset-no-major`, `check:changeset-fixed`, `check:published-files`, `check:manifest-repository-directory`, `check:turbo-task-graph`, `check:llms-txt` and `check:dual-build-cjs-loads` (after the full build). `--ran`: `77 derived famil(ies) accounted for — 77 run, 0 NOT-MEASURED`. Run beyond the derivation, exit 0 each: `check:published-readme-exports`, `check:published-readme-links`, `check:package-naming`, `check-published-list-mirrors.mjs`. - `Check Changeset` locally, with this body as the event (`GITHUB_EVENT_PATH`): exit 1 on `minor`, exit 0 on `patch` ("a non-protocol change at `patch`"). - Earlier rounds: on `2d69cfb4`, `check:llms-txt` red on the count (fixed in `c8b7eaf2`); on `9742af04`, CI Test Core timed out three spawn cases (fixed in `2eb91a69`, see Tests). - Dogfood battery: n/a (no `!` in the title, no `(narrowing)` arm). - Live model run: none (paused by the maintainer); the bin is exercised from the packed tarball up to the driver's login check. <details><summary>The 77 derived gates, as run</summary> ```text node scripts/check-adr-0087-registration.mjs --base origin/main :: exit 0 node scripts/check-adr-0087-registration.mjs --self-test :: exit 0 node scripts/check-changeset-no-major.mjs --base origin/main :: exit 0 node scripts/check-closing-keyword-parity.mjs :: exit 0 node scripts/check-closing-keyword-parity.mjs --self-test :: exit 0 node scripts/check-comment-mask-adoption.mjs :: exit 0 node scripts/check-comment-mask-adoption.mjs --self-test :: exit 0 node scripts/check-comment-mask-corpus.mjs :: exit 0 node scripts/check-console-intercept-disarm.mjs :: exit 0 node scripts/check-console-intercept-disarm.mjs --self-test :: exit 0 node scripts/check-dev-prereqs.mjs --self-test :: exit 0 node scripts/check-dts-emitted.mjs --self-test :: exit 0 node scripts/check-empty-changeset.mjs --base origin/main :: exit 0 node scripts/check-empty-changeset.mjs --self-test :: exit 0 node scripts/check-issue-citations.mjs :: exit 0 node scripts/check-osv-exemptions.mjs :: exit 0 node scripts/check-osv-exemptions.mjs --self-test :: exit 0 node scripts/check-plugin-teardown-shape.mjs :: exit 0 node scripts/check-plugin-teardown-shape.mjs --self-test :: exit 0 node scripts/ci/assert-turbo-remote-cache.mjs --self-test :: exit 0 node scripts/ci/plan-test-core-shards.mjs --self-test :: exit 0 node scripts/ci/prune-turbo-cache.mjs --self-test :: exit 0 node scripts/ci/snapshot-next.mjs --self-test :: exit 0 node scripts/docs-audit/check-affected-docs.mjs :: exit 0 node scripts/docs-audit/check-drift-comment.mjs :: exit 0 node scripts/osv-base-relative.mjs --self-test :: exit 0 node scripts/pm/release-rehearsal-clone.mjs --self-test :: exit 0 node scripts/release-pending-publish.mjs --self-test :: exit 0 pnpm --filter @objectstack/spec run check:duration-unit-keys :: exit 0 pnpm --filter @objectstack/spec run check:empty-state :: exit 0 pnpm --filter @objectstack/spec run check:liveness :: exit 0 pnpm --filter @objectstack/spec run check:llms-txt :: exit 0 pnpm --filter @objectstack/spec run check:strictness-ledger :: exit 0 pnpm --filter @objectstack/spec run check:variant-docs :: exit 0 pnpm check:changeset-fixed :: exit 0 pnpm check:changeset-gate-self-tests :: exit 0 pnpm check:ci-filter-parity :: exit 0 pnpm check:closed-ceremony-text :: exit 0 pnpm check:cross-package-test-inputs :: exit 0 pnpm check:doc-authoring :: exit 0 pnpm check:driver-memory-census :: exit 0 pnpm check:dts-closure :: exit 0 pnpm check:dual-build-cjs-loads :: exit 0 pnpm check:engine-double-contract :: exit 0 pnpm check:gitlink-declared :: exit 0 pnpm check:issue-citations :: exit 0 pnpm check:keyed-text-bounds :: exit 0 pnpm check:lean-entry-closure :: exit 0 pnpm check:logger-receiver-detach :: exit 0 pnpm check:manifest-repository-directory :: exit 0 pnpm check:merge-driver :: exit 0 pnpm check:nul-bytes :: exit 0 pnpm check:objectql-double-limit :: exit 0 pnpm check:org-identifier :: exit 0 pnpm check:override-consistency :: exit 0 pnpm check:page-declaration-shape :: exit 0 pnpm check:platform-object-tenancy-census :: exit 0 pnpm check:pnpm-flag-spellings :: exit 0 pnpm check:published-files :: exit 0 pnpm check:query-options-erasure :: exit 0 pnpm check:refd-timer-probe :: exit 0 pnpm check:registry-log-declared :: exit 0 pnpm check:rest-log-spy-declared :: exit 0 pnpm check:slot-lookup :: exit 0 pnpm check:sourcemap-no-sources-content :: exit 0 pnpm check:system-context-census :: exit 0 pnpm check:test-source-alias :: exit 0 pnpm check:tier-file-adoption :: exit 0 pnpm check:turbo-task-graph :: exit 0 pnpm check:type-check-coverage :: exit 0 pnpm check:type-check-debt :: exit 0 pnpm check:undeclared-dep-imports :: exit 0 pnpm check:vendor-export-contract-resolve :: exit 0 pnpm check:vouch-for-reference-callers :: exit 0 pnpm check:watch-hint-literal :: exit 0 pnpm check:where-matcher :: exit 0 pnpm check:workspace-manifest-cycles :: exit 0 ``` </details> ## Human Minimum - [x] 1. Schemas start as Zod; `packages/spec` holds no runtime logic. (Only `llms.txt` prose in spec.) - [x] 2. Every name I added uses the casing for its kind. - [x] 3. No formatter ran; `pnpm lint` passes (scoped to the package; the repo-wide run is CI's). - [x] 4. One issue, one branch off an up-to-date `main`, one pull request. - [x] 5. A change that publishes carries a changeset, and both declaration lines are filled in here and in the changeset. - [x] 6. `packages/spec` touched: its generated artifacts are regenerated and committed. (`llms.txt` is hand-kept; nothing generated reads it.) - [x] 7. No hand edits to `content/docs/references/`, `packages/*/CHANGELOG.md` or `content/docs/releases/`. - [x] 8. Title, body and comments are in English (the 维护者速读 block is the lane's Chinese summary). - [x] The *Before Submitting* list in CONTRIBUTING.md is done. --- _Generated by [Claude Code](https://claude.ai/code/session_01FEkQ5nGqwL3Ag8M8TdoRiC)_ <!-- signed: domain:devx#3 · os-project-manager · session_01FEkQ5njkh3xbcDN6EtSfw7 --> --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 0d0c377 commit fba4c00

14 files changed

Lines changed: 619 additions & 25 deletions

File tree

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
---
2+
'@objectstack/scenarios': patch
3+
---
4+
5+
feat(scenarios): `@objectstack/scenarios` is published, with an `objectstack-scenarios` command line: `npx @objectstack/scenarios run` runs the open scenario cards with no checkout of the platform
6+
7+
Clause-②: yes
8+
Metadata-protocol: none
9+
10+
The package was private. It is now released with the `@objectstack/*` packages, at their version, and it carries:
11+
12+
- **A `bin`, `objectstack-scenarios`.** `objectstack-scenarios run [flags]` is the scenario runner (`run-scenarios` in this repository) with the same flags, defaults and refusals: it drives your own local Claude Code login and refuses to start when `ANTHROPIC_API_KEY` or `ANTHROPIC_AUTH_TOKEN` is set or the CLI is logged in with an API key. `help` and `version` are the other two commands. The bin runs the built JavaScript in `dist/`.
13+
- **The open card library as package data** (`dist/library/`): the cards of `docs/qa/scenarios/` and the capability areas they are checked against, copied at build time. `run` reads them unless `--cards DIR` names another directory, which is loaded as the overlay. Run records go to `./scenario-runs` unless `--out` says otherwise.
14+
- **The library entries it already had**: the card schema and loader (`@objectstack/scenarios`), the scorer (`./score`, `./score/full`), the runner (`./runner`) and the gallery exporter (`./export`).
15+
16+
It is not an `objectstack` (`os`) subcommand, and nothing in `@objectstack/cli` changes.

‎.changeset/config.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,7 @@
8383
"create-objectstack",
8484
"@objectstack/connector-openapi",
8585
"@objectstack/verify",
86+
"@objectstack/scenarios",
8687
"@object-ui/auth",
8788
"@object-ui/cli",
8889
"@object-ui/collaboration",

‎packages/qa/scenarios/README.md‎

Lines changed: 32 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
# @objectstack/scenarios
22

3-
Private, unpublished. The schema (zod) and loader for the open scenario library under
3+
Published, with a command line: `npx @objectstack/scenarios run` (below, "Running from npm").
4+
The schema (zod) and loader for the open scenario library under
45
[`docs/qa/scenarios/`](../../../docs/qa/scenarios/README.md) — that README is the format's
56
reference.
67

@@ -27,10 +28,11 @@ The private overlay's owner migrates its cards to each new card shape (today: `o
2728
- `src/score/` — the deterministic scorer, entries `@objectstack/scenarios/score` and
2829
`@objectstack/scenarios/score/full` (below).
2930

30-
Not published. `pnpm --filter @objectstack/scenarios build` (tsup) writes `dist/` as ESM, CJS and
31-
`.d.ts`, and every `exports` subpath points there, so a consumer that links the framework tree
32-
and bundles its own entry can load the scorer under plain Node. The suites and `run-scenarios`
33-
read `src/` directly. Nothing here calls a model or the network.
31+
`pnpm --filter @objectstack/scenarios build` (tsup) writes `dist/` as ESM, CJS and `.d.ts`, and
32+
every `exports` subpath points there, so plain Node loads the package with no TypeScript resolver.
33+
The build then copies the open library into `dist/library/` (`scripts/pack-library.mjs`), which
34+
the published command line runs. The suites and `run-scenarios` read `src/` directly. The schema,
35+
loader and scorer call no model and no network; the runner drives a local Claude Code session.
3436

3537
## Scoring a run against a card — `@objectstack/scenarios/score`
3638

@@ -147,6 +149,31 @@ pnpm --filter @objectstack/scenarios run-scenarios --packages workspace # o
147149
Per card and mode the runner writes `docs/qa/platform-checklist/runs/<date>-scenario-<card>.json`
148150
(git-ignored) and prints the scorecard.
149151

152+
### Running from npm
153+
154+
No checkout needed: the package's `objectstack-scenarios` command is the same runner, with the same flags.
155+
156+
```bash
157+
npx @objectstack/scenarios run --only service_desk --modes scratch # one card, scratch mode
158+
npx @objectstack/scenarios run # every card, both modes
159+
npx @objectstack/scenarios run --cards ./my-cards # a directory of your own, loaded as the overlay
160+
npx @objectstack/scenarios help # the commands; `version` prints the package version
161+
```
162+
163+
- **Prerequisites**: Node 22 or later, and Claude Code installed and logged in with your own subscription
164+
(`claude auth login`). The runner refuses to start when `ANTHROPIC_API_KEY` or `ANTHROPIC_AUTH_TOKEN` is set, or the
165+
CLI is logged in with an API key. Each card scaffolds a project with `create-objectstack` (`--create-version` picks
166+
its version).
167+
- **The cards**: the package carries the open library of this repository (`docs/qa/scenarios/*.json` and the
168+
capability areas, copied into `dist/library/` at build time) and nothing else. `--cards DIR` runs another directory
169+
as the overlay, where a `generic` card is refused.
170+
- **The records**: one `scenario-runs/<date>-scenario-<card>.json` per card, under the directory you run it from,
171+
unless `--out DIR` says otherwise; the scorecard is printed as in a checkout. `--work DIR` keeps the scaffolded
172+
projects (default: a temp directory).
173+
- **The packages**: always the published ones (`--packages published`, the default). `--packages workspace` builds
174+
and packs a checkout's packages, so the published CLI refuses it; run it from a checkout.
175+
- It is not an `objectstack` (`os`) command: scenario scoring stays out of the customer CLI.
176+
150177
| Mode | Steps |
151178
|:---|:---|
152179
| `scratch` | `create-objectstack` → the agent builds from the card prompt and the runner's data contract (`prompts.ts`) → `objectstack validate` / `build` / `verify` (proofs chosen from the tags) → score → `objectstack dev --fresh` → seed, one user + permission-set grant + API key per persona → every question as every persona it names and every operation as its persona, over the app's MCP → the iterate change → rebuild, re-score, re-run |
Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
#!/usr/bin/env node
2+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
3+
4+
// `bin.objectstack-scenarios` in package.json: npm packs this file because it
5+
// is the `bin` target. It runs the BUILT CLI (`dist/bin.js`, from
6+
// `src/bin.ts`), never a `.ts` source; a refusal or failure prints its message
7+
// and exits 1, as `run-scenarios` does.
8+
import { runCli } from '../dist/bin.js';
9+
10+
runCli(process.argv.slice(2)).then(
11+
(code) => process.exit(code),
12+
(e) => {
13+
console.error(e instanceof Error ? e.message : e);
14+
process.exit(1);
15+
},
16+
);

‎packages/qa/scenarios/package.json‎

Lines changed: 17 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,14 @@
11
{
22
"name": "@objectstack/scenarios",
33
"version": "0.1.0",
4-
"private": true,
54
"license": "Apache-2.0",
6-
"description": "Scenario-card schema and loader for the open scenario library under docs/qa/scenarios/. A card states an authoring prompt, the structural expectation a generated app must meet, seed rows, data questions whose truth the loader COMPUTES from a declarative spec over those rows, personas, write operations with their guardrail checks, and one iteration request \u2014 plus the deterministic scorer that judges a built artifact, an observed write or an agent's answer against a loaded card (`@objectstack/scenarios/score`). Not published (a minimal tsup build to `dist/` so a linked consumer's bundled entry can load it under plain Node; `./score` is the narrow scorer, `./score/full` adds the capability-tag checks), no model calls: a devx instrument, not a protocol (packages/spec carries no obligation for this format).",
5+
"description": "Run the open ObjectStack scenario cards end to end from npm: `npx @objectstack/scenarios run` builds an app from each card's prompt with your local Claude Code login, scores it and writes one run record per card. Also the scenario-card schema and loader (computed truths over seed rows) and the deterministic scorer (`@objectstack/scenarios/score`, `./score/full`). Ships the open card library (docs/qa/scenarios) as package data, never a private overlay; refuses an API-billed login. A devx instrument, not a protocol: packages/spec carries no obligation for this format.",
76
"type": "module",
87
"main": "dist/index.js",
98
"types": "dist/index.d.ts",
9+
"bin": {
10+
"objectstack-scenarios": "./bin/objectstack-scenarios.js"
11+
},
1012
"exports": {
1113
".": {
1214
"import": {
@@ -60,7 +62,7 @@
6062
}
6163
},
6264
"scripts": {
63-
"build": "tsup && node ../../../scripts/check-dts-emitted.mjs",
65+
"build": "tsup && node scripts/pack-library.mjs && node ../../../scripts/check-dts-emitted.mjs",
6466
"test": "vitest run",
6567
"typecheck": "tsc --noEmit",
6668
"run-scenarios": "tsx src/runner/cli.ts",
@@ -93,5 +95,16 @@
9395
"directory": "packages/qa/scenarios"
9496
},
9597
"homepage": "https://objectstack.ai/docs",
96-
"bugs": "https://github.com/objectstack-ai/objectstack/issues"
98+
"bugs": "https://github.com/objectstack-ai/objectstack/issues",
99+
"publishConfig": {
100+
"access": "public"
101+
},
102+
"files": [
103+
"dist",
104+
"README.md",
105+
"CHANGELOG.md"
106+
],
107+
"engines": {
108+
"node": ">=22.0.0"
109+
}
97110
}
Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
#!/usr/bin/env node
2+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
3+
4+
// The package build's last step: copy the OPEN card library into
5+
// `dist/library/`, keeping the repository's layout, so the published CLI runs
6+
// the cards with no checkout (`src/runner/library.ts` reads it back):
7+
//
8+
// docs/qa/scenarios/*.json → dist/library/docs/qa/scenarios/
9+
// docs/qa/platform-checklist/areas/*.json → dist/library/docs/qa/platform-checklist/areas/
10+
//
11+
// ⛔ Only these two directories of THIS repository, one level deep, regular
12+
// `.json` files only. The private overlay (held-out and customer-derived
13+
// cards, objectstack-ai/cloud#3603) lives outside this repository and has no
14+
// path in here; `test/build/packed-tarball.test.ts` packs the package and
15+
// fails on any card in the tarball that is not byte for byte a card of
16+
// `docs/qa/scenarios/`.
17+
//
18+
// turbo re-runs the build when either directory changes: the package's
19+
// `turbo.json` adds them to the build's inputs.
20+
21+
import { copyFileSync, lstatSync, mkdirSync, readdirSync, rmSync } from 'node:fs';
22+
import { dirname, join, resolve } from 'node:path';
23+
import { fileURLToPath } from 'node:url';
24+
25+
const PKG = resolve(dirname(fileURLToPath(import.meta.url)), '..');
26+
const REPO = resolve(PKG, '../../..');
27+
const OUT = join(PKG, 'dist/library');
28+
const DIRS = ['docs/qa/scenarios', 'docs/qa/platform-checklist/areas'];
29+
30+
rmSync(OUT, { recursive: true, force: true });
31+
const counts = [];
32+
for (const dir of DIRS) {
33+
const from = join(REPO, dir);
34+
const to = join(OUT, dir);
35+
mkdirSync(to, { recursive: true });
36+
let copied = 0;
37+
for (const name of readdirSync(from).sort()) {
38+
if (!name.endsWith('.json') || !lstatSync(join(from, name)).isFile()) continue;
39+
copyFileSync(join(from, name), join(to, name));
40+
copied++;
41+
}
42+
if (copied === 0) {
43+
console.error(`pack-library: no .json file under ${dir}: the published CLI would have nothing to run`);
44+
process.exit(1);
45+
}
46+
counts.push(`${copied} from ${dir}`);
47+
}
48+
console.log(`pack-library: ${counts.join(', ')} → ${OUT}`);

‎packages/qa/scenarios/src/bin.ts‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
3+
/**
4+
* `objectstack-scenarios` — the published command line (`npx @objectstack/scenarios run …`).
5+
*
6+
* A thin dispatcher over the runner's own CLI (`runner/cli.ts`): `run` hands
7+
* every argument after it to the same `main` that
8+
* `pnpm --filter @objectstack/scenarios run-scenarios` runs, so the flags, the
9+
* defaults and the refusals are that CLI's (no API-billed login, the published
10+
* packages by default). Not an `os` subcommand, and it carries no scoring of
11+
* a customer's own app: the cards it runs are the open library the package
12+
* ships (`runner/library.ts`), or a `--cards` directory loaded as the overlay.
13+
*
14+
* `bin/objectstack-scenarios.js` imports the BUILT form of this file
15+
* (`dist/bin.js`), so npm runs JavaScript, never a `.ts` source.
16+
*
17+
* The runner is imported only for `run`, and the build keeps it out of this
18+
* file (`tsup.config.ts` turns the import into one of the built `dist/cli.js`):
19+
* its graph (the scorer's objectql and plugin-sharing among it) takes about a
20+
* second warm and several cold, which `help` and `version` must not pay.
21+
*/
22+
23+
import { readFileSync } from 'node:fs';
24+
import { dirname, join } from 'node:path';
25+
import { fileURLToPath } from 'node:url';
26+
27+
export const USAGE = `Usage: objectstack-scenarios <command> [flags]
28+
29+
Commands:
30+
run [flags] run the scenario cards end to end and write one run record per card
31+
(the flags of run-scenarios: --only, --modes, --driver, --cards, --out, …;
32+
see the package README, "Running from npm")
33+
help print this text
34+
version print the package version
35+
36+
Runs on your own Claude Code login: it refuses to start when an API key is configured.`;
37+
38+
export interface CliIo {
39+
out: (line: string) => void;
40+
err: (line: string) => void;
41+
}
42+
43+
const consoleIo: CliIo = { out: (l) => console.log(l), err: (l) => console.error(l) };
44+
45+
function packageVersion(): string {
46+
// `src/bin.ts` and `dist/bin.js` both sit one directory below the package root.
47+
const manifest = join(dirname(fileURLToPath(import.meta.url)), '..', 'package.json');
48+
return (JSON.parse(readFileSync(manifest, 'utf8')) as { version: string }).version;
49+
}
50+
51+
/** Run one command line; answers the process exit code. */
52+
export async function runCli(argv: readonly string[], io: CliIo = consoleIo): Promise<number> {
53+
const [command, ...rest] = argv;
54+
if (command === 'run') return (await import('./runner/cli.js')).main(rest);
55+
if (command === 'help' || command === '--help' || command === '-h') {
56+
io.out(USAGE);
57+
return 0;
58+
}
59+
if (command === 'version' || command === '--version' || command === '-v') {
60+
io.out(packageVersion());
61+
return 0;
62+
}
63+
io.err(command === undefined ? 'objectstack-scenarios: name a command' : `objectstack-scenarios: unknown command \`${command}\``);
64+
io.err(USAGE);
65+
return 2;
66+
}

‎packages/qa/scenarios/src/runner/cli.ts‎

Lines changed: 17 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,8 @@
44
* `pnpm --filter @objectstack/scenarios run-scenarios [flags]` — run every card
55
* under `docs/qa/scenarios/` end to end and write one run record per card.
66
*
7-
* --cards DIR the card directory (default: the repo's docs/qa/scenarios);
7+
* --cards DIR the card directory (default: the repo's docs/qa/scenarios,
8+
* or the copy the published package carries, `library.ts`);
89
* any other directory is loaded as the private overlay,
910
* where a `generic` card is refused (`libraryOf`)
1011
* --only ID[,ID] run only these cards
@@ -32,27 +33,30 @@
3233
* model each turn was served, and mark a mismatch
3334
* --template F a verified build (dist/objectstack.json) for template mode
3435
* --no-iterate-run skip the Run half after the iterate (scratch mode)
35-
* --out DIR where records go (default docs/qa/platform-checklist/runs, git-ignored)
36+
* --out DIR where records go (default docs/qa/platform-checklist/runs, git-ignored;
37+
* ./scenario-runs when run from the published package)
3638
* --work DIR where projects are scaffolded (default: a temp dir)
3739
* --create-version V the create-objectstack version (default latest)
3840
* --packages P where the @objectstack/* packages come from (`packages.ts`):
3941
* `published` (default) — npm, what a customer gets;
4042
* `workspace` — this checkout, built and packed as the
41-
* release would publish it (unreleased; never exported)
43+
* release would publish it (unreleased; never exported;
44+
* refused when run from the published package, which has no checkout)
4245
*
4346
* ⛔ No API-billed model: the live driver refuses to start when an API key is
4447
* configured (the cost ruling on cloud#3600).
4548
*/
4649

4750
import { execSync } from 'node:child_process';
48-
import { existsSync, mkdirSync, mkdtempSync, readFileSync, writeFileSync } from 'node:fs';
51+
import { mkdirSync, mkdtempSync, readFileSync, writeFileSync } from 'node:fs';
4952
import { tmpdir } from 'node:os';
5053
import { dirname, join, resolve } from 'node:path';
5154
import { fileURLToPath } from 'node:url';
5255

5356
import { formatRefusal, libraryOf, loadScenarios, readCapabilityAreas } from '../loader.js';
5457
import { HttpApp } from './app.js';
5558
import { ClaudeCodeDriver, RecordedDriver, RecordingDriver, type AgentDriver } from './driver.js';
59+
import { CAPABILITY_AREAS_DIR, locateLibraryRoot } from './library.js';
5660
import { minutesText, runCard, type ProjectOps } from './orchestrate.js';
5761
import { installedSpecVersion, PACKAGES_MODES, prepareWorkspacePackages, settleWorkspaceProject, workspaceScaffolder, type PackageSource, type PackagesMode } from './packages.js';
5862
import { bootDev, os, readArtifact, readNamespace, scaffold } from './project.js';
@@ -191,15 +195,19 @@ export function makeDriver(spec: string, opts: DriverLimits & DriverModels = {})
191195
export async function main(argv: readonly string[]): Promise<number> {
192196
const args = parseArgs(argv);
193197
const here = dirname(fileURLToPath(import.meta.url));
194-
const repo = findUp(here, (d) => existsSync(join(d, 'pnpm-workspace.yaml')));
198+
// The checkout, or the library root the published package carries (`library.ts`).
199+
const { root: repo, checkout } = locateLibraryRoot(here);
200+
if (args.packages === 'workspace' && !checkout) {
201+
throw new Error('--packages workspace builds and packs the packages of an objectstack checkout; the published CLI has none. Run it from a checkout (`pnpm --filter @objectstack/scenarios run-scenarios --packages workspace`), or use --packages published');
202+
}
195203
const cardsDir = resolve(args.cards ?? join(repo, 'docs/qa/scenarios'));
196-
const outDir = resolve(args.out ?? join(repo, 'docs/qa/platform-checklist/runs'));
204+
const outDir = resolve(args.out ?? (checkout ? join(repo, 'docs/qa/platform-checklist/runs') : 'scenario-runs'));
197205
const work = resolve(args.work ?? mkdtempSync(join(tmpdir(), 'scenario-run-')));
198206
mkdirSync(work, { recursive: true });
199207
mkdirSync(outDir, { recursive: true });
200208

201209
const { cards, refusals } = loadScenarios(cardsDir, {
202-
areas: readCapabilityAreas(join(repo, 'docs/qa/platform-checklist/areas')),
210+
areas: readCapabilityAreas(join(repo, CAPABILITY_AREAS_DIR)),
203211
library: libraryOf(cardsDir, repo),
204212
});
205213
for (const r of refusals) console.error(formatRefusal(r));
@@ -250,7 +258,8 @@ export async function main(argv: readonly string[]): Promise<number> {
250258
const template = args.template ? JSON.parse(readFileSync(args.template, 'utf8')) : undefined;
251259
let framework: string | undefined;
252260
try {
253-
framework = execSync('git rev-parse --short HEAD', { cwd: repo, encoding: 'utf8' }).trim();
261+
// Only a checkout has a framework commit: an installed package may sit in someone else's repository.
262+
if (checkout) framework = execSync('git rev-parse --short HEAD', { cwd: repo, encoding: 'utf8' }).trim();
254263
} catch {
255264
/* not a checkout */
256265
}

0 commit comments

Comments
 (0)