Skip to content

Commit 86e9ef0

Browse files
feat(scenarios): run-scenarios --packages published|workspace; the gallery exporter refuses workspace records (#31188)
<!-- claude-projects-attribution: {"github_login":"os-project-manager"} --> _Requested by **Leehom** · [project thread](https://claude.ai/code/project/chan_01TenzsRopZj4jMbvjX8uz1X?thread=cmsg_01TenzsRopZj4jMbvjX8uz1X5KEQhDJwoAWVnx1AGDucTA)_ Fixes #31147 Clause-②: no Metadata-protocol: none ## 维护者速读 - 改了什么:场景跑批器 `run-scenarios` 新增 `--packages published|workspace`,默认 `published`(行为不变)。`workspace` 先按发布的方式构建、`pnpm pack` 本仓所有会发布的包,再用打包出的 `create-objectstack` 原样脚手架项目,依赖与技能都装自这些 tarball;每份运行记录和计分卡首行写明包来源(`packages: published 18.3.0` 或 `packages: workspace @ SHA (unreleased)`)。画廊导出器只导出 `published` 的运行,`workspace` 记录(以及未写明来源的记录)一律拒绝。 - 为什么改:维护者裁决「两种都要」:发版后跑已发布的包,发版前用工作区未发布的包先测技能改动有没有让 AI 少犯错;未发布的结果绝不能进画廊。 - 第二轮(head `38562001`):CI 的 pnpm 是 Corepack 垫片,在仓外无 `packageManager` 的项目目录里会解析到 registry 最新版 pnpm 12,它不读 `npm_config_*`,于是装依赖时直奔 npm 而失败。现在 workspace 模式里脚手架和收尾安装用的每个 `pnpm` 都固定为本仓自己的版本(见下文 *pnpm is pinned to the workspace's*)。 - 风险与回滚:只动私有包 `@objectstack/scenarios`,不发布任何东西,`create-objectstack` 未改;`published` 路径的脚手架命令不变,只多读一次已装的 spec 版本写进记录。回滚即还原本 PR。 - 先看:`packages/qa/scenarios/src/runner/packages.ts`(打包、pnpmfile、overrides、安装后核验)、`src/export/record.ts` 的 `unpublishedRunRefusal` 与 `exporter.ts` 的调用点、`src/runner/cli.ts` 的 `--packages` 接线。 - 两分钟验证:`pnpm --filter @objectstack/scenarios exec vitest run --maxWorkers=2 test/runner/packages.test.ts test/export/exporter.test.ts`,预期 49/49 通过(含工作区独有技能标记落进 `.claude/skills/`、`workspace` 记录被导出器拒绝)。 - 需要你做的:无,席位落地。 ## What changes - **`run-scenarios --packages published|workspace`** (`runner/cli.ts`), default `published`, which keeps today's scaffold command (`npx -y create-objectstack@VERSION`) byte for byte. `--packages workspace` beside `--create-version` is refused: the version names a published scaffolder. - **`workspace` mode** (`runner/packages.ts`, new), built once per run before the first card: 1. Builds what the release builds for the packages it publishes: every non-private workspace package named `@objectstack/*` or `create-objectstack` (75 today), `turbo run build build:dist` filtered to them (`build:dist` is the Console copy the release's `console:build` makes). 2. `pnpm pack`s each one: the tarball npm would serve, `files[]` only, every `workspace:` range rewritten to the version. 3. Scaffolds with the packed `create-objectstack`, untouched: `npx -y file:TARBALL NAME`, its own install, its own skills sync, its own summary. Every `pnpm` it runs, and the install in step 4, is the workspace's own pinned pnpm (`pinnedPnpm`, see *Acceptance notes*). Its `pnpm install` is pointed at the tarballs by a global pnpmfile (`npm_config_global_pnpmfile`) whose `readPackage` hook rewrites every packed name in every dependency field of every manifest. The project's `package.json` is not rewritten: it pins the version, as a published scaffold does. 4. Persists the same map as `overrides` in the project's `pnpm-workspace.yaml` and installs again, so a later `pnpm add` by the agent stays on the tarballs. 5. Verifies, and refuses the scaffold (the mode is skipped with the reason, like a failed scaffold) unless every `@objectstack/*` / `create-objectstack` entry in `node_modules/.pnpm` is a tarball (`@file+…`) and `.claude/skills/` holds the workspace build's catalog (`packages/skills/dist/skills`), skill for skill and file for file. - **The package source in the record and the scorecard** (`runner/scorecard.ts`): `RunRecordMeta.packages` is required, so every record states it as `env.packages` (`{ source: 'published', versions }` or `{ source: 'workspace', commit, uncommitted }`) and `env.packagesLine`; the scorecard table's first line is `packages: published 18.3.0` or `packages: workspace @ SHA (unreleased)` (`+ uncommitted changes` when the tree had any). The published version is the `@objectstack/spec` each scaffolded project installed. - **⛔ The gallery exporter refuses every run not on published packages** (`export/record.ts` `unpublishedRunRefusal`, checked in `exporter.ts` before the pass judgement): a `workspace` record, a record that states no package source (written before this flag), and a `published` record with no version are refused per card, with the reason; as for any refusal, an app the previous index listed is removed when its newest record is refused. The export CLI offers no flag around it. - `runCommand` / `scaffold` (`runner/project.ts`) take an environment addition (the pnpmfile variable, and the `PATH` that leads with the pinned pnpm), merged over the existing environment. README: the `--packages` command line and a **Package source** paragraph. ## Why The maintainer's ruling on the card, verbatim: 「两种都要」 — run on the published packages after each release, and a runner option to run on the workspace's unreleased packages beforehand, so a skill change can be measured before it ships. A workspace run's `@objectstack/spec` pin still reads `18.3.0` (the workspace has not been versioned), which is why the exporter cannot trust the version and refuses on the stated source instead. ## Tests - **Workspace mode against a workspace of its own** (`test/runner/packages.test.ts`, new, 9 cases, ~3 s of tests): a temp pnpm workspace holding a skills catalog with a marker skill and a stand-in `create-objectstack` that does what the real one does in its order (write the manifest and `pnpm-workspace.yaml`, `pnpm install`, copy each skill). The pack, the `npx -y file:` scaffold, the pnpmfile and both installs are the real tools, at a version npm never published, so an install that reached for the registry fails instead of quietly finding something. Pins: only the released packages are built and packed (a `private` one is not) and the commit is stated; **the marker present only in the workspace build is in `.claude/skills/`** and the verification is empty; the installed package is the tarball (`@file+`), not a link, and a file outside `files` is absent; `package.json` keeps the version pin, `pnpm-workspace.yaml` gains the overrides, the lockfile carries no pnpmfile checksum; the hook rewrites only packed names, in every field; **every install ran the workspace's pinned pnpm** (the project's `node_modules/.modules.yaml` names the checkout's `pnpm --version`; the temp workspace declares that pin as the real root does). A failed scaffold reports its whole output. Negative pins: a published catalog in `.claude/skills/` (no marker, an extra skill) is refused file by file; a registry store entry is refused; a project with no pnpm store is refused. - **Exporter** (`test/export/exporter.test.ts`, new top-level describe, 4 cases): a `workspace` record that passed every half (`judgeRecord` passes it) is refused and nothing of it is written; a newer `workspace` record removes the published export it follows; a record with no package source, and a published one with no version, are refused; a published run exports and its `scorecard.md` states `packages: published 18.3.0`. - **Flag and line** (`test/runner/units.test.ts`, new describe, 3 cases): default `published`, `workspace` taken, an unknown value and `workspace` + `--create-version` refused; both line forms; the published version read from the installed spec. Existing record fixtures (`orchestrate.test.ts`, `export/fixture.ts`) now state `published 18.3.0`. <details><summary>Ablations (each restored: blob == HEAD, git diff HEAD empty)</summary> Ablations 1–3 run through `scripts/ablation-replace.mjs` (anchor x1 → x0, replacement x0 → x1); the suites import `src/` directly, so no dist leg. - Ablation 1, the exporter refusal disarmed (its `if (unpublished)` guard in `exporter.ts` ANDed with an always-false `String("ABL1").length` comparison): `exporter.test.ts -t 'only a run on published packages'` → `3 failed | 1 passed` (the workspace, newer-workspace and unstated/versionless cases red; the published case green). - Ablation 2, the skills file comparison disarmed (its `if (differ.length)` guard in `packages.ts` ANDed with an always-false `String("ABL2").length` comparison): `packages.test.ts` → `1 failed | 7 passed` (the published-catalog refusal). - Ablation 3, the pnpmfile variable renamed (`ABL3_npm_config_global_pnpmfile`): `packages.test.ts` → the `beforeAll` scaffold answers exit 1 (the install reaches for a version npm never published), the suite reds, 5 cases do not run. - Ablation 4 (round 2, on the `packages.ts` that `38562001` carries, by a saved copy of `packages.ts` restored with `cmp` clean), the shim dropped from the scaffold's `PATH` (`env.PATH` set to the inherited `PATH`), run under a CI-shaped Corepack (`COREPACK_HOME` set, `corepack enable` into a temp bin first on `PATH`, `CI=true`): `packages.test.ts` → the `beforeAll` scaffold answers exit 1 with CI's exact error, `No matching version found for @objectstack/skills@0.0.0-scenario-workspace-test while fetching it from https://registry.npmjs.org/`; restored, `9 passed`. </details> <details><summary>Live workspace scaffold, no model (the order's no-live-run rule held)</summary> Measured in this container on the branch after a release-shape build of the scaffold closure plus the Console (`turbo run build build:dist`, 98 tasks, 15 min 14 s, exit 0): - Packed 75 tarballs in 20 s; scaffold with the packed `create-objectstack` exit 0 in 14 s; settle (overrides + install, `Already up to date`) exit 0 in 7 s. - `node_modules/.pnpm`: 61 `@objectstack/*` entries, 0 not from a tarball; `@objectstack/spec` resolves to `.pnpm/@objectstack+spec@file+..+..+workspace-packages+objectstack-spec-18.3.0.tgz`. - `.claude/skills/`: 11 skills, byte-identical to `packages/skills/dist/skills` (the catalog's top-level `README.md` is not a skill and is not synced, measured with `diff -rq`); `skills-lock.json` entries `source: @objectstack/skills`, `sourceType: node_modules`. - `objectstack validate` in the project: exit 0. `objectstack dev --fresh` (`bootDev`): `/api/v1/health` 200 after 13 s, stopped. - Control: a published `create-objectstack@18.3.0` scaffold of the same name, run through the same verification, is refused on every `@objectstack/*` store entry (registry) and on 47 skill files that differ from the workspace catalog — the check tells the two apart on today's `main`. - A first measurement used `npx -y --package=TARBALL create-objectstack`: the scaffold exited 0 with no skills, because npx hands `--package` to its child as `npm_config_package` and the scaffolder's own `npx -y skills@1.7.2 …` then ran `skills@1.7.2` as a bin of the scaffolder's tarball (`sh: 1: skills@1.7.2: not found`). The verification refused it; the scaffold now names the tarball positionally (`npx -y file:TARBALL`), as the published one is named. - A recorded-driver end-to-end run (`--driver recorded:F`) is NOT MEASURED (listed below). </details> ## NOT MEASURED - A recorded-driver end-to-end run (`--driver recorded:F`). No recording of a card exists in this container, and making one needs a live model run, which is paused. - The agent's own later `pnpm add` in a workspace-mode project. It runs whatever pnpm that project directory resolves, because the pin covers only the runner's scaffold and settle steps. It stays on the tarballs through the `overrides` persisted in `pnpm-workspace.yaml`, but that mechanism is measured on pnpm 10.31 only. Under a Corepack whose unpinned default is a later major, it is not measured. ## Acceptance notes - **pnpm is pinned to the workspace's (round 2).** Round 1 went red on Test Core (5/8) for `packages.test.ts`, while it was green locally. CI's `pnpm` is a Corepack shim, and the scaffolded project lives in a temp directory outside the checkout with no `packageManager` field, so Corepack resolved the registry's `latest` there: pnpm 12.18.0 on 2026-11-07. pnpm 12 reads none of `npm_config_*`, so the global pnpmfile went unread and the install asked npm for a version it never published. The container measured green only because its `pnpm` is a standalone 10.31.0. A real run on a machine whose pnpm is Corepack's would have failed the same way, so the fix is in the runner and not only in the test. `prepareWorkspacePackages` now writes a `pnpm` shim (`pinnedPnpm`, `runner/packages.ts`). It runs the `pnpm` on `PATH` from inside the checkout, where Corepack (or pnpm's own version management) resolves the root's `packageManager` pin, and points it at the caller's directory with `--dir`. The shim is refused unless it answers the same `--version` the checkout does. The scaffold and the settle install both run under a `PATH` led by it. The settle install still runs without the pnpmfile, so the lockfile records no pnpmfile checksum. Measured under a CI-shaped Corepack: the Corepack store afterwards holds only the pinned 10.31.0 and no `lastKnownGood.json`, so the `setup-pnpm` store guard's incident shape is not armed. The shim is a POSIX `sh` script, so workspace mode runs on Linux and macOS, not on Windows. - **Why a pnpmfile and then overrides, not one of them.** The scaffolder writes `pnpm-workspace.yaml` and installs in the same process, so overrides cannot be in place before its install; a global pnpmfile is the one pnpm 10.31 reads from the environment (`npm_config_global_pnpmfile`; `pnpm_config_global_pnpmfile` is not read, measured: the install reached for the registry). A lockfile written under a pnpmfile records its checksum, and a later install without it would re-resolve from npm; the persisted overrides plus the second install remove that path. Running the scaffolder with `--skip-install` and installing in the runner was rejected: it would copy the scaffolder's skills command into the runner, where a later bump of its pinned `skills` CLI would not follow. - **Pack, not link**, as the card expected: a tarball holds `files[]` only (pinned: a file outside `files` is absent from the installed package). - **Scope of the overrides**: every package the release publishes, not only the scaffold's closure, so a package the agent adds later is the workspace's too. - **Workspace mode needs pnpm** for the scaffolder's install; with npm the verification finds no `.pnpm` store and refuses the scaffold, never a silent npm install. The runner already runs inside this pnpm workspace. - **A record written before this PR states no package source and is refused by the exporter**; run the card again. Records are local, git-ignored output, and the gallery is not live yet. - The build step is the release's own and runs through turbo; a second run of an unchanged tree is a cache hit. A cold build of the scaffold closure plus the Console took 15 min here. - No change to `create-objectstack`: the scaffolder needed no hook, so nothing published moves and no changeset is owed (`skip-changeset`). ## Verification **Round 2, head `38562001`** (one commit on `4551309a`, touching `runner/packages.ts` and `test/runner/packages.test.ts` only): - Reproduced CI's red under a CI-shaped Corepack (above). With the fix, `packages.test.ts` reports `9 passed` both under that Corepack and under the standalone pnpm. The whole package under Corepack reports `Test Files 24 passed (24)`, `Tests 471 passed (471)`. `tsc --noEmit` exits 0. - Derived gates: 51 commands, all `exit 0`; `--ran` reconciliation: `51 run, 0 NOT-MEASURED`. A first pass on the round-2 commit had `check:cross-package-test-inputs` exit 1, because the test read the repo-root `package.json` for the pin. The test now asks `pnpm --version` in its own directory instead, and the gate exits 0. **Round 1**: all on head `4551309a`, in worktree `objectstack-issue-31147`, base `8ef4d309`; `origin/main` has moved 8 commits since, none under `packages/qa/scenarios`, `packages/create-objectstack` or `packages/skills`, and `git merge-tree` is clean. - Package suite, under the verify lock after `pnpm --filter "@objectstack/scenarios..." build`: `pnpm --filter @objectstack/scenarios test` → `Test Files 24 passed (24)`, `Tests 470 passed (470)`; `pnpm --filter @objectstack/scenarios typecheck` (`tsc --noEmit`, tests included) exit 0; `VERDICT command-exit 0`. - Lint scoped to the package: `pnpm exec eslint packages/qa/scenarios --no-inline-config` exit 0. - Derived gates: `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` → 51 commands; all 51 run, each `exit 0`; `--ran` reconciliation: `51 derived, 51 run, 0 NOT-MEASURED, 0 UNRUN`. (A first round on an earlier head read `check-plugin-teardown-shape --self-test` and `check:dual-build-cjs-loads` exit 3, prerequisite not met; after `git fetch --depth=1` of the self-test's pinned commit and the package build, both exit 0 on this head.) - Changeset: none. The diff touches only the private `@objectstack/scenarios` (`"private": true`): `skip-changeset`. - Dogfood battery: n/a (no `!` in the title, no `(narrowing)` arm). <details><summary>The 51 derived gates, as run</summary> ```text node scripts/check-closing-keyword-parity.mjs :: exit 0 node scripts/check-closing-keyword-parity.mjs --self-test :: exit 0 node scripts/check-comment-mask-adoption.mjs :: exit 0 node scripts/check-comment-mask-adoption.mjs --self-test :: exit 0 node scripts/check-comment-mask-corpus.mjs :: exit 0 node scripts/check-dts-emitted.mjs --self-test :: exit 0 node scripts/check-issue-citations.mjs :: exit 0 node scripts/check-plugin-teardown-shape.mjs :: exit 0 node scripts/check-plugin-teardown-shape.mjs --self-test :: exit 0 node scripts/ci/assert-turbo-remote-cache.mjs --self-test :: exit 0 node scripts/ci/plan-test-core-shards.mjs --self-test :: exit 0 node scripts/ci/prune-turbo-cache.mjs --self-test :: exit 0 node scripts/docs-audit/check-affected-docs.mjs :: exit 0 node scripts/docs-audit/check-drift-comment.mjs :: exit 0 pnpm --filter @objectstack/spec run check:duration-unit-keys :: exit 0 pnpm check:ci-filter-parity :: exit 0 pnpm check:closed-ceremony-text :: exit 0 pnpm check:cross-package-test-inputs :: exit 0 pnpm check:dispatcher-error-vocabulary :: exit 0 pnpm check:doc-authoring :: exit 0 pnpm check:driver-memory-census :: exit 0 pnpm check:dts-closure :: exit 0 pnpm check:dual-build-cjs-loads :: exit 0 pnpm check:engine-double-contract :: exit 0 pnpm check:error-status-conformance :: exit 0 pnpm check:gitlink-declared :: exit 0 pnpm check:issue-citations :: exit 0 pnpm check:keyed-text-bounds :: exit 0 pnpm check:lean-entry-closure :: exit 0 pnpm check:logger-receiver-detach :: exit 0 pnpm check:nul-bytes :: exit 0 pnpm check:objectql-double-limit :: exit 0 pnpm check:org-identifier :: exit 0 pnpm check:page-declaration-shape :: exit 0 pnpm check:platform-object-tenancy-census :: exit 0 pnpm check:published-files :: exit 0 pnpm check:query-options-erasure :: exit 0 pnpm check:refd-timer-probe :: exit 0 pnpm check:registry-log-declared :: exit 0 pnpm check:rest-log-spy-declared :: exit 0 pnpm check:slot-lookup :: exit 0 pnpm check:sourcemap-no-sources-content :: exit 0 pnpm check:system-context-census :: exit 0 pnpm check:test-source-alias :: exit 0 pnpm check:tier-file-adoption :: exit 0 pnpm check:type-check-coverage :: exit 0 pnpm check:type-check-debt :: exit 0 pnpm check:undeclared-dep-imports :: exit 0 pnpm check:vouch-for-reference-callers :: exit 0 pnpm check:watch-hint-literal :: exit 0 pnpm check:where-matcher :: exit 0 ``` </details> ## Human Minimum - [x] 1. Schemas start as Zod; `packages/spec` holds no runtime logic. - [x] 2. Every name I added uses the casing for its kind. - [x] 3. No formatter ran; `pnpm lint` passes. - [x] 4. One issue, one branch off an up-to-date `main`, one pull request. - [x] 5. A change that publishes carries a changeset, and both declaration lines are filled in here and in the changeset. (Publishes nothing: `skip-changeset`.) - [x] 6. `packages/spec` touched: its generated artifacts are regenerated and committed. (Not touched.) - [x] 7. No hand edits to `content/docs/references/`, `packages/*/CHANGELOG.md` or `content/docs/releases/`. - [x] 8. Title, body and comments are in English. - [x] The *Before Submitting* list in CONTRIBUTING.md is done. --- _Generated by [Claude Code](https://claude.ai/code/session_01FEkQ5motx584vybgif8qQe)_ <!-- signed: domain:devx#3 · os-project-manager · session_01FEkQ5motx584vybgif8qQe --> --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent fe97d85 commit 86e9ef0

14 files changed

Lines changed: 746 additions & 30 deletions

File tree

‎packages/qa/scenarios/README.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -132,6 +132,7 @@ pnpm --filter @objectstack/scenarios run-scenarios --driver recorded:/tmp/rec.js
132132
pnpm --filter @objectstack/scenarios run-scenarios --author-timeout 90 # minutes per authoring turn (default 60)
133133
pnpm --filter @objectstack/scenarios run-scenarios --author-max-turns 400 # turns per authoring turn, every card (default by tier)
134134
pnpm --filter @objectstack/scenarios run-scenarios --ask-max-turns 20 # turns per question or operation turn (default 12)
135+
pnpm --filter @objectstack/scenarios run-scenarios --packages workspace # on this checkout's unreleased packages (default: published)
135136
```
136137

137138
Per card and mode the runner writes `docs/qa/platform-checklist/runs/<date>-scenario-<card>.json`
@@ -142,6 +143,17 @@ Per card and mode the runner writes `docs/qa/platform-checklist/runs/<date>-scen
142143
| `scratch` | `create-objectstack` → the agent builds from the card prompt and the runner's data contract (`prompts.ts`) → `objectstack validate` / `build` / `verify` (proofs chosen from the tags) → score → `objectstack dev --fresh` → seed, one user + permission-set grant + API key per persona → every question as every persona it names and every operation as its persona, over the app's MCP → the iterate change → rebuild, re-score, re-run |
143144
| `template` | a blank host project (`host<code>`) → boot → install the card's verified build **unmanaged** (`/api/v1/marketplace/install-local`, `installMode: 'unmanaged'`) → the agent applies the iterate change to the installed copy → score → seed → Run. A refused install is a finding in the record |
144145

146+
- **Package source** (`packages.ts`, `--packages`): `published` (the default) scaffolds with `npx -y
147+
create-objectstack@<version>` and installs from npm, what a customer gets. `workspace` measures the next release before
148+
it ships: it builds what the release builds for the packages it publishes (every non-private `@objectstack/*` and
149+
`create-objectstack`, `turbo run build build:dist`), `pnpm pack`s each — the tarball npm would serve, `files[]` only —
150+
and scaffolds with the packed `create-objectstack`, whose own install a global pnpmfile points at the tarballs. The
151+
tarballs are then kept as `overrides` in the project's `pnpm-workspace.yaml` (a later `pnpm add` stays on them), and the
152+
scaffold is refused unless every `@objectstack/*` package came from a tarball and `.claude/skills/` equals the
153+
workspace build's catalog (`packages/skills/dist/skills`) file for file. Every record (`env.packages`,
154+
`env.packagesLine`) and scorecard states the source: `packages: published 18.3.0`, or `packages: workspace @ <sha>
155+
(unreleased)`. ⛔ The gallery exporter exports only a `published` run: a `workspace` record, or one that states no
156+
source, is refused.
145157
- **Driver** (`driver.ts`): the local `claude` CLI on your own login (`claude -p --output-format stream-json`, the
146158
prompt written to its stdin: a build prompt carries the whole seed, and one argv string over 128 KiB makes `spawn`
147159
fail with `E2BIG` on Linux). It refuses to start when `ANTHROPIC_API_KEY` / `ANTHROPIC_AUTH_TOKEN` is set or the CLI

‎packages/qa/scenarios/src/export/cli.ts‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,9 @@
2323
*
2424
* It writes only under `--out`; it pushes nothing anywhere. ⛔ It exports only
2525
* a card whose `origin` is `generic` and whose `tier` is `starter` or
26-
* `delivery` (`generic.ts`) — and offers no flag to do otherwise.
26+
* `delivery` (`generic.ts`), and only a run on published packages — never a
27+
* `run-scenarios --packages workspace` record (`record.ts`) — and offers no
28+
* flag to do otherwise.
2729
*/
2830

2931
import { existsSync, readFileSync } from 'node:fs';

‎packages/qa/scenarios/src/export/exporter.ts‎

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -9,10 +9,12 @@
99
* date) decides, in this order, each a refusal with its reason:
1010
*
1111
* 1. the card is loaded, marked generic and of a gallery tier ({@link GalleryGuard});
12-
* 2. the record passed every half ({@link judgeRecord});
13-
* 3. the run's scratch project is still on disk, with its build;
14-
* 4. its transcript joins with the recording ({@link buildTranscript});
15-
* 5. its screenshots were taken, unless the caller turned them off.
12+
* 2. ⛔ the run was on published packages ({@link unpublishedRunRefusal}): a
13+
* `--packages workspace` run is never exported or shown;
14+
* 3. the record passed every half ({@link judgeRecord});
15+
* 4. the run's scratch project is still on disk, with its build;
16+
* 5. its transcript joins with the recording ({@link buildTranscript});
17+
* 6. its screenshots were taken, unless the caller turned them off.
1618
*
1719
* Output (`--out`), per exported app, written whole into a staging directory
1820
* and then swapped in, so a re-export first deletes the old directory and
@@ -52,7 +54,7 @@ import { renderScorecard, type CardScorecard } from '../runner/scorecard.js';
5254
import { GALLERY_GUARD, type GalleryGuard } from './generic.js';
5355
import { parseIndex, serializeIndex, type GalleryApp, type GalleryIndex } from './manifest.js';
5456
import { judgeRecord, type PassJudgement } from './pass.js';
55-
import { readRunRecords, redactWorkDir, type RecordFile } from './record.js';
57+
import { readRunRecords, redactWorkDir, unpublishedRunRefusal, type RecordFile } from './record.js';
5658
import type { CaptureScreenshots, PlannedShot, SkippedShot } from './screenshots.js';
5759
import { projectFrameworkVersion, readAuthoredSource } from './source.js';
5860
import { buildTranscript, mergeRecordings, renderTranscript } from './transcript.js';
@@ -273,6 +275,8 @@ async function exportOne(
273275
const verdict = guard(scenario);
274276
if (!verdict.eligible) throw new Refused(verdict.reasons);
275277
const { record } = rf;
278+
const unpublished = unpublishedRunRefusal(record);
279+
if (unpublished) throw new Refused([`${rf.file}: ${unpublished}`]);
276280
const judgement = judgeRecord(record);
277281
if (!judgement.passed) throw new Refused([`${rf.file} did not pass every half`, ...judgement.reasons]);
278282

‎packages/qa/scenarios/src/export/record.ts‎

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,12 @@ export const ScenarioRunRecordSchema = z.looseObject({
5252
date: z.iso.date(),
5353
app: z.string().min(1),
5454
runner: z.literal('scenario-runner'),
55-
env: z.looseObject({ framework: z.string().min(1), driver: z.looseObject({ driver: z.string().min(1) }) }),
55+
env: z.looseObject({
56+
framework: z.string().min(1),
57+
/** `runner/packages.ts` `PackageSource`; judged by {@link unpublishedRunRefusal}, so a record without it still reads. */
58+
packages: z.unknown().optional(),
59+
driver: z.looseObject({ driver: z.string().min(1) }),
60+
}),
5661
evidenceDir: z.string().min(1),
5762
results: z.array(z.looseObject({ id: z.string().min(1), verdict: VerdictSchema })),
5863
scorecard: z.looseObject({ card: z.string().min(1), modes: z.array(z.unknown()) }),
@@ -82,6 +87,23 @@ export interface ReadRecordsResult {
8287
ignored: string[];
8388
}
8489

90+
/**
91+
* Why a record's run may not be exported or shown, or `undefined` when it ran
92+
* on published packages. ⛔ Only a `published` run is gallery content: a
93+
* `--packages workspace` run measured an unreleased build — the record's
94+
* `@objectstack/spec` pin reads as a released version all the same — and a
95+
* record that does not state its source (written before `--packages`) proves
96+
* neither.
97+
*/
98+
export function unpublishedRunRefusal(record: ScenarioRunRecord): string | undefined {
99+
const p = record.env.packages as { source?: unknown; versions?: unknown; commit?: unknown } | undefined;
100+
if (p === undefined || p === null) return 'the record does not state its package source (`env.packages`): only a run on published packages is exported, and this one does not say it was — run the card again';
101+
if (p.source === 'workspace') return `the run was on this workspace's unreleased packages (\`packages: workspace @ ${String(p.commit)}\`): only a run on published packages is exported or shown`;
102+
if (p.source !== 'published') return `the record's package source is \`${String(p.source)}\`, not \`published\`: only a run on published packages is exported or shown`;
103+
if (!Array.isArray(p.versions) || p.versions.length === 0) return 'the record states published packages but no version: the run installed no project, so nothing proves what it ran on';
104+
return undefined;
105+
}
106+
85107
export class RunRecordError extends Error {
86108
constructor(file: string, detail: string) {
87109
super(`${file}: not a readable scenario run record: ${detail}`);

‎packages/qa/scenarios/src/runner/cli.ts‎

Lines changed: 36 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,10 @@
2727
* --out DIR where records go (default docs/qa/platform-checklist/runs, git-ignored)
2828
* --work DIR where projects are scaffolded (default: a temp dir)
2929
* --create-version V the create-objectstack version (default latest)
30+
* --packages P where the @objectstack/* packages come from (`packages.ts`):
31+
* `published` (default) — npm, what a customer gets;
32+
* `workspace` — this checkout, built and packed as the
33+
* release would publish it (unreleased; never exported)
3034
*
3135
* ⛔ No API-billed model: the live driver refuses to start when an API key is
3236
* configured (the cost ruling on cloud#3600).
@@ -42,6 +46,7 @@ import { formatRefusal, libraryOf, loadScenarios, readCapabilityAreas } from '..
4246
import { HttpApp } from './app.js';
4347
import { ClaudeCodeDriver, RecordedDriver, RecordingDriver, type AgentDriver } from './driver.js';
4448
import { minutesText, runCard, type ProjectOps } from './orchestrate.js';
49+
import { installedSpecVersion, PACKAGES_MODES, prepareWorkspacePackages, settleWorkspaceProject, workspaceScaffolder, type PackageSource, type PackagesMode } from './packages.js';
4550
import { bootDev, os, readArtifact, readNamespace, scaffold } from './project.js';
4651
import { cardScorecard, renderScorecard, runRecord } from './scorecard.js';
4752

@@ -75,10 +80,12 @@ export interface CliArgs {
7580
out?: string;
7681
work?: string;
7782
createVersion?: string;
83+
/** `--packages`: where the `@objectstack/*` packages come from. */
84+
packages: PackagesMode;
7885
}
7986

8087
export function parseArgs(argv: readonly string[]): CliArgs {
81-
const a: CliArgs = { modes: ['scratch', 'template'], driver: 'claude', iterateRun: true };
88+
const a: CliArgs = { modes: ['scratch', 'template'], driver: 'claude', iterateRun: true, packages: 'published' };
8289
for (let i = 0; i < argv.length; i++) {
8390
const flag = argv[i]!;
8491
const val = () => {
@@ -113,7 +120,14 @@ export function parseArgs(argv: readonly string[]): CliArgs {
113120
else if (flag === '--out') a.out = val();
114121
else if (flag === '--work') a.work = val();
115122
else if (flag === '--create-version') a.createVersion = val();
116-
else throw new Error(`unknown flag \`${flag}\``);
123+
else if (flag === '--packages') {
124+
const p = val();
125+
if (!(PACKAGES_MODES as readonly string[]).includes(p)) throw new Error(`unknown --packages \`${p}\` (${PACKAGES_MODES.join(', ')})`);
126+
a.packages = p as PackagesMode;
127+
} else throw new Error(`unknown flag \`${flag}\``);
128+
}
129+
if (a.packages === 'workspace' && a.createVersion !== undefined) {
130+
throw new Error('--create-version names a published create-objectstack; --packages workspace scaffolds with the one this checkout builds');
117131
}
118132
return a;
119133
}
@@ -174,11 +188,25 @@ export async function main(argv: readonly string[]): Promise<number> {
174188
(info.askMaxTurns !== undefined ? ` · ask max turns: ${info.askMaxTurns}` : ''),
175189
);
176190

191+
// Built and packed once, before the first card: every scaffold of this run installs the same tarballs.
192+
const workspace = args.packages === 'workspace' ? await prepareWorkspacePackages(repo, join(work, 'workspace-packages')) : undefined;
193+
if (workspace) console.log(`packages: workspace @ ${workspace.source.commit}${workspace.source.uncommitted ? ' + uncommitted changes' : ''} — ${workspace.tarballs.size} packed under ${join(work, 'workspace-packages')}`);
194+
/** The `@objectstack/spec` versions the current card's published scaffolds installed. */
195+
let publishedVersions = new Set<string>();
196+
177197
const ops: ProjectOps = {
178-
scaffold: (name, under) => {
198+
scaffold: async (name, under) => {
179199
const parent = join(work, under);
180200
mkdirSync(parent, { recursive: true });
181-
return scaffold(parent, name, args.createVersion ? { version: args.createVersion } : {});
201+
if (workspace) {
202+
const done = await scaffold(parent, name, workspaceScaffolder(workspace));
203+
if (done.result.exit !== 0) return done;
204+
return { dir: done.dir, result: await settleWorkspaceProject(done.dir, workspace) };
205+
}
206+
const done = await scaffold(parent, name, args.createVersion ? { version: args.createVersion } : {});
207+
const version = installedSpecVersion(done.dir);
208+
if (version) publishedVersions.add(version);
209+
return done;
182210
},
183211
os: (dir, a) => os(dir, a),
184212
readArtifact,
@@ -197,11 +225,13 @@ export async function main(argv: readonly string[]): Promise<number> {
197225

198226
for (const scenario of chosen) {
199227
console.log(`\n▶ ${scenario.card.id} (${args.modes.join(' + ')})`);
228+
publishedVersions = new Set();
200229
const cardRun = await runCard(scenario, driver, ops, { modes: args.modes, iterateRun: args.iterateRun, admin: ADMIN, ...(template ? { templateArtifact: template } : {}) });
201-
const record = runRecord({ run: `${date}-scenario-${scenario.card.id}`, date, ...(framework ? { framework } : {}), driver: info, runner: 'scenario-runner', evidenceDir: work }, cardRun);
230+
const packages: PackageSource = workspace ? workspace.source : { source: 'published', versions: [...publishedVersions].sort() };
231+
const record = runRecord({ run: `${date}-scenario-${scenario.card.id}`, date, ...(framework ? { framework } : {}), packages, driver: info, runner: 'scenario-runner', evidenceDir: work }, cardRun);
202232
const file = join(outDir, `${date}-scenario-${scenario.card.id}.json`);
203233
writeFileSync(file, JSON.stringify(record, null, 2));
204-
console.log(renderScorecard(cardScorecard(cardRun)));
234+
console.log(renderScorecard(cardScorecard(cardRun, packages)));
205235
console.log(`record: ${file}`);
206236
if (driver instanceof RecordingDriver && args.record) driver.save(args.record);
207237
}

‎packages/qa/scenarios/src/runner/index.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ export * from './driver.js';
1212
export * from './namespace.js';
1313
export * from './observe.js';
1414
export * from './orchestrate.js';
15+
export * from './packages.js';
1516
export * from './project.js';
1617
export * from './prompts.js';
1718
export * from './scorecard.js';

0 commit comments

Comments
 (0)