Repository navigation
Commit c571165
feat(scenarios): a gallery exporter writes verified, generic scenario apps to a local directory in the layout of objectstack-ai/gallery (#30930)
<!-- claude-projects-attribution: {"github_login":"marchtian"} -->
_Requested by **田騏** · [project
thread](https://claude.ai/code/project/chan_01VmuUhXU9SSUzqJYyXSmYSi?thread=cmsg_01VmuUhXU9SSUzqJYyXSmYSiNjyvCdXYygFqntMNR7S2dW)_
Part of #30349
Clause-②: no
Metadata-protocol: none
## 维护者速读
- **改了什么**:新增「画廊导出器」`export-gallery`:读本地跑分器写下的跑分记录和项目目录,把每一半都通过的场景应用按
objectstack-ai/gallery 的布局写到本地目录:应用源码、跑分记录与计分卡、每个角色的问数问答记录(Markdown 加
JSON),以及 Playwright 对启动后应用截的列表、看板/日历(卡片打了标签才截)、记录页和仪表盘截图,外加一份经 zod
校验、排序稳定的 `index.json`。重导会先删旧目录;不再通过的场景从目录和索引里移除。
- **为什么改**:画廊是主要引流渠道,内容必须是验证过、每次发版重建的应用;导出器是从跑分结果到画廊内容的那一步。
- **风险与回滚**:只动未发布的私有包 `@objectstack/scenarios`(QA 工具),只写本地目录,不向 gallery
仓库写任何东西。卡片目前没有「通用」标记,所以导出器现在拒绝所有卡片,画廊保持为空,直到你决定标记字段。回滚即还原本 PR。
- **第二轮**:按决定 4,记录里只要有一个「未测量」的轮次(种子行被拒,或记录标为 not measured
的任一问答或操作),这个应用就不导出,即使各半的结论都是 pass;已导出的旧版本会被移除。PR 改为 `Part of #30349`。
- **第三轮**(合同评审 FAIL 后):用 `--work` 时,记录里原机器的工作目录(`evidenceDir`)和 `--work`
两者都替换成 `$WORK`,输出里不再出现任何一方的绝对路径;页面没真正显示(非 2xx,或 Console
自己的「未找到」视图)的截图被拒绝并报告;种子写入中途抛错时,已写入/被拒的部分计数留在记录里(与 main 一致);`--runs`
里没有任何跑分记录而旧索引列着应用时拒绝执行、什么都不删;`--no-screenshots`
不会用无截图的条目替换原本有截图的条目,旧目录和条目保留,摘要里标为 kept。
-
**先看**:`packages/qa/scenarios/src/export/generic.ts`(只导出通用卡的守卫);`packages/qa/scenarios/src/export/pass.ts`(「每一半都通过」怎么从记录里读);`packages/qa/scenarios/src/export/exporter.ts`(删除与替换的规则)。
- **两分钟验证**:`pnpm --filter @objectstack/scenarios exec vitest run
test/export`,预期 44 个用例全部通过,其中 "generic only" 用例断言默认守卫拒绝开放库全部 9 张卡。
- **需要你做的**:在报告评论的 `write_path_options` 里选一条写入 gallery
的路径,并决定卡片的「通用」标记字段。
## What changes
A gallery exporter in `@objectstack/scenarios` (`src/export/`, CLI `pnpm
--filter @objectstack/scenarios export-gallery --runs DIR --out DIR`, in
the style of `src/runner/cli.ts`). It reads the run records and project
directories the local runner writes and writes verified apps to `--out`
in the layout of objectstack-ai/gallery. ⛔ It writes nothing to
objectstack-ai/gallery and adds no workflow; the write path is a design
item for the maintainer (the `write_path_options` in the report
comment).
**Which record, and "passed every half"** (`pass.ts`). The newest record
of each card decides (by `date`). Its per-half verdicts are read from
the record's `results`, the rows `runRecord` writes, and nothing is
re-scored. A half passes only on `pass`: `partial`, `fail`, `blocked`,
`not-run` and a missing row are not a pass. The record must hold scratch
mode, because that is the project the gallery ships. Scratch mode owes
all six halves (Build, Run-问数, Run-operations, iterate Build, iterate
Run-问数, iterate Run-operations), so a run with `--no-iterate-run` does
not pass. Template mode, when present, owes its three iterate halves.
Its first Build and Run rows are `not-run` by construction and are not
asked. ⛔ A record with any not-measured turn does not pass, even when
every half reads `pass` (round 2, decision 4). The scorecard counts a
turn that rests on a refused seed row in neither passed nor total, so
its half can still read `pass`. `judgeRecord` therefore reads every Run
half the record holds (scratch Run, both modes' iterate Run) and names
each question or operation marked `notMeasured` as a reason.
**Generic only** (`generic.ts`). A card is exported only when it is
explicitly marked generic. The card schema has no such marker:
`provenance` is free prose, and the open library's directory is not a
marker because the loader reads any directory, the private overlay
included. So `NO_GENERIC_MARKER`, the guard the CLI uses, refuses every
card, and the CLI has no flag to pass another guard. The exporter takes
the guard as a parameter so its tests can run the export path with a
double. `schema.ts` and `loader.ts` are untouched (#30846 is in flight
there).
**The `apps/SCENARIO/` layout** (`exporter.ts` `APP_LAYOUT`):
`README.md` (title, requirement, the verdict of each owed half, the
scorecard table, links), `source/` (the authored metadata source: the
allow-list `objectstack.config.ts`, `objectstack.manifest.json`,
`package.json`, `tsconfig.json` and the `src/` tree; never
`node_modules`, `dist`, `.agents`/`.claude` skills, `.env` or scaffold
boilerplate; a symbolic link refuses the export), `run-record.json` (the
record as read, with every work directory that names the run replaced by
`$WORK`, see below), `scorecard.json` and `scorecard.md`,
`transcript.json` and `transcript.md`, and `screenshots/KIND.png`.
**No local path** (`record.ts` `redactWorkDir`, `exporter.ts`
`workDirsOf`). A record names the work directory of the machine that ran
it: `evidenceDir`, every build command tail, any error text. `--work
DIR` names where the projects are read on this machine. Both, and the
real path of each that exists here, are replaced by `$WORK` in
`run-record.json`, `scorecard.json`, `scorecard.md`, `transcript.json`,
`transcript.md` and `README.md`. The longest is replaced first, and only
at a path-name boundary, so `/tmp` never rewrites `/tmpfoo` or `/x/tmp`.
The authored `source/` files are copied as written.
**The transcript** (`transcript.ts`). The record keeps each turn's grade
but not its reply. The reply is in the run's recording (`run-scenarios
--record F`, passed as `--recording F`). Every recorded reply is graded
again with `gradeCardAnswer`, exactly as the runner grades it, and a
grade that differs from the record's refuses the export. A recording
from another run therefore never reaches the gallery as this run's
answers. An asked turn with no recorded reply also refuses the export.
**Screenshots** (`screenshots.ts`). This is a separate step that
`--no-screenshots` skips. It boots the scratch project with the runner's
own `bootDev`, seeds it with the runner's own seed loop, and signs in
through the Console's sign-in door. `planShots` chooses the views from
the card and the build alone:
- list: the first nav-target object the build declares;
- kanban or calendar: only when a card capability tag names one (e.g.
`records-forms.kanban-drag-persistence`), showing a view of that type
the build declares, read through the spec's `deriveViewContainerObject`
and `expandViewContainer`;
- record: the list object's first seed row;
- dashboard: one the card expects, otherwise the build's first.
A tagged view the build lacks, or a refused seed row, is a skipped shot
with its reason. A page the Console did not show is never proof
(`pageRefusal`): a navigation that answers no response or a status
outside 2xx (a 304 revalidation counts as shown), or a page that draws
one of the Console's own not-found views (`CONSOLE_NOT_FOUND`: the
`object-not-found`, `object-package-disabled`, `record-page-not-found`
and `form-not-found` test ids, and the router's `Page not found` title),
is a skipped shot with the reason, and its file is removed. A screenshot
step that fails refuses the app instead of exporting it without its
proof.
**Re-export replaces** (`exporter.ts`). Each app is written whole into a
staging directory and swapped in after the old directory is deleted. The
exporter owns what the previous `index.json` lists. An app it listed
that does not export this time is deleted and leaves the index, and the
summary says why. That covers an app whose newest record no longer
passes, an app whose card is gone or not generic, and an app whose
project is gone. A directory the index never listed is left alone and
named. An invalid previous index stops the export before anything is
touched. So does a runs directory with no scenario record while the
previous index lists apps: that is a wrong `--runs`, and the CLI exits
non-zero. An export with `--no-screenshots` never replaces an app whose
previous export had screenshots: the old directory and entry are kept
and the summary lists the app as `kept`. An app that no longer passes is
still removed.
**`index.json`** (`manifest.ts`, zod, strict at every level, validated
before it is written). It holds `schemaVersion`, plus one entry per app,
sorted by id. Each entry has `id`, `code`, `namespace`, `path`,
`function` (the card's `category`), `title` (`en` from the card's
`title`; `zh` is `null` because the card schema has no Chinese title),
`summary` (`prompt.short`), `capabilities`, `screenshots`, `qa`
(`questions`, `turns`, `personas`), `operations`, `framework` (`version`
is the `@objectstack/spec` pin of the generated project; `runner` is the
record's `env.framework`), `runDate` and `run`. There is no export
timestamp, so a re-export with no change is a no-op diff.
**Runner touch-ups, no behaviour change.** `orchestrate.ts`: the seed
loop moves out of `runHalf` into an exported `seedApp` (the screenshot
step reuses it), which tallies into the half's own `seed` so an insert
that throws still leaves the partial tally in the record, as on main,
and `modeDir(card, mode)` names the `CARD-scratch` directory the
exporter finds the project in. `runner/cli.ts`: `ADMIN` and `findUp` are
exported. `package.json`: the `./export` entry, the `export-gallery`
script, and `playwright-core` 1.63.0, the version already in the
lockfile for dogfood and service-print. The lockfile change is the
three-line importer entry.
## Tests
- `test/export/exporter.test.ts` (28 tests) runs on a real `runCard` run
of the real `crm_account_with_contacts` card, in both modes. Its fixture
(`test/export/fixture.ts`) writes real project directories under a temp
work directory with in-memory app doubles, wrapped in the runner's own
`RecordingDriver`. It pins:
- the exact layout;
- the source allow-list, which leaves out `node_modules`, `.agents`,
`.claude`, `.objectstack`, `.env`, `README.md` and `Dockerfile`;
- the `$WORK` redaction;
- the full index entry;
- the transcript, with 9 asked turns per Run half, each with its reply
and grade, and the three Run halves;
- a re-export that is byte-identical, file for file, and leaves the
index unwritten;
- stale files gone after a re-export;
- a newer failing record that removes the app from `apps/` and the
index, while an unowned directory is kept;
- the newest record deciding;
- an invalid previous index refusing with the tree unchanged;
- an unreadable record refusing the whole export, while non-runner JSON
is ignored;
- the default guard refusing every card, both in `exportGallery` and end
to end through the CLI over the real card directory;
- a missing recording, and a recording whose reply grades differently,
each refused;
- a missing project and a missing card, each refused;
- the screenshot step's shots reaching the index and README, and a
failing screenshot step refusing the app;
- the index schema refusing unsorted, duplicate, wrong-path and
unknown-key input;
- the pass rule on clean, not-run, partial, fail, missing-row and
no-scratch records;
- under `workDir`, a record whose strings name another machine's
directory: neither that directory, nor `workDir`, nor its real path
appears in any file under `--out`, and the redaction's path-name
boundary;
- an empty runs directory over an index that lists apps refusing,
through `exportGallery` and the CLI, with the tree unchanged, while an
empty runs directory with no index writes an empty index;
- `--no-screenshots` keeping an app that had shots, byte for byte, and
reporting it as `kept`, while a newer failing record still removes it;
- a not-measured turn: a run whose app refuses one seed row (contact
`ct4`) keeps every half at `pass` and still does not pass, with one
reason per not-measured question and operation in each Run half; an
operation marked not measured is its own reason; and `exportGallery`
refuses the app and removes its earlier export from `apps/` and the
index.
- `test/export/screenshots.test.ts` (16 tests) tests view selection with
a double:
- untagged cards get no kanban or calendar;
- tagged cards get the declared view of that type;
- a missing view or dashboard is skipped with its reason;
- a build with no app plans no shots;
- a container is read in the shape `os build` writes;
- `captureScreenshots` with boot, harness and browser doubles runs boot,
then seed, then shots, then close, then stop;
- a refused record row is skipped;
- a browser that writes no file raises an error, and the app is still
stopped;
- `pageRefusal` on 2xx, 304, no response, other statuses and the
Console's not-found views, and `captureScreenshots` skipping a 500 page
and a not-found page with their reasons, their files removed.
- `test/runner/orchestrate.test.ts` (one more test): a seed insert that
throws mid-loop leaves the partial tally (2 inserted, 1 refused) in the
half next to its error. The same test fails against the round-2
`seedApp`.
## Acceptance notes
- The exporter refuses every card today because no card is marked
generic. The marker field and the Chinese title are open questions in
the report comment, so a merged PR leaves the gallery empty until the
marker lands.
- PNG bytes are not stable across boots, because user ids and relative
times differ. `index.json` and every text file are stable; a re-export
with screenshots rewrites the PNGs.
- The gallery's `apps/README.md` says it "is replaced when the first
export lands". The gallery's AGENTS.md lists it as hand-written, so the
exporter does not write it. The layout above is what it should describe.
- On the released 18.3 Console, an administrator's first app page opens
a default-currency dialog, once per browser session. The adapter visits
the first shot's page once without shooting it. That dialog is retired
on `main` (the first-run checklist changeset).
## Verification
<details><summary>Real boot reading: create-objectstack 18.3.0, a
hand-written fixture app, Chromium</summary>
No model was run, so no generated app exists here. Instead, a project
was scaffolded with `npx create-objectstack@latest democrm` (18.3.0). A
fixture app was written into it by hand: account and contact objects, a
kanban list view, a dataset and dashboard, and app `democrm_crm`. `os
build` passed. The project was placed at
`WORK/crm-account-with-contacts-scratch/democrm`, and `exportGallery`
ran with the real `captureScreenshots` (`bootDev`, `HttpApp`,
`playwrightBrowser` on `/opt/pw-browsers/chromium`), using the crm card
with a kanban tag added. Result: exported, four shots, none skipped —
`/_console/apps/democrm_crm/democrm_account`,
`…/democrm_account/view/democrm_account.board`,
`…/democrm_account/record/ac1`, `…/dashboard/democrm_overview`. The PNGs
show the seeded grid, the kanban grouped by industry, the Acme record
page with its two contacts, and the dashboard with 3 accounts by
industry. The first reading found the currency dialog over the list
shot; the warm-up visit was added and the reading repeated clean. A
first attempt from inside vitest got `401` on the admin sign-in, because
vitest's `NODE_ENV=test` reached `objectstack dev`, so no dev admin was
seeded. The CLI runs under tsx without it. This reading is a one-off and
is not a permanent test.
</details>
<details><summary>Gates (local subset, derived by dispatch-gates for the
final diff)</summary>
Head `30360bf2c6` (round 3). `node scripts/pm/dispatch-gates.mjs
--commands --repo objectstack-ai/objectstack` derived 64 commands, the
same list as rounds 1 and 2; each exit code is captured before any pipe.
```text
node scripts/check-closing-keyword-parity.mjs :: exit 0
node scripts/check-closing-keyword-parity.mjs --self-test :: exit 0
node scripts/check-comment-mask-adoption.mjs :: exit 0
node scripts/check-comment-mask-adoption.mjs --self-test :: exit 0
node scripts/check-comment-mask-corpus.mjs :: exit 0
node scripts/check-console-intercept-disarm.mjs :: exit 0
node scripts/check-console-intercept-disarm.mjs --self-test :: exit 0
node scripts/check-dts-emitted.mjs --self-test :: exit 0
node scripts/check-issue-citations.mjs :: exit 0
node scripts/check-osv-exemptions.mjs :: exit 0
node scripts/check-osv-exemptions.mjs --self-test :: exit 0
node scripts/check-plugin-teardown-shape.mjs :: exit 0
node scripts/check-plugin-teardown-shape.mjs --self-test :: exit 0
node scripts/ci/assert-turbo-remote-cache.mjs --self-test :: exit 0
node scripts/ci/plan-test-core-shards.mjs --self-test :: exit 0
node scripts/ci/prune-turbo-cache.mjs --self-test :: exit 0
node scripts/docs-audit/check-affected-docs.mjs :: exit 0
node scripts/docs-audit/check-drift-comment.mjs :: exit 0
node scripts/osv-base-relative.mjs --self-test :: exit 0
pnpm --filter @objectstack/spec run check:duration-unit-keys :: exit 0
pnpm --filter @objectstack/spec run check:llms-txt :: exit 0
pnpm check:changeset-fixed :: exit 0
pnpm check:ci-filter-parity :: exit 0
pnpm check:closed-ceremony-text :: exit 0
pnpm check:cross-package-test-inputs :: exit 0
pnpm check:dispatcher-error-vocabulary :: exit 0
pnpm check:doc-authoring :: exit 0
pnpm check:driver-memory-census :: exit 0
pnpm check:dts-closure :: exit 0
pnpm check:dual-build-cjs-loads :: exit 0
pnpm check:engine-double-contract :: exit 0
pnpm check:error-status-conformance :: exit 0
pnpm check:gitlink-declared :: exit 0
pnpm check:issue-citations :: exit 0
pnpm check:keyed-text-bounds :: exit 0
pnpm check:lean-entry-closure :: exit 0
pnpm check:logger-receiver-detach :: exit 0
pnpm check:manifest-repository-directory :: exit 0
pnpm check:merge-driver :: exit 0
pnpm check:nul-bytes :: exit 0
pnpm check:objectql-double-limit :: exit 0
pnpm check:org-identifier :: exit 0
pnpm check:override-consistency :: exit 0
pnpm check:page-declaration-shape :: exit 0
pnpm check:platform-object-tenancy-census :: exit 0
pnpm check:pnpm-flag-spellings :: exit 0
pnpm check:published-files :: exit 0
pnpm check:query-options-erasure :: exit 0
pnpm check:refd-timer-probe :: exit 0
pnpm check:registry-log-declared :: exit 0
pnpm check:rest-log-spy-declared :: exit 0
pnpm check:slot-lookup :: exit 0
pnpm check:sourcemap-no-sources-content :: exit 0
pnpm check:system-context-census :: exit 0
pnpm check:test-source-alias :: exit 0
pnpm check:tier-file-adoption :: exit 0
pnpm check:turbo-task-graph :: exit 0
pnpm check:type-check-coverage :: exit 0
pnpm check:type-check-debt :: exit 0
pnpm check:undeclared-dep-imports :: exit 0
pnpm check:vendor-export-contract-resolve :: exit 0
pnpm check:watch-hint-literal :: exit 0
pnpm check:where-matcher :: exit 0
pnpm check:workspace-manifest-cycles :: exit 0
```
`--ran` reconciliation: `✓ dispatch-gates --ran: 64 derived famil(ies)
accounted for — 64 run, 0 NOT-MEASURED (a DERIVED zero — all 64 recorded
an exit code and none of them is 3)`.
Package: `pnpm --filter @objectstack/scenarios` `tsc --noEmit` exit 0;
`vitest run` exit 0, `Test Files 20 passed (20)`, `Tests 290 passed
(290)`. `test:repo` is not run locally; it belongs to CI.
</details>
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_017ihL2ebbZkr2QAEG1x7UC8
<!-- signed: domain:devx#2 · marchtian ·
session_017ihL2ebbZkr2QAEG1x7UC8 -->
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 78eb352 commit c571165
18 files changed
Lines changed: 2509 additions & 37 deletions
File tree
- packages/qa/scenarios
- src
- export
- runner
- test
- export
- runner
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
11 | | - | |
| 11 | + | |
| 12 | + | |
12 | 13 | | |
13 | 14 | | |
14 | 15 | | |
15 | 16 | | |
16 | | - | |
| 17 | + | |
| 18 | + | |
17 | 19 | | |
18 | 20 | | |
19 | 21 | | |
20 | 22 | | |
21 | 23 | | |
22 | 24 | | |
| 25 | + | |
23 | 26 | | |
24 | 27 | | |
25 | 28 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
0 commit comments