Repository navigation
Commit 6ef7f58
feat(scenarios): the scenario scorer moves into @objectstack/scenarios and judges capability tags, agent-action exposure, write outcomes and per-persona answers (#30594)
Fixes #30291
Clause-②: no
Metadata-protocol: none
## 维护者速读
- 改了什么:确定性场景评分器整体从 `packages/cli` 迁入私有包 `@objectstack/scenarios`(裁决
D)。新增封闭的能力标签检查表、Agent 动作暴露检查、写操作结果判分和按角色判分,并提供给运行器用的入口
`@objectstack/scenarios/score`。
- 为什么改:维护者裁定方案 D(「同意 D」),评分器和卡片格式放在一起。已发布的 CLI 不再携带任何场景代码,也不再需要依赖私有包。
- 风险与回滚:CLI 只把 `MetadataEvalCase` 类型挪到 `corpus.ts` 并原名重导出,行为不变。评分器此前没有从
CLI 发布过。回滚即撤销本 PR。
-
先看:`packages/qa/scenarios/src/score/scenario-tags.ts`(标签表与动作暴露判定)、`packages/qa/scenarios/src/score/index.ts`(运行器入口)、`.changeset/30290-scenario-scorer-port.md`
的删除(见下方"需要你做的")。
- 两分钟验证:`pnpm --filter @objectstack/scenarios test`,预期 11 个文件、163
个用例全部通过。
- 需要你做的:确认删除尚未发布的 `.changeset/30290-scenario-scorer-port.md`。它写的是"CLI
新增评分器",迁移后这句话已不成立。`check-empty-changeset` 对这类有意更正会一直保持红色,直到有人确认。
## What changes
Per the maintainer's ruling D on #30291 (「同意 D」), the deterministic
scenario scorer lives in the private `@objectstack/scenarios` package
(`packages/qa/scenarios`), beside the card format it judges. No model
call anywhere (cost ruling on objectstack-ai/cloud#3600:
「目前阶段尽量不要用云端的token,api的更贵」).
- **Moved with `git mv`, so history follows:** `scenario-score`,
`scenario-graph` and `scenario-run-grade` (merged in #30332),
`scenario-tags` and `scenario-operation-grade`, and their tests. They go
from `packages/cli/src/lint/` to `packages/qa/scenarios/src/score/` and
`test/score/`. `@objectstack/cli` ends with no scenario code and no
mirrored card types.
- **The graders take the card schema's own types:** `ScenarioExpect`,
`z.infer` of `OutcomeSchema` and `PostStateSchema`, a `Pick` of
`ScenarioCard`, and the loader's `Truth`. The `expect`, outcome,
persona, agent-action and truth mirrors are gone.
- **`@objectstack/scenarios/score`** (private subpath) is the runner's
entry (#30292). It has three functions, and each takes a card the loader
returned:
- `scoreCardArtifact(card, artifact)`;
- `gradeCardOperation(card, operationId, observation)`;
- `gradeCardAnswer(card, { question, persona?, turn })`. The persona's
hidden objects and fields are read off the card.
- **Capability-tag checks** (`scenario-tags.ts`): a closed table of nine
rows, listed in the package README for the mid-tier card authors.
- A tag with no row is `unscored`, never a pass. So is a row whose card
gives it nothing to judge. Both are counted nowhere.
- A row that runs zero checks fails.
- Rows read the owning predicates (`resolveFlowTriggerKind`,
`APPROVAL_NODE_TYPE`, `SINGLE_OPTION_TYPES`,
`summaryFunctionReadsField`, `effectiveSharingModel`,
`standaloneActionOwnerKey`). Collections are read through
`resolveStackCollection` from its owner, `@objectstack/core`.
- **Agent-action exposure:** each declared action must exist on its
object, carry `ai.exposed: true` (the MCP doors' opt-in) and, where the
card says `confirm`, carry `ai.requiresConfirmation: true` (the flag the
confirmation gate enforces, not the delete-mode listing heuristic). A
missing action fails all of its checks.
- **Write-operation grader:**
- applied: every post-state is read back, one row per `where`, and the
audit actor matches;
- refused: the exact `ErrorCode`, with zero writes;
- pending_confirmation: zero writes.
An unobserved fact fails.
- **Per-persona Run grading:**
- a `withheld` truth is must-not-reveal: a decline is correct, and a
stated all-rows figure is a leak;
- an all-rows figure outside a scoped persona's rows is a leak when the
reply states it.
- **Dependencies** of the private package: `@objectstack/core`,
`@objectstack/objectql`, `@objectstack/plugin-sharing` (workspace). Its
vitest config aliases each to source (`check:test-source-alias`).
- **Gates re-homed:**
- the cross-package inputs move from `@objectstack/cli#test` to the
scenarios entry and task, which also declare
`packages/cli/src/lint/corpus.ts` (the golden builds and the drift pin
read it);
- the `second-object-read-exposure` pin row for `scenario-graph.ts`
follows the file (same `not-a-served-read` class).
- **CLI:** `MetadataEvalCase` is declared in `corpus.ts`, beside the
corpus it types, and `metadata-eval.ts` re-exports it under the same
name. That keeps the scenarios type-check program from pulling in the
whole lint command.
- **Changesets:** one changeset (`@objectstack/scenarios` and
`@objectstack/cli`, patch). The unreleased
`.changeset/30290-scenario-scorer-port.md` is **deleted on purpose**: it
announces a scorer in the CLI that this PR moves out, so it is a
deliberate correction of a pending release note. `check-empty-changeset`
stays red on it until a person confirms.
- Body item 2 (`os lint --eval` reads cards) is cancelled by the ruling.
## Tests
- `pnpm --filter @objectstack/scenarios test`: 11 files, 163 tests.
Moved suites unchanged in what they assert. Added
`test/score/entry.test.ts` (the runner entry over the real
`crm_account_with_contacts` card).
- `scenario-cards.test.ts` loads the control cards through the loader:
- **corpus drift pin.** Each control card's `expect` must equal what
`corpus.ts` derives, with a floor of exactly 3 controls;
- **end-to-end readings:** crm 10/18, invoice 18/27, project 19/26;
- crm completed with what it asks scores 1, and each removal scores
lower.
- `pnpm --filter @objectstack/scenarios typecheck`: exit 0.
- `pnpm --filter @objectstack/core exec vitest run
src/security/second-object-read-exposure.pin.test.ts`: 4/4.
- `pnpm --filter @objectstack/cli typecheck`, build, and `vitest run
test/metadata-eval.test.ts src/lint/`: 43/43.
## Acceptance notes
- **Tag names.** `travel_expense.json` (merged meanwhile) declares 12
tags. Five score: `master-detail-inline-edit`, `roll-up-summary`,
`select-options`, `automation.record-triggered-flow` and
`access-security.field-permissions`. The rest are `unscored`:
`records-forms.formula-field`, `records-forms.validation-rule`,
`approvals.multi-step-approval`, `access-security.row-level-security`,
`dashboards.report`, `ai.ask-data` and `ai.agent-actions`. The README
table is what the remaining five card authors align to, or extend.
- **Size.** `dispatch-gates` reads 4324 changed lines: it counts with
`--no-renames`, so every moved file counts twice. The rename-aware count
is 1806, and that is how GitHub counts. Two controls: #30375 is 6567
no-renames vs 1561 rename-aware vs GitHub 1561; #30550 is 3228 vs 2482
vs GitHub 2482.
- The round-2 gate-forced additions to
`scripts/cross-package-test-inputs.mjs` and `turbo.json` are recorded as
accepted on #30291. They now point at the scenarios package.
## Verification
### One ablation per moved module, at the new home
(scripts/ablation-replace.mjs; tests import src relatively, so no dist
leg)
Every mutation landed (anchor 1 → 0) and was restored (blob == HEAD,
`git diff HEAD` empty):
| module | mutation | result |
|:---|:---|:---|
| scenario-score | widget measure check → pass | 2 failed / 16 |
| scenario-graph | nav entry resolves → true | 2 failed / 5 |
| scenario-run-grade | withheld leak scan off | 1 failed / 19 |
| scenario-tags | master-detail inlineEdit → true | 1 failed / 30 |
| scenario-operation-grade | no-write → true | 2 failed / 13 |
| score/index | operation judged against the wrong outcome | 1 failed /
7 |
The full per-check battery of the previous round (18 mutations) ran
against the same assertions before the move.
### Gates
- `node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack
--commands` on the patch-round diff: 91 commands, all run, exit codes
captured before any pipe.
- `check-empty-changeset --base origin/main`: exit 1, the
deliberate-correction refusal on `30290-scenario-scorer-port.md`
(above).
- `check:dual-build-cjs-loads` and `check:i18n-coverage`: exit 3
`PREREQUISITE NOT MET`. They read every package's built output; NOT
MEASURED, CI owns them.
- Every other family: exit 0, including
`check:cross-package-test-inputs`, `check:ci-filter-parity`,
`check:turbo-task-graph`, `check:undeclared-dep-imports`,
`check:test-source-alias` and `check:workspace-manifest-cycles`.
- Clause-② closure: walking the relative imports of the CLI's
`dist/index.d.ts`, `dist/console.d.ts` and `dist/hook-body.d.ts` reaches
39 files, none a scenario file, and no bare `@objectstack/scenarios`
specifier. `dist/lint/` holds only `corpus`, `hook-body-lowering`,
`metadata-eval` and `score`.
- Dogfood: n/a. No `!` in the title and no `(narrowing)` arm.
---
_Generated by [Claude
Code](https://claude.ai/code/session_01Q4EjjPXwX1YfHyKJUh4agu)_
<!-- signed: epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu
-->
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 4261169 commit 6ef7f58
24 files changed
Lines changed: 1741 additions & 65 deletions
File tree
- .changeset
- packages
- cli/src/lint
- core/src/security
- qa/scenarios
- src/score
- test/score
- scripts
This file was deleted.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
19 | | - | |
20 | 19 | | |
21 | 20 | | |
22 | 21 | | |
| |||
34 | 33 | | |
35 | 34 | | |
36 | 35 | | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
37 | 50 | | |
38 | 51 | | |
39 | 52 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| 22 | + | |
22 | 23 | | |
23 | 24 | | |
24 | | - | |
25 | | - | |
26 | | - | |
27 | | - | |
28 | | - | |
29 | | - | |
30 | | - | |
31 | | - | |
32 | | - | |
33 | | - | |
34 | | - | |
35 | | - | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
36 | 30 | | |
37 | 31 | | |
38 | 32 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
189 | 189 | | |
190 | 190 | | |
191 | 191 | | |
192 | | - | |
| 192 | + | |
193 | 193 | | |
194 | 194 | | |
195 | 195 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
21 | 22 | | |
22 | 23 | | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
| 6 | + | |
7 | 7 | | |
8 | 8 | | |
9 | | - | |
| 9 | + | |
| 10 | + | |
10 | 11 | | |
11 | 12 | | |
12 | 13 | | |
13 | 14 | | |
14 | 15 | | |
15 | 16 | | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
16 | 20 | | |
17 | 21 | | |
18 | 22 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
12 | | - | |
13 | | - | |
| 12 | + | |
| 13 | + | |
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| |||
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
38 | | - | |
| 38 | + | |
39 | 39 | | |
40 | 40 | | |
41 | 41 | | |
| |||
0 commit comments