Repository navigation
Commit 31d23c3
feat(cli): port the deterministic scenario scorer — shape score, artifact graph reader and Run grader — with a discrimination self-test (#30332)
Fixes #30290
Clause-②: no
Metadata-protocol: none
## 维护者速读
- 改了什么:把云端评测里的确定性打分器搬进 `packages/cli/src/lint`(三个新文件:结构打分
`scenario-score.ts`、构建产物转打分图 `scenario-graph.ts`、对话回合判分
`scenario-run-grade.ts`),各带测试;不调用任何模型,不加命令行参数,不加包入口导出,目前没有调用方。
- 为什么改:cloud#3600 D2
裁决「打分器可以迁移到开源项目」,场景库需要一个开源、纯代码的打分器;成本裁决「目前阶段尽量不要用云端的token,api的更贵」排除了模型评审。搬运时用
5 个黄金样例做「区分度自测」,实测原代码有两处会给坏构建打满分(导航指向不存在的对象、仪表盘组件绑定不存在的数据集),已在本 PR 修掉。
- 风险与回滚:风险低。只新增文件,没有任何现有代码引用它们,发布面只多出 `dist/lint/scenario-*`
三个模块(包入口不可达);回滚即删这三个文件和它们的测试。
- 先看:`packages/cli/src/lint/scenario-score.ts`(看顶部「与云端源码的差异」一段和
`scoreGraph`
的导航检查、`measureChecks`);`packages/cli/src/lint/scenario-graph.ts`(只按 spec
的唯一形状读产物,读不懂就抛错);`packages/cli/src/lint/scenario-score.test.ts`(区分度自测)。
- 两分钟验证:`pnpm --filter @objectstack/cli exec vitest run
src/lint/scenario-score.test.ts src/lint/scenario-graph.test.ts
src/lint/scenario-run-grade.test.ts`,预期 3 个文件、33 个用例全绿。
- 需要你做的:无,席位落地。
## What changes
Ported under the cloud#3600 D2 ruling from the cloud eval harness
(`structural-score.ts`, the graph reader in `agent-e2e.ts`, and
`ask-accuracy/scorer.ts`), each file with the Apache-2.0 header and one
provenance line. No model call anywhere; no new CLI flag; no new
package-entry export.
- **`packages/cli/src/lint/scenario-score.ts`**: `ScenarioExpectation`
(field-for-field the card's `expect` block, mirrored from the sibling
contract and not imported), `EvalGraph`, `scoreGraph` (exact names),
`ShapeExpectation`, `deriveShape` and `scoreGraphShape` (name-agnostic).
The reference type set is the spec's `REFERENCE_VALUE_TYPES`.
- **`packages/cli/src/lint/scenario-graph.ts`**:
`artifactToEvalGraph(artifact)` reads a built `dist/objectstack.json`
into the graph. Collections come through `resolveStackCollection`
(`@objectstack/core`, already re-exported by
`utils/stack-collections.ts`), the rule the CLI and
`@objectstack/verify` share, so a multi-package artifact with its
collections only under `packages[]` (ADR-0130 option B) reads the same
as a single-package one. ⛔ No local walk of `packages[]`. Each key is
read in its one spec shape (`fields` record, `reference` via
`referenceTargetOf`, `navigation` + nested `children`, widget `dataset`
+ `values`); the cloud reader's alias chains (`navigation ?? nav ?? tabs
?? …`, `reference_to ?? referenceTo ?? reference`) are deliberately not
carried over. An object with no name or a `fields` that is not a record
throws a `TypeError` instead of being read as an empty (degraded)
object. A fixed-target type (`user`) gets no `reference` in the graph;
which types fix their target is asked of the spec (`referenceTargetOf({
type })`), not listed here.
- **`packages/cli/src/lint/scenario-run-grade.ts`**: `gradeRun(truth,
hidden, { replyText, toolCalls: [{ name, input, output }] })` returns `{
correct, cited, leaks, problems }`. The dataset and the hosted-chat
transcript type are cut out: the caller passes the truth figures and the
hidden scope (objects, fields, hidden values, hidden aggregates). A tool
call answered over a hidden field is detected structurally (every key
and string in the input), not through a list of tool parameter names, so
a raw `sort` / `group_by` spelling is still seen. Records-in-order and
refusal-wording grading belonged to the cloud dataset's question kinds
and are not ported.
### Two deviations from the cloud source, both measured
The discrimination self-test degrades each golden build one defect at a
time and requires a strictly lower score. With the cloud semantics
restored, two degradations score exactly as high as the intact build:
1. `nav:TARGET->exists` asked whether an object/dashboard/app of that
name exists, not whether any app navigates to it. Now it passes only
when an app nav entry targets the name and resolves, and every app nav
entry is scored for resolving (`nav-resolves:TARGET`), as the shape
scorer already did.
2. A widget bound to a dataset the graph does not carry passed its
measure check (`!ds ||`). A local artifact is the whole build, so that
binding dangles and now fails, in both scorers.
Smaller: the field target key is `reference` (the spec's) rather than
`reference_to`, and the platform-object prefix is `sys_` / `ai_` (no
`cloud_`).
## Tests
All readings at `c0a6cf98` (`git rev-parse --short HEAD`), the final
commit, under `scripts/pm/os-verify-lock.sh` (shared-box seconds).
- New suites: `pnpm --filter @objectstack/cli exec vitest run
--maxWorkers=2 src/lint/scenario-score.test.ts
src/lint/scenario-graph.test.ts src/lint/scenario-run-grade.test.ts`
gave `Test Files 3 passed (3)` · `Tests 33 passed (33)`.
- Package unit tier: `pnpm --filter @objectstack/cli exec vitest run
--project unit --maxWorkers=2` gave `Test Files 387 passed (387)` ·
`Tests 5460 passed (5460)`. The `integration` tier is left to CI: the
diff touches no integration file, spawn entry, or driver/kernel boot
path.
- `pnpm --filter @objectstack/cli typecheck` exited 0 (`tsc --noEmit`,
then `check:test-typecheck: OK`). `eslint --no-inline-config
packages/cli/src/lint/scenario-*.ts` exited 0.
<details><summary>Discrimination self-test and the two ablations
(one-time proof, not kept as tests)</summary>
Every intact golden build scores 1 on both scorers. Each variant
(missing lookup, optionless select, dangling nav target, dangling
dashboard measure, dangling dashboard dataset) scores strictly lower on
both `scoreGraph` and `scoreGraphShape`, over 5 builds (optionless
select over 4).
The cloud semantics were put back with `node
scripts/ablation-replace.mjs` (the anchor must hit, and it restores to
`blob == HEAD` with an empty `git diff HEAD`):
1. Nav check reverted to `targets.has(norm(navName))` with the per-entry
`nav-resolves` loop removed. Result: 3 failed / 13 passed in
`scenario-score.test.ts`. That includes `dangling nav target` with
`invoice_with_line_items exact: expected 1 to be less than 1`, which
means the degraded build scored the same 1.00 as the intact one.
Restored: blob `eed52bc0d3ca` == HEAD.
2. Measure check reverted to `!ds ||`. Result: 1 failed / 15 passed,
`dangling dashboard dataset` with `invoice_with_line_items exact:
expected 1 to be less than 1`. Restored: blob `eed52bc0d3ca` == HEAD.
</details>
<details><summary>Clause-② measurement</summary>
After `pnpm --filter @objectstack/cli build`, I walked the declaration
closure of the three package entries (`dist/index.d.ts`,
`dist/console.d.ts`, `dist/hook-body.d.ts`). It holds 39 declaration
files, and 0 of them are under `dist/lint/`. Positive control: the same
walker started at `dist/lint/scenario-graph.d.ts` finds 2 files under
`dist/lint/`. Reading: `Clause-②: no`.
</details>
## Acceptance notes
- PM mechanism assumption on the artifact reader: the CLI already has
the reader, `resolveStackCollection` (re-exported by
`packages/cli/src/utils/stack-collections.ts`), and it is reused.
`packages/verify/src/artifact-collections.ts`'s `declaredCollection` was
not used: it answers a different question (a truthy top level wins, so a
declared-empty array behaves differently) and the CLI does not reach
into `@objectstack/verify` for a read it already owns.
- PM mechanism assumption on the corpus: the 5 fixtures in `corpus.ts`
carry objects only (no app, dataset or dashboard), so the self-test
completes each with an app (nav over every object and the dashboard),
one dataset and one dashboard, then parses the result with
`ObjectStackDefinitionSchema` (the parse `os build` runs before writing
the artifact). Every intact build scores 1 on both scorers; the
reproduction of the "always 1.00" blind spot is the two ablations below,
not the intact scores.
- `blog_post_with_comments` declares no select, so the optionless-select
variant runs on 4 of 5 builds; the test pins that count.
- The two scorer blind spots also live in the cloud source this was
ported from (`carrier: cloud`, reported to the seat, not filed here).
## Verification
Gate list from `node scripts/pm/dispatch-gates.mjs --repo
objectstack-ai/objectstack --commands` (no paths: the script takes the
change set from the merge-base). Each command ran at `c0a6cf98`, with
its exit code captured before any pipe. `--ran` reconciliation: `59
derived famil(ies) accounted for — 57 run, 2 NOT-MEASURED (2 DERIVED
from a recorded exit 3)`.
- 57 exited 0. `node scripts/check-plugin-teardown-shape.mjs
--self-test` first exited 3 because its pinned positive-control commit
was missing on a shallow checkout. After `git fetch --depth=1 origin
621a487` it exited 0.
- NOT MEASURED: `check:dual-build-cjs-loads`, reason: PREREQUISITE NOT
MET (it reads `dist/` of 10 packages outside this diff's build closure).
- NOT MEASURED: `check:i18n-coverage`, reason: PREREQUISITE NOT MET (the
showcase config loads packages without build output, e.g.
`@objectstack/service-print`).
- Both are declared narrowings, and CI runs them on its full build. The
diff touches neither dual-build output nor i18n.
---
_Generated by [Claude
Code](https://claude.ai/code/session_01Q4EjjPXwX1YfHyKJUh4agu)_
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent a545a04 commit 31d23c3
8 files changed
Lines changed: 1313 additions & 0 deletions
File tree
- .changeset
- packages
- cli/src/lint
- core/src/security
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
0 commit comments