Repository navigation
Commit 58085ff
feat(scenarios): scenario-card schema + loader (private packages/qa package) and nine open seed cards (#30340)
Fixes #30289
Clause-②: no
Metadata-protocol: none
## 维护者速读
- **改了什么:** 新增私有包
`@objectstack/scenarios`(`packages/qa/scenarios`,不发布、无构建步骤、不调用任何模型),内含场景卡的
zod schema 与加载器。按 v3 场景库的地基改造:五项 schema 扩展(跨对象关系路径、卡级 `asOf`
加相对时间窗、按角色的行范围与逐角色真值、"种子答不了"的真值类型、写操作声明预期结果),外加可选的
`run.agentActions`。三张简单对照卡(发票、项目任务、CRM)升级到新格式,每张 8–9 个问数、5 个写操作。删除 6
张退役卡。README 写明 v3 场景库形态和中档卡的能力底线。
- **为什么改:** cloud#3600 Q1 = v3(「同意这9个场景」)与执行参数裁决;席位选定方案 A:本 PR 只做地基,6
张中档卡各自成单(#30476–#30481,均 Blocked-by 本 PR,合并后并行派发),以免单个 PR 超过 3000
行人工合并线。
- **风险与回滚:** 不触 `packages/spec`、不改公开面、包为 `private: true`、`expect`
形状未动(与已合并的评分器共用)。仓级改动只有已登记的三处(cross-package 声明、turbo 输入、ci-filter-parity
自测按名记录)。回滚 = revert 本 PR。
- **先看:** ① `packages/qa/scenarios/src/loader.ts` 的 `checkPath` /
`resolverFor`(关系路径)和逐角色真值那一段;② `packages/qa/scenarios/src/schema.ts` 的
`OutcomeSchema` 与 guardrail 对 outcome 的要求(在 `loader.ts`);③
`docs/qa/scenarios/invoice_with_line_items.json`(新格式的完整样子)。
- **两分钟验证:** `pnpm --filter @objectstack/scenarios test`,预期 `Test Files
3 passed`、`Tests 59 passed`;把 `invoice_with_line_items.json` 里 sales_rep
的 `scope` 删掉再跑,library 用例转红,`amount_per_invoice` 的逐角色真值对不上。
- **需要你做的:** 无,席位落地。
## What changes
**Rescope (seat decision, option A).** This PR is now the v3 library's
foundation. The six mid-tier cards (travel expense, order-to-cash,
service desk, leave and balances, requisition-to-receipt, asset
lifecycle) are sub-issues #30476, #30477, #30478, #30479, #30480 and
#30481. Each is Blocked-by this PR and each lands as its own PR. No
mid-tier card content is written here.
**New private package `@objectstack/scenarios`**
(`packages/qa/scenarios`), named after its siblings (`private: true`,
`exports` at `src`, no build step, `zod` its only runtime dependency).
It has three files:
- `src/schema.ts`: the card shape, with strict objects throughout.
- `src/truth.ts`: the declarative truth evaluator.
- `src/loader.ts`: `loadScenarios(dir, { areas })` returns typed cards
with `truths` and `personaTruths`, or `{ file, path, message }`
refusals. A card with any refusal is not returned.
**The five schema extensions.** Each one extends a closed enum or shape;
nothing is loosened.
1. **Relation paths.** `filter[].field` and `groupBy` accept
`lookup.field` and deeper paths, such as `invoice.status` or
`account.industry`. Every hop except the last must be a lookup declared
in `expect.lookups`, and the path resolves through the seed rows' lookup
ids. A hop that is not a declared lookup is refused at the exact path,
and so is a final field that no seed row of the reached object carries.
2. **Time windows.** `asOf` is a card-level fixed date, validated as an
ISO date. The new filter op `in_window` takes a value of `{ kind:
last_days, days }` or `{ kind: calendar, period: this_month |
previous_month | this_year | previous_year }`, and the window is
inclusive and computed against `asOf`. The loader refuses three things:
a window when the card has no `asOf`, a window over a field that holds
no date, and a window value given to any op other than `in_window`.
3. **Persona row scope.** A persona can now declare `scope: [{ object,
field, value }]`, where `field` may be a path (`invoice.owner` scopes
invoice lines through their invoice). A question can list `personas`,
and the loader then also computes that question's truth as each listed
persona. Before evaluating, it applies the persona's scope rules for the
question's object. If the question reads an object or field the persona
hides, the persona's truth is `{ kind: withheld }`, meaning a correct
reply declines.
4. **Unanswerable.** A new truth kind `{ kind: unanswerable, reason }`
says the seed cannot answer, so a correct reply says so instead of
giving a figure. It joins the closed enum.
5. **Declared write outcomes.** Every operation now names the persona it
acts for (`actingAs`) and declares an `outcome`, so it can be graded by
code alone. The outcome is one of:
- `applied`, with the post-state rows: each named by a seeded `id`, or
by `where` for a row the write creates, and carrying either the `values`
the data API must read back or `absent`. It can also name an
`auditActor`.
- `refused`, with a code from the spec's closed `ErrorCode` catalogue
(`@objectstack/spec/api`: StandardErrorCode plus the registered ledger,
ADR-0112). The catalogue is read from the spec export and never restated
here, so the package now depends on `@objectstack/spec` and its vitest
config aliases that import to spec source.
- `pending_confirmation`.
The guardrail checks are now held to the outcome: `write_validation`
needs a refused create or update, `confirm_irreversible` needs
`pending_confirmation`, and `audit_actor` needs `applied` with an
`auditActor`.
Two smaller additions:
- The optional `run.agentActions` lists at least 3 agent-callable
business actions, `{ name, object, summary, confirm }`, with at least
one confirm-gated.
- The counts band: a card needs at least 5 questions and 2 operations.
Below the default of 8 questions or 4 operations it must state why in
`run.countsReason`, or it is refused.
**The three controls, upgraded.** The `expect` blocks are untouched: the
corpus derivation still equals the cards.
| Card | Questions | Operations | New-arm questions |
|:---|:---|:---|:---|
| `invoice_with_line_items` | 9 | 5 | cross-object `invoice.status`;
`issued_on` this month (`asOf` 2026-10-31); 2 per-persona (sales_rep row
scope, warehouse_staff withheld); 1 unanswerable |
| `project_with_tasks` | 8 | 5 | cross-object filter and groupBy through
`project.status`; 2 per-persona (contractor scope by assignee,
client_viewer and contractor withheld); 1 unanswerable |
| `crm_account_with_contacts` | 8 | 5 | cross-object filter and groupBy
through `account.industry`; 2 per-persona (sdr scope via
`account.owner`, partner_user withheld); 1 unanswerable |
Each control's five operations are the same set of outcomes: one applied
create, one applied update with an audit actor, one confirm-gated
delete, one refused validation violation (`VALIDATION_FAILED`, the code
the engine's record validator answers for a per-field refusal), and one
refused overstep by a persona without the right (`PERMISSION_DENIED`).
Each control also declares 3 agent actions, one of them confirm-gated.
**Retired:** the six earlier generic cards are removed with their test
pins. `docs/qa/scenarios/README.md` now describes the v3 shape, the new
arms, and the mid-tier capability floor, including the declarative
floor: flows, formulas and validation rules are preferred over code
hooks, so a graduated app can install unmanaged.
**Repo wiring forced by existing gates**, unchanged from the first
round:
- `scripts/cross-package-test-inputs.mjs` and `turbo.json` declare that
this suite reads `docs/qa/scenarios/*.json` and the checklist area
files.
- `scripts/check-ci-filter-parity.mjs` names that glob in its rollback
pin, which moves from 20 to 21.
## Tests
- `test/library.test.ts`: the three controls load with zero refusals.
All 25 truths are asserted against hand-worked values, and so are the 10
persona truths.
- `test/refusals.test.ts`: 38 refusal cases in a table, each asserting
the single refusal's `file`, exact `path` and message class. There are
also 2 controls (the unmutated card loads clean, and a below-band card
loads once it states `countsReason`), plus id/basename and invalid-JSON
cases. The new arms are covered by:
- relation paths: 2 cases;
- time windows: 4 cases;
- persona scope and personas: 2 cases;
- unanswerable: 1 case;
- outcomes and actingAs: 5 cases;
- guardrail-to-outcome matching: 2 cases;
- agent actions: 2 cases;
- countsReason: 1 case.
- `test/truth.test.ts`: the evaluator, now including unanswerable, the
resolver for paths, and window ranges across month and year edges
(including a leap February).
## Acceptance notes
- **Size:** 2532 changed lines vs the merge base (+2529 / -3, 20 files),
under the 3000 human-merge line. This was measured before every push.
- **Persona rows:** a persona must still hide at least one object or
field (the earlier rule). So no persona is all-seeing, and an operation
acts as a scoped persona. That fits the per-persona grading, but it
means a card has no admin persona. If the mid-tier cards need one,
relaxing that rule is a follow-up decision, not made here.
- **Withheld semantics:** a question reading an object or field a
persona hides answers `withheld` for that persona. That includes a
hidden field reached through a relation path, and the lookup field used
as a hop counts as read.
- Not wired to the CLI or the scorer, which is sibling work. The
`expect` shape is untouched.
## Verification
<details><summary>Package suite and typecheck (head c1affd3)</summary>
```
pnpm --filter @objectstack/scenarios test Test Files 3 passed (3) · Tests 59 passed (59)
pnpm --filter @objectstack/scenarios typecheck exit 0
os-verify-lock: VERDICT command-exit 0
```
</details>
<details><summary>Ablations: one per new arm, through
scripts/ablation-replace.mjs, each restored byte-identical (git
hash-object == HEAD blob), git diff HEAD empty after</summary>
- `relation_paths`: mutation landed (blob 73dcb24->3fb8af58), 32
test(s) red, e.g. ['loads every card with no refusal', 'holds exactly
the three control cards (the mid-tier cards land on their own)'];
restored: True
- `time_windows`: mutation landed (blob d562f0d->9254d982), 2 test(s)
red, e.g. ["invoice_with_line_items: every question's computed truth is
the hand-worked one", 'computes windows against asOf, inclusive, across
month and year edges']; restored: True
- `persona_row_scope`: mutation landed (blob 73dcb24->c801f0c3), 3
test(s) red, e.g. ["invoice_with_line_items: every question's computed
truth is the hand-worked one", "project_with_tasks: every question's
computed truth is the hand-worked one"]; restored: True
- `persona_withheld`: mutation landed (blob 73dcb24->39a838a3), 3
test(s) red, e.g. ["invoice_with_line_items: every question's computed
truth is the hand-worked one", "project_with_tasks: every question's
computed truth is the hand-worked one"]; restored: True
- `unanswerable`: mutation landed (blob d562f0d->f9a664ff), 4 test(s)
red, e.g. ["invoice_with_line_items: every question's computed truth is
the hand-worked one", "project_with_tasks: every question's computed
truth is the hand-worked one"]; restored: True
- `outcome_guardrail_match`: mutation landed (blob 73dcb24->d93bd217),
1 test(s) red, e.g. ['refuses a confirmation check on an operation that
does not wait']; restored: True
- `outcome_post_state_id`: mutation landed (blob 73dcb24->f1d67cba), 1
test(s) red, e.g. ['refuses a post-state naming a created row by an id
the seed lacks']; restored: True
- `agent_actions_confirm`: mutation landed (blob 73dcb24->58d94865), 1
test(s) red, e.g. ['refuses agent actions with none confirm-gated'];
restored: True
- `counts_reason`: mutation landed (blob 73dcb24->313cb397), 1 test(s)
red, e.g. ['refuses fewer than 8 questions without a stated reason'];
restored: True
- `refusal_code_catalogue`: mutation landed (blob 47f63c7->163daac4), 1
test(s) red, e.g. ['refuses a refusal code outside the platform
error-code catalogue']; restored: True
- git diff HEAD empty: True
</details>
<details><summary>Derived gates (dispatch-gates --commands, reconciled
with --ran)</summary>
Tree: objectstack-ai/objectstack at c1affd3 (merge base f506b79).
`dispatch-gates --ran` reported `92 derived famil(ies) accounted for —
88 run, 4 NOT-MEASURED (3 DERIVED from a recorded exit 3, 1 claimed)`
and exited 0.
- 88 run, every one exit 0. Among them:
`check:cross-package-test-inputs`, `check:ci-filter-parity`,
`check:pm-dispatch-gates`, `check:test-source-alias`,
`check:type-check-coverage`, `check:undeclared-dep-imports`,
`check:doc-authoring`, `check:issue-citations`, `check:nul-bytes`,
`check:changeset-fixed`, `check-changeset-no-major` and
`check-empty-changeset`.
- `check:error-code-casing` (ADR-0112 D1) was red in CI on c3a257d
because the refusal test used a lowercase literal. Reproduced locally
with `pnpm check:error-code-casing`: exit 1,
`packages/qa/scenarios/test/refusals.test.ts:83 'validation_error'
(assignment)`. On c1affd3 it exits 0 (`no unlisted lowercase error
codes in 11258 scanned file(s)`). The fix is a closed vocabulary rather
than an exemption: the refusal-code field is now the spec's `ErrorCode`,
and the test probes a SCREAMING code outside the catalogue
(`VALIDATION_BROKEN`).
- NOT MEASURED, exit 3 PREREQUISITE NOT MET (built `dist/` of packages
this diff does not build): `check:dual-build-cjs-loads`,
`check:lean-entry-closure`, `@objectstack/lint
check:doc-formula-expressions`.
- NOT MEASURED, timed out under the local 420 s wrapper (CI-owned
whole-workspace census): `check:type-check-debt`.
- CI on c1affd3 at report time: 58 check-runs success and the rest
skipped. None red.
</details>
---
_Generated by [Claude
Code](https://claude.ai/code/session_01Q4EjjPXwX1YfHyKJUh4agu)_
<!-- signed: epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu
-->
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent c7840ea commit 58085ff
20 files changed
Lines changed: 2529 additions & 3 deletions
File tree
- .changeset
- docs/qa/scenarios
- packages/qa/scenarios
- src
- test
- scripts
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
0 commit comments