Skip to content

Commit 5b22ee7

Browse files
fix(scenarios): run grading reads the card's today, app validation codes, overstep probes, scope figures and served keys (#31214)
<!-- claude-projects-attribution: {"github_login":"os-project-manager"} --> _Requested by **Leehom** · [project thread](https://claude.ai/code/project/chan_01TenzsRopZj4jMbvjX8uz1X?thread=cmsg_01TenzsRopZj4jMbvjX8uz1XUxCbRpVSZrhh4aWPUL2UTj)_ Fixes #31145 Clause-②: no Metadata-protocol: none ## 维护者速读 - 改了什么:场景跑测的评分与探针按首次中档跑测暴露的七处误判修正:操作提示词也写明“Today is 卡片日期”,平台时钟自己盖章(agent 未写)的日期字段才接受“跑测当天”,agent 自己写了跑测当天仍判错;应用自定义的校验拒绝码(400 类)可满足 `VALIDATION_FAILED`;越权写入同样由跑测器亲自探测;范围数字只有在本人可见行算不出时才算泄露;隐藏字段按工具返回的键判泄露,不按请求的 `fields`。 - 为什么改:首次完整中档跑测里,表现正确的 agent 和平台因为评分器的这些读法被判失败,分数反映的是评分器而不是被测对象。 - 风险与回滚:只动私有包 `@objectstack/scenarios` 与两份 README,不发布、卡片 JSON 未改;回滚即 revert 本 PR。代价是评分规则放宽三处(应用码、跑测日期、本人范围数字),每处都是封闭规则并有反向用例钉住。 - 先看:`packages/qa/scenarios/src/score/scenario-operation-grade.ts`(`refusalMeets`、`runDateReading`)、`packages/qa/scenarios/src/score/scenario-run-grade.ts`(范围数字豁免、`PROJECTION_KEYS`)、`packages/qa/scenarios/src/loader.ts`(`figuresInScope`)。 - 两分钟验证:`pnpm --filter @objectstack/scenarios exec vitest run test/runner/mid-tier-run-readings.test.ts`,预期 21 个用例全绿。 - 需要你做的:无,席位落地。 ## What changes Each item is pinned on the real card it was found on, in the new `test/runner/mid-tier-run-readings.test.ts`. 1. **The card's today.** - Measured on base `13c6d8ff`: `questionPrompt` already said ` Today is ASOF.`; `operationPrompt` did not. Both now render it through one helper, `todayOf` (`src/runner/prompts.ts`). - Expected states that pin a date are judged against the date the agent was told. Only under the card's conditional ("if the platform clock itself must match") does the run date also meet a declared `asOf`: in a field none of the agent's own write calls on that object set, so the platform's clock stamped it (a default or a flow). The runner records the fields each write call set (`agentSetOf` / `agentSetFields` in `observe.ts`: a create's or update's `data`, a `run_action`'s `params`) on the observation as `agentSet`, and the run dates as each clock instant's UTC and local date at the turn's start and end (`clockDates`). - The grader (`runDateReading` in `scenario-operation-grade.ts`) reads a run date in place of `asOf` three ways: platform-stamped passes, `read "2026-11-07", declared "2026-10-31" (the card's today), the run date the platform stamped`; agent-written fails, `…: the agent wrote the run date where the prompt told it the card's today`; and an observation that does not say which write set it fails as not observed. The post-state `where` lookup (`postStateRows`) applies the same rule, so a created row whose date the agent wrote as the run date is not found. - So an agent that ignores "Today is 2026-10-31" and writes the real date fails, which is the failure the run reported; the four measured rows (`decided_on`, `approved_on`, `order_date`, `checked_out_on`) were the agent stamping the real date. 2. **App-defined validation codes: decided, a closed rule, applied to every card.** A declared `VALIDATION_FAILED` is met by an answered code that (a) is not in the spec's `ErrorCode` catalogue, so it is the app's own (a hook's declared refusal, for example `OVER_RECEIPT`), and (b) carries a 4xx status the spec files under `VALIDATION_ERROR` (`standardErrorCodeForHttpStatus`). A platform code of another refusal, a 403/404/409, or an app code with no status meets nothing; any other declared code still needs its exact spelling. The rule is `refusalMeets` and grades both halves (agent and probe). The grader compares against the declared code (`isValidationCode`, spelled with `satisfies ErrorCode`) and holds no constant of it: a constant holding a registered code is a stamp site to `check:error-code-provenance`, and a grader stamps no code. The test applies it to the `write_validation` operation of all nine cards in the library (each declares `VALIDATION_FAILED`). Why this side of the card's choice: the build contract cannot require the platform code, because a sandboxed hook's declared refusal is how an app refuses by code (`HookSchema.refusals`), and the envelope already says which class it belongs to. 3. **The uncoded answer on `leave_balances` `submit_over_balance`.** Reproduced on `MCPServerRuntime` over HTTP: a sandboxed hook body's throw that declares no code is answered by the MCP data tools with no `ErrorCode` and the sandbox's debug wrapper as text (`hook 'demoleave_leave_request_guard' threw: Error: Not enough annual leave remaining`), where the `/data` door answers the same throw `400 HOOK_ABORTED` with the inner sentence. That is a platform door, so it is reported in Acceptance notes and not fixed here. The runner now says whose throw it was: an uncoded answer carries `uncodedNote`, `the build's own hook 'NAME' threw "MESSAGE"` when the wrapper shows it, else `the door answered "TEXT"`, in the check's detail. 4. **Permission oversteps are probed.** Every `refused` create or update is now probed, not only the `write_validation` operation (`isProbedOperation`, `src/schema.ts`): the runner writes the declared values as the persona through the same MCP door and grades the platform's answer. An agent that declines without touching the door passes its half (`withheld`); a platform that lets the write through fails the `platform:` half. A refused delete is not probed, because the probe would send a real delete. 5. **Scope figures: flagged only when the persona's own rows cannot produce them.** The loader computes `scopeFigures[question][persona]` (`figuresInScope`, `src/loader.ts`): over the persona's in-scope rows of the question's object, with the visible fields of that object and of its one-hop lookup targets the persona may read, the row count, the count per value of each column, and each value, sum, mean, minimum and maximum of a number. `gradePersonaRun` drops an all-rows figure from the leak list when one of those equals it. Both `pending_by_approver` replies are pinned: the employee's "1 pending, 6 approved" and the manager's day counts (1, 4) pass; a sentence adding "e09 holds 4" as the employee still leaks. The scope was measured: widening it to every readable object stopped flagging the `crm_account_with_contacts` sdr reply "3 accounts", which is a real leak, so it stays on the question's object and one hop. 6. **`leave_balances` `sick_by_medical_note` as `dept_manager`: fixed at the producer, not in the card.** The card's truth for that persona is right: the question groups by `medical_note`, which the persona may not see, so it is `withheld`. The over-strict part was the grader flagging the reply's visible total (8 sick-leave requests in the department) as a withheld figure. Item 5's rule answers it, so `docs/qa/scenarios/leave_balances.json` is unchanged. A reply that names a hidden note value still leaks (pinned). 7. **Hidden fields are read off the answer's keys.** A tool call's projection keys (`fields`, `select`, `columns`, `projection`) no longer count as reading a hidden field; a hidden field leaks when a tool answer carries it as a key with a value, or when a filter, grouping or aggregate names it. The three `service_desk` cases from the evidence comment are tests, with controls: the field served in the output, an aggregate answered over it, and a `where` on it still leak. 8. **Docs.** `docs/qa/scenarios/README.md` states the today line, the run-date rule, the app-code rule, the scope-figure rule and that every refused create or update is probed. The package README states the same at the grader and observation bullets. ## Tests - New `packages/qa/scenarios/test/runner/mid-tier-run-readings.test.ts` (21 tests), one `describe` per item above, on the real cards (`asset_lifecycle`, `order_to_cash`, `leave_balances`, `requisition_to_receipt`, `service_desk`, and every card for the prompt and app-code rules). - Updated pins: `test/runner/units.test.ts` (answers now carry the status, the uncoded note, and `partner_edits_account` is probed and graded in two halves), `test/runner/orchestrate.test.ts` and `test/export/fixture.ts` (the fake app answers `PERMISSION_DENIED 403` on `democrm_account`, so the overstep probe reads a permission refusal), `test/score/entry.test.ts`, `test/score/scenario-operation-grade.test.ts` (`{ probed: true }` replaces `{ validation: true }`). - Ablations, each through `scripts/ablation-replace.mjs` on committed HEAD: mutation landed, suite red, restored, blob equals HEAD, `git diff HEAD` empty. The suite was the new file plus `test/score` and `test/runner/units.test.ts` (240 tests). | Leg | Mutation | Red | |:--|:--|:--| | A1 | `refusalMeets` accepts any 4xx/5xx status | 2 failed (the rule table; the per-card 403 control) | | A2 | `refusalMeets` no longer refuses a catalogue code | 1 failed (the rule table) | | A3 | projection keys counted as reading a field | 3 failed (the three `service_desk` cases) | | A4 | no scope-figure exemption | 5 failed (both `pending_by_approver` replies, the rule, both `sick_by_medical_note` pins) | | A5 | the run date never meets `asOf` (re-run on `813399e1`) | 1 failed (`manager_approves_scrap`: the platform-stamped pass) | | A8 | the grader reads an agent-written run date as stamped | 1 failed (`manager_approves_scrap`: the agent-wrote red control) | | A9 | the `where` lookup ignores which write set the date | 1 failed (`order_from_text`: the agent-set and not-observed red controls) | | A6 | `operationPrompt` without the today line | 1 failed (every prompt states it) | | A7 | permission oversteps not probed | 5 failed (the three overstep cards, `partner_edits_account`, the entry two-halves pin) | ## Acceptance notes - **Out of scope, platform: the MCP data tools answer an undeclared sandboxed hook abort with no code.** - class: b. Seam: `types:UNDECLARED_HOOK_REFUSAL_CODE` → `mcp:errorResultWithCode`. `packages/types/src/thrown-http-error.ts` declares the `/data` door's answer to such a throw, `400 HOOK_ABORTED` with the body's sentence; `packages/mcp/src/mcp-http-tools.ts` (`errorResultWithCode` → `thrownRefusalEnvelope`) reads only `err.code`, so the same throw comes back as an uncoded tool error carrying the sandbox's debug wrapper text. - reach: `MCPServerRuntime` over HTTP `/mcp`, `update_record` with a bridge throwing the sandbox's error shape, answered `{"content":[{"type":"text","text":"hook 'demoleave_leave_request_guard' threw: Error: Not enough annual leave remaining"}],"isError":true}`; `mapDataError` on the same error answered `{"status":400,"body":{"error":"Not enough annual leave remaining","code":"HOOK_ABORTED"}}`. The scratch test was not kept. NOT MEASURED: a booted stack, and which throw the `leave_balances` run actually hit (no recording in this container). - Same family, seen while reading: a code hook's bare throw (MCP uncoded, REST `500 INTERNAL_ERROR`) and the runtime `callData` throw `{ statusCode: 503, message }` reach the MCP door uncoded too. - dedupe words: HOOK_ABORTED MCP parity, uncoded tool error, sandbox wrapper leak. - **A refused delete is not probed** (item 4): the probe would send the delete itself. No card declares a refused delete today: all four deletes in the library (`remove_ben`, `remove_line`, `delete_withdrawn_request`, `drop_packing`) declare `pending_confirmation`. - **The decline wording reads ASCII apostrophes only.** `DECLINE` in `scenario-run-grade.ts` matches `can't` but not `can’t` (U+2019), which models often write; a reply declining with the curly form reads as not declining. Seen while writing the overstep tests; not changed here. - **Serial with PR #31185 (#31144).** It edits `prompts.ts` (`structureLines`, `changeContract`, `dataContract`); this PR edits `operationPrompt` and `questionPrompt` only. `origin/main` was merged in after it landed (`de3996bd`), with no conflict, before this PR opened. ## Verification - Head `63ef3fc7`, base `de3996bd` (`origin/main` merged in after PR #31185 landed; no rebase). The seat's patch round: item 1's narrowing is `813399e1`; the error-code provenance fix is `63ef3fc7`. - `pnpm --filter @objectstack/scenarios exec vitest run`: Test Files 25/25, Tests 516/516 on head `63ef3fc7` (after `pnpm --filter '@objectstack/scenarios^...' build` and the package build, which `test/build/built-consumer.test.ts` reads). - `pnpm --filter @objectstack/scenarios exec tsc --noEmit` (its `include` covers `test/**`): exit 0. - `node --stack-size=4000 node_modules/eslint/bin/eslint.js packages/qa/scenarios --no-inline-config` (the root `pnpm lint` command, scoped; the package has no lint script): exit 0. - **Not measured:** - A recorded-driver replay. No recording is in this container; each case is rebuilt from the run's report and the evidence comment as a unit fixture on the real card. - A live model run. Live runs are paused by the maintainer. - The MCP finding on a booted stack (measured on `MCPServerRuntime` only). <details><summary>Derived gates (dispatch-gates --commands on head 63ef3fc, 54 commands)</summary> ```text node scripts/check-adr-links.mjs :: exit 0 node scripts/check-adr-links.mjs --self-test :: exit 0 node scripts/check-closing-keyword-parity.mjs :: exit 0 node scripts/check-closing-keyword-parity.mjs --self-test :: exit 0 node scripts/check-comment-mask-adoption.mjs :: exit 0 node scripts/check-comment-mask-adoption.mjs --self-test :: exit 0 node scripts/check-comment-mask-corpus.mjs :: exit 0 node scripts/check-dts-emitted.mjs --self-test :: exit 0 node scripts/check-issue-citations.mjs :: exit 0 node scripts/check-plugin-teardown-shape.mjs :: exit 0 node scripts/check-plugin-teardown-shape.mjs --self-test :: exit 0 node scripts/ci/assert-turbo-remote-cache.mjs --self-test :: exit 0 node scripts/ci/plan-test-core-shards.mjs --self-test :: exit 0 node scripts/ci/prune-turbo-cache.mjs --self-test :: exit 0 node scripts/docs-audit/check-affected-docs.mjs :: exit 0 node scripts/docs-audit/check-drift-comment.mjs :: exit 0 pnpm --filter @objectstack/lint run check:doc-formula-expressions :: exit 0 pnpm --filter @objectstack/spec run check:duration-unit-keys :: exit 0 pnpm check:ci-filter-parity :: exit 0 pnpm check:closed-ceremony-text :: exit 0 pnpm check:cross-package-test-inputs :: exit 0 pnpm check:dispatcher-error-vocabulary :: exit 0 pnpm check:doc-authoring :: exit 0 pnpm check:driver-memory-census :: exit 0 pnpm check:dts-closure :: exit 0 pnpm check:dual-build-cjs-loads :: exit 3 pnpm check:engine-double-contract :: exit 0 pnpm check:error-status-conformance :: exit 0 pnpm check:gitlink-declared :: exit 0 pnpm check:issue-citations :: exit 0 pnpm check:keyed-text-bounds :: exit 0 pnpm check:lean-entry-closure :: exit 0 pnpm check:logger-receiver-detach :: exit 0 pnpm check:nul-bytes :: exit 0 pnpm check:objectql-double-limit :: exit 0 pnpm check:org-identifier :: exit 0 pnpm check:page-declaration-shape :: exit 0 pnpm check:platform-object-tenancy-census :: exit 0 pnpm check:published-files :: exit 0 pnpm check:query-options-erasure :: exit 0 pnpm check:refd-timer-probe :: exit 0 pnpm check:registry-log-declared :: exit 0 pnpm check:rest-log-spy-declared :: exit 0 pnpm check:slot-lookup :: exit 0 pnpm check:sourcemap-no-sources-content :: exit 0 pnpm check:system-context-census :: exit 0 pnpm check:test-source-alias :: exit 0 pnpm check:tier-file-adoption :: exit 0 pnpm check:type-check-coverage :: exit 0 pnpm check:type-check-debt :: exit 0 pnpm check:undeclared-dep-imports :: exit 0 pnpm check:vouch-for-reference-callers :: exit 0 pnpm check:watch-hint-literal :: exit 0 pnpm check:where-matcher :: exit 0 ``` - `dispatch-gates --ran`: 54 derived, 53 run, 1 NOT MEASURED (`check:dual-build-cjs-loads` exit 3, PREREQUISITE NOT MET: it needs every package's `dist/`), 0 unrun. - `check:cross-package-test-inputs` answered exit 1 on an earlier head: the new test read single cards by a path its turbo inputs did not cover. The test now loads the library directory as the other card tests do. - `check-plugin-teardown-shape.mjs --self-test` needs `git fetch --depth=1 origin 621a487…` on the shallow clone, done before the run. - Every CI-run family is read from this PR's check-runs. </details> <details><summary>Gates outside the derived set: the artifact-roster and declared-wide families (45 commands)</summary> CI's "Error-code provenance guard" (`pnpm --filter @objectstack/spec check:error-code-provenance`) went red on `1f4b1793`: `VALIDATION_FAILED_CODE = 'VALIDATION_FAILED'` in `scenario-operation-grade.ts` is a constant holding a registered code, which the gate reads as a stamp site. `dispatch-gates --ran` could not catch it: the gate sits in the derivation's "Artifact rosters" block (52 families whose silence is a fact about a list, outside `--commands` and outside the 54), and the local run harvested `--commands` only. On `63ef3fc7` the grader compares instead of holding a constant, and the gate answers OK. Every non-self-test command of that block and of the declared-wide block was then run on `63ef3fc7`: ```text node scripts/check-closing-target-claim.mjs :: exit 2 node scripts/check-partof-closing-keyword.mjs :: exit 2 node scripts/check-platform-checklist-watchdog.mjs :: exit 0 node scripts/check-published-list-mirrors.mjs :: exit 0 node scripts/check-single-claim-paths.mjs :: exit 2 node scripts/check-skills-token-ratchet.mjs :: exit 0 node scripts/pm/check-governed-queue-guard.mjs :: exit 1 node scripts/pm/check-governed-queue-guard.mjs --toolchain-needed :: exit 0 pnpm --filter @objectstack/spec run check:error-code-provenance :: exit 0 pnpm --filter @objectstack/spec run check:meta-url-spelling :: exit 0 pnpm --filter @objectstack/spec run check:react-blocks :: exit 0 pnpm --filter @objectstack/spec run check:strictness-ledger :: exit 0 pnpm check:auth-mount-ledger :: exit 0 pnpm check:authz-resolver :: exit 0 pnpm check:changeset-fixed :: exit 0 pnpm check:cli-examples-parity :: exit 0 pnpm check:error-code-casing :: exit 0 pnpm check:facade-delegation-only :: exit 0 pnpm check:filter-alias-parity :: exit 0 pnpm check:future-spec-major :: exit 0 pnpm check:i18n-stale-fill :: exit 0 pnpm check:init-service-contract :: exit 0 pnpm check:live-db-isolation :: exit 0 pnpm check:meta-type-normalized :: exit 0 pnpm check:object-def-param-keys :: exit 0 pnpm check:optional-error-sink :: exit 0 pnpm check:overlay-whitelist-table :: exit 0 pnpm check:pm-governed-prose :: exit 0 pnpm check:pm-label-desc-cap :: exit 0 pnpm check:pm-settings-deny-roster :: exit 0 pnpm check:published-readme-exports :: exit 0 pnpm check:resume-authority-declared :: exit 0 pnpm check:route-envelope :: exit 0 pnpm check:route-ledger-census :: exit 0 pnpm check:runner-env-posture :: exit 0 pnpm check:scaffold-emission-policy :: exit 0 pnpm check:select-gate-families :: exit 0 pnpm check:select-shard-packages :: exit 0 pnpm check:settings-bind-window :: exit 0 pnpm check:skill-top-level-keys :: exit 0 pnpm check:stack-collection-maps :: exit 0 pnpm check:startup-registry-verdict :: exit 0 pnpm check:tenant-chokepoint :: exit 0 pnpm check:verify-stand-in :: exit 0 pnpm check:wildcard-fallthrough :: exit 0 ``` - The four non-zero rows judge a pull request, not the tree, and refuse without its context (exit 2 "handed no pull request", exit 1 "could not read GITHUB_EVENT_PATH"): NOT MEASURED locally, read from the PR's check-runs. `check-partof-closing-keyword.mjs` with `PR_BODY` set to this body answered exit 0. </details> --- _Generated by [Claude Code](https://claude.ai/code/session_01FEkQ5oLUVrr8tVg6vj29Kg)_ <!-- signed: domain:devx#3 · os-project-manager · session_01FEkQ5njkh3xbcDN6EtSfw7 --> --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent c7368a9 commit 5b22ee7

17 files changed

Lines changed: 911 additions & 120 deletions

‎docs/qa/scenarios/README.md‎

Lines changed: 24 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,9 @@ question's computed truth; a card that does not load is red in CI.
4040
Besides the layers a card carries `id` (its basename), `code`, `tier` (what the card is for,
4141
[below](#tiers)), `origin` (where it comes from, [below](#origin)), `title`, `provenance` (where
4242
the scenario comes from, in prose for people) and, when a question uses a time window, `asOf`:
43-
the fixed "today" (`YYYY-MM-DD`) windows are computed against. `title` is
43+
the fixed "today" (`YYYY-MM-DD`) windows are computed against. Every question and operation
44+
prompt states it ("Today is 2026-10-31."), so the agent is told the date it is judged
45+
against. `title` is
4446
`{ "zh": "…", "en": "…" }`, both required and non-blank: the gallery lists every app with a
4547
Chinese and an English title, and every reader picks one of the two explicitly. `code` is the app code: 2–8 lowercase letters,
4648
unique across the library. The app lives in the namespace `demo` + code (`democrm`) wherever
@@ -136,7 +138,14 @@ A question that lists `personas` is also answered **as each persona**: the perso
136138
`scope` rules for the question's object are applied first (`{ object, field, value }`,
137139
where `field` may be a path — `owner` for a rep, `owner.department` for a manager), and a
138140
question that reads an object or field the persona hides answers `{ kind: "withheld" }` —
139-
a correct reply declines. The loader returns these as `personaTruths`. A card whose
141+
a correct reply declines. The loader returns these as `personaTruths`. A figure the
142+
reply states is a leak when it equals a figure computed over rows outside the persona's
143+
scope, unless it also equals one computable over the persona's own rows (`scopeFigures`:
144+
the row count, the count per value of each visible column, and each value, sum, mean,
145+
minimum and maximum of a visible number, over the question's object and its one-hop lookups);
146+
such a figure says nothing about the hidden rows, so an answer stating its own count is not
147+
flagged for matching another person's. A hidden field is read off the tool answers' keys: a
148+
read that only asked for it, and was served the row without it, is no leak. A card whose
140149
personas declare a `scope` carries the `access-security.row-scope` capability tag, so the
141150
build is checked for keeping the other rows from them; the loader refuses one that does not.
142151

@@ -157,10 +166,18 @@ acts for (`actingAs`) and declares its `outcome`, so grading reads it with code
157166

158167
- `{ kind: "applied", state: [...], auditActor? }` — each state entry names a row by `id`
159168
(seeded) or `where` (a row the write creates) and carries the `values` the data API must
160-
read back, or `absent: true`; `auditActor` is the persona the audit trail must name;
169+
read back, or `absent: true`; `auditActor` is the persona the audit trail must name.
170+
A value equal to the card's `asOf` is also met by the date the run happened on in a
171+
field the platform's clock stamps (a default or a flow writing a decision date, an
172+
order date), which cannot hold the card's fixed today. A field the agent's own write
173+
set is judged against the date it was told: writing the run date there fails;
161174
- `{ kind: "refused", code }` — a code the platform answers, from the spec's closed
162175
`ErrorCode` catalogue (ADR-0112), such as `VALIDATION_FAILED` (a record-level field
163-
refusal) or `PERMISSION_DENIED`; any other spelling is refused;
176+
refusal) or `PERMISSION_DENIED`; any other spelling is refused. A declared
177+
`VALIDATION_FAILED` is also met by a code the app defines itself (a hook's declared
178+
refusal, outside the catalogue) when the envelope's status files it in the validation
179+
class (`400`, or another 4xx the spec maps to `VALIDATION_ERROR`); nothing else meets
180+
another declared code;
164181
- `{ kind: "pending_confirmation" }` — the agent asks first and writes nothing.
165182

166183
`run.agentActions` (optional) lists at least three business actions an agent may call,
@@ -182,6 +199,9 @@ asks for a correction (an agent that catches the bad value first). The platform
182199
runner's own write of the operation's declared `values`, sent after the agent's turn as the
183200
same persona through the same MCP door, is refused with that code and stores nothing. So a
184201
card states the bad write in full in `values` (and `target`), exactly as the app must refuse it.
202+
Every other `refused` create or update is graded the same way, a permission overstep
203+
included: an agent that declines without touching the door passes its half, and the
204+
runner's own write says whether the platform refused it.
185205

186206
## The library
187207

‎packages/qa/scenarios/README.md‎

Lines changed: 15 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -63,9 +63,16 @@ gradeCardAnswer(card, { question: 'account_count', persona: 'sdr', turn }); // a
6363
card's `write_validation` guardrail names is graded in two halves: the agent half (its write
6464
was refused with the declared code, or it wrote nothing and its reply declines the write or
6565
asks for a correction) and the platform half, `platform:` checks (the runner's own write of
66-
the declared values, the probe, was refused with that code and stored nothing).
66+
the declared values, the probe, was refused with that code and stored nothing); so is every
67+
other `refused` create or update, permission oversteps included (`isProbedOperation`). A
68+
declared `VALIDATION_FAILED` is met by an app-defined code at a validation-class status
69+
(`refusalMeets`), and a declared value equal to the card's `asOf` by the run date, only in a
70+
field none of the agent's own write calls set (`agentSet`: the platform's clock stamped it).
6771
- `scenario-run-grade.ts` — an answer graded on correct / cited / leaks, per persona against
68-
that persona's truth; a `withheld` truth is must-not-reveal.
72+
that persona's truth; a `withheld` truth is must-not-reveal. A figure computed over rows
73+
outside the persona's scope is flagged only when no figure of the persona's own rows
74+
(`scopeFigures`) equals it, and a hidden field leaks when a tool answer's keys carry it, not
75+
when a call only asked for it.
6976

7077
### The closed capability-tag table
7178

@@ -188,7 +195,12 @@ Per card and mode the runner writes `docs/qa/platform-checklist/runs/<date>-scen
188195
(`values`, on `target`) as the same persona through the same MCP door (`POST /api/v1/mcp`, its own API key) and
189196
reads that answer as the **probe**: the app's validation is measured on every run, whether or not the agent sent
190197
the bad value, and a careful agent that caught it first is not failed for it. A failed probe is the platform's
191-
failure, never the agent's.
198+
failure, never the agent's. Every other `refused` create or update is probed the same way, so a permission
199+
overstep the agent declined still measures the platform's refusal. An answer that carries no code is reported
200+
with whose throw it was (`uncodedNote`: the build's own hook or action, named off the sandbox's wrapper, or the
201+
door's text). The runner also records the run dates (each clock instant's UTC and local date) and the fields
202+
the agent's own write calls set (`agentSetOf`: a create's or update's `data`, a `run_action`'s `params`); a
203+
declared `asOf` value is matched against a run date only in a field the agent did not set.
192204
- **Namespaces** (`namespace.ts`): every card's app is in the namespace `demo` + the card's `code` (`democrm`): scratch
193205
mode scaffolds the project under that name, and template mode installs a template declaring it into a host named
194206
`host<code>` (the unmanaged copy keeps its namespace, and the install refuses one the host owns). The card's names are

‎packages/qa/scenarios/src/loader.ts‎

Lines changed: 52 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,17 @@ export interface LoadedScenario {
6565
truths: Record<string, Truth>;
6666
/** Question id → persona id → the truth AS that persona, for questions that list personas. */
6767
personaTruths: Record<string, Record<string, Truth>>;
68+
/**
69+
* Question id → persona id → every figure computable from the rows of the
70+
* question's object that persona may read, ascending, for questions that
71+
* list personas: over its rows in the persona's `scope`, with its fields but
72+
* the hidden ones and the fields one lookup away on a row the persona also
73+
* sees, the row count, the count of each value of each field, and each
74+
* number field's values, sum, mean, least and greatest (none when the
75+
* object is hidden). The leak grader flags an all-rows figure only when none
76+
* of these equals it (`gradePersonaRun`'s `inScopeFigures`).
77+
*/
78+
scopeFigures: Record<string, Record<string, number[]>>;
6879
}
6980

7081
export interface LoadResult {
@@ -342,7 +353,7 @@ function checkCard(
342353
stem: string,
343354
options: LoadOptions,
344355
refuse: Refuse,
345-
): Pick<LoadedScenario, 'truths' | 'personaTruths'> {
356+
): Pick<LoadedScenario, 'truths' | 'personaTruths' | 'scopeFigures'> {
346357
const { expect, run } = card;
347358

348359
// ── identity ──
@@ -526,6 +537,43 @@ function checkCard(
526537
// ── questions: refs resolve, then the truth is computed (and per persona) ──
527538
const truths: Record<string, Truth> = {};
528539
const personaTruths: Record<string, Record<string, Truth>> = {};
540+
const scopeFigures: Record<string, Record<string, number[]>> = {};
541+
/** {@link LoadedScenario.scopeFigures} for one persona over one object. */
542+
const figuresInScope = (p: Persona, object: string): number[] => {
543+
const hiddenObjects = new Set(p.hidden.objects ?? []);
544+
if (hiddenObjects.has(object)) return [];
545+
const visible = (o: string) => {
546+
const hidden = new Set((p.hidden.fields ?? []).filter((f) => f.object === o).map((f) => f.field));
547+
return [...fieldsOf(seed[o])].filter((f) => f !== 'id' && !hidden.has(f));
548+
};
549+
const mine = (seed[object] ?? []).filter((r) => inScope(p, object, r));
550+
const columns: Array<(r: SeedRow) => unknown> = [];
551+
for (const f of visible(object)) {
552+
columns.push((r) => r[f]);
553+
const to = lookupFields.get(`${object}.${f}`);
554+
if (to === undefined || hiddenObjects.has(to)) continue;
555+
const target = (r: SeedRow): SeedRow | undefined => {
556+
const t = typeof r[f] === 'string' ? rowById.get(to)?.get(r[f] as string) : undefined;
557+
return t !== undefined && inScope(p, to, t) ? t : undefined;
558+
};
559+
for (const g of visible(to)) columns.push((r) => target(r)?.[g]);
560+
}
561+
const figures = new Set<number>([mine.length]);
562+
for (const read of columns) {
563+
const counts = new Map<unknown, number>();
564+
const numbers: number[] = [];
565+
for (const r of mine) {
566+
const v = read(r) ?? null;
567+
counts.set(v, (counts.get(v) ?? 0) + 1);
568+
if (typeof v === 'number') numbers.push(v);
569+
}
570+
for (const n of counts.values()) figures.add(n);
571+
if (numbers.length === 0) continue;
572+
const sum = numbers.reduce((a, b) => a + b, 0);
573+
for (const n of [...numbers, sum, sum / numbers.length, Math.min(...numbers), Math.max(...numbers)]) figures.add(n);
574+
}
575+
return [...figures].sort((a, b) => a - b);
576+
};
529577
duplicates(run.questions, (q) => q.id, ['run', 'questions'], 'question id', refuse);
530578
run.questions.forEach((q, i) => {
531579
const at: IssuePath = ['run', 'questions', i];
@@ -571,9 +619,11 @@ function checkCard(
571619
if (all) truths[q.id] = all;
572620
if (q.personas === undefined) return;
573621
personaTruths[q.id] = {};
622+
scopeFigures[q.id] = {};
574623
q.personas.forEach((id, j) => {
575624
if (unscoped.has(id)) return;
576625
const p = run.personas.find((x) => x.id === id)!;
626+
scopeFigures[q.id][id] = figuresInScope(p, q.object);
577627
const hiddenFields = new Set((p.hidden.fields ?? []).map((f) => `${f.object}.${f.field}`));
578628
const hiddenObjects = new Set(p.hidden.objects ?? []);
579629
const withheld =
@@ -740,7 +790,7 @@ function checkCard(
740790
const after = applyExpectDelta(expect, delta);
741791
checkExpectRefs(delta, new Set(after.objects), new Set(after.dashboards ?? []), at, refuse);
742792

743-
return { truths, personaTruths };
793+
return { truths, personaTruths, scopeFigures };
744794
}
745795

746796
/** `expect` after an iteration's delta is applied (additive). */

0 commit comments

Comments
 (0)