Scenario runner: tell the persona agent the card's date, grade app-defined validation codes, probe permission oversteps, fix scope-figure false positives and an over-strict persona truth

epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

Filing-gate category: ① defects with measured reproductions. Found by the first complete mid-tier run, 2026-11-07, main 5f003744, scratch mode, --author-timeout 60, tier caps 300 turns, driver claude-code-cli 2.1.319 · oauth_token · API-billed: no. Every build finished: asset_lifecycle 96 turns / 23 min; requisition_to_receipt 71 turns / 19 min; leave_balances 110 turns / 22 min; order_to_cash 149 turns / 28 min. Every seed loaded in full (559 / 637 / 617 / 619 rows). Routing, verbatim: 「能交给车队开发的你不要自己开发」. Epic: objectstack-ai/cloud#3600.

Card	Build	问数	Operations
asset_lifecycle	136/140	11/15	2/6
requisition_to_receipt	133/140	11/13	3/6
leave_balances	109/114	11/15	2/6
order_to_cash	143/147	11/12	1/6

Most of what remains is the runner's and the cards', as below. The epic seat holds the recordings and run records.

Card 2 of 2: Run grading and the operation contract

"Today" (affects at least 5 operations). Expected states pin the card's asOf (2026-10-31), but the persona agent stamps the real date:
- asset_lifecycle manager_approves_scrap: decided_on read "2026-11-07", declared "2026-10-31";
- order_to_cash approve_discount: approved_on, the same;
- order_from_text: the row with order_date="2026-10-31" was not found;
- asset_lifecycle receive_and_issue_from_text: checked_out_on, the same.
One reply even notes "The app's own clock says 2026-11-07". Ask: the operation and question prompts state "Today is ". Expected states that pin a date are judged against the date the agent was told. If the platform clock itself must match (defaults, flows that stamp NOW()), boot the app with a fixed clock, or compare such fields as "the run date".
App-defined refusal codes. The runner's direct probe got the build's own rule codes: ASSET_NOT_IN_STOCK (asset_lifecycle check_out_asset_in_repair), OVER_RECEIPT (requisition_to_receipt receive_more_than_ordered) and NOT_REQUESTER (leave_balances delete_withdrawn_request). The cards declare VALIDATION_FAILED. Ask: decide whether an app-defined validation code satisfies a validation operation (a closed rule: a refusal of the validation class, read from the platform's error envelope), or whether the build contract requires the platform code. State it, and apply it to every card.
An error with no code. leave_balances submit_over_balance: the probe got "(tool error without a code)". Find which door answered without an ErrorCode. If a platform door (MCP, data API, a flow or hook throw surfacing through MCP) emits an uncoded error, report it as a platform defect with the reproduction. If it is the build's own throw, the probe should say so.
An overstep the agent declines without touching the door (asset_lifecycle employee_approves_own_scrap, requisition_to_receipt manager_approves_finance_level, leave_balances raise_own_entitlement): graded (no write attempted). Extend the runner's direct probe (already used for validation operations) to permission oversteps: write as the persona and grade the platform's answer, so the agent's good behaviour does not hide the platform's.
Scope-figure false positives (the leak detector's "figure outside the persona's scope"; Scenario grading: an overstep refused on read, an agent that pre-validates, and a plain decline all grade as failures; cards with persona scope lack the row-scope tag #30883 fixed the hidden-value half only). In leave_balances pending_by_approver, employee answered "1 pending step" correctly and was flagged for "6", its own visible approved count. dept_manager was flagged for "1" and "4", day counts in its own table. Ask: flag a scope figure only when it equals a figure computable solely from out-of-scope rows and not from in-scope rows, and pin both replies.
An over-strict persona truth (leave_balances sick_by_medical_note, dept_manager): the platform correctly refused grouping by the hidden medical_note. The agent then gave the total sick-leave count (8), which reads only visible fields, and the card withholds it. Fix the card's persona truth.

Reader: the domain:devx lane seat, from pm:queue.

triage#1 · os-zhuang · session_01AavokzJ5DndAwitDXvKy4U

Path: none

Triage: audit · p1 kept · `domain:devx` · 2026-11-07T15:01Z

⛔ Not a claim.

An epic objectstack-ai/cloud#3600 card (scenario runner and cards), from the first complete mid-tier run or the maintainer's 「两种都要」. Routing per 「能交给车队开发的你不要自己开发」.
p1, domain:devx, area:devpath and pm:queue kept.

Generated by Claude Code

epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

More evidence from service_desk (same run, main 5f003744: build 85 turns / 18 min, seed 496/496, 问数 10/16, operations 3/6):

The "requested-but-stripped hidden field" false positive persists after Scenario grading: an overstep refused on read, an agent that pre-validates, and a plain decline all grade as failures; cards with persona scope lack the row-scope tag #30883 (PR fix(scenarios): grade read-path refusals, plain declines, pre-validating agents and RLS row scope as the right answers they are #30972). contract_value_by_tier as support_agent and as portal_customer, and minutes_by_group as portal_customer, are each flagged "query_records answered over hidden field(s) …". Yet the recorded outputs carry no contract_value / minutes key: the platform stripped it, and the matching aggregate_records answered 403 "Field read denied". The agent then declined correctly. Add these three as tests. The leak check must read the output keys, not the requested fields.
"Today", again: log_ticket_from_call: no row with opened_on="2026-10-31"; reply_then_resolve: resolved_on read "2026-11-07", declared "2026-10-31". This is item 1 of this card.
Scope-figure false positive, again: csat_by_group was flagged for "2" as support_agent and as portal_customer (item 5).

Live runs are paused by the maintainer, so verify with the recorded driver; the epic seat holds the recording.

domain:devx#3 · os-project-manager · session_01FEkQ5njkh3xbcDN6EtSfw7

Claim: PM loop round 1
Session: session_01FEkQ5njkh3xbcDN6EtSfw7
Account: os-project-manager
Branch: claude/project-thread-dbw7av (a project card thread, session_01FEkQ5oLUVrr8tVg6vj29Kg; the thread can push only its harness branch, a measured platform exception to claude/issue-<n>-<slug>)
Worktree: objectstack-issue-31145
Domain: domain:devx
Seat: domain:devx#3
File surface: packages/qa/scenarios/src/** (operation and question prompts' date line in runner/prompts.ts; the direct probe in runner/orchestrate.ts / runner/observe.ts; the leak and scope-figure checks under src/score/** or runner/observe.ts), their tests, and docs/qa/scenarios/leave_balances.json (persona truth) plus any card the validation-code rule (item 2) touches. A platform door found emitting an uncoded error is reported, ⛔ not fixed here.
Container & model: M, mode:thread, model: default — dispatch-gates --tier: no path-derived mandate, effort: high
Clause-②: no
Metadata-protocol: none
Responsibility: n/a — private QA runner and cards; item 3 may surface a platform defect, which the seat files
Thread-read: 6708802169
Premise: holds. Measured by the first complete mid-tier run (main 5f003744) and the epic seat's service_desk evidence 6708802169; the dev re-measures each item on current main.
Serial constraints cleared: PR #31185 (#31144) edits runner/prompts.ts and leave_balances.json; the order has the dev do the score/probe work first and merge origin/main after #31185 lands, before touching those two files. #31147 (in flight) touches runner/cli.ts / project.ts / export/**: disjoint at file level.

Order: pm-orders/devx/order-31145.md (project files), handed to the card thread in its opening message.

domain:devx#3 · os-project-manager · session_01FEkQ5oLUVrr8tVg6vj29Kg
os-dev-report
{
"issue": 31145,
"status": "done",
"branch": "claude/project-thread-dbw7av",
"pr": "#31214",
"session": "session_01FEkQ5oLUVrr8tVg6vj29Kg",
"premise_still_valid": true,
"summary": "All seven items land in the private scenario runner. (1) operationPrompt now states the card's today like questionPrompt (one helper, todayOf); a declared value equal to asOf is also met by the run date the platform stamped (isRunDate, clockDates, postStateRows where-matching). (2) Decided inside the ruling: a declared VALIDATION_FAILED is met by an app-defined code (not in ErrorCode) at a 4xx status the spec files under VALIDATION_ERROR (refusalMeets), applied to both halves and pinned on all nine cards. (3) The uncoded answer is the MCP data door's, reproduced on MCPServerRuntime: reported as an out-of-scope platform finding, and the runner's check detail now names whose throw it was (uncodedNote). (4) Every refused create or update is probed (isProbedOperation), so a declined permission overstep is graded on the platform's answer. (5) An all-rows figure is a leak only when no figure of the persona's own rows (loader scopeFigures over the question object and one-hop lookups) equals it; both pending_by_approver replies pinned. (6) leave_balances sick_by_medical_note: the card's withheld truth is right; the over-strict part was the grader flagging the visible total 8, fixed at the producer by item 5, so leave_balances.json is unchanged. (7) Projection keys (fields/select/columns/projection) no longer count as reading a hidden field; output keys do; the three service_desk cases are tests with controls.",
"tests": "Head 1f4b179: pnpm --filter @objectstack/scenarios exec vitest run: Test Files 25/25, Tests 516/516 (after the scenarios dependency closure build). tsc --noEmit exit 0. New test/runner/mid-tier-run-readings.test.ts: 21/21. Ablations through scripts/ablation-replace.mjs on committed HEAD, each restored with blob == HEAD and git diff HEAD empty, suite = new file + test/score + test/runner/units.test.ts (240 tests): A1 refusalMeets any 4xx/5xx status: 2 failed; A2 catalogue codes accepted: 1 failed; A3 projection keys read as fields: 3 failed; A4 no scope-figure exemption: 5 failed; A5 run date never meets asOf: 1 failed; A6 operationPrompt without the today line: 1 failed; A7 permission oversteps not probed: 5 failed. No dist preflight: the package's tests import src.",
"gates": "dispatch-gates --commands on head 1f4b179: 54 commands, each run with its exit code captured before any pipe; 52 exit 0; check:dual-build-cjs-loads exit 3 (PREREQUISITE NOT MET, every package's dist needed): NOT MEASURED; check:dts-closure exit 1 naming eight packages outside the diff whose local dist held JS without declarations (driver-memory, mcp, plugin-hono-server, service-cache, service-cluster, service-i18n, service-push, trigger-api); after rebuilding those eight it answered exit 0 (145/145). dispatch-gates --ran: 54 derived, 53 run, 1 NOT-MEASURED, 0 UNRUN. check:cross-package-test-inputs was exit 1 on an earlier head (the new test read single card paths); fixed by loading the library directory, exit 0 on head. Scoped lint: node --stack-size=4000 node_modules/eslint/bin/eslint.js packages/qa/scenarios --no-inline-config exit 0. CI families: in_progress on the PR's check-runs at report time.",
"dogfood_battery": "n/a — no ! in the PR title and no (narrowing) arm",
"mcp_calls": "6 — all reads: issue_read get and get_comments (#31145); pull_request_read get and get_files (#31185); pull_request_read get (#31185 merge state); pull_request_read get (#31214 read-back).",
"api_writes": "2 — relay pr_create (repository_dispatch fw-20261107T180105Z-593963, run 42399141743, read-back identical, labels skip-changeset, assignees none) and this os-dev-report comment via post-stamped. git push is not a REST write.",
"open_questions": [],
"security_findings": "none",
"out_of_scope_findings": [
"class: b · Seam: types:UNDECLARED_HOOK_REFUSAL_CODE → mcp:errorResultWithCode · reach: MCPServerRuntime HTTP /mcp update_record with a bridge throwing the sandbox's error shape answered {content:[{type:text,text:"hook 'demoleave_leave_request_guard' threw: Error: Not enough annual leave remaining"}],isError:true}, no ErrorCode and the sandbox's debug wrapper as text; mapDataError on the same error answers 400 HOOK_ABORTED with the inner sentence (packages/types/src/thrown-http-error.ts). packages/mcp/src/mcp-http-tools.ts errorResultWithCode → thrownRefusalEnvelope reads only err.code. Same family: a code hook's bare throw (MCP uncoded vs REST 500 INTERNAL_ERROR) and the runtime callData throw {statusCode:503,message}. NOT MEASURED: a booted stack; which throw the leave_balances run hit (no recording). · dedupe words: HOOK_ABORTED MCP parity, uncoded tool error, sandbox wrapper leak",
"carrier: none · noted, not filed: the reply-decline wording (DECLINE in packages/qa/scenarios/src/score/scenario-run-grade.ts) matches can't but not the curly can’t (U+2019) that models often write, so such a reply reads as not declining; seen while writing the overstep tests, not changed (PR Acceptance notes)."
]
}

Generated by Claude Code

domain:devx#3 · os-project-manager · session_01FEkQ5oLUVrr8tVg6vj29Kg
os-dev-report
{
"issue": 31145,
"status": "done",
"branch": "claude/project-thread-dbw7av",
"pr": "#31214",
"session": "session_01FEkQ5oLUVrr8tVg6vj29Kg",
"premise_still_valid": true,
"summary": "Patch round on the seat's review of 1f4b179 (item 1 and a CI red), head 63ef3fc. (1) The run date meets a declared asOf only in a field none of the agent's own write calls on that object set, so the platform's clock stamped it: the observation carries agentSet (agentSetOf / agentSetFields in observe.ts: a create's or update's data, a run_action's params), runDateReading in the grader answers stamped (pass), agent (fail: 'the agent wrote the run date where the prompt told it the card's today') or unknown (fail), and postStateRows' where match applies the same rule. An agent that ignores 'Today is 2026-10-31' and writes the real date now fails. (2) CI 'Error-code provenance guard' was red on 1f4b179 and 813399e: VALIDATION_FAILED_CODE = 'VALIDATION_FAILED' is a constant holding a registered code, a stamp site to the gate; the grader now compares (isValidationCode, spelled with satisfies ErrorCode) and holds no constant, and the gate answers OK. dispatch-gates --ran missed it because the gate is in the derivation's 'Artifact rosters' block, outside --commands and the 54 derived families; this round ran every non-self-test command of that block and of the declared-wide block too (45). Items 2-7 unchanged.",
"tests": "Head 63ef3fc: pnpm --filter @objectstack/scenarios exec vitest run: Test Files 25/25, Tests 516/516 (after the dependency closure build and the package build, which test/build/built-consumer.test.ts reads). tsc --noEmit exit 0. New pins in test/runner/mid-tier-run-readings.test.ts: manager_approves_scrap platform-stamped pass plus agent-wrote, other-date, no-run-date and not-observed red controls (via observeOperation from a turn); order_from_text where lookup found when stamped, not found when the agent's create set order_date or agentSet is absent (via postStateRows and observeOperation). Ablations through scripts/ablation-replace.mjs, each restored with blob == HEAD and git diff HEAD empty, suite = new file + test/score + test/runner/units.test.ts (240): A5 run date never meets asOf: 1 failed (stamped pass); A8 agent-written run date read as stamped: 1 failed (agent-wrote control); A9 where lookup ignores which write set the date: 1 failed (order_from_text). Earlier legs A1-A4, A6, A7 stand (lines unchanged).",
"gates": "dispatch-gates --commands on head 63ef3fc: 54 commands, exit codes captured before any pipe: 53 exit 0, check:dual-build-cjs-loads exit 3 (PREREQUISITE NOT MET): NOT MEASURED. dispatch-gates --ran: 54 derived, 53 run, 1 NOT-MEASURED, 0 UNRUN. Outside the derived set, every non-self-test command of the Artifact-rosters and declared-wide blocks (45) on 63ef3fc: 41 exit 0, including pnpm --filter @objectstack/spec run check:error-code-provenance; check-closing-target-claim exit 2, check-partof-closing-keyword exit 2, check-single-claim-paths exit 2, check-governed-queue-guard exit 1 all refuse without a pull request context: NOT MEASURED locally (check-partof-closing-keyword with PR_BODY set to the new body: exit 0). Scoped lint exit 0. CI: in_progress on 63ef3fc at report time.",
"dogfood_battery": "n/a — no ! in the PR title and no (narrowing) arm",
"mcp_calls": "6 — unchanged, all reads (no MCP call this round).",
"api_writes": "3 — relay pr_create (fw-20261107T180105Z-593963, run 42399141743), the first os-dev-report comment (6711051465), and this superseding report. git push is not a REST write. PR body NOT patched (os-dev.md: the dev writes the body only at pr_create); the seat-ready replacement body is /mnt/project-files/pm-orders/devx/31145-pr-body.md (item 1, item 2 code comparison, A5/A8/A9 rows, Verification head 63ef3fc, the gates blocks).",
"open_questions": [],
"security_findings": "none",
"out_of_scope_findings": [
"class: b · Seam: types:UNDECLARED_HOOK_REFUSAL_CODE → mcp:errorResultWithCode · reach: MCPServerRuntime HTTP /mcp update_record with a bridge throwing the sandbox's error shape answered {content:[{type:text,text:"hook 'demoleave_leave_request_guard' threw: Error: Not enough annual leave remaining"}],isError:true}, no ErrorCode and the sandbox's debug wrapper as text; mapDataError on the same error answers 400 HOOK_ABORTED with the inner sentence (packages/types/src/thrown-http-error.ts). packages/mcp/src/mcp-http-tools.ts errorResultWithCode → thrownRefusalEnvelope reads only err.code. Same family: a code hook's bare throw (MCP uncoded vs REST 500 INTERNAL_ERROR) and the runtime callData throw {statusCode:503,message}. NOT MEASURED: a booted stack; which throw the leave_balances run hit (no recording). · dedupe words: HOOK_ABORTED MCP parity, uncoded tool error, sandbox wrapper leak · carrier: the devx seat, filing it (its message on this round)",
"carrier: none · noted, not filed: the reply-decline wording (DECLINE in packages/qa/scenarios/src/score/scenario-run-grade.ts) matches can't but not the curly can’t (U+2019) that models often write, so such a reply reads as not declining; seen while writing the overstep tests, not changed (PR Acceptance notes)."
]
}

Generated by Claude Code

domain:devx#3 · os-project-manager · session_01FEkQ5njkh3xbcDN6EtSfw7

Landed 2026-11-07T19:23Z · PR #31214 merged through the merge queue at 2026-11-07T19:22Z, as 5b22ee759364cb675ff5d4281b797c3684323f69.

Reading	Value
PR #31214	`merged: true`, merge commit `5b22ee7593`, head reviewed `63ef3fc7da` (ACCEPT on the PR, round 2)
#31145	closed `completed` by `Fixes #31145`

State change: pm:dispatched removed (card closed). Out of scope: #31218 filed (MCP data tools answer an undeclared sandboxed hook abort uncoded); the --commands roster-block miss went to #30763 (in flight). Acceptance notes: DECLINE misses the curly can’t; a run_action param named unlike the field it sets reads as platform-stamped.

Navigation Menu

Repository navigation

Scenario runner: tell the persona agent the card's date, grade app-defined validation codes, probe permission oversteps, fix scope-figure false positives and an over-strict persona truth #31145

Card 2 of 2: Run grading and the operation contract

Triage: audit · p1 kept · `domain:devx` · 2026-11-07T15:01Z

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Scenario runner: tell the persona agent the card's date, grade app-defined validation codes, probe permission oversteps, fix scope-figure false positives and an over-strict persona truth #31145

Description

Card 2 of 2: Run grading and the operation contract

Activity

objectstack-fleet commented on Nov 7, 2026

Triage: audit · p1 kept · domain:devx · 2026-11-07T15:01Z

objectstack-fleet commented on Nov 7, 2026

objectstack-fleet commented on Nov 7, 2026

objectstack-fleet commented on Nov 7, 2026

objectstack-fleet commented on Nov 7, 2026

objectstack-fleet commented on Nov 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Triage: audit · p1 kept · `domain:devx` · 2026-11-07T15:01Z