Repository navigation
Commit d8b2295
fix(scenarios): the seed loads a user-handle field with a user id, and both author loops run verify (#30903)
<!-- claude-projects-attribution: {"github_login":"marchtian"} -->
_Requested by **田騏** · [project
thread](https://claude.ai/code/project/chan_01VmuUhXU9SSUzqJYyXSmYSi?thread=cmsg_01VmuUhXU9SSUzqJYyXSmYSiNjyvCdXYygFqntMNR7S2dW)_
Fixes #30882
Clause-②: no
Metadata-protocol: none
## 维护者速读
- **改了什么**:场景跑分器灌种子数据时,把"负责人"一类字段里的人员代号(如 `u_sam`)换成该角色真实用户的 id(字段本身和
`owner_id`
都换);没有对应角色的代号(`u_tia`)记在管理员名下,提示词照此写明。种子有被拒的行时,依赖这些行的问数和写操作记为「未测」而非失败,计分卡显示
`⚠ seed: N refused`。构建与迭代两轮的提示词也让 AI 自己跑一遍跑分时用的同一条 `verify` 命令。
- **为什么改**:第一次真跑时 8 条种子只进了 1
条,后面的问数和写操作量的是跑分器自己的缺陷,不是被测应用,计分卡却把它们算成失败;AI 也从未被告知要过
`verify`,于是构建"降级"并不说明它修不好。
- **风险与回滚**:只动未发布的私有包 `@objectstack/scenarios`(QA 工具),不影响产品运行时;回滚即还原本
PR。其他卡片中指向记录或选项的范围字段照旧原样发送。
- **先看**:`packages/qa/scenarios/src/runner/orchestrate.ts` 的种子循环与
`seedDependence`;`packages/qa/scenarios/src/runner/scorecard.ts`
的未测计数;`packages/qa/scenarios/src/runner/prompts.ts` 的 `userHandleFields`
与 Row ownership 文案。
- **两分钟验证**:`pnpm --filter @objectstack/scenarios exec vitest run
test/runner`,预期全部通过,其中"loads all 8 rows"断言种子 8/8 入库,"a refused seed
row"两条断言 `⚠ seed` 警告与未测标记。
- **需要你做的**:无,席位落地。
## What changes
**Defect 1 — the seed lost every owner-scoped row.** The seed loop set
`owner_id` for a row whose persona-scope field named a persona's handle,
but sent the field itself as the handle text (`owner: "u_sam"`). A
faithful build models that field as a user lookup and refuses the
handle; a handle no persona holds (`u_tia`) had no user at all, and
every contact under a refused account failed as a cascade.
- `prompts.ts` `userHandleFields(card)`: the direct scope fields whose
seed values are user handles — a field a persona's direct `scope` rule
names that the card declares as neither a lookup nor a select, and that
is not `id`. Across the nine cards in `docs/qa/scenarios/` that is
exactly `account.owner` (crm), `invoice.owner` (invoice) and
`task.assignee` (project). Every other direct scope field names a seeded
row (`asset.custodian` = `e06`, `department.id`) or an option
(`ticket_comment.visibility` = `public`) and is still sent verbatim —
replacing those values would have broken the lookups on six other cards.
- `orchestrate.ts` seed loop: a user-handle field now carries the
persona's user id in the field itself and in `owner_id`. `owner_id`
stays mapped: it is the platform's canonical, reassignable record owner
(`packages/objectql/src/registry.ts`), the column owner-based sharing
reads. A handle no persona holds is loaded as the admin user (field and
`owner_id`), and the run record lists it under `seed.handlesAsAdmin`.
- `prompts.ts` `dataContract`: the Row ownership paragraph now states
the contract the seed keeps. The handle field and `owner_id` both get
the user's id: the persona's user for a held handle, and the admin the
runner signs in as for an unheld one. For crm it reads: "In
`democrm_account.owner` the value is a handle that stands for a user,
and the handle text is never stored: the runner loads that field, and
the row's owner (`owner_id`), with that user's id (`u_sam` → the `sdr`
user; `u_tia`, which no person below holds → the admin user the runner
signs in as). Make that field accept a user id; a rule on it matches
that user." This is the card's "state another contract in the build
prompt" branch.
**Defect 1, ask 3 — a refused seed row makes the Run half not measured,
never a fail.**
- `orchestrate.ts` `seedDependence(card, refused)` takes out the refused
rows and, through the card's declared lookups (`expect.lookups`), every
seed row under one, transitively. A contact of a refused account is out
whether or not the app refused it too.
- A question is not measured when its `object` holds a row that is out,
because its truth was computed over rows the app does not hold. An
operation is not measured when its `target`, a lookup value it writes,
or a post-state row (`id`, or a lookup value in its `values`) names a
row that is out.
- `runHalf` does not ask those turns. Each is recorded with
`notMeasured` and the reason, for example `seed: needs democrm_contact
ct2, a child of democrm_account ac1, which the app refused`. Everything
else is asked and graded as before.
- `scorecard.ts` adds a Seed row and an Iterate · Seed row, for example
`7/8 ⚠ seed: 1 refused`. The ask and operation tallies leave
not-measured turns out of both passed and total, and their cells name
the count, for example `4/4 · 5 not measured`. A half with nothing
measured stays `not-run` in the record, never `fail`.
**Defect 2 — the author loop never ran verify.** `verifyArgs(card)` is
`['verify', ...verifyFlags(card)]`. `buildHalf` runs exactly that, and
the build prompt and the iterate prompt both name it next to `validate`
and `build`: for the crm card, `npx objectstack validate`, `npx
objectstack build` and `npx objectstack verify --reference-shapes`,
"until all three pass". The template-mode iterate prompt is unchanged:
that mode edits the installed copy through the metadata API and runs no
`verify`.
## Tests
- `test/runner/orchestrate.test.ts`, new top-level describe `runCard — a
refused seed row makes what reads it not measured, never a fail`, with
`RefusingApp`, a double that refuses the seed rows it is told to:
- `shows ⚠ seed: 1 refused, leaves the questions on that object unasked
and unscored, and grades the rest as usual`. It refuses ct4: the
scorecard reads `| Seed | 7/8 ⚠ seed: 1 refused |`; the five contact
question turns are not measured and were not asked; the four account
turns are graded (4/4); the five operations are graded (5/5); and the
record's run-ask is `pass` with notes `4/4 · 5 not measured`.
- `an operation whose target hangs off a refused row is not run; a half
with nothing measured is not-run, not fail`. It refuses all three
accounts: only `contact_bad_email` runs, all 9 question turns are out,
run-ask is `not-run` and run-operations is `pass`.
- The clean run now also asserts `| Seed | 8/8 | — |`, with no warning.
- `test/runner/units.test.ts` `a refused seed row takes out its children
and what the card names on them, and nothing else` pins the rule on ac1:
its target, a child, and the account an operation writes are out;
`add_contact` and `contact_bad_email` are not.
- `test/runner/orchestrate.test.ts`, new top-level describe `runCard —
the seed against a build whose owner field is a user lookup`. It uses
`UserLookupApp`, an app double that refuses an `owner`/`owner_id` naming
no user and a contact whose `account` names no loaded account, as the
first live run's app did.
- `the double refuses the handle the old seed sent, so it can fail`.
- `loads all 8 rows: the owner field carries the persona user id, and a
handle no persona holds is the admin`: `seed` equals `{ inserted: 8,
refused: [], handlesAsAdmin: ['u_tia'] }`; ac1 and ac3 carry the sdr
user in `owner` and `owner_id`, ac2 the admin; all five contacts load;
the five operations pass.
- The existing ownership test now expects ac2's `owner_id` to be the
admin (it was undefined).
- `test/runner/units.test.ts`:
- `both authoring loops name the verify invocation the runner scores
with, next to validate and build`.
- `a user-handle field is a scope field the card declares as neither
lookup nor select; the contract says it is loaded with a user id`, which
also pins that `service_desk` has none.
- The regression test is built from the card in the repo
(`crm_account_with_contacts`), not from the first run's recording, which
the epic seat holds. No live model was run.
## Acceptance notes
- **Non-persona handles become the admin, not a provisioned user.** The
seat accepted this as the card's "state another contract in the build
prompt" branch. The admin is a user the runner already holds, it grants
no new permission, and it keeps all 8 rows loadable with no new HTTP
path. The cost: two unheld handles on one card would both map to the
admin, so a truth grouped by owner could not tell them apart. No card
today has one.
- **The not-measured rule is object-level for questions.** A question is
out when any row of its own object is out, even if its filter would not
have read that row. That is the reading the card states ("depends on a
refused row"), and it never scores a truth the app could not have
answered. The cost is that one refused contact takes out every contact
question.
- **Not-measured turns are not asked.** That saves model turns, and a
grade against rows the app lacks would mean nothing.
- **The package README is not updated.** It does not describe the Seed
row or the not-measured marks, because #30846 has the README in flight.
- **Out of scope, seen while reading other cards.** `personaHandles`
treats every direct scope string value as a handle, so the seed loop's
`owner_id` mapping (unchanged here) also fires on fields that name rows
or options. On `service_desk`, 237 public `ticket_comment` rows are
owned by the portal_customer user (`visibility = public`), and 15
tickets name two personas, so the last field read wins `owner_id`. This
is reported to the seat, not fixed here.
## Verification
<details><summary>Package tests and typecheck (head
b1f0628)</summary>
- `pnpm --filter @objectstack/scenarios test` :: exit 0 — Test Files 18
passed (18), Tests 245 passed (245)
- `pnpm --filter @objectstack/scenarios typecheck` :: exit 0 (the
tsconfig includes `test/**/*`)
</details>
<details><summary>Ablations (each restored, blob equal to HEAD, git diff
HEAD empty)</summary>
- Deleting `body[k] = userId;` in the seed loop: the 8-row pin goes red.
`inserted` is not 8 and `refused` is not empty: the double refuses the
handle.
- Replacing the verify-naming checks line with the old
validate-and-build text: the verify-invocation pin goes red.
- Feeding `seedDependence` an empty refused list in `runHalf`: both
refused-seed run tests go red.
- Deleting the `⚠ seed: N refused` suffix in `seedCell`: both
refused-seed run tests go red. The first attempt used an empty
`--replacement`, which the tool refused before running anything; it was
redone with `--delete`.
</details>
<details><summary>Derived gates on the final diff: 48 of 48 run, every
one exit 0 (head b1f0628, merge base 98f640d, origin/main not
moved since round 1)</summary>
- `node scripts/pm/dispatch-gates.mjs --commands --repo
objectstack-ai/objectstack` derived 48 commands for the 5 changed paths.
- Round 2: all 48 exited 0 (the round-1 `pnpm build` output is still in
place for `check:dual-build-cjs-loads`).
- `dispatch-gates --ran` with the exit-coded record: "48 derived
famil(ies) accounted for — 48 run, 0 NOT-MEASURED (a DERIVED zero — all
48 recorded an exit code and none of them is 3)".
- Changeset: none. `@objectstack/scenarios` is `private: true`, so
nothing publishes; this PR carries `skip-changeset`, as this package's
recent PRs (#30837, #30801) did.
- Not run locally, left to CI: the repo-wide `pnpm test`, `pnpm lint`
and the CI-only families the derivation lists outside its 48.
</details>
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- signed: domain:devx#2 · marchtian ·
session_017ihL2ebbZkr2QAEG1x7UC8 -->
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 45a085a commit d8b2295
5 files changed
Lines changed: 380 additions & 36 deletions
File tree
- packages/qa/scenarios
- src/runner
- test/runner
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
41 | 41 | | |
42 | 42 | | |
43 | 43 | | |
44 | | - | |
| 44 | + | |
45 | 45 | | |
46 | 46 | | |
47 | 47 | | |
| |||
97 | 97 | | |
98 | 98 | | |
99 | 99 | | |
| 100 | + | |
| 101 | + | |
100 | 102 | | |
101 | 103 | | |
102 | 104 | | |
| |||
105 | 107 | | |
106 | 108 | | |
107 | 109 | | |
| 110 | + | |
| 111 | + | |
108 | 112 | | |
109 | 113 | | |
110 | 114 | | |
| |||
113 | 117 | | |
114 | 118 | | |
115 | 119 | | |
116 | | - | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
117 | 126 | | |
118 | 127 | | |
119 | 128 | | |
| |||
236 | 245 | | |
237 | 246 | | |
238 | 247 | | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
239 | 320 | | |
240 | 321 | | |
241 | 322 | | |
| |||
276 | 357 | | |
277 | 358 | | |
278 | 359 | | |
279 | | - | |
| 360 | + | |
280 | 361 | | |
281 | 362 | | |
282 | 363 | | |
| |||
311 | 392 | | |
312 | 393 | | |
313 | 394 | | |
314 | | - | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
315 | 403 | | |
316 | 404 | | |
| 405 | + | |
317 | 406 | | |
318 | 407 | | |
319 | 408 | | |
320 | 409 | | |
321 | | - | |
| 410 | + | |
| 411 | + | |
322 | 412 | | |
323 | 413 | | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
324 | 420 | | |
325 | 421 | | |
326 | 422 | | |
| |||
329 | 425 | | |
330 | 426 | | |
331 | 427 | | |
| 428 | + | |
| 429 | + | |
332 | 430 | | |
333 | 431 | | |
| 432 | + | |
334 | 433 | | |
335 | 434 | | |
336 | 435 | | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
337 | 440 | | |
338 | 441 | | |
339 | 442 | | |
| |||
352 | 455 | | |
353 | 456 | | |
354 | 457 | | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
355 | 463 | | |
356 | 464 | | |
357 | 465 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
53 | 90 | | |
54 | 91 | | |
55 | 92 | | |
| |||
102 | 139 | | |
103 | 140 | | |
104 | 141 | | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
105 | 158 | | |
106 | 159 | | |
107 | 160 | | |
| |||
111 | 164 | | |
112 | 165 | | |
113 | 166 | | |
114 | | - | |
115 | | - | |
116 | | - | |
117 | | - | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
118 | 187 | | |
119 | 188 | | |
120 | | - | |
| 189 | + | |
121 | 190 | | |
122 | 191 | | |
123 | 192 | | |
124 | 193 | | |
125 | | - | |
126 | | - | |
| 194 | + | |
| 195 | + | |
127 | 196 | | |
128 | 197 | | |
129 | 198 | | |
| |||
0 commit comments