Skip to content

Commit d8b2295

Browse files
fix(scenarios): the seed loads a user-handle field with a user id, and both author loops run verify (#30903)
<!-- claude-projects-attribution: {"github_login":"marchtian"} --> _Requested by **田騏** · [project thread](https://claude.ai/code/project/chan_01VmuUhXU9SSUzqJYyXSmYSi?thread=cmsg_01VmuUhXU9SSUzqJYyXSmYSiNjyvCdXYygFqntMNR7S2dW)_ Fixes #30882 Clause-②: no Metadata-protocol: none ## 维护者速读 - **改了什么**:场景跑分器灌种子数据时,把"负责人"一类字段里的人员代号(如 `u_sam`)换成该角色真实用户的 id(字段本身和 `owner_id` 都换);没有对应角色的代号(`u_tia`)记在管理员名下,提示词照此写明。种子有被拒的行时,依赖这些行的问数和写操作记为「未测」而非失败,计分卡显示 `⚠ seed: N refused`。构建与迭代两轮的提示词也让 AI 自己跑一遍跑分时用的同一条 `verify` 命令。 - **为什么改**:第一次真跑时 8 条种子只进了 1 条,后面的问数和写操作量的是跑分器自己的缺陷,不是被测应用,计分卡却把它们算成失败;AI 也从未被告知要过 `verify`,于是构建"降级"并不说明它修不好。 - **风险与回滚**:只动未发布的私有包 `@objectstack/scenarios`(QA 工具),不影响产品运行时;回滚即还原本 PR。其他卡片中指向记录或选项的范围字段照旧原样发送。 - **先看**:`packages/qa/scenarios/src/runner/orchestrate.ts` 的种子循环与 `seedDependence`;`packages/qa/scenarios/src/runner/scorecard.ts` 的未测计数;`packages/qa/scenarios/src/runner/prompts.ts` 的 `userHandleFields` 与 Row ownership 文案。 - **两分钟验证**:`pnpm --filter @objectstack/scenarios exec vitest run test/runner`,预期全部通过,其中"loads all 8 rows"断言种子 8/8 入库,"a refused seed row"两条断言 `⚠ seed` 警告与未测标记。 - **需要你做的**:无,席位落地。 ## What changes **Defect 1 — the seed lost every owner-scoped row.** The seed loop set `owner_id` for a row whose persona-scope field named a persona's handle, but sent the field itself as the handle text (`owner: "u_sam"`). A faithful build models that field as a user lookup and refuses the handle; a handle no persona holds (`u_tia`) had no user at all, and every contact under a refused account failed as a cascade. - `prompts.ts` `userHandleFields(card)`: the direct scope fields whose seed values are user handles — a field a persona's direct `scope` rule names that the card declares as neither a lookup nor a select, and that is not `id`. Across the nine cards in `docs/qa/scenarios/` that is exactly `account.owner` (crm), `invoice.owner` (invoice) and `task.assignee` (project). Every other direct scope field names a seeded row (`asset.custodian` = `e06`, `department.id`) or an option (`ticket_comment.visibility` = `public`) and is still sent verbatim — replacing those values would have broken the lookups on six other cards. - `orchestrate.ts` seed loop: a user-handle field now carries the persona's user id in the field itself and in `owner_id`. `owner_id` stays mapped: it is the platform's canonical, reassignable record owner (`packages/objectql/src/registry.ts`), the column owner-based sharing reads. A handle no persona holds is loaded as the admin user (field and `owner_id`), and the run record lists it under `seed.handlesAsAdmin`. - `prompts.ts` `dataContract`: the Row ownership paragraph now states the contract the seed keeps. The handle field and `owner_id` both get the user's id: the persona's user for a held handle, and the admin the runner signs in as for an unheld one. For crm it reads: "In `democrm_account.owner` the value is a handle that stands for a user, and the handle text is never stored: the runner loads that field, and the row's owner (`owner_id`), with that user's id (`u_sam` → the `sdr` user; `u_tia`, which no person below holds → the admin user the runner signs in as). Make that field accept a user id; a rule on it matches that user." This is the card's "state another contract in the build prompt" branch. **Defect 1, ask 3 — a refused seed row makes the Run half not measured, never a fail.** - `orchestrate.ts` `seedDependence(card, refused)` takes out the refused rows and, through the card's declared lookups (`expect.lookups`), every seed row under one, transitively. A contact of a refused account is out whether or not the app refused it too. - A question is not measured when its `object` holds a row that is out, because its truth was computed over rows the app does not hold. An operation is not measured when its `target`, a lookup value it writes, or a post-state row (`id`, or a lookup value in its `values`) names a row that is out. - `runHalf` does not ask those turns. Each is recorded with `notMeasured` and the reason, for example `seed: needs democrm_contact ct2, a child of democrm_account ac1, which the app refused`. Everything else is asked and graded as before. - `scorecard.ts` adds a Seed row and an Iterate · Seed row, for example `7/8 ⚠ seed: 1 refused`. The ask and operation tallies leave not-measured turns out of both passed and total, and their cells name the count, for example `4/4 · 5 not measured`. A half with nothing measured stays `not-run` in the record, never `fail`. **Defect 2 — the author loop never ran verify.** `verifyArgs(card)` is `['verify', ...verifyFlags(card)]`. `buildHalf` runs exactly that, and the build prompt and the iterate prompt both name it next to `validate` and `build`: for the crm card, `npx objectstack validate`, `npx objectstack build` and `npx objectstack verify --reference-shapes`, "until all three pass". The template-mode iterate prompt is unchanged: that mode edits the installed copy through the metadata API and runs no `verify`. ## Tests - `test/runner/orchestrate.test.ts`, new top-level describe `runCard — a refused seed row makes what reads it not measured, never a fail`, with `RefusingApp`, a double that refuses the seed rows it is told to: - `shows ⚠ seed: 1 refused, leaves the questions on that object unasked and unscored, and grades the rest as usual`. It refuses ct4: the scorecard reads `| Seed | 7/8 ⚠ seed: 1 refused |`; the five contact question turns are not measured and were not asked; the four account turns are graded (4/4); the five operations are graded (5/5); and the record's run-ask is `pass` with notes `4/4 · 5 not measured`. - `an operation whose target hangs off a refused row is not run; a half with nothing measured is not-run, not fail`. It refuses all three accounts: only `contact_bad_email` runs, all 9 question turns are out, run-ask is `not-run` and run-operations is `pass`. - The clean run now also asserts `| Seed | 8/8 | — |`, with no warning. - `test/runner/units.test.ts` `a refused seed row takes out its children and what the card names on them, and nothing else` pins the rule on ac1: its target, a child, and the account an operation writes are out; `add_contact` and `contact_bad_email` are not. - `test/runner/orchestrate.test.ts`, new top-level describe `runCard — the seed against a build whose owner field is a user lookup`. It uses `UserLookupApp`, an app double that refuses an `owner`/`owner_id` naming no user and a contact whose `account` names no loaded account, as the first live run's app did. - `the double refuses the handle the old seed sent, so it can fail`. - `loads all 8 rows: the owner field carries the persona user id, and a handle no persona holds is the admin`: `seed` equals `{ inserted: 8, refused: [], handlesAsAdmin: ['u_tia'] }`; ac1 and ac3 carry the sdr user in `owner` and `owner_id`, ac2 the admin; all five contacts load; the five operations pass. - The existing ownership test now expects ac2's `owner_id` to be the admin (it was undefined). - `test/runner/units.test.ts`: - `both authoring loops name the verify invocation the runner scores with, next to validate and build`. - `a user-handle field is a scope field the card declares as neither lookup nor select; the contract says it is loaded with a user id`, which also pins that `service_desk` has none. - The regression test is built from the card in the repo (`crm_account_with_contacts`), not from the first run's recording, which the epic seat holds. No live model was run. ## Acceptance notes - **Non-persona handles become the admin, not a provisioned user.** The seat accepted this as the card's "state another contract in the build prompt" branch. The admin is a user the runner already holds, it grants no new permission, and it keeps all 8 rows loadable with no new HTTP path. The cost: two unheld handles on one card would both map to the admin, so a truth grouped by owner could not tell them apart. No card today has one. - **The not-measured rule is object-level for questions.** A question is out when any row of its own object is out, even if its filter would not have read that row. That is the reading the card states ("depends on a refused row"), and it never scores a truth the app could not have answered. The cost is that one refused contact takes out every contact question. - **Not-measured turns are not asked.** That saves model turns, and a grade against rows the app lacks would mean nothing. - **The package README is not updated.** It does not describe the Seed row or the not-measured marks, because #30846 has the README in flight. - **Out of scope, seen while reading other cards.** `personaHandles` treats every direct scope string value as a handle, so the seed loop's `owner_id` mapping (unchanged here) also fires on fields that name rows or options. On `service_desk`, 237 public `ticket_comment` rows are owned by the portal_customer user (`visibility = public`), and 15 tickets name two personas, so the last field read wins `owner_id`. This is reported to the seat, not fixed here. ## Verification <details><summary>Package tests and typecheck (head b1f0628)</summary> - `pnpm --filter @objectstack/scenarios test` :: exit 0 — Test Files 18 passed (18), Tests 245 passed (245) - `pnpm --filter @objectstack/scenarios typecheck` :: exit 0 (the tsconfig includes `test/**/*`) </details> <details><summary>Ablations (each restored, blob equal to HEAD, git diff HEAD empty)</summary> - Deleting `body[k] = userId;` in the seed loop: the 8-row pin goes red. `inserted` is not 8 and `refused` is not empty: the double refuses the handle. - Replacing the verify-naming checks line with the old validate-and-build text: the verify-invocation pin goes red. - Feeding `seedDependence` an empty refused list in `runHalf`: both refused-seed run tests go red. - Deleting the `⚠ seed: N refused` suffix in `seedCell`: both refused-seed run tests go red. The first attempt used an empty `--replacement`, which the tool refused before running anything; it was redone with `--delete`. </details> <details><summary>Derived gates on the final diff: 48 of 48 run, every one exit 0 (head b1f0628, merge base 98f640d, origin/main not moved since round 1)</summary> - `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` derived 48 commands for the 5 changed paths. - Round 2: all 48 exited 0 (the round-1 `pnpm build` output is still in place for `check:dual-build-cjs-loads`). - `dispatch-gates --ran` with the exit-coded record: "48 derived famil(ies) accounted for — 48 run, 0 NOT-MEASURED (a DERIVED zero — all 48 recorded an exit code and none of them is 3)". - Changeset: none. `@objectstack/scenarios` is `private: true`, so nothing publishes; this PR carries `skip-changeset`, as this package's recent PRs (#30837, #30801) did. - Not run locally, left to CI: the repo-wide `pnpm test`, `pnpm lint` and the CI-only families the derivation lists outside its 48. </details> 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- signed: domain:devx#2 · marchtian · session_017ihL2ebbZkr2QAEG1x7UC8 --> --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 45a085a commit d8b2295

5 files changed

Lines changed: 380 additions & 36 deletions

File tree

‎packages/qa/scenarios/src/runner/orchestrate.ts‎

Lines changed: 113 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ import type { AgentDriver, StepRef } from './driver.js';
4141
import { citationLabels, namespaceOfArtifact, namespacedScenario, permissionSetName, templateHostName } from './namespace.js';
4242
import { observeOperation, touchedRowId, type Row } from './observe.js';
4343
import type { CommandResult } from './project.js';
44-
import { buildPrompt, iteratePrompt, operationPrompt, personaHandles, questionPrompt, templateIteratePrompt, verifyFlags } from './prompts.js';
44+
import { buildPrompt, iteratePrompt, operationPrompt, personaHandles, questionPrompt, templateIteratePrompt, userHandleFields, verifyArgs } from './prompts.js';
4545
import type { ParsedTurn } from './transcript.js';
4646

4747
// ── What the orchestrator needs from the outside world ──────────────────────
@@ -97,6 +97,8 @@ export interface BuildHalf {
9797
export interface QuestionResult {
9898
question: string;
9999
persona: string | null;
100+
/** Why this turn was not asked: it reads a row the seed could not load ({@link seedDependence}). Not measured, never a fail. */
101+
notMeasured?: string;
100102
grade?: PersonaRunGrade;
101103
turn?: TurnSummary;
102104
error?: string;
@@ -105,6 +107,8 @@ export interface QuestionResult {
105107
export interface OperationResult {
106108
operation: string;
107109
persona: string;
110+
/** Why this operation was not run: it acts on a row the seed could not load ({@link seedDependence}). Not measured, never a fail. */
111+
notMeasured?: string;
108112
observation?: OperationObservation;
109113
grade?: OperationGrade;
110114
turn?: TurnSummary;
@@ -113,7 +117,12 @@ export interface OperationResult {
113117

114118
export interface RunHalf {
115119
personas: Array<Omit<PersonaAccount, 'apiKey'>>;
116-
seed: { inserted: number; refused: Array<{ object: string; id: string; code?: string; message?: string }> };
120+
seed: {
121+
inserted: number;
122+
refused: Array<{ object: string; id: string; code?: string; message?: string }>;
123+
/** Handles a user-handle field named that no persona holds, each loaded as the admin user (`u_tia`). */
124+
handlesAsAdmin?: string[];
125+
};
117126
questions: QuestionResult[];
118127
operations: OperationResult[];
119128
error?: string;
@@ -236,6 +245,78 @@ export function hiddenValues(scenario: LoadedScenario, personaId: string): Array
236245
return [...out];
237246
}
238247

248+
type Card = LoadedScenario['card'];
249+
type Question = Card['run']['questions'][number];
250+
type Operation = Card['run']['operations'][number];
251+
252+
/**
253+
* What a refused seed row takes out of the Run half, read from what the card
254+
* names. The rows out are the refused ones plus, transitively, every seed row
255+
* whose declared lookup (`expect.lookups`) names a row that is out — a contact
256+
* of a refused account, whether or not the app refused it too. Then:
257+
*
258+
* - a question is not measured when its `object` has a row that is out: its
259+
* truth was computed over rows the app does not hold;
260+
* - an operation is not measured when its `target`, a lookup value it writes,
261+
* or a post-state row (`id`, or a lookup value in its `values`) names a row
262+
* that is out.
263+
*
264+
* Everything else grades as usual. A not-measured turn is not asked and counts
265+
* in neither passed nor total.
266+
*/
267+
export function seedDependence(
268+
card: Card,
269+
refused: ReadonlyArray<{ object: string; id: string }>,
270+
): { out: Map<string, string>; question(q: Question): string | undefined; operation(op: Operation): string | undefined } {
271+
/** `object/id` → the refused row it hangs off, as `object id`. */
272+
const out = new Map<string, string>(refused.map((r) => [`${r.object}/${r.id}`, `${r.object} ${r.id}`]));
273+
const lookups = card.expect.lookups ?? [];
274+
for (let grew = true; grew; ) {
275+
grew = false;
276+
for (const l of lookups) {
277+
for (const row of card.run.seed[l.object] ?? []) {
278+
const parent = out.get(`${l.to}/${String(row[l.field])}`);
279+
if (parent !== undefined && !out.has(`${l.object}/${row.id}`)) {
280+
out.set(`${l.object}/${row.id}`, parent);
281+
grew = true;
282+
}
283+
}
284+
}
285+
}
286+
const why = (object: string, id: unknown): string | undefined => {
287+
const root = typeof id === 'string' ? out.get(`${object}/${id}`) : undefined;
288+
if (root === undefined) return undefined;
289+
return root === `${object} ${String(id)}` ? `seed: needs ${root}, which the app refused` : `seed: needs ${object} ${String(id)}, a child of ${root}, which the app refused`;
290+
};
291+
const viaLookups = (object: string, values: Record<string, unknown> | undefined): string | undefined => {
292+
for (const l of lookups) {
293+
const w = l.object === object && values && l.field in values ? why(l.to, values[l.field]) : undefined;
294+
if (w) return w;
295+
}
296+
return undefined;
297+
};
298+
return {
299+
out,
300+
question: (q) => {
301+
for (const row of card.run.seed[q.object] ?? []) {
302+
const w = why(q.object, row.id);
303+
if (w) return w;
304+
}
305+
return undefined;
306+
},
307+
operation: (op) => {
308+
const direct = (op.target !== undefined ? why(op.object, op.target) : undefined) ?? viaLookups(op.object, op.values);
309+
if (direct) return direct;
310+
if (op.outcome.kind !== 'applied') return undefined;
311+
for (const st of op.outcome.state) {
312+
const w = (st.id !== undefined ? why(st.object, st.id) : undefined) ?? viaLookups(st.object, st.values);
313+
if (w) return w;
314+
}
315+
return undefined;
316+
},
317+
};
318+
}
319+
239320
/** The persona (not a card artifact) a scenario's audit trail names, from the user id the app recorded. */
240321
function personaOfUser(userId: string | null, accounts: readonly PersonaAccount[], adminId: string): string | null {
241322
if (userId === null) return null;
@@ -276,7 +357,7 @@ async function buildHalf(
276357
}
277358
half.commands.push(await ops.os(dir, ['validate']));
278359
half.commands.push(await ops.os(dir, ['build']));
279-
half.commands.push(await ops.os(dir, ['verify', ...verifyFlags(scenario.card)]));
360+
half.commands.push(await ops.os(dir, verifyArgs(scenario.card))); // the invocation every authoring prompt names
280361
const artifact = ops.readArtifact(dir);
281362
if (artifact === undefined) {
282363
half.error = half.error ?? 'no dist/objectstack.json after `objectstack build`';
@@ -311,16 +392,31 @@ export async function runHalf(
311392
}
312393
const adminKey = await app.adminApiKey();
313394

314-
// Seed, with row ownership mapped from the persona handles.
395+
// Seed, with row ownership mapped from the persona handles. A row whose
396+
// direct scope field names a persona's handle is owned (`owner_id`, the
397+
// platform's record owner) by that persona's user. A user-handle field
398+
// (`account.owner`, `userHandleFields`) is itself loaded with a user id —
399+
// the persona's, or the admin's for a handle no persona holds (`u_tia`) —
400+
// because a faithful build models it as a user lookup, which refuses the
401+
// handle text. Any other scope field names a row or an option and is sent
402+
// verbatim.
315403
const handles = personaHandles(card);
316404
const ownerFields = new Set(card.run.personas.flatMap((p) => (p.scope ?? []).filter((s) => !s.field.includes('.')).map((s) => `${s.object}.${s.field}`)));
405+
const userFields = userHandleFields(card);
317406
for (const object of seedOrder(scenario)) {
318407
for (const row of card.run.seed[object] ?? []) {
319408
const body: Record<string, unknown> = { ...row };
320409
for (const [k, v] of Object.entries(row)) {
321-
const persona = typeof v === 'string' && ownerFields.has(`${object}.${k}`) ? handles.get(v) : undefined;
410+
if (typeof v !== 'string' || !ownerFields.has(`${object}.${k}`)) continue;
411+
const persona = handles.get(v);
322412
const acct = persona ? accounts.find((a) => a.persona === persona) : undefined;
323413
if (acct) body.owner_id = acct.userId;
414+
if (userFields.has(`${object}.${k}`)) {
415+
const userId = acct?.userId ?? app.adminUserId();
416+
body[k] = userId;
417+
body.owner_id = userId;
418+
if (!acct) half.seed.handlesAsAdmin = [...new Set([...(half.seed.handlesAsAdmin ?? []), v])].sort();
419+
}
324420
}
325421
const r = await app.insertRow(object, body);
326422
if (r.ok) half.seed.inserted++;
@@ -329,11 +425,18 @@ export async function runHalf(
329425
}
330426

331427
const keyOf = (persona: string | null) => (persona === null ? adminKey : accounts.find((a) => a.persona === persona)!.apiKey);
428+
// A refused seed row makes what reads it not measured — never a fail of the app's answer.
429+
const dependence = seedDependence(card, half.seed.refused);
332430

333431
for (const q of card.run.questions) {
432+
const notMeasured = dependence.question(q);
334433
for (const persona of [null, ...(q.personas ?? [])]) {
335434
if (persona === null && q.personas?.length) continue; // a persona-scoped question is asked as its personas
336435
const res: QuestionResult = { question: q.id, persona };
436+
if (notMeasured) {
437+
half.questions.push({ ...res, notMeasured });
438+
continue;
439+
}
337440
try {
338441
const turn = await driver.ask({ ...ref, step: q.id, ...(persona ? { persona } : {}), prompt: questionPrompt(scenario, q.id), mcp: { url: app.mcpUrl, apiKey: keyOf(persona) } });
339442
res.turn = summarize(turn);
@@ -352,6 +455,11 @@ export async function runHalf(
352455

353456
for (const op of card.run.operations) {
354457
const res: OperationResult = { operation: op.id, persona: op.actingAs };
458+
const notMeasured = dependence.operation(op);
459+
if (notMeasured) {
460+
half.operations.push({ ...res, notMeasured });
461+
continue;
462+
}
355463
try {
356464
const before = (await app.listRows(op.object)).rows as Row[];
357465
const turn = await driver.ask({ ...ref, step: op.id, persona: op.actingAs, prompt: operationPrompt(card, op.id), mcp: { url: app.mcpUrl, apiKey: keyOf(op.actingAs) } });

‎packages/qa/scenarios/src/runner/prompts.ts‎

Lines changed: 76 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,43 @@ export function personaHandles(card: Card): Map<string, string> {
5050
return out;
5151
}
5252

53+
/**
54+
* The scope fields whose seed values are user handles, `object.field`: a
55+
* field a persona's direct (non-path) `scope` rule names that the card does
56+
* not declare as a lookup or a select, and that is not `id`. In the crm card
57+
* that is `account.owner` (`u_sam`, `u_tia`). Every other direct scope field
58+
* holds a row id or an option value (`asset.custodian` = `e06`, an employee
59+
* row; `ticket_comment.visibility` = `public`), which the seed loads verbatim.
60+
*/
61+
export function userHandleFields(card: Card): Set<string> {
62+
const declared = new Set([
63+
...(card.expect.lookups ?? []).map((l) => `${l.object}.${l.field}`),
64+
...(card.expect.selectFields ?? []).map((s) => `${s.object}.${s.field}`),
65+
]);
66+
const out = new Set<string>();
67+
for (const p of card.run.personas) {
68+
for (const sc of p.scope ?? []) {
69+
const key = `${sc.object}.${sc.field}`;
70+
if (!sc.field.includes('.') && sc.field !== 'id' && !declared.has(key)) out.add(key);
71+
}
72+
}
73+
return out;
74+
}
75+
76+
/** The handles a seed row names in a {@link userHandleFields} field that no persona signs in as (`u_tia`): the seed loads each as the admin user. */
77+
export function unheldHandles(card: Card): string[] {
78+
const held = personaHandles(card);
79+
const out = new Set<string>();
80+
for (const key of userHandleFields(card)) {
81+
const [object, field] = key.split('.') as [string, string];
82+
for (const row of card.run.seed[object] ?? []) {
83+
const v = row[field];
84+
if (typeof v === 'string' && !held.has(v)) out.add(v);
85+
}
86+
}
87+
return [...out].sort();
88+
}
89+
5390
/** The every-item-prefixed rule, in every authoring prompt. */
5491
export function namingRule(card: Card): string {
5592
const ns = scenarioNamespace(card);
@@ -102,6 +139,22 @@ export function dataContract(card: Card): string {
102139
`Row ownership: a row whose ${[...new Set(card.run.personas.flatMap((p) => (p.scope ?? []).filter((s) => !s.field.includes('.')).map((s) => `\`${s.field}\``)))].join(' / ')} names a handle ` +
103140
`(${[...handles].map(([h, id]) => `\`${h}\` = \`${id}\``).join(', ')}) is loaded as owned by that person’s user.`,
104141
);
142+
const userFields = [...userHandleFields(card)];
143+
if (userFields.length) {
144+
const unheld = unheldHandles(card);
145+
const held = [
146+
...new Set(
147+
card.run.personas.flatMap((p) => (p.scope ?? []).filter((sc) => userHandleFields(card).has(`${sc.object}.${sc.field}`)).map((sc) => `\`${String(sc.value)}\` → the \`${p.id}\` user`)),
148+
),
149+
];
150+
lines.push(
151+
`In ${userFields.map((k) => `\`${k}\``).join(' / ')} the value is a handle that stands for a user, and the handle text is never stored: ` +
152+
`the runner loads that field, and the row’s owner (\`owner_id\`), with that user’s id (${[
153+
...held,
154+
...(unheld.length ? [`${unheld.map((h) => `\`${h}\``).join(', ')}, which no person below holds → the admin user the runner signs in as`] : []),
155+
].join('; ')}). Make that field accept a user id; a rule on it matches that user.`,
156+
);
157+
}
105158
}
106159
if (card.run.agentActions?.length) {
107160
lines.push('');
@@ -111,19 +164,35 @@ export function dataContract(card: Card): string {
111164
return lines.join('\n');
112165
}
113166

114-
const BUILD_PREAMBLE =
115-
'You are building an ObjectStack app in this project (scaffolded by `create-objectstack`; follow the project’s AGENTS.md ' +
116-
'and its objectstack skills). Author the metadata, then run `npx objectstack validate` and `npx objectstack build` ' +
117-
'and fix what they report until both pass. Work without asking questions.';
167+
/**
168+
* The `objectstack verify` arguments the runner scores a build with
169+
* (`buildHalf` in `orchestrate.ts` runs exactly these), so every authoring
170+
* loop can be told to run the same proof before it stops.
171+
*/
172+
export function verifyArgs(card: Card): string[] {
173+
return ['verify', ...verifyFlags(card)];
174+
}
175+
176+
/** The three commands a scratch build is scored on, as the agent runs them: `validate`, `build`, then {@link verifyArgs}. */
177+
function authoringChecks(card: Card): string {
178+
return `run \`npx objectstack validate\`, \`npx objectstack build\` and \`npx objectstack ${verifyArgs(card).join(' ')}\` and fix what they report until all three pass`;
179+
}
180+
181+
function buildPreamble(card: Card): string {
182+
return (
183+
'You are building an ObjectStack app in this project (scaffolded by `create-objectstack`; follow the project’s AGENTS.md ' +
184+
`and its objectstack skills). Author the metadata, then ${authoringChecks(card)}. Work without asking questions.`
185+
);
186+
}
118187

119188
export function buildPrompt(card: Card): string {
120-
return [BUILD_PREAMBLE, '', '## The app', '', text(card.prompt.detailed), '', dataContract(card)].join('\n');
189+
return [buildPreamble(card), '', '## The app', '', text(card.prompt.detailed), '', dataContract(card)].join('\n');
121190
}
122191

123192
export function iteratePrompt(card: Card): string {
124193
return [
125-
'This project holds a working ObjectStack app. Apply the change below, then run `npx objectstack validate` and ' +
126-
'`npx objectstack build` and fix what they report until both pass. Keep everything else working. Work without asking questions.',
194+
`This project holds a working ObjectStack app. Apply the change below, then ${authoringChecks(card)}. ` +
195+
'Keep everything else working. Work without asking questions.',
127196
'',
128197
namingRule(card),
129198
'',

0 commit comments

Comments
 (0)