Skip to content

Commit b393496

Browse files
committed
qa(checklist): clauses for three defects round 3 found outside the ledger, and two RUNNER rules
api-backend.batch-transactional-discovery (empty-string batch id, #24781), records-forms.validation-rule-type-matrix (unevaluable warning rule logs a rejection, #24782; closes round 2's unevaluable-CEL fixture gap) and access-security.no-active-org-session-semantics (walled group leg, #24784; automated.ref names the layer its pin drives). Each clause stays red until its card's fix lands. RUNNER: rule 6 gains "a pin scores only the layer it drives", and rule 8 keeps an off-clause probe budget in every run. Claude-Session: https://claude.ai/code/session_01MRkuLGosywLTY6EMcP6fGK Co-authored-by: Claude <noreply@anthropic.com>
1 parent a43405d commit b393496

4 files changed

Lines changed: 107 additions & 17 deletions

File tree

‎docs/qa/platform-checklist/RUNNER.md‎

Lines changed: 41 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -110,10 +110,49 @@ selector handed you has a verdict; planned ids are listed, not chased.
110110
and cite its output as the evidence; drive the browser only for what the pin doesn't
111111
cover. The reverse also holds: when a sweep hand-proves something repeatedly,
112112
propose promoting it to a permanent test and set `automated` in a revision.
113+
114+
**A pin scores only the layer it drives.** Before a pin scores a clause whose oracle
115+
is a public door's answer (an HTTP status, an error envelope, a row the door left
116+
behind), name the layer the pin reaches — a **plugin unit** (one plugin's hooks or
117+
middleware over a stub), the **engine** (a real ObjectQL engine and driver, no HTTP),
118+
or **HTTP** (a booted server answering the door) — and write that layer in the
119+
clause's evidence. Read it off the pin's own harness, never off its file name or its
120+
`automated.ref` prose: a stub terminal standing in for the engine means "lands" is
121+
"the middleware called `next()`", and nothing below it ran. A pin that stops above
122+
the layer the clause's oracle sits at scores that clause **`partial`**, naming the
123+
missing layer, and the live leg is driven, or recorded `blocked` with its fixture
124+
cited. It never scores `pass`. The instance that made this a rule is
125+
`access-security.no-active-org-session-semantics`: the `group` CONTROL of
126+
`no-active-organization-write-refusal.test.ts` "lands" a write by a member with no
127+
active organization at the plugin-security layer, two rounds scored the walled legs
128+
from it while a walled boot was refused, and on a full walled `group` boot the engine
129+
answers that same write with a 500 (#24784).
113130
7. **Verify pass for high-stakes claims.** For P0 `fail`s and any finding that would
114131
ship or block a release: a second, independent agent re-derives the verdict from the
115132
captured evidence alone (not from the first agent's narrative) before it is acted
116133
on. Disagreement → re-run the item.
134+
8. **Keep an off-clause probe budget.** Clauses only ask what someone already thought
135+
to ask. Every run keeps time AFTER its clauses have verdicts for probing the doors its
136+
selector covers past those clauses — on the same boot, as the same personas. Probe
137+
where a clause stops:
138+
- **boundary values** on the inputs the clauses drive: the empty string, a missing
139+
id, an unknown id;
140+
- **the opposite severity or posture** of a path a clause asserts: the `warning`
141+
twin of an `error` rule, `group` beside `isolated`;
142+
- **recipients or readers outside the asserted scope**: who else a notification,
143+
share or read reaches.
144+
145+
A probe is not a clause, so it scores nothing and no item verdict moves on it. A
146+
finding that passes the filing gate is extracted like any other product defect (see
147+
"Extraction obligation" below), and its card carries
148+
`QA-source: #<run record> · <area.item> · observed beside the clauses`, naming the
149+
item whose doors were probed. A probe that finds a gap the ledger should have asked
150+
about is also a checklist-accuracy finding for the wave's anchor card: the clause it
151+
was missing. The run record states how long the probes took and what they covered
152+
(doors, personas, the boundary classes tried), and says so when the budget was
153+
spent on nothing. A record that omits the probe line reads as a run that skipped it.
154+
Round 3 of the 18.0 sweep (QA run #24779) found #24781 and #24782 this way, beside
155+
items whose clauses all passed.
117156

118157
### Environment facts the runner should not re-derive
119158

@@ -648,7 +687,8 @@ The in-environment JSON scratch (RUNNER shape, never committed):
648687

649688
The issue body is: env fingerprint · scope (selector + per-item `revision`) · the
650689
per-clause verdict table (text oracle evidence) · a reproduction rule per `fail` (rule 2's
651-
carve-out excepted) · derived item verdicts + fixture gaps. The durable,
690+
carve-out excepted) · derived item verdicts + fixture gaps · the off-clause probe line
691+
(rule 8: how long, which doors and boundary classes, what it found). The durable,
652692
version-controlled truth is still the checklist under `areas/`; a run is a dated assertion
653693
about one build, and it lives in its issue, not the tree.
654694

‎docs/qa/platform-checklist/areas/access-security.json‎

Lines changed: 20 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -2347,7 +2347,7 @@
23472347
"title": "A session with no active organization is a LEGAL state with fail-closed semantics: tenant-scoped reads resolve to nothing, tenant-scoped writes are refused loudly naming the missing org, and nothing is ever NULL-stamped",
23482348
"since": "v17",
23492349
"status": "active",
2350-
"revision": 5,
2350+
"revision": 6,
23512351
"priority": "P0",
23522352
"surface": "api",
23532353
"personas": [
@@ -2364,13 +2364,14 @@
23642364
"knownGaps": [
23652365
"The state is timing-derived on the signup path (better-auth defers the membership write past the signup transaction, ADR-0093), so reproducing it by racing signup is flaky. The DURABLE reproduction is the removed-member path, or a multi-organization deployment whose reconciler binds nobody (ADR-0093 D1 `no-target-org`). Record WHICH producer the run used — a clause proven only on a racy producer is weaker evidence and must say so.",
23662366
"No stock showcase fixture is known to configure `invite-only` or SSO-JIT-pending; those two producers are blocked(fixture) unless the run configures them.",
2367-
"requires[0]'s tenancy-walled deployment is a walled boot. On 18.0.0 os serve — and os dev / os start, which boot through it — boots OS_TENANCY_POSTURE=group|isolated (and the legacy OS_MULTI_ORG_ENABLED=true): 18.x step 4 (#24580) deleted the walled boot refusal (serve.ts walledPostureRefusal) before 18.0.0, and of the published prereleases only 18.0.0-next.5 carries it. The walled boot is provisioned per RUNNER.md's environment-facts entry 'A walled boot is a fixture of its own': the multi-org runtime declared by the app (no example app declares it), OS_PLATFORM_OWNER_EMAIL, and a declared OS_AUTH_MEMBERSHIP_POLICY; each one absent refuses the boot naming itself. The item stays blocked (fixture, #9334) for the producer reasons above, not for the posture; its pin (automated.ref) boots the posture in-process."
2367+
"requires[0]'s tenancy-walled deployment is a walled boot. On 18.0.0 os serve — and os dev / os start, which boot through it — boots OS_TENANCY_POSTURE=group|isolated (and the legacy OS_MULTI_ORG_ENABLED=true): 18.x step 4 (#24580) deleted the walled boot refusal (serve.ts walledPostureRefusal) before 18.0.0, and of the published prereleases only 18.0.0-next.5 carries it. The walled boot is provisioned per RUNNER.md's environment-facts entry 'A walled boot is a fixture of its own': the multi-org runtime declared by the app (no example app declares it), OS_PLATFORM_OWNER_EMAIL, and a declared OS_AUTH_MEMBERSHIP_POLICY; each one absent refuses the boot naming itself. The item stays blocked (fixture, #9334) for the producer reasons above, not for the posture; its pin (automated.ref) boots the posture in-process.",
2368+
"acceptance[7] (the walled `group` leg for a member with no active organization) stays red until #24784 lands: on a walled group boot at 580fd4ed (QA run #24783, re-derived under RUNNER rule 7) the create answers 500 ERR_SYSTEM_WRITE_ORGANIZATION_REQUIRED and writes no row. It is scored only on a live walled group boot; a run without one records it blocked(fixture) citing RUNNER.md's 'A walled boot is a fixture of its own' — ⛔ never from automated.ref, whose group CONTROL stops at the plugin-security layer (RUNNER rule 6, a pin scores only the layer it drives). When #24784's fix merges, re-run the clause and drop this line"
23682369
]
23692370
},
23702371
"blocked": { "by": "fixture", "ref": "#9334" },
23712372
"automated": {
23722373
"kind": "unit",
2373-
"ref": "packages/plugins/plugin-security/src/no-active-organization-write-refusal.test.ts — pins every clause of this item at unit level, INCLUDING a control asserting that the `single` posture has no wall, i.e. the test itself agrees the stock showcase configuration makes the item vacuous"
2374+
"ref": "packages/plugins/plugin-security/src/no-active-organization-write-refusal.test.ts — pins every clause of this item at unit level, INCLUDING a control asserting that the `single` posture has no wall, i.e. the test itself agrees the stock showcase configuration makes the item vacuous. The layer it drives is a plugin unit: SecurityPlugin's middleware over a stub terminal that stands in for the engine, so a case that 'lands' there means the middleware called next(), and the engine's own insert path never runs. Its `group` describe's CONTROL 'a NON-EMPTY membership set lands, even with no active organization' therefore proves the plugin-security wall lets that write through and nothing past it: on a full walled group boot the engine refuses the same write with 500 ERR_SYSTEM_WRITE_ORGANIZATION_REQUIRED (#24784). Under RUNNER rule 6 the pin scores a clause whose oracle is the HTTP answer partial at most, naming the engine and HTTP layers as missing, and it cannot score acceptance[7] at all"
23742375
},
23752376
"steps": [
23762377
"establish the state and PROVE it: obtain a session whose `activeOrganizationId` is null, and cite the evidence (session row / session endpoint payload) — do not infer the state from the symptom you are about to test",
@@ -2380,7 +2381,8 @@
23802381
"query the underlying table as admin for rows whose tenant column is NULL, before and after the write attempt",
23812382
"repeat the read and the write against a NON-tenant-scoped object to bound the blast radius",
23822383
"sign in and out with the state in force, then inspect the audit ledger rows the session produced (plugin-audit auth-session-audit login/logout)",
2383-
"run the same read+write as the control admin persona to show the refusal is state-derived, not a blanket denial"
2384+
"run the same read+write as the control admin persona to show the refusal is state-derived, not a blanket denial",
2385+
"walled `group` leg (acceptance[7]), on a walled boot with OS_TENANCY_POSTURE=group provisioned per RUNNER.md's 'A walled boot is a fixture of its own': a member of organization A (non-empty membership set) sets A active with POST /api/v1/auth/organization/set-active, creates a showcase_account and a showcase_project in A, then clears it with POST /api/v1/auth/organization/set-active {organizationId: null} and proves the state (GET /api/v1/auth/get-session reads session.activeOrganizationId null). As that member, POST /api/v1/data/showcase_task {title, project: the project in A}; take the admin-side count of showcase_task rows whose organization_id is NULL or empty before and after, and read back any row the create produced"
23842386
],
23852387
"acceptance": [
23862388
{
@@ -2424,6 +2426,12 @@
24242426
"oracle": "api",
24252427
"verify": "the run record states each subsystem's observed behavior side by side; any disagreement is the finding, whichever way it leans",
24262428
"evidence": "the three-way comparison table"
2429+
},
2430+
{
2431+
"clause": "walled `group` leg: a member whose membership set is NON-EMPTY and whose session has no active organization never gets a 5xx on a tenant-scoped create, and nothing is stamped with a NULL organization_id. Which answer `group` gives — the write lands in a member organization, or a 4xx refusal — is #24784's to decide; this clause asserts only those two facts. Red until #24784 lands",
2432+
"oracle": "api",
2433+
"verify": "step-9 create on the live walled group boot: status is NOT 5xx (500 ERR_SYSTEM_WRITE_ORGANIZATION_REQUIRED, logged '[REST] Unhandled error: SystemWriteOrganizationRequiredError', is the #24784 shape); the admin-side NULL-or-empty organization_id count for showcase_task is identical before and after; and if the create answered 2xx, the row it produced carries a real organization_id the member belongs to. Score it only from that live boot: the automated.ref pin's group CONTROL stops above the engine (RUNNER rule 6), so it cannot score this clause",
2434+
"evidence": "the get-session payload + the create response, quoted + both NULL counts + the produced row's organization_id when the create landed"
24272435
}
24282436
],
24292437
"negative": [
@@ -2446,7 +2454,8 @@
24462454
"PR #8471 (ADR-0123 + enforcement)",
24472455
"@objectstack/plugin-security security-plugin.ts step 3.7 + tenant-layer.ts (computeTenantLayer0Filter, RLS_DENY_FILTER)",
24482456
"@objectstack/plugin-auth auth-manager.ts composeDatabaseHooks + reconcile-membership.ts",
2449-
"@objectstack/plugin-audit auth-session-audit.ts (login/logout rows)"
2457+
"@objectstack/plugin-audit auth-session-audit.ts (login/logout rows)",
2458+
"QA-source: #24783 · access-security.no-active-org-session-semantics · acceptance[1] (the walled group leg; #24784)"
24502459
],
24512460
"history": [
24522461
{
@@ -2473,6 +2482,12 @@
24732482
"date": "2026-10-20",
24742483
"change": "fixtures.knownGaps[2]'s release attribution corrected — round 10 of wave anchor #24579. The line said the walled boot refusal was 18.0's ('the 18.0 refusal') and that 18.x step 4 (#24580) deleted it, so a runner on 18.0.0 could not tell whether its release boots the walled posture. Under ruling record 6282882793 on the v18 epic (items 1 and 2), 18.0.0 is published only after step 4 has merged, and a walled posture boots on it. Of the published @objectstack/cli prereleases only 18.0.0-next.5 carries serve.ts walledPostureRefusal: next.0 to next.3 predate it, measured in each npm tarball and at each version tag, and no next.4 was published. Only that attribution changed: no clause, step or other knownGap moved",
24752484
"ref": "#24788"
2485+
},
2486+
{
2487+
"revision": 6,
2488+
"date": "2026-10-20",
2489+
"change": "acceptance[7], steps[8] and knownGaps[3] added, and automated.ref now names the layer its pin drives — for #24784, which the walled pass of round 3 (QA run #24783, 580fd4ed) found. Rounds 1 and 2 scored the walled legs from the pin because a walled boot was refused then; the pin's group CONTROL 'lands' a write by a member with no active organization at the plugin-security layer, and on a full walled group boot the engine refuses that write with a 500. The new clause asserts only no 5xx and no NULL stamping (which answer group gives is #24784's to decide) and stays red until #24784's fix lands. acceptance[0]'s and [1]'s group wording and knownGaps[0]-[1]'s producers (round 3 accuracy finding F3) belong to wave anchor #24579 and are not touched here",
2490+
"ref": "#24802"
24762491
}
24772492
]
24782493
},

0 commit comments

Comments
 (0)