Skip to content

Commit b1ee77f

Browse files
committed
fix(conformance): close round six — the gate declined to count, but did not reject (PNI-204)
Two findings, both in material the previous round added, both about the gate rather than the corpus. Declining to count a wrongly-typed value is not the same as rejecting it: an expectation like eventPaths: [] beside a valid outcome passed the vacuity check, because the outcome carried it, while the payload assertion it looks like did nothing on either lane. The gate now validates every expectation's shape against what a runner can actually read, and that check was itself verified by injecting exactly that value and watching it fail. And a comment overclaimed: era-current-peer-keeps-modern-content is not the only fixture that kills the 0.0.57 gate — an unpinned subagent fixture fails too if that shim installs unconditionally. It protects two gates, not three, and now says so.
1 parent d8a89b9 commit b1ee77f

1 file changed

Lines changed: 54 additions & 2 deletions

File tree

spec/harness/conformance.test.ts

Lines changed: 54 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -193,6 +193,56 @@ describe("the conformance fixture corpus", () => {
193193
)
194194
.map(([key]) => key);
195195

196+
/** The type each expectation must hold for a runner to read it at all. */
197+
const KEY_SHAPES: Record<string, (value: unknown) => boolean> = {
198+
outcome: (v) => v === "completed" || v === "failed",
199+
errorContains: (v) => typeof v === "string",
200+
runError: (v) => typeof v === "boolean" || typeof v === "string",
201+
eventTypes: (v) => Array.isArray(v) && v.every((e) => typeof e === "string"),
202+
eventTypesAbsent: (v) =>
203+
Array.isArray(v) && v.every((e) => typeof e === "string"),
204+
eventPaths: (v) => v !== null && typeof v === "object" && !Array.isArray(v),
205+
eventAbsentPaths: (v) =>
206+
Array.isArray(v) && v.every((e) => typeof e === "string"),
207+
warnings: (v) => Array.isArray(v) && v.every((e) => typeof e === "string"),
208+
noWarnings: (v) => typeof v === "boolean",
209+
messageCount: (v) => typeof v === "number",
210+
messages: (v) => Array.isArray(v),
211+
state: () => true,
212+
request: (v) => v !== null && typeof v === "object" && !Array.isArray(v),
213+
requestAbsentPaths: (v) =>
214+
Array.isArray(v) && v.every((e) => typeof e === "string"),
215+
};
216+
217+
it.each(fixtures)("$file gives every expectation a readable shape", ({
218+
fixture,
219+
}) => {
220+
// Declining to COUNT a wrongly-typed value is not the same as rejecting
221+
// it: `eventPaths: []` beside a valid outcome would pass the vacuity
222+
// check while the payload assertion it looks like silently does nothing.
223+
const blocks: Array<[string, Record<string, unknown>]> = [
224+
["expect", (fixture.expect ?? {}) as Record<string, unknown>],
225+
];
226+
for (const [lane, override] of Object.entries(
227+
fixture.expectOverrides ?? {},
228+
))
229+
blocks.push([
230+
`expectOverrides.${lane}`,
231+
(override ?? {}) as Record<string, unknown>,
232+
]);
233+
for (const [where, block] of blocks) {
234+
for (const [key, value] of Object.entries(block)) {
235+
if (key === "intentional") continue;
236+
const shape = KEY_SHAPES[key];
237+
if (shape === undefined) continue; // the unknown-key gate covers this
238+
expect(
239+
shape(value),
240+
`${where}.${key} holds a value no runner can read: ${JSON.stringify(value)}`,
241+
).toBe(true);
242+
}
243+
}
244+
});
245+
196246
it.each(fixtures)("$file asserts something", ({ fixture }) => {
197247
const base = (fixture.expect ?? {}) as Record<string, unknown>;
198248
expect(
@@ -267,8 +317,10 @@ describe("the conformance fixture corpus", () => {
267317
],
268318
["a conformant 1.0 stream stays quiet", "conformant-run-is-quiet"],
269319
// The 0.0.39 and 0.0.47 fixtures delegate their version-gate coverage
270-
// here, and it is the only fixture that kills the 0.0.57 gate — so
271-
// deleting it would silently remove three gate checks.
320+
// here, so deleting this one removes those two checks entirely. The
321+
// 0.0.57 gate is also covered incidentally elsewhere — an unpinned
322+
// subagent fixture fails if that shim installs unconditionally — so this
323+
// entry protects two gates, not three.
272324
[
273325
"the era version gates are killable",
274326
"era-current-peer-keeps-modern-content",

0 commit comments

Comments
 (0)