Skip to content

Commit d00dc1e

Browse files
committed
Count every sibling array, and stop calling a closed gap a loss
Two bugs in the recall guard, both found by pointing it at a real report rather than at a fixture. Every element of an array shares one path key, and the guard was overwriting instead of summing — so it measured only the LAST sibling and went blind to a loss in any of the others. It now adds, and `findings[].path 5 -> 2` is visible where `3 -> 2` was all it could see before. And the "fewer is better" exemption matched whole words, so `knownGaps` shrinking — a gap closing, which is the tool improving — was reported as recall lost. It matches as a case-insensitive suffix now. A red on good news is a red people learn to ignore.
1 parent fe8ea61 commit d00dc1e

1 file changed

Lines changed: 14 additions & 8 deletions

File tree

scripts/check-artifact-recall.mjs

Lines changed: 14 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -50,8 +50,12 @@ const TRACKED = [
5050
"assets/example-srd",
5151
];
5252

53-
// Arrays whose shrinking is an improvement, not a regression.
54-
const FEWER_IS_BETTER = new Set(["warnings", "errors", "todo", "todos", "unresolved", "skipped", "failures", "gaps"]);
53+
// Arrays whose shrinking is an improvement, not a regression. Matched as a
54+
// SUFFIX on the last path segment, case-insensitively: a repo calls them
55+
// `knownGaps`, `parseErrors` or `unresolvedRefs`, and a guard that only knew the
56+
// bare words would report every closed gap as a loss — a red on good news is a
57+
// red people learn to ignore.
58+
const FEWER_IS_BETTER = ["warning", "error", "todo", "unresolved", "skipped", "failure", "gap", "miss"];
5559

5660
const refIdx = process.argv.indexOf("--ref");
5761
const ref = refIdx === -1 ? "HEAD" : (process.argv[refIdx + 1] ?? "HEAD");
@@ -95,18 +99,20 @@ function measureJson(text) {
9599
} catch {
96100
return null; // unparseable: fall back to lines rather than guess
97101
}
102+
// ADD, never set: every element of an array shares one path (`cases[].sites`),
103+
// so overwriting would keep only the last sibling's length and go blind to a
104+
// loss in any of the others. Summing makes the key mean "sites across all
105+
// cases", which is the number that must not go down.
106+
const add = (path, n) => m.set(path, (m.get(path) ?? 0) + n);
98107
const visit = (node, path) => {
99108
if (Array.isArray(node)) {
100-
m.set(path, node.length);
109+
add(path, node.length);
101110
node.forEach((v) => visit(v, `${path}[]`));
102111
return;
103112
}
104113
if (node && typeof node === "object") for (const [k, v] of Object.entries(node)) visit(v, path ? `${path}.${k}` : k);
105114
};
106115
visit(data, "");
107-
// `path[]` collapses every element of an array onto one key, so a nested
108-
// array's counts add up across siblings — which is what we want: ten findings
109-
// each losing one path step is a loss, even if no single array got shorter.
110116
return m;
111117
}
112118

@@ -129,8 +135,8 @@ const measure = (rel, text) => {
129135
};
130136

131137
const exempt = (what) => {
132-
const last = what.split(/[.[]/).filter(Boolean).pop() ?? what;
133-
return FEWER_IS_BETTER.has(last);
138+
const last = (what.split(/[.[]/).filter(Boolean).pop() ?? what).toLowerCase().replace(/s$/, "");
139+
return FEWER_IS_BETTER.some((w) => last.endsWith(w));
134140
};
135141

136142
// ---------------------------------------------------------------------------

0 commit comments

Comments
 (0)