Skip to content

Commit 6cc5c52

Browse files
authored
test(diagnostics): close the rule-4 acceptance gap and three ledger nitpicks (#323)
P-022 rule 4 states two lines as normative acceptance; the executable guard implemented only the first. This brings the guard up to the norm accepted in #322, and no further. _insertion_effect() returns churn / added / removed. Each of the three mutations is caught by exactly one number, with the other two structurally blind to it: seed = sorted index churn=42 added=1 removed=0 iterate the golden's codes churn=0 added=0 removed=0 cap ledger at previous size churn=0 added=1 removed=1 (baseline) churn=0 added=1 removed=0 The third number came from a Codex finding during review: a generator capped at its previous size emits the probe AND drops an existing code. The dropped record never enters the churn comparison and the addition still counts one, so both original numbers read clean while the ledger had not grown at all. This does not widen rule 4 — a delta is a gain, and counting only additions was measuring the stated norm wrong. The churn gate sits behind the staleness check, so it was additionally verified after regenerating under the mutation, where it fires with its own message. Three nitpicks from the #321 review, also mutation-checked: * analyzer_corpus read with expect, matching every other accessor in the file; dropping the key from one case now fails naming it, where unwrap_or(false) would have silently under-counted coverage. * changed == unexplained asserted explicitly, since they move together by construction and an added explanation channel must split them deliberately. Recorded honestly: on a green tree this is 0 == 0 and proves nothing. Shown to have teeth by a compound mutation (corrupt one recorded text so a divergence exists, drop the unexplained increment) — it fires first, naming the counter split rather than the divergence. * every_optional_field_shape_is_still_exercised guards the nine arms the generator rotates. Collapsing the two-evidence arm and regenerating fails only this test: the divergence check still passes 47/47, because it asks whether each case matches its own recorded text, never whether a shape disappeared. Test-only; no production code touched on either side. Refs #255, #250.
1 parent 3736ec0 commit 6cc5c52

2 files changed

Lines changed: 186 additions & 25 deletions

File tree

‎rust/crates/own-diagnostics/tests/ledger_replay.rs‎

Lines changed: 96 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -8,12 +8,16 @@
88
//! [`own_diagnostics::TITLES`] must have a case, so a family cannot be added
99
//! without a fixture.
1010
//!
11-
//! The three failure modes, each with its own test so a red build names the
12-
//! actual problem:
11+
//! The failure modes, each with its own test so a red build names the actual
12+
//! problem:
1313
//!
1414
//! * **missing** — a `TITLES` code with no ledger case;
1515
//! * **orphan** — a ledger case naming a code absent from `TITLES`;
16-
//! * **divergence** — the rendered text disagrees with the reference.
16+
//! * **divergence** — the rendered text disagrees with the reference;
17+
//! * **collapsed shape** — the generator still covers every code, but stopped
18+
//! producing one of the optional-field arms it exists to pin. Invisible to
19+
//! the divergence check, which only asks whether each case matches its *own*
20+
//! recorded text: drop the two-evidence arm and that test still passes 47/47.
1721
//!
1822
//! What this does NOT claim: that the analyzer emits a given code. That is step
1923
//! 4's contract, pinned over the real `.own` corpus by `diag_parity.json`. The
@@ -25,7 +29,7 @@
2529

2630
use std::collections::BTreeSet;
2731

28-
use own_diagnostics::{title, Diagnostic, TITLES};
32+
use own_diagnostics::{title, Diagnostic, Severity, TITLES};
2933
use serde_json::Value;
3034

3135
const FIXTURE: &str = concat!(
@@ -183,6 +187,29 @@ fn rendered_text_matches_the_reference_for_every_family() {
183187
}
184188
}
185189

190+
// `changed` and `unexplained` move together **by construction**: the loop
191+
// above increments both on the same branch, because this replay has no
192+
// mechanism for marking a difference as *explained*. That is deliberate —
193+
// an explained divergence would need a reviewed entry in the fixture, and
194+
// no such entry exists. Asserting the identity records the fact instead of
195+
// leaving two counters that merely look independent: if an explanation
196+
// channel is ever added, this line fails and forces the two to be split on
197+
// purpose rather than letting `unexplained` quietly over-report.
198+
//
199+
// Honest about its own reach: while the tree is green this is `0 == 0` and
200+
// proves nothing — the divergence branch never runs. It is a tripwire that
201+
// arms only once something actually diverges, which is exactly when a
202+
// silently over-reporting `unexplained` would mislead. Verified by a
203+
// compound mutation (corrupt one case's recorded text so a divergence
204+
// exists, and drop the `unexplained` increment): this fires first, naming
205+
// the counter split rather than the divergence.
206+
assert_eq!(
207+
counters.changed, counters.unexplained,
208+
"changed and unexplained are equal by construction; they diverged, so an \
209+
explanation mechanism was introduced without separating the counters: \
210+
{counters:?}"
211+
);
212+
186213
assert_eq!(
187214
counters.unexplained,
188215
0,
@@ -195,6 +222,66 @@ fn rendered_text_matches_the_reference_for_every_family() {
195222
assert_eq!(counters.changed, 0, "counters: {counters:?}");
196223
}
197224

225+
/// Every optional-field arm the generator can produce must still appear.
226+
///
227+
/// The ledger's value is that it renders *shapes*, not 47 copies of one shape:
228+
/// `_shape_for` rotates subject, resource kind, severity and evidence length by
229+
/// each code's own seed. Nothing so far would notice that rotation collapsing.
230+
/// Changing the vocabulary changes every seed-derived arm at once, so a future
231+
/// edit could quietly leave, say, no two-evidence case anywhere — and the
232+
/// rendering replay would still pass 47/47, because it only checks that each
233+
/// case matches its own recorded text.
234+
///
235+
/// This asserts presence, not counts: the exact split is a property of the hash
236+
/// and would make the test a second golden with no extra signal.
237+
#[test]
238+
fn every_optional_field_shape_is_still_exercised() {
239+
let root = load();
240+
let diagnostics: Vec<Diagnostic> = cases(&root)
241+
.iter()
242+
.map(|case| {
243+
let code = code_of(case);
244+
serde_json::from_value(case.get("diagnostic").expect("'diagnostic'").clone())
245+
.unwrap_or_else(|e| panic!("code {code:?}: diagnostic does not load: {e}"))
246+
})
247+
.collect();
248+
249+
let count = |f: &dyn Fn(&Diagnostic) -> bool| diagnostics.iter().filter(|d| f(d)).count();
250+
let arms: [(&str, usize); 9] = [
251+
("subject: present", count(&|d| d.subject.is_some())),
252+
("subject: absent", count(&|d| d.subject.is_none())),
253+
(
254+
"resource_kind: present",
255+
count(&|d| d.resource_kind.is_some()),
256+
),
257+
(
258+
"resource_kind: absent",
259+
count(&|d| d.resource_kind.is_none()),
260+
),
261+
(
262+
"severity: warning",
263+
count(&|d| d.severity == Severity::Warning),
264+
),
265+
("severity: error", count(&|d| d.severity == Severity::Error)),
266+
("evidence: none", count(&|d| d.evidence.is_empty())),
267+
("evidence: one", count(&|d| d.evidence.len() == 1)),
268+
("evidence: two", count(&|d| d.evidence.len() == 2)),
269+
];
270+
271+
let unexercised: Vec<&str> = arms
272+
.iter()
273+
.filter(|(_, n)| *n == 0)
274+
.map(|(name, _)| *name)
275+
.collect();
276+
assert!(
277+
unexercised.is_empty(),
278+
"{} optional-field shape arm(s) are no longer exercised by any ledger case: \
279+
{unexercised:?}. The ledger stopped covering a shape it is supposed to pin — \
280+
restore the rotation rather than deleting the arm. Full census: {arms:?}",
281+
unexercised.len()
282+
);
283+
}
284+
198285
#[test]
199286
fn analyzer_corpus_coverage_is_recorded_not_assumed() {
200287
// Rendering coverage is total; analyzer-corpus coverage is not, and the
@@ -212,7 +299,11 @@ fn analyzer_corpus_coverage_is_recorded_not_assumed() {
212299
.filter(|c| {
213300
c.get("analyzer_corpus")
214301
.and_then(Value::as_bool)
215-
.unwrap_or(false)
302+
// `expect`, not `unwrap_or(false)`: every other accessor in this
303+
// file treats a missing key as a broken fixture, and defaulting
304+
// here would silently under-count coverage if the generator ever
305+
// stopped emitting the flag.
306+
.expect("case 'analyzer_corpus'")
216307
})
217308
.count();
218309
assert_eq!(

‎tests/test_diag_ledger_fixtures.py‎

Lines changed: 90 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -26,14 +26,18 @@
2626
reach. Silently rendering all 47 and calling it "full parity" would overstate
2727
the evidence, so the coverage flag is carried per code and summarised.
2828
29-
## The three failure modes it closes
29+
## The failure modes it closes
3030
3131
* **missing** — a code in `TITLES` with no case. The replay fails naming it, so
3232
adding a diagnostic without a fixture is a red build.
3333
* **orphan** — a case whose code is not in `TITLES`. Catches a removed or
3434
renamed code leaving a fixture behind.
3535
* **stale** — the generated file differs from what the current reference
3636
produces (the standing `--write` discipline, as in the other two slices).
37+
* **unstable under insertion** — adding one vocabulary member must rewrite no
38+
existing record, add exactly one, and remove none. This is P-022 rule 4's
39+
normative acceptance, enforced by [`_insertion_effect`]; the three numbers fail
40+
in different directions and none implies the others.
3741
3842
Run: python tests/test_diag_ledger_fixtures.py (verify)
3943
python tests/test_diag_ledger_fixtures.py --write (regenerate)
@@ -47,6 +51,7 @@
4751
import os
4852
import re
4953
import sys
54+
from typing import NamedTuple
5055

5156
sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
5257

@@ -218,44 +223,109 @@ def run() -> int:
218223
f"{orphans}")
219224
return 1
220225

221-
churn = _insertion_churn()
222-
if churn:
223-
print(f"FAIL: inserting one code rewrote {churn} existing ledger record(s). "
224-
f"Shapes must derive from the code itself, never from its position in "
225-
f"the sorted vocabulary — adding a family has to be a ONE-record diff, "
226-
f"or a real renderer change hides in the churn")
226+
effect = _insertion_effect()
227+
if effect.churn:
228+
print(f"FAIL: inserting one code rewrote {effect.churn} existing ledger "
229+
f"record(s); rule 4 requires 0. Shapes must derive from the code itself, "
230+
f"never from its position in the sorted vocabulary — adding a family has "
231+
f"to be a ONE-record diff, or a real renderer change hides in the churn")
232+
return 1
233+
if effect.added != 1:
234+
print(f"FAIL: inserting one code produced {effect.added} new ledger record(s); "
235+
f"rule 4 requires exactly 1. Zero means the generator does not cover the "
236+
f"new family at all — the ledger would go quietly incomplete. More than "
237+
f"one means a record count is derived from the vocabulary's SIZE, which "
238+
f"is the churn defect wearing a different hat")
239+
return 1
240+
if effect.removed:
241+
print(f"FAIL: inserting one code REMOVED {effect.removed} existing ledger "
242+
f"record(s); rule 4 requires 0. A delta is a gain: a generator that adds "
243+
f"the new record while dropping another has not grown, and the dropped "
244+
f"record leaves the churn comparison entirely, so neither of the other "
245+
f"two numbers can see it")
227246
return 1
228247

229248
totals = data["totals"]
230249
print(f"diagnostic ledger OK: {totals['codes']}/{len(TITLES)} codes covered "
231250
f"({totals['by_family']}), "
232251
f"{totals['analyzer_corpus']} of them also produced by the .own corpus sweep; "
233-
f"a new code rewrites {churn} existing record(s)")
252+
f"inserting a code rewrites {effect.churn} existing record(s), "
253+
f"adds {effect.added} and removes {effect.removed}")
234254
return 0
235255

236256

237-
def _insertion_churn() -> int:
238-
"""How many EXISTING records change when one new code joins the vocabulary.
257+
class _InsertionEffect(NamedTuple):
258+
"""What inserting one vocabulary member does to the generated ledger.
259+
260+
P-022 rule 4 states the acceptance as:
239261
240-
Must be zero. This is the property that makes the ledger readable: the whole
241-
point is that adding a diagnostic family shows up as a single new record a
242-
reviewer can check, not as a wall of unrelated edits. An index-derived shape
243-
scored 42 of 47 here before the seed was made position-independent.
262+
insert one synthetic vocabulary member
263+
existing-record churn == 0
264+
new-record delta == 1
244265
245-
A probe code is inserted mid-vocabulary (so it shifts sorted positions) and
246-
removed again in a `finally`, so the live TITLES the rest of the suite sees
247-
is untouched."""
266+
`added` and `removed` are both halves of what that second line *means*: a
267+
delta is a gain, and a run that adds one record while quietly dropping
268+
another has gained nothing. Counting only additions would accept exactly
269+
that, so the measurement is split rather than the norm widened.
270+
"""
271+
272+
churn: int
273+
"""EXISTING records whose content changed. Must be 0."""
274+
added: int
275+
"""Records that did not exist before. Must be exactly 1."""
276+
removed: int
277+
"""Records that existed before and are now gone. Must be 0."""
278+
279+
280+
def _insertion_effect() -> _InsertionEffect:
281+
"""Measure rule 4's acceptance against the live generator.
282+
283+
`churn` is the half that was paid for: an index-derived shape scored 42 of
284+
47 here before the seed was made position-independent. Adding a family has
285+
to read as a single new record, or a real renderer change hides in the noise.
286+
287+
`added` is the other half, and it fails in the opposite direction. Zero means
288+
the generator did not emit the new family at all — the ledger goes quietly
289+
incomplete, which is precisely the gap `churn` alone cannot see, because a
290+
generator that skips the new code has perfect churn. Above one means some
291+
record count is derived from the vocabulary's *size* rather than its members,
292+
which is the churn defect in different clothing.
293+
294+
`removed` closes the hole the other two leave between them. A generator
295+
refactored to cap the ledger at its previous size emits the probe *and*
296+
drops an existing code: the dropped record is absent from `after`, so it
297+
never enters the churn comparison, and the addition still counts one. Both
298+
original numbers read clean while the ledger did not grow at all — measured,
299+
not hypothesised (probe added, `OWN052` silently gone).
300+
301+
None of the three is the mechanism. A content hash satisfies them today; any
302+
stable, cross-process-reproducible mapping that holds the acceptance conforms,
303+
and swapping it is not a violation.
304+
305+
The probe is inserted **mid-vocabulary**, not appended: `DI999` sorts at
306+
position 5 of 48, between `DI005` and `EFF001`, shifting the sorted index of
307+
the remaining 42 codes — which is what makes an index-derived shape score
308+
churn 42 here. It is removed again in a `finally`, so the live TITLES the
309+
rest of the suite sees is untouched."""
248310
probe = "DI999"
249311
if probe in TITLES: # pragma: no cover - defensive
250-
return 0
312+
raise AssertionError(
313+
f"{probe} is a real diagnostic code now, so it cannot act as the "
314+
f"insertion probe; pick an unused code rather than letting this "
315+
f"check silently pass"
316+
)
251317
before = {case["code"]: case for case in build()["cases"]}
252318
TITLES[probe] = "ledger insertion probe (not a real diagnostic)"
253319
try:
254320
after = {case["code"]: case for case in build()["cases"]}
255321
finally:
256322
del TITLES[probe]
257-
shared = set(before) & set(after) - {probe}
258-
return sum(1 for code in shared if before[code] != after[code])
323+
shared = (set(before) & set(after)) - {probe}
324+
return _InsertionEffect(
325+
churn=sum(1 for code in shared if before[code] != after[code]),
326+
added=len(set(after) - set(before)),
327+
removed=len(set(before) - set(after)),
328+
)
259329

260330

261331
if __name__ == "__main__":

0 commit comments

Comments
 (0)