fix: count failure categories per example; key the narrative cache on its facts

[babaliauskas]

What this changes

Two report-correctness fixes found while reading a hosted run (r_20261013_converter_ab3ce8).

1. Failure categories count examples, not rows (7ff042f)

• _failure_categories (analysis/policy.py) counted every scored row carrying a label.
  • tool_selection stamps TOOL_SELECTION_DRIFT on both its conformance row and its divergence row.
  • So one example that picked another tool showed up twice: "Different tools chosen (2 examples)" in the narrative, and 2 in Top regression causes.
• Each label is now counted once per distinct (prompt_id, example_id). This is the rule INVARIANT_VIOLATION already followed.
• INVARIANT_VIOLATION keeps its narrower blocking-only count.

2. The narrative cache is keyed on the facts it describes (2142d0f)

• insights.json was keyed on state.config_hash plus the insights model.
  • state.config_hash is fixed when the run is created.
  • evaluate and analyze re-read evalshift.yaml on every call.
  • So any re-score or re-analysis of the same run served the old paragraph next to new figures.
• The key now also includes a digest of the Facts block, which is built from the scores, the analysis and the persisted decision.
  • A change to any of them regenerates the narrative on the next report.
  • A plain re-render still reuses the cache.
• Existing caches miss once and regenerate, which costs one model call per run.
• Not covered: push still ships whatever the last report cached. Running analyze and then push with no report in between uploads the previous paragraph. This is documented, not changed.

Docs: DOCS.md, docs/configuration.md, llms-full.txt and CHANGELOG (Unreleased → Fixed).

No change to the config, suite, state.json, scores.jsonl, insights.json envelope or bundle schemas.

Why

• Fix 1: a converter run with one regressed example (cap_20587…, the target called convert_units km→kg where the source refused) reported TOOL_SELECTION_DRIFT · 2.
• Fix 2: reproduced without a CLI upgrade.
  1. Loosen max_latency_increase from 0.30 to 0.60 on a run that failed only on latency.
  2. Run analyze + report on that same run.
  3. migration_decision.json says pass, but the HTML narrative still says "FAIL under the configured policy. 7 of 8 budgets passed…".
• Fix 2 affects any flow that re-runs a stage on an existing run directory:
  • tuning a budget;
  • editing an evaluator;
  • exporting a missing judge or embedding key, then running evaluate again;
  • upgrading the CLI.
• Fresh compare and CI runs were unaffected.

Verified locally (CLI from this branch, on copies of testApp runs)

Check	Before	After
r_20261012_converter_c80f30: decision + HTML Top regression causes	Different tools chosen 2	1
r_20261012_planner_664d3c (control: 3 drift rows on 3 examples)	3	3
Converter, templated narrative (fallback_insight)	"Different tools chosen (2 examples)"	"Meaning of the answer changed (1 examples)" (now a 1–1 tie)
Converter: cached narrative, unchanged run, plain report	reused	reused, no API key needed
Converter: cached narrative, budget loosened, analyze + report	stale "FAIL…" beside a PASS verdict	cache miss, stale paragraph gone
Facts digest under PYTHONHASHSEED 0/1/2/3/random	n/a	identical (the cache is stable across processes)

Gates:

• ruff check, ruff format --check and mypy --strict are clean.
• pytest: 3095 passed, coverage 95.03%.
• The pre-push CI mirror passed.

New tests:

• TestFailureCategoryCountsExamples::test_one_example_flagged_by_two_axes_counts_once
• test_a_reanalysed_decision_invalidates_the_cache
• test_rescored_records_invalidate_the_cache

All three failed before the change.

Not in this PR

• Pre-existing test: tests/unit/test_cli_capture_remote.py::test_fetch_without_config_file_errors depends on terminal width. Rich wraps "not found" across lines when the temp path is long. It fails at the default width on main too and passes at COLUMNS=200.
• Plural: "(1 examples)" in insights/templates.py:147.
• Hosted client:
  • The advisory bar shows a bare "Regressed 100%" for 1 of 1 judge score.
  • The per-example "critical" badge is a display threshold (|Δ| > 0.1), not the statistical severity the verdict counts.

🤖 Generated with Claude Code