fix: semantic drift within min_similarity no longer fails runs; tool evaluators skip tool-free suites

[babaliauskas]

What this changes

• Semantic gate. A semantic row whose similarity stays above min_similarity (it carries no SEMANTIC_REGRESSION flag) now enters the paired significance test as a delta of 0. That is the line the policy budgets already drew, and both now read it from one predicate: analysis/tolerance.py::within_tolerance. delta_avg_score, the slice aggregates and the per-example rows stay raw. A comparison with forgiven rows carries the note k of n rows within the evaluator's own threshold (min_similarity) — tested as no change.
• A fail now says why.
  • When a run fails only because a blocking comparison is critical or high, decision.reason now names each such comparison. The report hero no longer shows "7 of 7 budgets within policy" under FAIL.
  • The templated recommendation names the actual cause: breached overall or slice budgets, blocking evaluators on the overall slice, or a generic pointer. It no longer always claims that a budget was breached.
  • The summary's critical count is now counted from the blocking regressions.
• Tool-free suites.
  • Tool evaluators (including a top-level trace_invariants that every suite inherits) are not attempted on a pair where both calls are error-free and neither has a tool trace. Such a pair gets no row and no coverage entry, so it no longer demotes the run to conditional_pass.
  • evaluate warns once that the evaluator checked nothing because no example offered tools, and compare now shows that warning too.
  • Errored pairs keep their error rows, and a trace on only one side is still booked as unmeasured.
• Docs: docs/methodology.md, docs/configuration.md, docs/agents.md, DOCS.md, llms-full.txt, and CHANGELOG.md under [Unreleased].

No changes to the config, suite, state.json, scores.jsonl or bundle schemas. ComparisonResult has no new field; the new information goes in notes.

Why

Run r_20261011_triage_e14cda (suite triage, anthropic/claude-sonnet-5 → anthropic/claude-sonnet-5-5, CLI 1.4.1) came out FAIL, but nothing in it had regressed. Every budget passed and every semantic.cosine row was above min_similarity: 0.9 (the lowest was 0.904). The semantic evaluator pins the source score at 1.0, so any rewording produces a small negative delta. Nine small drops that all go the same way have almost no variance, so the paired t-test on the raw deltas called them critical (adjusted p 0.0004, d −2.15), and that failed the run.

Fixing that alone would still have left the run at conditional_pass. The top-level agent_contract invariants had been written for another suite's tools, and they booked every pair in this tool-free suite as unmeasured.

Before → after for this run: FAIL → PASS. Re-analysing the existing scores already gives semantic.cosine severity none (Δ −0.0524, note "9 of 9 rows within…"). Clearing the stale agent_contract coverage needs a fresh evaluate, which is still pending: it needs an embedding key.

Accepted consequence: if only a few rows in a comparison are flagged and many are forgiven, the comparison may not reach significance on its own. The rate budgets (max_overall_regression_rate, min_equivalence_rate) then decide, which is what docs/methodology.md already described. To pick up the tool-free fix, an old run needs evaluate re-run; analyze alone is not enough.

Follow-ups, deliberately out of scope:

• max_critical_regressions is inert. _metrics reads a metadata["severity"] key that nothing writes, and fixing it is a cross-repo change with evalshift-server.
• When a slice budget alone fails the run, the hero still shows "N of N budgets within policy".
• The LLM insight prompt does not receive failed_budgets.
• compare still swallows evaluate's other warnings that only print outside quiet mode.
• severity_floor is written but never read.
• tool_arguments has the same pinned-source shape as semantic.
• When samples_per_example > 1, a reduced row can carry a semantic flag that disagrees with its mean score.

Test plan

• pytest passes: 3035 passed, 94.95% coverage (make ci)
• ruff check, ruff format --check clean
• mypy --strict src/evalshift_cli clean
• CHANGELOG.md updated under ## [Unreleased]
• (If user-visible) docs updated
• Re-run evaluate → analyze → report on r_20261011_triage_e14cda and confirm pass, the agent_contract "checked nothing" warning, and that no agent_contract comparison remains

New tests pin these cases:

• Nine forgiven rows test as none, and nine flagged rows still test as critical.
• One flagged row among forgiven ones runs through Wilcoxon without warnings, and its effect size and CI come from the adjusted deltas.
• Legacy rows with an empty kind, and renamed semantic evaluators, are both forgiven.
• For the pipeline as a whole, the triage-run regression passes, and the same drop below min_similarity still fails.
• A budget breach plus a critical comparison keeps reason = None and names the budget.
• A slice-budget fail names the slice budget.
• For tool-free pairs: an errored pair keeps its error row, a one-sided trace stays unmeasured, and an applies_to miss keeps its own warning wording.

🤖 Generated with Claude Code