Repository navigation
Commit 7f5043f
fix(eval): sync machine-readable artifacts with the headline; add consistency gate
The audit found the machine and the README disagreeing:
- v31 report's top-level `overall` said 185/0.925 while its own records recomputed
to 188/0.94 — refreshed to 188/0.94 (records are the source of truth).
- GET /eval/latest hardcoded a stale 60.5% hybrid baseline. It now serves a new
committed reproducible baseline (eval/baselines/reproducible_n200.json: config E,
codestral, no voting, no hints = 57.5% EA / 115 of 200), i.e. the number this
API's own pipeline actually achieves.
- FastAPI description "57% hybrid" → honest reproducible/hint-assisted split.
- runtime.txt python-3.12 → 3.13 (CI/ruff/mypy already target 3.13).
New regression test pins the invariant that every canonical report's `overall`
equals the recomputation from its records[], so summaries can't silently drift again.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>1 parent 742647e commit 7f5043f
5 files changed
Lines changed: 6943 additions & 6 deletions
File tree
- eval
- baselines
- reports/2026-05-26
- src/nl_sql/api
- tests/eval
0 commit comments