Repository navigation
Commit f7d5f6f
refactor(eval): freeze 9 voting scripts on the unsafe comparator into scripts/archive/
Nine voting/rescore scripts compare results with the row-level compare_results,
which scores an exec-failed prediction as a match when the gold result is empty
(the "qid 518" class defect). Voting is parked, so rather than migrate them they
are frozen under scripts/archive/ with a README stating the defect and pointing
new work at safe_compare_pred.
scripts/check_no_raw_compare.py is a CI gate (AST-based, so docstring mentions
don't trip it) that fails if any non-archived script reuses compare_results;
eval_demo.py is an explicit, documented allowlist (a demo, not a vote-merging
path). Test import paths updated to scripts.archive.*; tests/scripts stays green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>1 parent ad35531 commit f7d5f6f
15 files changed
Lines changed: 128 additions & 6 deletions
File tree
- scripts
- archive
- tests/scripts
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
Whitespace-only changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
0 commit comments