Skip to content

Commit f7d5f6f

Browse files
JuliaEdomclaude
andcommitted
refactor(eval): freeze 9 voting scripts on the unsafe comparator into scripts/archive/
Nine voting/rescore scripts compare results with the row-level compare_results, which scores an exec-failed prediction as a match when the gold result is empty (the "qid 518" class defect). Voting is parked, so rather than migrate them they are frozen under scripts/archive/ with a README stating the defect and pointing new work at safe_compare_pred. scripts/check_no_raw_compare.py is a CI gate (AST-based, so docstring mentions don't trip it) that fails if any non-archived script reuses compare_results; eval_demo.py is an explicit, documented allowlist (a demo, not a vote-merging path). Test import paths updated to scripts.archive.*; tests/scripts stays green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent ad35531 commit f7d5f6f

15 files changed

Lines changed: 128 additions & 6 deletions

‎scripts/archive/README.md‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# scripts/archive — FROZEN eval scripts, do not use
2+
3+
The nine scripts in this directory are frozen. They score predictions with the
4+
raw `compare_results` comparator from `nl_sql.eval.metrics.execution_accuracy`,
5+
which has a known defect: a prediction whose execution **failed** can still be
6+
scored as a match when the gold query returns an **empty** result (the
7+
"qid 518" class of false positives). Any accuracy numbers produced by these
8+
scripts are suspect for that case.
9+
10+
Do not run or import these scripts for new eval work, and do not copy their
11+
comparison logic. For rescoring, use the safe path instead:
12+
13+
- `scripts/audit_rescore.py`, which uses
14+
`nl_sql.eval.metrics.execution_accuracy.safe_compare_pred` (treats an
15+
exec-failed prediction as a non-match regardless of the gold result).
16+
17+
Frozen files:
18+
19+
- `archive_sweep.py`
20+
- `ensemble_vote.py`
21+
- `run_critique_retry.py`
22+
- `run_groq_voting.py`
23+
- `run_openrouter_voting.py`
24+
- `run_planner_eval.py`
25+
- `run_selfcon_retry.py`
26+
- `run_sonnet_voting.py`
27+
- `run_wide_schema_retry.py`
28+
29+
They are kept only as a record of the parked voting/retry experiments. If one
30+
of them is ever needed again, migrate it to `safe_compare_pred` first.

‎scripts/archive/__init__.py‎

Whitespace-only changes.

0 commit comments

Comments
 (0)