You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit ebf0fb3
Browse filesBrowse the repository at this point in the historyBrowse files
Mirror of the qid 518 pred-side bug already fixed in c74b46c, but on the
gold side. `_execute_gold` historically returned `([], [])` when BIRD's
gold SQL crashed (both `execute_readonly` and the raw-connection fallback
raised SQLAlchemyError/MemoryError — ~1% of BIRD gold SQLs). If a pred
happened to also return zero rows (e.g. `SELECT * WHERE 1=0`, or a wrong
filter that filtered out everything), `compare_results([], [])` blessed
match=True under BIRD set-equality scoring.
Changes:
- `_execute_gold_with_status` returns `(rows, cols, gold_failed: bool)`;
the legacy 2-tuple `_execute_gold` is retained as a thin wrapper so the
dozen+ scripts that import it keep working (their migration is Codex
audit #2-4, separate commit).
- `_compare_outcome` accepts a `gold_failed` kwarg and short-circuits to
`match=False, reason='gold execution failed'` when set.
- `safe_compare_pred` gains a symmetric `gold_failed: bool = False` kwarg
for voting/rescoring scripts that bypass the runner.
- All 3 runner-internal callsites (run_config_a / run_config_c-via-pipeline /
self-consistency voter) now route via `_execute_gold_with_status` and
pass the flag through to `_compare_outcome`.
- Regression tests:
- `tests/eval/test_metrics.py::TestSafeComparePred::test_gold_failed_*` (2)
- `tests/eval/test_runner.py::test_run_config_a_gold_fail_with_empty_pred_does_not_false_positive`
Gates: ruff check ✓ · ruff format --check ✓ · mypy strict ✓ · 347 tests
pass (was 344, +3 new). No behaviour change for the 99% of BIRD gold SQLs
that execute cleanly; only the gold-fail + empty-pred quirk path closes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
0 commit comments