Repository navigation
Commit e40e4da
fix: route voting/rescore through safe_compare_pred (Codex audit #2-4)
Three same-pattern findings: pred/gold execution exceptions get swallowed
into empty rows, then `compare_results` direct-call blesses empty-empty
as match=True. Mirrors of the qid 518 bug surface area:
#2 `scripts/merge_voting_rescues.py:73` — flipped baseline match=True from
stored alt_match booleans without re-executing SQL. Pre-fix voting
JSONs (with stale empty-empty match=True) could silently inflate EA.
Fix: default `--reverify` re-executes pred+gold via safe_compare_pred
(pred_failed + gold_failed flags) and rejects unverified candidates.
`--no-reverify` escape hatch retained for trusted legacy merges.
#3 `scripts/run_helallao_voting.py:189` — pred execution exceptions sank
alt_rows to [], then compare_results called directly. Fix: track
pred_failed + gold_failed via _execute_gold_with_status, route through
safe_compare_pred.
#4 `scripts/rescore_arcwise.py:127` — corrected-gold execution exceptions
sank gold_rows to [] then safe_compare_pred(pred_failed=...) — but
gold_failed wasn't tracked. Fix: switch to _execute_gold_with_status,
pass gold_failed through to safe_compare_pred.
Tests:
- tests/scripts/test_merge_voting_rescues.py: 4 new tests (reverify
rejects pred-fail / accepts real match / rejects gold-fail / no-reverify
flag skips path)
- tests/scripts/test_rescore_arcwise.py: 2 existing tests updated to
monkeypatch _execute_gold_with_status (3-tuple return)
Gates: ruff check ✓ · ruff format --check ✓ · mypy strict ✓ · 351 tests
pass (was 347, +4 new). Existing voting JSONs on disk are not retroactively
re-scored by this commit — historical baselines stay as-committed; only
new merges through merge_voting_rescues benefit from the gate.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>1 parent ebf0fb3 commit e40e4da
5 files changed
Lines changed: 317 additions & 14 deletions
File tree
- scripts
- tests/scripts
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
32 | 92 | | |
33 | 93 | | |
34 | 94 | | |
35 | 95 | | |
36 | 96 | | |
37 | 97 | | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
38 | 108 | | |
39 | 109 | | |
40 | 110 | | |
| |||
62 | 132 | | |
63 | 133 | | |
64 | 134 | | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
65 | 138 | | |
| 139 | + | |
66 | 140 | | |
| 141 | + | |
67 | 142 | | |
68 | 143 | | |
69 | 144 | | |
70 | 145 | | |
71 | | - | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
72 | 162 | | |
73 | 163 | | |
74 | 164 | | |
| |||
113 | 203 | | |
114 | 204 | | |
115 | 205 | | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
116 | 211 | | |
117 | 212 | | |
118 | 213 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
38 | | - | |
| 38 | + | |
39 | 39 | | |
40 | 40 | | |
41 | 41 | | |
| |||
117 | 117 | | |
118 | 118 | | |
119 | 119 | | |
| 120 | + | |
120 | 121 | | |
121 | | - | |
| 122 | + | |
122 | 123 | | |
123 | 124 | | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
124 | 129 | | |
125 | 130 | | |
| 131 | + | |
126 | 132 | | |
127 | 133 | | |
128 | | - | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
129 | 139 | | |
130 | 140 | | |
131 | 141 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
33 | | - | |
34 | | - | |
| 33 | + | |
| 34 | + | |
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
| |||
180 | 180 | | |
181 | 181 | | |
182 | 182 | | |
| 183 | + | |
183 | 184 | | |
184 | 185 | | |
185 | 186 | | |
| |||
190 | 191 | | |
191 | 192 | | |
192 | 193 | | |
| 194 | + | |
| 195 | + | |
193 | 196 | | |
194 | | - | |
| 197 | + | |
| 198 | + | |
195 | 199 | | |
196 | | - | |
| 200 | + | |
197 | 201 | | |
198 | 202 | | |
199 | 203 | | |
200 | 204 | | |
201 | | - | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
202 | 213 | | |
203 | 214 | | |
204 | 215 | | |
| |||
0 commit comments