Skip to content

Commit a7c617f

Browse files
JuliaEdomclaude
andcommitted
fix: rescore_arcwise transitions use fresh original_match (Codex #7)
`transitions["gained"]` / `transitions["lost"]` compared the variant match against the STORED `rec["match"]` from the input JSON instead of the FRESHLY re-executed `out_entry["original_match"]` written by the variant loop. On records where the two disagree, transitions would be computed under hybrid scoring semantics (stored EA bits mixed with current scoring). Reachability verified 2026-05-26 EOD-6: 0/200 stale-vs-fresh disagreements on `eval/reports/2026-05-24/v29-arcwise-rescored.json` (canonical: gained=7, lost=91 — unchanged by this fix). Future-proofing change with no observable difference in committed reports. Codex audit 2026-05-25 #7, reachability analysis in `docs/NEXT_SESSION.md` Open Audit Items table. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent c7a2ee0 commit a7c617f

1 file changed

Lines changed: 12 additions & 2 deletions

File tree

‎scripts/rescore_arcwise.py‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -147,6 +147,16 @@ def main() -> int:
147147
per_diff[variant][difficulty][0] += int(is_match)
148148

149149
# Transitions vs sql_only and vs full.
150+
# Use the FRESHLY-re-executed original-variant match (written at
151+
# `out_entry["original_match"]` by the variant loop above) rather
152+
# than the stale `orig_match` snapshot from the input JSON — the
153+
# whole point of rescore is to apply current scoring semantics, so
154+
# mixing in stored EA bits would produce a hybrid that doesn't
155+
# match either run cleanly (Codex audit 2026-05-25 #7). On the
156+
# current canonical v22-v30 baselines the two values agree on
157+
# every record, so this is a future-proofing change; no
158+
# observable difference in committed reports.
159+
fresh_orig = bool(out_entry.get("original_match", orig_match))
150160
for variant in ("sql_only", "full"):
151161
v_match = out_entry.get(f"{variant}_match")
152162
if v_match is None:
@@ -158,11 +168,11 @@ def main() -> int:
158168
)
159169
if gold_changed:
160170
out_entry[f"{variant}_gold_changed"] = True
161-
if orig_match and not v_match:
171+
if fresh_orig and not v_match:
162172
transitions["lost"].append(
163173
{"qid": qid, "variant": variant, "difficulty": difficulty}
164174
)
165-
elif (not orig_match) and v_match:
175+
elif (not fresh_orig) and v_match:
166176
transitions["gained"].append(
167177
{"qid": qid, "variant": variant, "difficulty": difficulty}
168178
)

0 commit comments

Comments
 (0)