Skip to content

Commit 03ad6ae

Browse files
JuliaEdomclaude
andcommitted
chore+fix: ruff format pass + regenerate stale baseline-summary headers
Closes Kimi audit P1.1 (CI format gate red since 071e385) and Codex audit #5/#6 (stale `overall.ea`/`matched` in 8 merged baseline reports; README v29 row/lift-trace endpoint advertising pre-audit 93.0% while headline claims audit-corrected 92.5%). Changes: - ruff format applied to 15 files (scripts/{archive_sweep,audit_rescore, rescore_arcwise,run_openrouter_voting,run_selfcon_retry, run_wide_schema_retry}.py, src/.../generate_sql.py, src/.../ execution_accuracy.py, 7 test files); 333 tests still green - scripts/refresh_baseline_summary.py: regenerates `overall.ea`/ `overall.matched`/`overall.n` from `records[]`; idempotent - tests/scripts/test_refresh_baseline_summary.py: 4 tests, including a regression guard that walks v22-v29 merged JSONs and asserts the summary block matches the per-record truth - eval/reports/2026-05-2[34]/v2[2-9]*-merged.json: 8 files regenerated (each -1/-0.005 — qid 518 v13 false positive surgically patched in c74b46c but not propagated to summary headers) - README.md: lift-trace endpoint and v29 table row now show audit- corrected 92.5% with explicit `safe_compare_pred` provenance and a new row documenting the 2026-05-25 audit correction - audit_kimi_25_05_26.md: independent Kimi audit captured at HEAD 071e385 Gates: ruff check ✓ · ruff format --check ✓ · mypy strict ✓ · 337 tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 071e385 commit 03ad6ae

27 files changed

Lines changed: 749 additions & 150 deletions

‎README.md‎

Lines changed: 3 additions & 2 deletions
Large diffs are not rendered by default.

‎audit_kimi_25_05_26.md‎

Lines changed: 404 additions & 0 deletions
Large diffs are not rendered by default.

‎eval/reports/2026-05-23/v22-v21-plus-p3f-207-1404-merged.json‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@
22
"configuration": "G_hybrid+multi-vote+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+p3f-targeted-hints",
33
"sql_model": "codestral+Sonnet challenging+gpt-oss-120b/20b voting+llama4-scout voting + meta-llama/llama-4-scout-17b-16e-instruct + qwen/qwen3-32b + codestral+grounded_critique + codestral+self-consistency + perplexity:claude-sonnet-4-6 + codestral+grounded_critique + groq:llama-3.3-70b-versatile+grounded_critique+fewshot3 + groq:qwen/qwen3-32b+grounded_critique+fewshot3 + openai/gpt-oss-20b + mistral:codestral-latest+grounded_critique+fewshot3 + mistral:codestral-latest+grounded_critique+fewshot3 + helallao:gpt-5.2 + helallao:grok-4.1 + helallao:gpt-5.2-thinking + helallao:grok-4.1-reasoning + helallao:kimi-k2-thinking + helallao:gpt-5.2 + helallao:kimi-k2-thinking + helallao:gpt-5.2-thinking + helallao:gpt-5.2 + helallao:claude-4.5-sonnet-thinking + helallao:kimi-k2-thinking + orchestrator-browser:claude-sonnet-4-6:ultrashort-birdgrain + config-c-p3f-schema-link-hints",
44
"overall": {
5-
"ea": 0.89,
5+
"ea": 0.885,
66
"n": 200,
7-
"matched": 178,
7+
"matched": 177,
88
"rescued_via_voting": 64
99
},
1010
"records": [
@@ -1449,7 +1449,7 @@
14491449
"pred_row_count": 4,
14501450
"gold_row_count": 1,
14511451
"comparison_reason": "row count mismatch: gold=1, pred=4",
1452-
"audit_note": "BIRD-official set-semantics audit (compare_results Counter\u2192set, see commit notes)"
1452+
"audit_note": "BIRD-official set-semantics audit (compare_results Counter→set, see commit notes)"
14531453
},
14541454
{
14551455
"question_id": 366,

‎eval/reports/2026-05-23/v23-v22-plus-archive-1205-merged.json‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@
22
"configuration": "G_hybrid+multi-vote+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+p3f-targeted-hints+archive-sweep",
33
"sql_model": "codestral+Sonnet challenging+gpt-oss-120b/20b voting+llama4-scout voting + meta-llama/llama-4-scout-17b-16e-instruct + qwen/qwen3-32b + codestral+grounded_critique + codestral+self-consistency + perplexity:claude-sonnet-4-6 + codestral+grounded_critique + groq:llama-3.3-70b-versatile+grounded_critique+fewshot3 + groq:qwen/qwen3-32b+grounded_critique+fewshot3 + openai/gpt-oss-20b + mistral:codestral-latest+grounded_critique+fewshot3 + mistral:codestral-latest+grounded_critique+fewshot3 + helallao:gpt-5.2 + helallao:grok-4.1 + helallao:gpt-5.2-thinking + helallao:grok-4.1-reasoning + helallao:kimi-k2-thinking + helallao:gpt-5.2 + helallao:kimi-k2-thinking + helallao:gpt-5.2-thinking + helallao:gpt-5.2 + helallao:claude-4.5-sonnet-thinking + helallao:kimi-k2-thinking + orchestrator-browser:claude-sonnet-4-6:ultrashort-birdgrain + config-c-p3f-schema-link-hints + archive-sweep",
44
"overall": {
5-
"ea": 0.895,
5+
"ea": 0.89,
66
"n": 200,
7-
"matched": 179,
7+
"matched": 178,
88
"rescued_via_voting": 65
99
},
1010
"records": [
@@ -1449,7 +1449,7 @@
14491449
"pred_row_count": 4,
14501450
"gold_row_count": 1,
14511451
"comparison_reason": "row count mismatch: gold=1, pred=4",
1452-
"audit_note": "BIRD-official set-semantics audit (compare_results Counter\u2192set, see commit notes)"
1452+
"audit_note": "BIRD-official set-semantics audit (compare_results Counter→set, see commit notes)"
14531453
},
14541454
{
14551455
"question_id": 366,

‎eval/reports/2026-05-23/v24-v23-plus-archive-rescore-959-merged.json‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@
22
"configuration": "G_hybrid+multi-vote+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+p3f-targeted-hints+archive-sweep+archive-rescore",
33
"sql_model": "codestral+Sonnet challenging+gpt-oss-120b/20b voting+llama4-scout voting + meta-llama/llama-4-scout-17b-16e-instruct + qwen/qwen3-32b + codestral+grounded_critique + codestral+self-consistency + perplexity:claude-sonnet-4-6 + codestral+grounded_critique + groq:llama-3.3-70b-versatile+grounded_critique+fewshot3 + groq:qwen/qwen3-32b+grounded_critique+fewshot3 + openai/gpt-oss-20b + mistral:codestral-latest+grounded_critique+fewshot3 + mistral:codestral-latest+grounded_critique+fewshot3 + helallao:gpt-5.2 + helallao:grok-4.1 + helallao:gpt-5.2-thinking + helallao:grok-4.1-reasoning + helallao:kimi-k2-thinking + helallao:gpt-5.2 + helallao:kimi-k2-thinking + helallao:gpt-5.2-thinking + helallao:gpt-5.2 + helallao:claude-4.5-sonnet-thinking + helallao:kimi-k2-thinking + orchestrator-browser:claude-sonnet-4-6:ultrashort-birdgrain + config-c-p3f-schema-link-hints + archive-sweep + archive-rescore",
44
"overall": {
5-
"ea": 0.9,
5+
"ea": 0.895,
66
"n": 200,
7-
"matched": 180,
7+
"matched": 179,
88
"rescued_via_voting": 66
99
},
1010
"records": [
@@ -1449,7 +1449,7 @@
14491449
"pred_row_count": 4,
14501450
"gold_row_count": 1,
14511451
"comparison_reason": "row count mismatch: gold=1, pred=4",
1452-
"audit_note": "BIRD-official set-semantics audit (compare_results Counter\u2192set, see commit notes)"
1452+
"audit_note": "BIRD-official set-semantics audit (compare_results Counter→set, see commit notes)"
14531453
},
14541454
{
14551455
"question_id": 366,

‎eval/reports/2026-05-24/v25-v24-plus-p3f-q902-merged.json‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@
22
"configuration": "G_hybrid+multi-vote+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+p3f-targeted-hints+archive-sweep+archive-rescore+p3f-q902",
33
"sql_model": "codestral+Sonnet challenging+gpt-oss-120b/20b voting+llama4-scout voting + meta-llama/llama-4-scout-17b-16e-instruct + qwen/qwen3-32b + codestral+grounded_critique + codestral+self-consistency + perplexity:claude-sonnet-4-6 + codestral+grounded_critique + groq:llama-3.3-70b-versatile+grounded_critique+fewshot3 + groq:qwen/qwen3-32b+grounded_critique+fewshot3 + openai/gpt-oss-20b + mistral:codestral-latest+grounded_critique+fewshot3 + mistral:codestral-latest+grounded_critique+fewshot3 + helallao:gpt-5.2 + helallao:grok-4.1 + helallao:gpt-5.2-thinking + helallao:grok-4.1-reasoning + helallao:kimi-k2-thinking + helallao:gpt-5.2 + helallao:kimi-k2-thinking + helallao:gpt-5.2-thinking + helallao:gpt-5.2 + helallao:claude-4.5-sonnet-thinking + helallao:kimi-k2-thinking + orchestrator-browser:claude-sonnet-4-6:ultrashort-birdgrain + config-c-p3f-schema-link-hints + archive-sweep + archive-rescore + p3f-q902-driverstandings-hint",
44
"overall": {
5-
"ea": 0.905,
5+
"ea": 0.9,
66
"n": 200,
7-
"matched": 181,
7+
"matched": 180,
88
"rescued_via_voting": 67
99
},
1010
"records": [
@@ -1449,7 +1449,7 @@
14491449
"pred_row_count": 4,
14501450
"gold_row_count": 1,
14511451
"comparison_reason": "row count mismatch: gold=1, pred=4",
1452-
"audit_note": "BIRD-official set-semantics audit (compare_results Counter\u2192set, see commit notes)"
1452+
"audit_note": "BIRD-official set-semantics audit (compare_results Counter→set, see commit notes)"
14531453
},
14541454
{
14551455
"question_id": 366,

‎eval/reports/2026-05-24/v26-v25-plus-p3f-q1531-merged.json‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@
22
"configuration": "G_hybrid+multi-vote+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+p3f-targeted-hints+archive-sweep+archive-rescore+p3f-q902+p3f-q1531",
33
"sql_model": "codestral+Sonnet challenging+gpt-oss-120b/20b voting+llama4-scout voting + meta-llama/llama-4-scout-17b-16e-instruct + qwen/qwen3-32b + codestral+grounded_critique + codestral+self-consistency + perplexity:claude-sonnet-4-6 + codestral+grounded_critique + groq:llama-3.3-70b-versatile+grounded_critique+fewshot3 + groq:qwen/qwen3-32b+grounded_critique+fewshot3 + openai/gpt-oss-20b + mistral:codestral-latest+grounded_critique+fewshot3 + mistral:codestral-latest+grounded_critique+fewshot3 + helallao:gpt-5.2 + helallao:grok-4.1 + helallao:gpt-5.2-thinking + helallao:grok-4.1-reasoning + helallao:kimi-k2-thinking + helallao:gpt-5.2 + helallao:kimi-k2-thinking + helallao:gpt-5.2-thinking + helallao:gpt-5.2 + helallao:claude-4.5-sonnet-thinking + helallao:kimi-k2-thinking + orchestrator-browser:claude-sonnet-4-6:ultrashort-birdgrain + config-c-p3f-schema-link-hints + archive-sweep + archive-rescore + p3f-q902-driverstandings-hint",
44
"overall": {
5-
"ea": 0.91,
5+
"ea": 0.905,
66
"n": 200,
7-
"matched": 182,
7+
"matched": 181,
88
"rescued_via_voting": 68
99
},
1010
"records": [
@@ -1449,7 +1449,7 @@
14491449
"pred_row_count": 4,
14501450
"gold_row_count": 1,
14511451
"comparison_reason": "row count mismatch: gold=1, pred=4",
1452-
"audit_note": "BIRD-official set-semantics audit (compare_results Counter\u2192set, see commit notes)"
1452+
"audit_note": "BIRD-official set-semantics audit (compare_results Counter→set, see commit notes)"
14531453
},
14541454
{
14551455
"question_id": 366,

‎eval/reports/2026-05-24/v27-v26-plus-p3f-q894-q1251-merged.json‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@
22
"configuration": "G_hybrid+multi-vote+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+p3f-targeted-hints+archive-sweep+archive-rescore+p3f-q902+p3f-q1531+p3f-q894+p3f-q1251",
33
"sql_model": "codestral+Sonnet challenging+gpt-oss-120b/20b voting+llama4-scout voting + meta-llama/llama-4-scout-17b-16e-instruct + qwen/qwen3-32b + codestral+grounded_critique + codestral+self-consistency + perplexity:claude-sonnet-4-6 + codestral+grounded_critique + groq:llama-3.3-70b-versatile+grounded_critique+fewshot3 + groq:qwen/qwen3-32b+grounded_critique+fewshot3 + openai/gpt-oss-20b + mistral:codestral-latest+grounded_critique+fewshot3 + mistral:codestral-latest+grounded_critique+fewshot3 + helallao:gpt-5.2 + helallao:grok-4.1 + helallao:gpt-5.2-thinking + helallao:grok-4.1-reasoning + helallao:kimi-k2-thinking + helallao:gpt-5.2 + helallao:kimi-k2-thinking + helallao:gpt-5.2-thinking + helallao:gpt-5.2 + helallao:claude-4.5-sonnet-thinking + helallao:kimi-k2-thinking + orchestrator-browser:claude-sonnet-4-6:ultrashort-birdgrain + config-c-p3f-schema-link-hints + archive-sweep + archive-rescore + p3f-q902-driverstandings-hint",
44
"overall": {
5-
"ea": 0.92,
5+
"ea": 0.915,
66
"n": 200,
7-
"matched": 184,
7+
"matched": 183,
88
"rescued_via_voting": 70
99
},
1010
"records": [
@@ -1449,7 +1449,7 @@
14491449
"pred_row_count": 4,
14501450
"gold_row_count": 1,
14511451
"comparison_reason": "row count mismatch: gold=1, pred=4",
1452-
"audit_note": "BIRD-official set-semantics audit (compare_results Counter\u2192set, see commit notes)"
1452+
"audit_note": "BIRD-official set-semantics audit (compare_results Counter→set, see commit notes)"
14531453
},
14541454
{
14551455
"question_id": 366,

‎eval/reports/2026-05-24/v28-v27-plus-p3f-q408-merged.json‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@
22
"configuration": "G_hybrid+multi-vote+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+p3f-targeted-hints+archive-sweep+archive-rescore+p3f-q902+p3f-q1531+p3f-q894+p3f-q1251+p3f-q408",
33
"sql_model": "codestral+Sonnet challenging+gpt-oss-120b/20b voting+llama4-scout voting + meta-llama/llama-4-scout-17b-16e-instruct + qwen/qwen3-32b + codestral+grounded_critique + codestral+self-consistency + perplexity:claude-sonnet-4-6 + codestral+grounded_critique + groq:llama-3.3-70b-versatile+grounded_critique+fewshot3 + groq:qwen/qwen3-32b+grounded_critique+fewshot3 + openai/gpt-oss-20b + mistral:codestral-latest+grounded_critique+fewshot3 + mistral:codestral-latest+grounded_critique+fewshot3 + helallao:gpt-5.2 + helallao:grok-4.1 + helallao:gpt-5.2-thinking + helallao:grok-4.1-reasoning + helallao:kimi-k2-thinking + helallao:gpt-5.2 + helallao:kimi-k2-thinking + helallao:gpt-5.2-thinking + helallao:gpt-5.2 + helallao:claude-4.5-sonnet-thinking + helallao:kimi-k2-thinking + orchestrator-browser:claude-sonnet-4-6:ultrashort-birdgrain + config-c-p3f-schema-link-hints + archive-sweep + archive-rescore + p3f-q902-driverstandings-hint + p3f-q408-rulings-hint",
44
"overall": {
5-
"ea": 0.925,
5+
"ea": 0.92,
66
"n": 200,
7-
"matched": 185,
7+
"matched": 184,
88
"rescued_via_voting": 70
99
},
1010
"records": [
@@ -1449,7 +1449,7 @@
14491449
"pred_row_count": 4,
14501450
"gold_row_count": 1,
14511451
"comparison_reason": "row count mismatch: gold=1, pred=4",
1452-
"audit_note": "BIRD-official set-semantics audit (compare_results Counter\u2192set, see commit notes)"
1452+
"audit_note": "BIRD-official set-semantics audit (compare_results Counter→set, see commit notes)"
14531453
},
14541454
{
14551455
"question_id": 366,

‎eval/reports/2026-05-24/v29-v28-plus-p3f-q1275-merged.json‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@
22
"configuration": "G_hybrid+multi-vote+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+merged+p3f-targeted-hints+archive-sweep+archive-rescore+p3f-q902+p3f-q1531+p3f-q894+p3f-q1251+p3f-q408+p3f-q1275",
33
"sql_model": "codestral+Sonnet challenging+gpt-oss-120b/20b voting+llama4-scout voting + meta-llama/llama-4-scout-17b-16e-instruct + qwen/qwen3-32b + codestral+grounded_critique + codestral+self-consistency + perplexity:claude-sonnet-4-6 + codestral+grounded_critique + groq:llama-3.3-70b-versatile+grounded_critique+fewshot3 + groq:qwen/qwen3-32b+grounded_critique+fewshot3 + openai/gpt-oss-20b + mistral:codestral-latest+grounded_critique+fewshot3 + mistral:codestral-latest+grounded_critique+fewshot3 + helallao:gpt-5.2 + helallao:grok-4.1 + helallao:gpt-5.2-thinking + helallao:grok-4.1-reasoning + helallao:kimi-k2-thinking + helallao:gpt-5.2 + helallao:kimi-k2-thinking + helallao:gpt-5.2-thinking + helallao:gpt-5.2 + helallao:claude-4.5-sonnet-thinking + helallao:kimi-k2-thinking + orchestrator-browser:claude-sonnet-4-6:ultrashort-birdgrain + config-c-p3f-schema-link-hints + archive-sweep + archive-rescore + p3f-q902-driverstandings-hint + p3f-q408-rulings-hint + p3f-q1275-laboratory-vocab-hint",
44
"overall": {
5-
"ea": 0.93,
5+
"ea": 0.925,
66
"n": 200,
7-
"matched": 186,
7+
"matched": 185,
88
"rescued_via_voting": 70
99
},
1010
"records": [
@@ -1449,7 +1449,7 @@
14491449
"pred_row_count": 4,
14501450
"gold_row_count": 1,
14511451
"comparison_reason": "row count mismatch: gold=1, pred=4",
1452-
"audit_note": "BIRD-official set-semantics audit (compare_results Counter\u2192set, see commit notes)"
1452+
"audit_note": "BIRD-official set-semantics audit (compare_results Counter→set, see commit notes)"
14531453
},
14541454
{
14551455
"question_id": 366,

0 commit comments

Comments
 (0)