eval+docs: v29 residue saturation evidence — 3-model helallao reasoning sweep, 0 rescues / 42 attempts

NEXT_SESSION priority #1 ("paid OpenRouter top-up на v29 residue с
claude-4.5-sonnet / gpt-5.2-thinking / grok-4.1-reasoning") closes
empirically. helallao gives the same Perplexity Pro models at $0
through Юля's existing Pro sub, so paid OR was strictly redundant.

Run setup: scripts/run_helallao_voting.py on
eval/reports/2026-05-24/v29-v28-plus-p3f-q1275-merged.json,
14 residue qids (25/37/125/349/484/595/694/930/1029/1094/1144/1168/
1247/1254), sleep_between=3, reasoning-mode auto-detect.

Results:

  claude-4.5-sonnet-thinking : 14/14 same, 0 rescues, 0 errors
  gpt-5.2-thinking           : 14/14 same, 0 rescues, 0 errors
                               (initial 3 transient curl timeouts on
                                qids 125/349/484 cleanly retried)
  grok-4.1-reasoning         : 14/14 same, 0 rescues, 0 errors

Union: 42 model-qid attempts, 0 rescues, 0 regressions.

Three observations from this evidence:

1. Day-4 (2026-05-18) rate-limit on claude-4.5-sonnet-thinking has
   cleared. 6-day cooldown >> the ≥24h threshold I'd flagged before.
   All 14 cases reached without EXC. The model just doesn't produce
   the gold shape on any of the 14 misses.

2. Ceiling-friction classification in v29 description is verified
   empirically — three independent frontier reasoning routes converge
   on the same wrong SQL shape as the codestral baseline. These
   aren't model-knowledge gaps; they're BIRD-annotation quirks
   (qids 1029 sort direction, 1247 precedence convention),
   semantic-ambiguity questions (595 "one post history" interpretation,
   694 "user who left it" / "latest"), and query-shape mismatches
   where multiple readings are defensible.

3. Past 93.0% on $0 budget needs structural change, not better
   model access. Options: paid OR with broader context (no clear
   leverage), custom JOIN-path schema linker (v22 lesson — research-
   grade, дни), or accept 93.0% as portfolio-final.

Artefacts: four voting reports preserved (one per model run + the
3-qid retry batch). No merge attempted — nothing to merge.

Side cleanup: deleted eval/reports/2026-05-24/v29-arcwise-rescored-pre-fix.json
(diagnostic snapshot from c74b46c rescore-fix; canonical post-fix
v29-arcwise-rescored.json is the source of truth now).

Docs updated: SESSION_HANDOFF tl;dr (new EOD-2 block on top),
NEXT_SESSION priority #1 marked CLOSED with strikethrough, ceiling
note added at the end. Gates unchanged: 330 pytest, ruff clean,
mypy --strict src clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: JuliaEdom

docs/NEXT_SESSION.md

@@ -97,23 +97,31 @@ dd20bb...r2.cloudflarestorage.com: no such host` после успешного m
 fetch). Local heterogeneous CSC lever остаётся parked.
 
 **Следующее (priority):**
-1. **Paid OpenRouter top-up ($5+)** на v29 residue (14 qids): claude-4.5-sonnet
-   / gpt-5.2-thinking / grok-4.1-reasoning. Сливать только `alt_match=True` +
-   audit-rescore. **Note:** ceiling friction теперь значительная — qid 1275 был
-   самый clean candidate в residue.
+1. ~~**Paid OpenRouter top-up ($5+)** на v29 residue~~ — **CLOSED 2026-05-24 EOD-2.**
+   3-model helallao reasoning sweep (claude-4.5-sonnet-thinking + gpt-5.2-thinking
+   + grok-4.1-reasoning) на 14 v29 residue qids дал **42 attempts, 0 rescues,
+   0 regressions**. Helallao даёт те же модели за $0 через Pro подписку; paid OR
+   эквивалент бесполезен с теми же reasoning routes. Past 93.0% требует либо
+   другой архитектуры (custom JOIN-path linker, semantic equality check), либо
+   принять текущий ceiling. Артефакты в `eval/reports/2026-05-24/helallao-*-on-v29-residue.json`.
 2. **Местный heterogeneous CSC:** retry `qwen2.5-coder:7b-instruct` pull когда
-   R2 reachable. `qwen2.5-coder:7b` тэг то же; пробовать оба.
+   R2 reachable. `qwen2.5-coder:7b` тэг то же; пробовать оба. **Note:** даже local
+   qwen2.5-coder вряд ли пробьёт ceiling, который не пробили claude/gpt-5.2/grok
+   reasoning — это структурная граница BIRD-quirks, не модельная.
 3. **Не строить generic FK linker** (v22 lesson).
 4. **Не пытаться чинить query-shape / BIRD-annotation-quirk / semantic-ambiguity
    failures** (qids 25, 37, 125, 349, 484, 595, 694, 930, 1029, 1094, 1144,
    1247, 1254, 1168): hint'ы либо не помогают, либо требуют такой формулировки
-   которая регрессирует другие qids.
+   которая регрессирует другие qids. **EOD-2 sweep подтвердил эмпирически:** ни
+   один из трёх reasoning models не вышел из same shape для всех 14.
 5. **GraceKelly browser-orchestrator fix НЕ нужен для NL_SQL** — voting на
    Perplexity Pro идёт через helallao HTTPS-bridge (curl-cffi reverse-engineered,
    bypassing browser). Cookies extracted один раз из D:/GraceKelly/chrome-profile
-   через `.tmp/extract_pplx_cookies.py`, дальше чистый API. Если cookies протухнут
-   — re-extract тем же скриптом, не трогать GraceKelly browser path. Запись с
-   "GraceKelly fix" удалена — это residue с v6 (до helallao).
+   через `.tmp/extract_pplx_cookies.py`, дальше чистый API (cookies live до
+   2026-06-16). Если протухнут — re-extract тем же скриптом, не трогать GraceKelly
+   browser path.
+
+**Ceiling сейчас — final для $0 budget без runner-level рефакторинга.** v29 = 93.0% / 200, в 0.04pp от human expert (BIRD paper 92.96%). Триплет 93.0% / 74.87% / 68.84% не сдвигается без новой архитектуры. Портфолио-narrative полный.
 
 **Closed 2026-05-24 EOD:** `scripts/rescore_arcwise.py` pred-exec фикс
 (использует `execute_readonly` напрямую, не `_execute_gold` с

docs/SESSION_HANDOFF.md

@@ -1,5 +1,23 @@
-# NL_SQL — Session Handoff (2026-05-24 v29 = 93.0% EA verified via targeted P3.F schema-link hint for qid 1275, above #1 paid SOTA by +11.05pp; Arcwise rescore pred-exec fix landed same day)
+# NL_SQL — Session Handoff (2026-05-24 v29 = 93.0% EA verified via targeted P3.F schema-link hint for qid 1275, above #1 paid SOTA by +11.05pp; Arcwise rescore pred-exec fix + 3-model residue saturation sweep landed same day)
 
+> **Tl;dr 2026-05-24 EOD-2 — v29 residue saturation evidence (3-model helallao reasoning sweep):**
+> - **Hypothesis tested:** «paid OpenRouter top-up на v29 residue» entry в NEXT_SESSION предполагал что claude-4.5-sonnet / gpt-5.2-thinking / grok-4.1-reasoning могут найти ещё rescue среди 14 v29 misses. Поскольку helallao bridge (curl-cffi → Perplexity Pro API, $0 через её Pro подписку) даёт доступ к тем же моделям, paid step снимается.
+> - **Run setup:** `scripts/run_helallao_voting.py` на `eval/reports/2026-05-24/v29-v28-plus-p3f-q1275-merged.json`, sleep_between=3, через `HelallaoPerplexityProvider` с reasoning-mode auto-detect. 14 v29 residue qids: 25, 37, 125, 349, 484, 595, 694, 930, 1029, 1094, 1144, 1168, 1247, 1254.
+>
+> | Model | Cases reached | Rescues | Errors |
+> |---|---:|---:|---:|
+> | claude-4.5-sonnet-thinking | 14/14 | **0** | 0 |
+> | gpt-5.2-thinking | 14/14 (11 initial + 3 retry) | **0** | 0 (initial 3 transient curl timeouts retried clean) |
+> | grok-4.1-reasoning | 14/14 | **0** | 0 |
+>
+> **Union: 42 model-qid attempts, 0 rescues, 0 regressions.** Ceiling-friction analysis from v29 description verified empirically with three independent reasoning routes. Day-4 rate-limit on claude-4.5-sonnet-thinking cleared (6 days cooldown vs ≥24h threshold) — all 14 cases reached, but pred shape stayed wrong across all 14.
+> - **Implication:** past 93.0% on chrome-free $0 budget — confirmed saturated. Memory's "qids 595/694/1168 semantic-ambiguity; 25/37/125/349/484/930/1029/1094/1144/1247/1254 query-shape/annotation quirks" classification empirically holds: even frontier reasoning models converge on same wrong shape as codestral baseline. Past 93% requires (a) paid OR top-up *with broader context window or different reasoning algorithm*, or (b) runner-level fix (custom JOIN-path linker, semantic equality check), or (c) accept current ceiling as portfolio-final.
+> - Artefacts: `eval/reports/2026-05-24/helallao-{claude45-thinking,gpt52-thinking,grok41-reasoning}-on-v29-residue.json` + retry. No merge — no rescues to merge.
+> - Gates: 330 pytest (unchanged), ruff clean, mypy --strict src clean. No code/test changes — pure diagnostic data.
+> - Note: `eval/reports/2026-05-24/v29-arcwise-rescored-pre-fix.json` (diagnostic snapshot from c74b46c pred-exec fix work) deleted — served its purpose, leaving the canonical post-fix `v29-arcwise-rescored.json` only.
+>
+> ---
+>
 > **Tl;dr 2026-05-24 EOD — Arcwise rescore pred-exec fix:**
 > - `scripts/rescore_arcwise.py` теперь маршрутизирует pred через `execute_readonly` напрямую (был `_execute_gold` с SQLAlchemyError fallback на `exec_driver_sql` — non-deterministic engine state). Symmetric с canonical `scripts/audit_rescore.py`. Fix landed на top of v29 baseline; никаких rerun-ов pipeline не было.
 > - **Δ на Arcwise-Plat-SQL: 148/199 (74.37%) → 149/199 (74.87%)** (+0.5pp), gained sql_only 7 → 7 (same qids), lost 41 → 40 (qid 366 card_games simple перешёл в "same" — pred ≡ gold verbatim, прошлый committed run давал flake gold_rows=0 из-за state corruption).