Skip to content

Commit 4a114fd

Browse files
JuliaEdomclaude
andcommitted
docs: EOD-5 handoff — Kimi+Codex audits closed (4 P1 commits + open P2 backlog)
- SESSION_HANDOFF.md prepended with EOD-5 tl;dr (commits 03ad6ae → e40e4da, gates green, 351 tests pass, push gated к юзеру) - NEXT_SESSION.md cold-pickup переписан: EOD-4 housekeeping снят (rescue evidence уже в 071e385), 8-item open backlog добавлен с estimate'ами (Kimi P1.3/P1.4/P1.6 + Codex #7-#10 P2) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent e40e4da commit 4a114fd

2 files changed

Lines changed: 80 additions & 34 deletions

File tree

‎docs/NEXT_SESSION.md‎

Lines changed: 44 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -5,56 +5,52 @@
55
66
## Cold-pickup checklist (orient в 2 минуты)
77

8-
**ВАЖНО при старте новой сессии — две unfinished housekeeping задачи от EOD-4:**
8+
**Open housekeeping (EOD-5):** push 4 local commits на origin когда юзер даст явное add. Иначе ничего.
99

1010
```powershell
1111
cd D:/NL_SQL
1212
13-
# A. Проверь не висят ли python.exe от прошлой сессии (helallao curl-cffi waits, etc).
13+
# 1. Что сейчас в репо?
14+
git log --oneline -8
15+
# Expected top 4 local (push gated к юзеру):
16+
# e40e4da fix: route voting/rescore through safe_compare_pred (Codex audit #2-4)
17+
# ebf0fb3 fix: gold-fail empty-empty false positive (Codex audit 2026-05-25 #1)
18+
# 4a79ecb refactor: NLSQL_M_SCHEMA / NLSQL_DAC env reads → PipelineConfig fields
19+
# 03ad6ae chore+fix: ruff format pass + regenerate stale baseline-summary headers
20+
# Origin tip: 071e385
21+
22+
# 2. Push когда захочешь (origin/main гейтится явным запросом юзера)
23+
# git push origin main
24+
25+
# 3. Orphan python procs от прошлых helallao runs (CPU guard)
1426
Get-Process python -ErrorAction SilentlyContinue |
1527
Where-Object { (Get-Date) - $_.StartTime -gt (New-TimeSpan -Minutes 30) } |
1628
Format-Table Id,StartTime,CPU,WS
1729
# Если есть orphans >30мин: Stop-Process -Id <pid> -Force
1830
19-
# B. Закоммить EOD-4 rescue evidence (3 untracked JSON в eval/reports/2026-05-25/).
20-
git status --short
21-
# Expected: ?? eval/reports/2026-05-25/helallao-q518-{rescue-attempt,grok,gpt52}.json
22-
# M chroma_data/* (runtime drift, ignore)
23-
git add eval/reports/2026-05-25/
24-
git commit -m "evidence: qid 518 rescue attempts closed (3 reasoning models, 0 alt_match) — gold returns 0 rows, v13 rescue bogus"
25-
git push origin main
26-
```
27-
28-
**Standard cold-pickup orientation:**
29-
30-
```powershell
31-
# 1. Что сейчас в репо?
32-
cd D:/NL_SQL
33-
git log --oneline -8
34-
# Expected top: 85fe388 audit-fix qid 518 / fbb9e24 HF redeploy / b6aed93 GraceKelly entry drop / 40ac2a1 helallao sweep / c6e5394 test+style / c74b46c rescore-arcwise fix / b208af4 v29 Arcwise rescore
35-
36-
# 2. Где actual baseline merged report?
37-
ls eval/reports/2026-05-24/v29-v28-plus-p3f-q1275-merged.json
38-
39-
# 3. Verify baseline после audit-correction (replay every stored pred under current runner)
31+
# 4. Verify baseline всё ещё консистентен после refresh_baseline_summary.py регенерации
4032
uv run python scripts/audit_rescore.py --report eval/reports/2026-05-24/v29-v28-plus-p3f-q1275-merged.json
4133
# Expected: stored 185 / true 185 / 0 mismatches
4234
43-
# 4. Verify все 8 P3.F gates ещё PASS (acceptance harness независим от audit-fix)
35+
# 5. Все 8 P3.F gates PASS
4436
uv run python scripts/p3f_acceptance.py --report eval/reports/2026-05-24/v29-v28-plus-p3f-q1275-merged.json --require-pass
4537
# Expected: 8 PASS, exit 0
4638
47-
# 5. Tests + lint + type
39+
# 6. Gates
4840
uv run pytest -q
4941
uv run ruff check src tests scripts app
42+
uv run ruff format --check src tests scripts app
5043
uv run mypy --strict src
51-
# Expected: 333 pass (+3 TestSafeComparePred regression tests) / clean / clean
44+
# Expected: 351 pass (was 333 + 18 EOD-5 new: 4 refresh_summary + 7 generate_sql_flags + 3 metrics gold_failed + 1 runner gold-fail end-to-end + 4 merge_voting reverify − 1 helallao_voting test unchanged) / ruff clean / format clean / mypy clean
5245
```
5346

54-
**Текущее состояние (HEAD `85fe388` pushed 2026-05-25 EOD-3, EOD-4 rescue evidence pending manual commit):**
55-
- **Repo + Streamlit + README + UI captions + live HF Space = v29 92.5%** (185/200) после CC-CX-KM /cxkm audit-correction (qid 518 v13 false positive исправлен через `safe_compare_pred` short-circuit).
56-
- HF redeploy: <https://liovina-nl-sql.hf.space>, E2E verified Playwright `92.5%` (EN) / `92,5%` (RU).
57-
- Все surface синхронизированы — gap нулевой.
47+
**Текущее состояние (HEAD `e40e4da` local, +4 ahead of origin `071e385`):**
48+
- **v29 = 92.5% (185/200) headline final** на $0 budget. Repo + Streamlit + README + UI captions + HF Space всё ещё 92.5% (deploy synced на EOD-3).
49+
- **Scoring integrity fully propagated:** `safe_compare_pred` теперь покрывает оба направления (pred-fail и gold-fail) и применяется во всех 3 voting/rescore путях. `merge_voting_rescues` имеет `--reverify` gate против stale pre-fix JSON.
50+
- **CI разблокирован** (был красным с `071e385` из-за format-check; fix landed в `03ad6ae`).
51+
- **Все baseline JSON summary headers** консистентны с per-record state (Codex #5 fix через `scripts/refresh_baseline_summary.py`).
52+
- **Test infra:** 351 pytest pass, mypy strict 0 issues, ruff check/format clean.
53+
- HF Spaces: <https://liovina-nl-sql.hf.space>, E2E verified Playwright `92.5%` (EN) / `92,5%` (RU) на EOD-3.
5854

5955
**Final triplet (final для $0 budget):**
6056

@@ -133,9 +129,25 @@ voted_by tag и delta, inline Python даёт control + audit trail. Не вын
133129
dd20bb...r2.cloudflarestorage.com: no such host` после успешного manifest
134130
fetch). Local heterogeneous CSC lever остаётся parked.
135131

136-
**Следующее (priority):**
132+
**Следующее (priority, EOD-5 → next sprint):**
133+
134+
0. **Push 4 EOD-5 commits** на `origin/main` когда юзер захочет (gated per CLAUDE.md). HEAD `e40e4da`, +4 ahead.
135+
136+
1. **Open audit items (Kimi + Codex, не закрыто автономно):**
137+
138+
| # | Severity | Scope | Estimate |
139+
|---|---|---|---|
140+
| Kimi P1.3 | P1 | `app/streamlit_app.py` 1184 lines → split (`components/`, `theme.py`, `i18n/`) | 1.5h |
141+
| Kimi P1.4 | P1 | `src/nl_sql/agent/nodes/_support.py` 17KB → split (`render_schema.py`, `parse_output.py`, `schema_hints.py`) | 1h |
142+
| Kimi P1.6 | P1 | API coverage 58% → DI для `_make_singletons` + mock provider в API tests | 1.5h |
143+
| Codex #7 | P2 | `scripts/rescore_arcwise.py:82` transition buckets используют stale `rec["match"]` вместо recomputed match | 30min |
144+
| Codex #8 | P2 | `execution_accuracy.py:206` unordered float comparison через hash-bucket — два tolerant-equivalent rows могут попасть в разные buckets, set-mode EA false-negative | 1h (replace hash-bucket с tolerance-aware row matching) |
145+
| Codex #9 | P2 | `cache.py:77` cache key omits `GenerateRequest.json_mode` — JSON-mode call может replay non-JSON cached response | 30min (bump cache version key + include json_mode + other provider-affecting params) |
146+
| Codex #10 | P2 | `cache.py:88` cache miss/fill не locked — parallel eval workers могут race, duplicate paid calls, last-writer-wins | 1h (per-key diskcache lock или atomic memoization) |
147+
148+
2. **HF Spaces redeploy** — на EOD-3 был synced на 92.5%, ничего не сдвинулось. Если юзер захочет регрес-проверить — `D:/NL_SQL/.deploy_hf.py` (gitignored, локальный).
137149

138-
0. **EOD-4 housekeeping** (см. cold-pickup checklist выше): commit 3 untracked `eval/reports/2026-05-25/helallao-q518-*.json`; kill orphan python processes если есть.
150+
3. **Past 92.5% headline (gated к юзеру, см. EOD-4):** runner-level CTE/SchemaAware Lite или paid OR with broader-context reasoning. Headroom ~0.5pp (next clean qid). Принципиальное решение оставлено за юзером — saturation подтверждена 3-моделями reasoning sweep + Pro retries на residue.
139151

140152
1. ~~**Paid OpenRouter top-up ($5+)** на v29 residue~~ — **CLOSED 2026-05-24 EOD-2.**
141153
3-model helallao reasoning sweep на 14 v29 residue qids: 42 attempts, 0 rescues.

‎docs/SESSION_HANDOFF.md‎

Lines changed: 36 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,39 @@
1-
# NL_SQL — Session Handoff (2026-05-25 EOD-4 close: v29 = **92.5% EA** final on $0 budget; qid 518 rescue attempts exhausted; bash session bricked late, **3 helallao rescue JSONs in `eval/reports/2026-05-25/` остались untracked — нужен ручной commit при старте новой сессии**)
2-
1+
# NL_SQL — Session Handoff (2026-05-25 EOD-5 close: Kimi + Codex audits ingested; 4 commits fixing CI-format + audit-correction inconsistencies + env→config refactor + same-pattern empty-empty bugs in 3 voting/rescore scripts + runner gold-side mirror; HEAD `e40e4da`, +4 ahead of origin, **push gated к юзеру**)
2+
3+
> **Tl;dr 2026-05-25 EOD-5 — Kimi+Codex dual audit closed P1 cluster, CI разблокирован, scoring-pattern fixes propagated:**
4+
> - **Two independent audits ingested:** Kimi (overall A grade, full report in `audit_kimi_25_05_26.md`) + Codex via `codex:codex-rescue` subagent (10 delta findings, no overlap with Kimi). Direct `codex exec` через Bash отбился permission gate → переключилась на Agent subagent (см. memory `feedback_no_codex_exec.md`).
5+
> - **CI был красным с `071e385`** (Kimi P1.1: 15 файлов не отформатированы; CI gate уже стоял на `.github/workflows/ci.yml:31`, но Kimi его не заметила → false positive в её action list). Fixed via `make format`.
6+
> - **Codex #5 audit-correction inconsistency:** все 8 v22-v29 merged baseline JSONs имели `overall.ea` / `overall.matched` +1 inflated после `safe_compare_pred` surgical patch — записи в `records[]` корректные (qid 518 = `match: False`), но summary headers не пересчитаны. Regenerated через новый `scripts/refresh_baseline_summary.py` (idempotent helper + 4 regression tests включая sweep guard на canonical baselines).
7+
> - **Codex #6 README headline:** lift-trace endpoint и v29 row показывали 93.0% pre-audit при headline 92.5%. Fixed: lift-trace оканчивается на 92.5% audit-corrected с explicit `−1 qid 518 v13` provenance + new table row документирует audit correction отдельно (preserves narrative history of v29 pre-audit number).
8+
> - **Kimi P1.5 testability:** `NLSQL_M_SCHEMA` / `NLSQL_DAC` reads вынесены из `src/nl_sql/agent/nodes/generate_sql.py` (был `import os` + `os.environ.get(...)` внутри node body) в typed `PipelineConfig.use_m_schema` / `use_dac_prompt` fields. `api/main.py::_make_singletons` и `scripts/run_helallao_voting.py` (единственный documented eval driver с этими envs) bootstrap env once. 7 новых unit tests на flag plumbing.
9+
> - **Codex #1 gold-side mirror of qid 518 bug:** `src/nl_sql/eval/runner.py::_execute_gold` возвращал `([], [])` когда BIRD gold SQL крашился (~1% случаев); если pred тоже возвращал `[]` (e.g. `SELECT * WHERE 1=0`), `compare_results([], [])` blessed match=True. Fixed: new `_execute_gold_with_status` returns `(rows, cols, gold_failed)`; `_compare_outcome` + `safe_compare_pred` accept `gold_failed` kwarg и short-circuit `match=False, reason='gold execution failed'`. Legacy `_execute_gold` retained как 2-tuple wrapper для 12+ скриптов которые ещё импортируют его. 3 новых regression tests.
10+
> - **Codex #2-4 same-pattern в скриптах:**
11+
> - `scripts/run_helallao_voting.py:189` — pred exec exceptions сваливались в `alt_rows=[]`; теперь tracks `pred_failed` + `gold_failed` flags, routes через `safe_compare_pred`.
12+
> - `scripts/rescore_arcwise.py:127` — corrected-gold exec exceptions сваливались в `gold_rows=[]`; теперь `_execute_gold_with_status` + `safe_compare_pred(gold_failed=...)`.
13+
> - `scripts/merge_voting_rescues.py:73` — флипал baseline `match=True` из stored `alt_match` без re-execution. Pre-fix voting JSONs могли silently inflate EA. Fixed: default `--reverify` re-executes pred+gold через `safe_compare_pred`; `--no-reverify` escape hatch для trusted legacy merges. 4 новых reverify tests.
14+
> - **4 commits на main (local-only, push gated):**
15+
> - `03ad6ae` chore+fix: ruff format + 8 stale baseline summaries + README lift trace + v29 table row
16+
> - `4a79ecb` refactor: NLSQL_M_SCHEMA / NLSQL_DAC env → PipelineConfig
17+
> - `ebf0fb3` fix: gold-fail empty-empty false positive (Codex #1)
18+
> - `e40e4da` fix: route voting/rescore through safe_compare_pred (Codex #2-4)
19+
> - **Gates green:** ruff check + format-check + mypy --strict + 351 pytest (was 333; +18 new tests).
20+
> - **HEAD `e40e4da` local; origin `071e385`** — **push не делался** per CLAUDE.md ("DO NOT push unless explicitly asked"). Cold-pickup: см. § `Cold-pickup checklist` ниже + `docs/NEXT_SESSION.md`.
21+
>
22+
> **Не закрыто автономно (требует решения / большой scope):**
23+
> - Kimi P1.3 `app/streamlit_app.py` 1184 lines → split (1.5h refactor)
24+
> - Kimi P1.4 `src/nl_sql/agent/nodes/_support.py` 17KB → split (1h refactor)
25+
> - Kimi P1.6 API coverage 58% → DI для `_make_singletons` (moderate refactor)
26+
> - Codex #7 transition buckets stale (P2 stylistic, low impact)
27+
> - Codex #8 hash-bucket float tolerance (P2 math bug в `compare_results` set mode)
28+
> - Codex #9 `cache.py:77` cache key omits `GenerateRequest.json_mode` (P2 correctness)
29+
> - Codex #10 `cache.py:88` cache miss/fill race без lock (P2 concurrency, parallel eval workers)
30+
>
31+
> **Memory updates:**
32+
> - new: `feedback_no_codex_exec.md` (CODEX EXEC через Bash запрещён, only Agent `codex:codex-rescue` subagent)
33+
> - deprecated: `feedback_codex_exec_direct.md` (старое правило про direct > subagent отменено)
34+
>
35+
> ---
36+
>
337
> **Tl;dr 2026-05-25 EOD-4 — qid 518 rescue attempts closed (all alt_match=False) + session end:**
438
> - **Goal:** после EOD-3 (audit-correction 93.0% → 92.5%) попытались legitimately rescue qid 518 через helallao reasoning, чтобы вернуть 93.0% с integrity.
539
> - **3 reasoning models attempted** (claude-4.5-sonnet-thinking, grok-4.1-reasoning, gpt-5.2-thinking) на qid 518 baseline=False через `scripts/run_helallao_voting.py --only-qids 518`. Все three generated clean alt_pred (e.g., grok: `SELECT format, name FROM legalities INNER JOIN cards USING (uuid) WHERE status='Banned' AND format=(SELECT format FROM legalities GROUP BY format ORDER BY COUNT(*) DESC LIMIT 1)`), но все **alt_match=False**.

0 commit comments

Comments
 (0)