Skip to content

Commit 4c1e0ed

Browse files
JuliaEdomclaude
andcommitted
refactor: split _support.py into _hints + _text_utils (Kimi P1.4) + Codex P2 reachability audit
Kimi audit P1.4 refactor: src/nl_sql/agent/nodes/_support.py 483 lines split into three focused modules without changing public API or behavior: - _support.py (184 lines) — public surface only: parse_generate_sql_output, render_m_schema, render_schema_block, render_fewshot_block. Imports internal helpers from the two siblings below. - _text_utils.py (53 lines, new) — JSON-fence stripping, safe-loads, NaN-safe float coerce, best-effort SELECT extraction. Used only by parse_generate_sql_output. - _hints.py (302 lines, new) — schema-block appendices: M-Schema regexes (_M_COL_RE, _M_FK_RE), join-hints, 10 P3.F schema-link if-blocks (one per landed BIRD-quirk rescue), extended-samples appendix. All 7 external import sites preserved (generate_sql.py, repair_once.py, plan_query.py, eval/runner.py, scripts/wider_sc_poc.py, tests/test_agent_support.py, tests/agent/nodes/test_schema_link_hints.py). No circular imports — _support → _hints/_text_utils, nothing flows back. 355 pytest pass unchanged; ruff check + format clean; mypy strict 0 issues. Codex P2 backlog reachability audit (housekeeping, no code change): triggered by a mis-attempt at P2 #9 (cache key omits json_mode) which turned out to be a false-positive — groq.py:44 force-sets json_mode=True on every Groq call via req.model_copy, Mistral codestral ignores the field, so per (provider, model) the value is constant and collision is impossible to trigger. Diff reverted after independent Codex + Kimi review. Then verified remaining P2 items have 0 production impact on the current state: - #7 (rescore_arcwise transition buckets stale): 0/200 stale-vs-fresh disagreements in eval/reports/2026-05-24/v29-arcwise-rescored.json. - #8 (_hashable float bucketing): 0 set-mismatch records in v22-v30 baselines (200 each); 8 in demo runs 2026-05-11, all honest column-count diff, not float-bucket. - #9 closed false-positive. - #10 (cache miss/fill race): latent, current eval pipeline serial per qid; fires only with parallel workers (not currently used). Per-item findings recorded in docs/NEXT_SESSION.md Open Audit Items table. Lesson: before touching audit findings, grep call-sites + reachability-check eval reports first. docs/SESSION_HANDOFF.md additionally received stale `_support.py::*` reference cleanups (now pointing at `_hints.py::*` after the split). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 3c82e37 commit 4c1e0ed

5 files changed

Lines changed: 435 additions & 348 deletions

File tree

‎docs/NEXT_SESSION.md‎

Lines changed: 23 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,18 @@
33
> Один лист, без воды. Берёшь, делаешь, обновляешь `SESSION_HANDOFF.md`,
44
> переписываешь этот файл под следующий sprint.
55
6+
## 2026-05-26 — Codex P2 backlog reachability audit (housekeeping, no code changes)
7+
8+
Triggered by mis-attempt at "small safe item" Codex P2 #9 (json_mode cache key) — landed fix + regression test, then independent Codex + Kimi review verdict = busywork (collision impossible per `groq.py:44` force-set). Diff reverted, HEAD `3c82e37` unchanged.
9+
10+
Verified remaining P2 items have **0 production impact** on current state:
11+
- #7 (rescore_arcwise transition buckets): `0/200` stale-vs-fresh disagreements в `v29-arcwise-rescored.json`. Transitions output unchanged if fixed.
12+
- #8 (`_hashable` float bucketing): `0` set-mismatch records в v22-v30 baselines (8 в demo runs 2026-05-11, all honest column-diff, not float-bucket).
13+
- #9 (json_mode cache key): false positive, closed (see counterfactual в backlog table).
14+
- #10 (cache miss/fill race): latent — текущий eval pipeline serial per qid; fires only при parallel workers (not currently used).
15+
16+
**Lesson:** before touching any backlog item, grep call-sites + reachability-check eval reports first. Codex audits may flag patterns без verifying they fire in actual runtime paths. Memory `feedback_no_shipping_blind_ci` extends to "verify P2 audit findings reachable before fixing".
17+
618
## 2026-05-25 EOD-6 — **v30 = 93.5% EA** verified, выше human-expert baseline
719

820
**Headline:** 92.5% (v29) → **93.5% / 200 (v30)** через два targeted P3.F schema-link hint'а на residue. **Выше human-expert baseline 92.96% (BIRD paper) на +0.54pp.** Per-tier v30: simple **97.0%**, moderate **91.9%** (90→91), challenging **91.2%** (30→31).
@@ -97,7 +109,7 @@ Per-tier v29 (post-EOD-3 correction): simple 97.0% (65/67) / **moderate 90.9%**
97109
error), повторить эти 8 шагов:
98110

99111
1. **Verify uniqueness** in n=200: `python -c "import json; r=json.load(open('eval/reports/2026-05-24/v29-v28-plus-p3f-q1275-merged.json',encoding='utf-8')); print([(x['question_id'], x['db_id']) for x in r['records'] if 'YOUR_PHRASE' in x['question'].lower()])"`. Phrase должна возвращать ТОЛЬКО target qid.
100-
2. **Add hint** в `src/nl_sql/agent/nodes/_support.py::_render_schema_link_hints_appendix`. Триггер = db_id + phrase(s) + table set. По шаблону существующих 8 if-блоков.
112+
2. **Add hint** в `src/nl_sql/agent/nodes/_hints.py::_render_schema_link_hints_appendix`. Триггер = db_id + phrase(s) + table set. По шаблону существующих 8 if-блоков.
101113
3. **Add target** в `scripts/p3f_acceptance.py::TARGETS` — required_columns + forbidden_columns (опционально).
102114
4. **Probe** `uv run python scripts/eval_baseline.py --config C --only-qids <NEW>,1275,408,894,1251,1531,902,1404,207 --report-suffix p3f-<new>-v1`. Все 8 prior targets должны PASS + новый match=True.
103115
5. **Merge** — inline Python (см. commit `99bae66` или `v28`/`v29` для шаблона; примерно 30 строк). Load baseline, swap pred_sql + match=True для new qid'ов, recompute summary + per_difficulty, write `v<N+1>-v<N>-plus-p3f-q<X>-merged.json`.
@@ -114,7 +126,7 @@ voted_by tag и delta, inline Python даёт control + audit trail. Не вын
114126
**Сделано:**
115127
- Расширен `scripts/p3f_acceptance.py` восьмым target'ом: qid `1275` moderate
116128
thrombosis_prediction, требует `Laboratory.CENTROMEA` + `Laboratory.SSB`.
117-
- В `src/nl_sql/agent/nodes/_support.py::_render_schema_link_hints_appendix`
129+
- В `src/nl_sql/agent/nodes/_hints.py::_render_schema_link_hints_appendix`
118130
добавлен узкий hint: db_id `thrombosis_prediction` + фраза
119131
`"anti-centromere"` или `"anti-SSB"` в вопросе + таблицы `{Patient,
120132
Laboratory}` в retrieved. Hint указывает что CENTROMEA/SSB **живут на
@@ -162,12 +174,12 @@ fetch). Local heterogeneous CSC lever остаётся parked.
162174
| # | Severity | Scope | Estimate |
163175
|---|---|---|---|
164176
| Kimi P1.3 | P1 | `app/streamlit_app.py` 1184 lines → split (`components/`, `theme.py`, `i18n/`) | 1.5h |
165-
| Kimi P1.4 | P1 | `src/nl_sql/agent/nodes/_support.py` 17KB → split (`render_schema.py`, `parse_output.py`, `schema_hints.py`) | 1h |
177+
| ~~Kimi P1.4~~ | **Done 2026-05-26** | `src/nl_sql/agent/nodes/_support.py` 483 lines → `_support.py` (public API, 184 lines) + `_text_utils.py` (JSON parsing, 53 lines) + `_hints.py` (schema appendices, 302 lines). Zero behavior change, 355 pytest pass, ruff + mypy strict clean. | 1h |
166178
| Kimi P1.6 | P1 | API coverage 58% → DI для `_make_singletons` + mock provider в API tests | 1.5h |
167-
| Codex #7 | P2 | `scripts/rescore_arcwise.py:82` transition buckets используют stale `rec["match"]` вместо recomputed match | 30min |
168-
| Codex #8 | P2 | `execution_accuracy.py:206` unordered float comparison через hash-bucket — два tolerant-equivalent rows могут попасть в разные buckets, set-mode EA false-negative | 1h (replace hash-bucket с tolerance-aware row matching) |
169-
| Codex #9 | P2 | `cache.py:77` cache key omits `GenerateRequest.json_mode` — JSON-mode call может replay non-JSON cached response | 30min (bump cache version key + include json_mode + other provider-affecting params) |
170-
| Codex #10 | P2 | `cache.py:88` cache miss/fill не locked — parallel eval workers могут race, duplicate paid calls, last-writer-wins | 1h (per-key diskcache lock или atomic memoization) |
179+
| Codex #7 | P2 latent | `scripts/rescore_arcwise.py:82` transition buckets используют stale `rec["match"]` вместо recomputed `out_entry["original_match"]` (line 141 overwrite). **Reachability verified 2026-05-26: 0/200 stale-vs-fresh disagreements в `eval/reports/2026-05-24/v29-arcwise-rescored.json`** — bug latent, transitions counts (7 gained / 91 lost) honest. Fix = 1-line swap, no observable change в output. | 30min, deferred |
180+
| Codex #8 | P2 latent | `execution_accuracy.py:209-221` `_hashable` bucketing через `round(v / 1e-6)` может развести два tolerance-equivalent rows (diff ~9e-7, banker's rounding edge) в разные buckets → set-mode false negative. **Reachability verified 2026-05-26: 0 set-mismatch records в v22-v30 baselines (200 records each); 8 set-mismatch в demo runs 2026-05-11, все honest column-count diff не float-bucket.** Fix = replace `_hashable` с pair-wise tolerance match (O(n²)). | 1h, deferred |
181+
| ~~Codex #9~~ | **false positive 2026-05-26** | `cache.py:77` cache key omits `req.json_mode`. **Не достижимо в текущем коде:** `src/nl_sql/llm/providers/groq.py:44` force-set'ит `json_mode=True` через `req.model_copy` на каждом Groq call; Mistral codestral игнорирует поле (`base.py:21` docstring). Per (provider, model) пара `json_mode` имеет константное значение → collision impossible. Не трогать (попытка fix landed 2026-05-26, reverted после Codex+Kimi independent review). | closed |
182+
| Codex #10 | P2 latent | `cache.py:88` cache miss/fill race без lock — parallel eval workers могут race, duplicate paid calls, last-writer-wins. **Reachability: текущий eval pipeline serial per qid (см. `runner.py::_run_one`). Latent до момента запуска parallel workers.** Fix = per-key diskcache lock или atomic memoization (`Cache.add` semantic). | 1h, deferred |
171183

172184
2. **HF Spaces redeploy** — на EOD-3 был synced на 92.5%, ничего не сдвинулось. Если юзер захочет регрес-проверить — `D:/NL_SQL/.deploy_hf.py` (gitignored, локальный).
173185

@@ -228,7 +240,7 @@ runner-level fix.
228240
**Сделано:**
229241
- Расширен `scripts/p3f_acceptance.py` седьмым target'ом: qid `408` moderate
230242
card_games, требует `rulings.text` + `rulings.uuid`, запрещает `cards.text`.
231-
- В `src/nl_sql/agent/nodes/_support.py::_render_schema_link_hints_appendix`
243+
- В `src/nl_sql/agent/nodes/_hints.py::_render_schema_link_hints_appendix`
232244
добавлен узкий hint: db_id `card_games` + фраза `"triggered ability"` в
233245
вопросе + таблицы `{cards, rulings}` в retrieved. Hint объясняет, что
234246
ruling-style abilities живут в `rulings.text` (не `cards.text`), требует
@@ -303,7 +315,7 @@ Past 93% — paid territory.
303315
- Расширен `scripts/p3f_acceptance.py` пятым и шестым target'ами:
304316
- qid `894` moderate formula_1, требует `lapTimes.milliseconds` в pred.
305317
- qid `1251` simple thrombosis_prediction, требует `Examination.ID` в pred.
306-
- В `src/nl_sql/agent/nodes/_support.py::_render_schema_link_hints_appendix`
318+
- В `src/nl_sql/agent/nodes/_hints.py::_render_schema_link_hints_appendix`
307319
добавлены два узких hint'а:
308320
- **qid 894 formula_1.** Триггер: db_id `formula_1` + фраза `"lap time recorded"`
309321
либо `"recorded lap time"` в вопросе + таблицы `{lapTimes, drivers, races}`
@@ -381,7 +393,7 @@ baseline 92.96%. Past 93% — paid territory.
381393
**Сделано:**
382394
- Расширен `scripts/p3f_acceptance.py` четвёртым target'ом: qid `1531` moderate
383395
debit_card_specializing, требует `yearmonth.consumption` column ref в pred.
384-
- В `src/nl_sql/agent/nodes/_support.py::_render_schema_link_hints_appendix`
396+
- В `src/nl_sql/agent/nodes/_hints.py::_render_schema_link_hints_appendix`
385397
добавлен узкий hint: db_id `debit_card_specializing`, фразы "top spending" и
386398
"average price" в вопросе, `{yearmonth, transactions_1k, customers}` все в
387399
retrieved-таблицах → многострочная подсказка с фрагментом готового SQL,
@@ -433,7 +445,7 @@ baseline 92.96%. Past 93% — paid territory.
433445
- Расширен `scripts/p3f_acceptance.py` третьим target'ом: qid `902` simple
434446
formula_1, требует `driverStandings.position`, запрещает `results.position` /
435447
`results.positionOrder`.
436-
- В `src/nl_sql/agent/nodes/_support.py::_render_schema_link_hints_appendix`
448+
- В `src/nl_sql/agent/nodes/_hints.py::_render_schema_link_hints_appendix`
437449
добавлен узкий hint: db_id `formula_1`, фраза "track number" в вопросе,
438450
`driverStandings` в таблицах → одна строка в Schema-link hints о
439451
`driverStandings.position` vs `results.position`. qid 902 — единственный

0 commit comments

Comments
 (0)