Skip to content

Commit 2ac852b

Browse files
JuliaEdomclaude
andcommitted
docs+chore: add BACKLOG tracker, untrack SESSION_HANDOFF, finalize methodology; tidy-ups
Audit follow-through (D1 + A5 + C6): - docs/BACKLOG.md — one living tracker (done / gated / won't-fix) so audits convert to status instead of rotting as parallel lists. - docs/SESSION_HANDOFF.md un-tracked (gitignored) — working kitchen, not for the public repo/Space; its eval trace now lives in methodology §13 (three honest tiers + reproduction + reproducibility caveat), which supersedes the stale planning tail. - C6: `import os` hoisted to module scope in api/main.py; GraceKelly typo fixed; untracked scratch eval reports removed. - uv.lock realigned to the Python 3.13 floor. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 0fe469b commit 2ac852b

8 files changed

Lines changed: 78 additions & 2277 deletions

File tree

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -117,6 +117,7 @@ reviews/*.err
117117
/_ref_*
118118
/[0-9][0-9]_[0-9][0-9]_[0-9][0-9].md
119119
/docs/NEXT_SESSION.md
120+
/docs/SESSION_HANDOFF.md
120121
/docs/data_flow.*
121122

122123
# Secrets — никогда не коммитить

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,7 @@ The UI reads `MISTRAL_API_KEY` from `.env`; copy `.env.example` first.
8181

8282
| Файл | Содержание |
8383
|---|---|
84+
| [docs/BACKLOG.md](docs/BACKLOG.md) | Живой трекер: что сделано, что gated, что won't-fix |
8485
| [docs/00_task.md](docs/00_task.md) | Постановка задачи (что / почему / scope / DoD) |
8586
| [docs/01_architecture.md](docs/01_architecture.md) | v1 — superseded, оставлен как исторический |
8687
| [docs/02_architecture_v2.md](docs/02_architecture_v2.md) | **Active baseline** — lean архитектура после CX+KM review |

‎docs/03_eval_methodology.md‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -380,3 +380,32 @@ Config B (BM25 cards) is intentionally absent from the shipped pipeline — dens
380380
- [ ] Bakeoff на 30 вопросов × 3 providers работает
381381
- [x] README результат-таблицы заполнены реальными числами (2026-05-12)
382382
- [ ] Hard checkpoint week 3 пройден (EA ≥35% или scope-down принят)
383+
384+
---
385+
386+
## 13. Итоговые результаты — три честных уровня (2026-07-11)
387+
388+
Одно число вводит в заблуждение, поэтому метрика BIRD Mini-Dev (n=200, SQLite, $0 free-tier) разложена по слоям. Только **уровень 1** включён в продукт (Streamlit UI и `/ask`); уровни 2–3 — eval-конфигурация.
389+
390+
| Уровень | EA | Что это | В продукте? | Воспроизводимость |
391+
|---|---:|---|:--:|---|
392+
| 1. Reproducible single-run | **57.5%** (115/200) | Чистый пайплайн на codestral, один прогон, config E (dense + repair), без голосования и подсказок. First-pass 56.0%; simple 71.6% / moderate 53.5% / challenging 41.2%. | да | одна команда, детерминированно |
393+
| 2. + multi-provider voting | 85.5% | Само-консистентность + голосование по провайдерам (Groq / OpenRouter / Perplexity-bridge). | eval | архив; не «с нуля» (Perplexity-cookie-bridge истёк 16.06) |
394+
| 3. + per-question schema-link hints | **94.0%** (188/200) | 11 hint-блоков под конкретные BIRD-вопросы (`_hints.py`); адаптация под annotation-quirks. simple 97.0% / moderate 92.9% / challenging 91.2%. | eval (`enable_bird_rescue_hints`, off) | архив-merge v22→v31; consistency проверяется `audit_rescore.py` (0 mismatches) |
395+
396+
### Воспроизвести
397+
398+
```powershell
399+
# Уровень 1 (продуктовый пайплайн, seed=0, детерминированный dev_split):
400+
uv run python scripts/eval_baseline.py --config E --n 200
401+
# Уровень 3 (hint-assisted, eval-only):
402+
uv run python scripts/eval_baseline.py --config E --n 200 --bird-rescue-hints
403+
```
404+
405+
Уровень 1 воспроизводится одной командой на free-tier Mistral. Уровень 3 воспроизводит hint-assisted число на том же прогоне с флагом. Уровень 2 — архивная трасса: часть rescues шла через Perplexity-cookie-bridge, которого больше нет; `audit_rescore.py` подтверждает консистентность архива, но «с нуля» этот слой сейчас не пересобирается.
406+
407+
### Честность витрины
408+
409+
94.0% выше human-expert baseline (BIRD paper, 92.96%) на +1.04pp — но это **hint-assisted слой**, кодирующий ответы к конкретным вопросам теста, а не провайдер-уровневая победа. Поэтому он выключен в продукте по умолчанию: живое демо и API отдают уровень 1 (57.5%). `GET /eval/latest` отдаёт именно уровень 1 — число, которого достигает сам API. Разделение на слои показывает и инженерию (воспроизводимые 57.5%), и научную честность (какой слой что даёт).
410+
411+
На Arcwise-corrected gold (Jin et al., CIDR/VLDB 2026) — 74.37% (148/199): noise-floor после исправления annotation-ошибок BIRD.

‎docs/BACKLOG.md‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# Backlog
2+
3+
Single living tracker for code/eval work items. Supersedes the scattered per-audit
4+
lists — new findings land here with a status so nothing rots undone.
5+
6+
Status: ✅ done · ⛔ gated (needs a human decision or an external host) · 🔎 won't-fix (verified)
7+
8+
## Done (2026-07-11)
9+
10+
Security / public surface
11+
- ✅ AST guard: `SELECT … INTO` denied; `pg_sleep_for/until`, `set_config`, `setval` banned; adversarial tests per vector.
12+
- ✅ Streamlit `Sentence` renderer HTML-escaped (stored-XSS class closed).
13+
- ✅ API: rate limit unconditional (anonymous bucketed by IP); key via `Settings` (`.env` honoured); `secrets.compare_digest`; 500s redacted to a `trace_id`; `truncated` surfaced in `AskResponse`.
14+
15+
Metric integrity
16+
- ✅ BIRD rescue hints gated behind `PipelineConfig.enable_bird_rescue_hints` (default **off**); product path no longer serves canned per-question answers. `eval_baseline.py --bird-rescue-hints` reproduces the hint-assisted number.
17+
- ✅ Reproducible single-run measured and committed (config E, codestral, 57.5% — `eval/baselines/reproducible_n200.json`).
18+
- ✅ Three-tier metric in README + methodology §13 (57.5% product / 85.5% voting / 94.0% hint-assisted), each labelled by whether it runs in the product.
19+
- ✅ Machine-readable artifacts synced: v31 report summary 185/0.925 → 188/0.94; `/eval/latest` serves the reproducible baseline (not a stale 60.5%); FastAPI description honest; regression test pins `overall == recompute(records)`.
20+
21+
Postgres path
22+
- ✅ Read-only made real: `postgresql_readonly` execution option (was a no-op `SET` inside an open transaction); DSN normalised to `psycopg` v3 (path had never run under psycopg2). Loader `scripts/load_postgres.py`; registry registers PG from `NL_SQL_PG_DSN`. Live PG16 proof in CI (`tests/test_postgres_readonly.py`).
23+
24+
Docs / hygiene / CI
25+
- ✅ README/DEPLOY truthful: Langfuse + vcr.py claims removed; unused langfuse compose service dropped; DEPLOY.md rewritten for the real HF Space (tracked-only + prune).
26+
- ✅ CWD-independent paths (`nl_sql.paths`); `ProviderName` synced with the factory (+`perplexity`, +`gracekelly`).
27+
- ✅ Nine voting scripts on the unsafe comparator frozen in `scripts/archive/` + CI gate (`check_no_raw_compare.py`).
28+
- ✅ CI: mypy over `src app scripts`; coverage floor (`--cov-fail-under=85`, currently 91%); aligned on Python 3.13.
29+
- ✅ `docs/SESSION_HANDOFF.md` un-tracked (kitchen); eval trace moved into methodology §13.
30+
31+
## Gated (needs a decision or an external host)
32+
33+
- ⛔ **Sustained-load / PG data migration.** The 460 MB `codebase_community` (StackExchange) load + a PG-backed eval need a real Postgres host; not runnable on the Windows dev box. Loader + CI read-only proof are in place; the full data load is the remaining step.
34+
- ⛔ **Space cleanup + git history.** `SESSION_HANDOFF.md` is un-tracked now, so future deploys won't republish it; removing it from the live Space and any git-history rewrite are force/deploy actions — owner's call.
35+
- ⛔ **External pen-test of the public API.** Self-attestation is not a substitute; needs an outside pass.
36+
37+
## Won't-fix (verified)
38+
39+
- 🔎 `execution_accuracy._hashable` float bucketing — 0 set-mismatch across v22–v31 baselines; O(n²) replacement not justified.
40+
- 🔎 `cache.py` miss/fill race — fires only under parallel eval workers, which aren't used.
41+
- 🔎 Chasing BIRD annotation-quirk residue above 94% — saturation shown by 3-model reasoning sweeps.

0 commit comments

Comments
 (0)