Skip to content

Commit a6f8ae7

Browse files
JuliaEdomclaude
andcommitted
feat(pipeline): gate BIRD rescue hints behind a flag (off in product); honest 3-tier metric
The 94.0% BIRD headline rested on _hints.py — 11 per-question schema-link hints that encode gold answers to specific benchmark questions and ran in the LIVE Streamlit UI and API. A reviewer opening _hints.py would (fairly) discount the whole showcase. - New PipelineConfig.enable_bird_rescue_hints, default False, threaded through the generate_sql / repair_once / plan_query nodes into render_schema_block. The product path (Streamlit, /ask) no longer injects canned per-question answers. - scripts/eval_baseline.py --bird-rescue-hints turns them on to reproduce the hint-assisted number; all five run_config_* helpers accept the flag. - Measured the reproducible single-run: config E, codestral, no voting/hints = 57.5%. - README replaces the single "94.0% > human expert" headline with a three-tier table: 57.5% reproducible (product) / 85.5% voting (eval) / 94.0% hint-assisted (eval), each labelled by whether it runs in the product, plus a one-command reproduction. Badges: live CI badge, python 3.13, dual metric. - Streamlit UI (i18n EN+RU) now leads with 57.5% (what the demo actually does) and marks 94.0% as the eval-only layer, not the product number. Tests: hint fixtures opt into the flag; a new test pins hints OFF by default. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 7f5043f commit a6f8ae7

12 files changed

Lines changed: 202 additions & 64 deletions

File tree

‎README.md‎

Lines changed: 40 additions & 27 deletions
Large diffs are not rendered by default.

‎app/i18n.py‎

Lines changed: 15 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -27,18 +27,22 @@
2727
"metric_percent": "100%",
2828
"metric_caption": "30 dev + 30 held-out, balanced split, all ten query categories at 100% on the free-tier codestral pipeline.",
2929
"research_kicker": "BIRD Mini-Dev research benchmark",
30-
"research_value": "94.0% / 200",
31-
"research_short": "Above the human-expert baseline · $0 external cost",
30+
"research_value": "57.5% / 200",
31+
"research_short": "Reproducible single-run (codestral, $0). Hint-assisted eval tier reaches 94.0%.",
3232
"methodology_label": "How the score was reached",
3333
"research_caption": (
34-
"Hybrid pipeline: "
34+
"This demo runs the reproducible pipeline — 57.5% EA, one free-tier codestral pass, "
35+
"no benchmark rescue hints (they are off by default). The 94.0% headline is an "
36+
"eval-only layer that adds per-question schema-link hints; it is documented below for "
37+
"transparency, not served here. "
38+
"Eval hybrid pipeline: "
3539
"<span class='nl-term' title='Mistral codestral-latest — SQL-specialised generation model, free tier'>codestral</span> + "
3640
"<span class='nl-term' title='Anthropic Claude 4.5 Sonnet via Perplexity Pro browser bridge — used on the hard tier'>Sonnet 4.6 bridge</span> + "
3741
"<span class='nl-term' title='Per-failure re-prompt with executable-shape feedback — only on frozen failures, no T=0 noise'>grounded-critique retry</span> + "
3842
"<span class='nl-term' title='helallao reverse-engineered HTTPS bridge to Perplexity backend — Grok 4.1, GPT-5.2, Claude 4.5 Sonnet, kimi-k2-thinking, gpt-5.2-thinking + DAC on residue, claude-4.5-sonnet-thinking on v18 residue, plain kimi-k2-thinking on v19 residue, reasoning + Pro modes'>helallao multi-model voting</span>. "
3943
"Scored under "
4044
"<span class='nl-term' title='bird-bench/mini_dev evaluation_ex.py — set-equality on row tuples, the methodology used by the BIRD leaderboard and by AskData/CHESS/XiYan in their reported numbers'>BIRD-official set semantics</span>. "
41-
"+46.2pp over the GPT-4 zero-shot reference (47.8%), $0 external cost. **Above human-expert baseline 92.96% (BIRD paper) by +1.04pp.** "
45+
"The hint-assisted tier is +46.2pp over the GPT-4 zero-shot reference (47.8%) at $0 external cost, and edges the human-expert baseline 92.96% (BIRD paper) by +1.04pp — but on a per-question rescue layer, not a provider-level win. "
4246
"On <span class='nl-term' title='Jin et al., CIDR/VLDB 2026, arXiv:2601.08778 — corrected BIRD gold annotations'>Arcwise-Plat corrected gold</span>: 74.37% (148/199) — honest noise-floor; +7 sql_only catches where our prediction is correct under Arcwise's corrected gold but BIRD's original gold disagrees. "
4347
"Seven late-stage model rescues on v16→v22, two archive-audit rescores on v23/v24 (qid 1205 via archive sweep, qid 959 via archive-rescore after the day-5 bind-bug fix), and nine targeted P3.F schema-link hints on v25→v31: qid 902 (driverStandings.position vs results.position), qid 1531 (yearmonth.Consumption subquery + SUM(Price/Amount) row-wise), qid 894 (lapTimes.milliseconds first SELECT column), qid 1251 (Patient ⋈ Laboratory ⋈ Examination semi-join), qid 408 (rulings.text filter via cards.uuid join + COUNT(DISTINCT cards.id)), qid 1275 (Laboratory.CENTROMEA/SSB IN ('negative','0') instead of fabricated tokens against Examination), qid 1168 (override projection-discipline: include Patient.Birthday as third SELECT column + ORDER BY Birthday ASC LIMIT 1 on JOIN), qid 1029 (european_football_2 positional inversion: 'highest buildUpPlaySpeed' = lower numeric value, sort ASC + INNER JOIN Team), qid 37 (california_schools 'lowest excellence rate' — BIRD inverts question word-order 'Street, City, Zip and State' to SELECT (Street, City, State, Zip); 'excellence rate' = NumGE1500 / NumTstTakr ASC LIMIT 1 directly on JOIN). Every cell verified via audit_rescore.py — 0 mismatches."
4448
),
@@ -110,18 +114,21 @@
110114
"metric_percent": "100%",
111115
"metric_caption": "30 dev + 30 held-out, сбалансированный сплит, все десять категорий запросов на 100% через бесплатный codestral.",
112116
"research_kicker": "Исследовательский бенчмарк BIRD Mini-Dev",
113-
"research_value": "94,0% / 200",
114-
"research_short": "Выше human-expert baseline · внешние расходы — ноль",
117+
"research_value": "57,5% / 200",
118+
"research_short": "Воспроизводимый single-run (codestral, $0). Hint-assisted eval-уровень — 94,0%.",
115119
"methodology_label": "Как получен результат",
116120
"research_caption": (
117-
"Гибридный пайплайн: "
121+
"Это демо гоняет воспроизводимый пайплайн — 57,5% EA, один прогон на free-tier codestral, "
122+
"без benchmark-подсказок (они выключены по умолчанию). 94,0% — eval-only слой с per-question "
123+
"schema-link подсказками; ниже он раскрыт для прозрачности, но здесь не подаётся. "
124+
"Eval-гибрид: "
118125
"<span class='nl-term' title='Mistral codestral-latest — модель, специализированная под генерацию SQL, бесплатный тариф'>codestral</span> + "
119126
"<span class='nl-term' title='Anthropic Claude 4.5 Sonnet через браузерный мост Perplexity Pro — на сложных кейсах'>мост к Sonnet 4.6</span> + "
120127
"<span class='nl-term' title='Повторный prompt со shape-фидбэком исполнения — только на зафиксированных фейлах, без шума T=0'>directed-critique retry</span> + "
121128
"<span class='nl-term' title='Реверс-инжиниринг HTTPS моста к бэкенду Perplexity — Grok 4.1, GPT-5.2, Claude 4.5 Sonnet, kimi-k2-thinking, gpt-5.2-thinking + DAC на residue, claude-4.5-sonnet-thinking на v18 residue, plain kimi-k2-thinking на v19 residue; режимы reasoning + Pro'>multi-model voting через helallao</span>. "
122129
"Scoring — "
123130
"<span class='nl-term' title='bird-bench/mini_dev evaluation_ex.py — set-равенство на результирующих кортежах. Тот же метод считает BIRD leaderboard и SOTA-числа AskData/CHESS/XiYan'>BIRD-official set-семантика</span>. "
124-
"+46,2 п.п. над zero-shot GPT-4 (47,8%), внешние расходы — ноль. **Выше human-expert baseline 92,96% (BIRD paper) на +1,04 п.п.** "
131+
"Hint-assisted уровень — +46,2 п.п. над zero-shot GPT-4 (47,8%) при нулевых внешних расходах, и на +1,04 п.п. выше human-expert baseline 92,96% (BIRD paper) — но за счёт per-question rescue-слоя, не провайдер-уровневой победы. "
125132
"На <span class='nl-term' title='Jin et al., CIDR/VLDB 2026, arXiv:2601.08778 — исправленные аннотации gold BIRD'>исправленном gold Arcwise-Plat</span>: 74,37% (148/199) — честный noise-floor; +7 sql_only catches, где наш ответ правильнее эталона BIRD согласно Arcwise. "
126133
"Семь late-stage rescue по моделям на пути v16→v22, плюс v23/v24 — archive-sweep и archive-rescore (qid 1205 / qid 959 после day-5 bind-bug fix), плюс v25→v31 — девять узких P3.F schema-link hint'ов: qid 902 (driverStandings.position вместо results.position), qid 1531 (subquery по yearmonth.Consumption + SUM(Price/Amount) построчно), qid 894 (lapTimes.milliseconds первой колонкой), qid 1251 (полу-джойн Patient ⋈ Laboratory ⋈ Examination), qid 408 (фильтр по rulings.text через join cards.uuid + COUNT(DISTINCT cards.id)), qid 1275 (Laboratory.CENTROMEA/SSB IN ('negative','0') вместо несуществующих Examination columns + invented '-'/'+-' tokens), qid 1168 (override projection-discipline: Patient.Birthday как 3-я колонка SELECT + ORDER BY Birthday ASC LIMIT 1 прямо на JOIN), qid 1029 (european_football_2 positional inversion: 'highest buildUpPlaySpeed' = меньшее число, sort ASC + INNER JOIN Team), qid 37 (california_schools 'lowest excellence rate' — BIRD инвертирует word-order вопроса 'Street, City, Zip and State' в SELECT (Street, City, State, Zip); 'excellence rate' = NumGE1500 / NumTstTakr ASC LIMIT 1 прямо на JOIN). Каждая ячейка верифицирована через audit_rescore.py — 0 mismatches."
127134
),

‎app/streamlit_app.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
"""Streamlit UI for the NL→SQL assistant.
22
3-
Modern light surface: white background, one calm indigo accent, soft
4-
shadows and rounded corners (Inter for text, JetBrains Mono for code).
3+
Modern light surface: warm stone palette on white, one terracotta accent,
4+
soft shadows and rounded corners (Manrope for text, JetBrains Mono for code).
55
Controls live in a top bar — database, mode and the EN/RU language
66
toggle; the sidebar keeps only the schema explorer, advanced retrieval
77
knobs and clear-chat. Bilingual chrome (data questions stay in their

‎scripts/eval_baseline.py‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -153,6 +153,16 @@ def main(argv: list[str] | None = None) -> int:
153153
"moderate-tier BIRD questions."
154154
),
155155
)
156+
parser.add_argument(
157+
"--bird-rescue-hints",
158+
action="store_true",
159+
help=(
160+
"inject the per-question BIRD schema-link rescue hints (the "
161+
"annotation-quirk adaptation layer that lifts EA to 94.0%%). Eval-only "
162+
"and OFF by default — the product pipeline never serves these. Turn on "
163+
"to reproduce the hint-assisted headline number."
164+
),
165+
)
156166
parser.add_argument(
157167
"--primary-sample-size",
158168
type=int,
@@ -339,6 +349,7 @@ def _on_progress(idx: int, total: int, rec: EvalRecord) -> None:
339349
fk_hops=args.fk_hops,
340350
table_budget=args.table_budget,
341351
sort_schema_block=args.sort_schema_block,
352+
enable_bird_rescue_hints=args.bird_rescue_hints,
342353
primary_sample_size=args.primary_sample_size,
343354
extended_sample_size=args.extended_sample_size,
344355
sql_candidate_temperatures=temps,
@@ -357,6 +368,7 @@ def _on_progress(idx: int, total: int, rec: EvalRecord) -> None:
357368
fk_hops=args.fk_hops,
358369
table_budget=args.table_budget,
359370
sort_schema_block=args.sort_schema_block,
371+
enable_bird_rescue_hints=args.bird_rescue_hints,
360372
primary_sample_size=args.primary_sample_size,
361373
extended_sample_size=args.extended_sample_size,
362374
progress=_on_progress,
@@ -373,6 +385,7 @@ def _on_progress(idx: int, total: int, rec: EvalRecord) -> None:
373385
fk_hops=args.fk_hops,
374386
table_budget=args.table_budget,
375387
sort_schema_block=args.sort_schema_block,
388+
enable_bird_rescue_hints=args.bird_rescue_hints,
376389
primary_sample_size=args.primary_sample_size,
377390
extended_sample_size=args.extended_sample_size,
378391
progress=_on_progress,
@@ -389,6 +402,7 @@ def _on_progress(idx: int, total: int, rec: EvalRecord) -> None:
389402
fk_hops=args.fk_hops,
390403
table_budget=args.table_budget,
391404
sort_schema_block=args.sort_schema_block,
405+
enable_bird_rescue_hints=args.bird_rescue_hints,
392406
primary_sample_size=args.primary_sample_size,
393407
extended_sample_size=args.extended_sample_size,
394408
progress=_on_progress,

‎src/nl_sql/agent/graph.py‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -140,6 +140,14 @@ class PipelineConfig:
140140
(`generate_sql_dac.txt`) which decomposes multi-clause questions into
141141
sub-questions before composing SQL. Replaces the legacy `NLSQL_DAC=1`
142142
env toggle; `api/main.py` reads the env once at boot and threads it here."""
143+
enable_bird_rescue_hints: bool = False
144+
"""When True, inject the per-question BIRD schema-link "rescue" hints
145+
(`_hints.py::_render_schema_link_hints_appendix`) into the schema block.
146+
These are annotation-quirk adaptations tied to specific BIRD Mini-Dev
147+
questions and lift the benchmark to 94.0% EA — so they belong ONLY in the
148+
eval configuration, never in the product path. Default False: the live
149+
Streamlit/API pipeline must not serve a canned answer to a benchmark
150+
question. `scripts/eval_baseline.py --bird-rescue-hints` turns them on."""
143151

144152

145153
@dataclass(slots=True)
@@ -185,11 +193,13 @@ def build_pipeline(config: PipelineConfig) -> CompiledStateGraph[Any, Any, Any,
185193
temperature=config.sql_temperature,
186194
use_m_schema=config.use_m_schema,
187195
use_dac_prompt=config.use_dac_prompt,
196+
enable_bird_rescue_hints=config.enable_bird_rescue_hints,
188197
),
189198
"validate": make_validate_node(),
190199
"repair_once": make_repair_once_node(
191200
config.sql_provider,
192201
sort_schema_block=config.sort_schema_block,
202+
enable_bird_rescue_hints=config.enable_bird_rescue_hints,
193203
),
194204
"execute": make_execute_node(
195205
registry=config.registry,
@@ -204,6 +214,7 @@ def build_pipeline(config: PipelineConfig) -> CompiledStateGraph[Any, Any, Any,
204214
config.sql_provider,
205215
sort_schema_block=config.sort_schema_block,
206216
temperature=config.sql_temperature,
217+
enable_bird_rescue_hints=config.enable_bird_rescue_hints,
207218
)
208219
if config.enable_grounded_critique:
209220
nodes["grounded_critique"] = make_grounded_critique_node()

‎src/nl_sql/agent/nodes/_support.py‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -133,6 +133,7 @@ def render_schema_block(
133133
context: ContextBundle | None,
134134
*,
135135
sort_alphabetically: bool = False,
136+
enable_bird_rescue_hints: bool = False,
136137
) -> str:
137138
"""Render schema chunks + FK neighbours into a single text block.
138139
@@ -164,9 +165,13 @@ def render_schema_block(
164165
join_hints = _render_join_hints_appendix(all_hits)
165166
if join_hints:
166167
blocks.append(join_hints)
167-
schema_link_hints = _render_schema_link_hints_appendix(context, all_hits)
168-
if schema_link_hints:
169-
blocks.append(schema_link_hints)
168+
# BIRD per-question rescue hints are eval-only (they encode gold answers for
169+
# specific benchmark questions). Off by default so the product path never
170+
# serves a canned answer — see PipelineConfig.enable_bird_rescue_hints.
171+
if enable_bird_rescue_hints:
172+
schema_link_hints = _render_schema_link_hints_appendix(context, all_hits)
173+
if schema_link_hints:
174+
blocks.append(schema_link_hints)
170175
appendix = _render_extended_samples_appendix(context.extended_samples)
171176
if appendix:
172177
blocks.append(appendix)

‎src/nl_sql/agent/nodes/generate_sql.py‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ def make_generate_sql_node(
2929
sort_schema_block: bool = False,
3030
use_m_schema: bool = False,
3131
use_dac_prompt: bool = False,
32+
enable_bird_rescue_hints: bool = False,
3233
) -> Callable[[PipelineState], PipelineState]:
3334
def node(state: PipelineState) -> PipelineState:
3435
question = state.get("question", "")
@@ -42,7 +43,11 @@ def node(state: PipelineState) -> PipelineState:
4243
if use_m_schema:
4344
schema_text = render_m_schema(context)
4445
else:
45-
schema_text = render_schema_block(context, sort_alphabetically=sort_schema_block)
46+
schema_text = render_schema_block(
47+
context,
48+
sort_alphabetically=sort_schema_block,
49+
enable_bird_rescue_hints=enable_bird_rescue_hints,
50+
)
4651
# CHASE-SQL divide-and-conquer prompt — decomposes multi-clause questions
4752
# into sub-questions before composing SQL. Driven by
4853
# `PipelineConfig.use_dac_prompt`; api/main.py bootstraps from `NLSQL_DAC=1`.

‎src/nl_sql/agent/nodes/plan_query.py‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ def make_plan_node(
2828
max_tokens: int = 600,
2929
temperature: float = 0.0,
3030
sort_schema_block: bool = False,
31+
enable_bird_rescue_hints: bool = False,
3132
) -> Callable[[PipelineState], PipelineState]:
3233
def node(state: PipelineState) -> PipelineState:
3334
question = state.get("question", "")
@@ -36,7 +37,11 @@ def node(state: PipelineState) -> PipelineState:
3637
prompt = load_prompt(
3738
"plan",
3839
dialect=dialect,
39-
schema_block=render_schema_block(context, sort_alphabetically=sort_schema_block),
40+
schema_block=render_schema_block(
41+
context,
42+
sort_alphabetically=sort_schema_block,
43+
enable_bird_rescue_hints=enable_bird_rescue_hints,
44+
),
4045
fewshot_block=render_fewshot_block(context),
4146
question=question,
4247
)

‎src/nl_sql/agent/nodes/repair_once.py‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ def make_repair_once_node(
2525
max_tokens: int = 1024,
2626
temperature: float = 0.0,
2727
sort_schema_block: bool = False,
28+
enable_bird_rescue_hints: bool = False,
2829
) -> Callable[[PipelineState], PipelineState]:
2930
def node(state: PipelineState) -> PipelineState:
3031
generated = state.get("generated")
@@ -39,7 +40,11 @@ def node(state: PipelineState) -> PipelineState:
3940
prompt = load_prompt(
4041
"repair_sql",
4142
dialect=dialect,
42-
schema_block=render_schema_block(context, sort_alphabetically=sort_schema_block),
43+
schema_block=render_schema_block(
44+
context,
45+
sort_alphabetically=sort_schema_block,
46+
enable_bird_rescue_hints=enable_bird_rescue_hints,
47+
),
4348
question=question,
4449
previous_sql=previous_sql,
4550
error_context=error_context,

0 commit comments

Comments
 (0)