Skip to content

Commit 742647e

Browse files
JuliaEdomclaude
andcommitted
fix(db): make the Postgres path real and genuinely read-only; anchor paths; sync ProviderName
The Postgres path was doubly broken and had never actually run: - Its read-only "defence" was a placebo. `SET default_transaction_read_only = on` was issued from inside an already-open transaction (SQLAlchemy autobegins before the first statement), and Postgres fixes a transaction's read-only mode at BEGIN — so the current transaction stayed read-write. Real enforcement now comes from the engine's `postgresql_readonly` execution option, applied at transaction start, which holds even for a superuser DSN. Proven by tests/test_postgres_readonly.py running against a live postgres:16 service in CI (skipped locally without a DSN): INSERT/UPDATE/DELETE/CREATE/SELECT-INTO all rejected, SELECT works. - A bare `postgresql://` DSN resolves to the psycopg2 driver, which isn't installed (only psycopg 3 is) — every connection would have died with ModuleNotFoundError. DSNs are now normalised to `postgresql+psycopg://`. scripts/load_postgres.py loads a shipped SQLite DB into Postgres (codebase_community is the StackExchange-derived slice), and the registry registers a PG database when NL_SQL_PG_DSN is set (point it at the read-only role for defence in depth). Also folds in the CWD-independence fix (C1) and ProviderName sync (C2): - nl_sql.paths.PROJECT_ROOT anchors chroma_data / data / eval baseline paths to the repo root, so launching from any CWD no longer silently yields "0 databases". - ProviderName gains perplexity + gracekelly (the factory already supported both); NL_SQL_DEFAULT_PROVIDER=gracekelly no longer crashes Settings validation. CI: new pg-integration job (postgres:16 service) runs the read-only proof; the frozen-scripts gate is wired into repo-hygiene. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent f7d5f6f commit 742647e

10 files changed

Lines changed: 450 additions & 9 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,3 +50,47 @@ jobs:
5050

5151
- name: Check no internal notes are tracked
5252
run: python3 scripts/check_repo_hygiene.py
53+
54+
- name: Verify compare_results gate detector
55+
run: python3 scripts/check_no_raw_compare.py --self-test
56+
57+
- name: No raw compare_results outside scripts/archive/
58+
run: python3 scripts/check_no_raw_compare.py
59+
60+
pg-integration:
61+
# Live Postgres 16 proof that the pipeline connection is genuinely read-only.
62+
# These tests skip locally (no NL_SQL_PG_TEST_DSN); here they run for real.
63+
runs-on: ubuntu-latest
64+
timeout-minutes: 10
65+
services:
66+
postgres:
67+
image: postgres:16-alpine
68+
env:
69+
POSTGRES_DB: nl_sql_demo
70+
POSTGRES_USER: postgres
71+
POSTGRES_PASSWORD: postgres
72+
ports:
73+
- 5432:5432
74+
options: >-
75+
--health-cmd "pg_isready -U postgres -d nl_sql_demo"
76+
--health-interval 5s
77+
--health-timeout 3s
78+
--health-retries 10
79+
steps:
80+
- uses: actions/checkout@v4
81+
82+
- name: Install uv
83+
uses: astral-sh/setup-uv@v3
84+
with:
85+
version: "0.8.23"
86+
87+
- name: Set up Python
88+
run: uv python install 3.13
89+
90+
- name: Sync dependencies
91+
run: uv sync --extra dev
92+
93+
- name: Read-only Postgres proof
94+
env:
95+
NL_SQL_PG_TEST_DSN: postgresql://postgres:postgres@localhost:5432/nl_sql_demo
96+
run: uv run pytest tests/test_postgres_readonly.py -v

‎scripts/load_postgres.py‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
"""Load a SQLite target database into Postgres 16.
2+
3+
Makes the "Postgres 16 + StackExchange-mini" claim real: it copies a shipped
4+
SQLite database (Chinook, or any BIRD Mini-Dev slice — ``codebase_community`` is
5+
the StackExchange-derived one) table-by-table into a Postgres schema the pipeline
6+
can then query through its read-only connection.
7+
8+
Run against the docker-compose Postgres (see docker-compose.yml, profile
9+
``postgres``). Loading requires WRITE privileges, so pass the OWNER DSN here
10+
(``postgres`` superuser), NOT the ``nl_sql_ro`` role the pipeline uses at query
11+
time:
12+
13+
docker compose --profile postgres up -d
14+
uv run python scripts/load_postgres.py \
15+
--sqlite data/bird_mini_dev/MINIDEV/dev_databases/codebase_community/codebase_community.sqlite \
16+
--pg-dsn postgresql://postgres:postgres@localhost:5433/nl_sql_demo
17+
18+
The tables are created with ``if_exists=replace`` (idempotent re-runs). Type
19+
mapping is delegated to pandas/SQLAlchemy, which is sufficient for these demo
20+
databases; this is a load tool, not a general SQLite→Postgres migrator.
21+
"""
22+
23+
from __future__ import annotations
24+
25+
import argparse
26+
import sqlite3
27+
import sys
28+
from pathlib import Path
29+
30+
import pandas as pd
31+
from sqlalchemy import Engine, create_engine, text
32+
33+
from nl_sql.db.connection import _normalise_pg_driver
34+
35+
36+
def _sqlite_tables(sqlite_path: Path) -> list[str]:
37+
con = sqlite3.connect(f"file:{sqlite_path}?mode=ro", uri=True)
38+
try:
39+
rows = con.execute(
40+
"SELECT name FROM sqlite_master WHERE type = 'table' "
41+
"AND name NOT LIKE 'sqlite_%' ORDER BY name"
42+
).fetchall()
43+
finally:
44+
con.close()
45+
return [r[0] for r in rows]
46+
47+
48+
def load_sqlite_into_postgres(
49+
sqlite_path: Path,
50+
pg_dsn: str,
51+
*,
52+
only_tables: list[str] | None = None,
53+
chunk_size: int = 5_000,
54+
) -> dict[str, int]:
55+
"""Copy every table from the SQLite file into Postgres. Returns row counts."""
56+
if not sqlite_path.exists():
57+
raise FileNotFoundError(f"SQLite database not found: {sqlite_path}")
58+
59+
tables = only_tables or _sqlite_tables(sqlite_path)
60+
src = sqlite3.connect(f"file:{sqlite_path}?mode=ro", uri=True)
61+
dst = create_engine(_normalise_pg_driver(pg_dsn), future=True)
62+
loaded: dict[str, int] = {}
63+
try:
64+
for table in tables:
65+
total = 0
66+
first = True
67+
for frame in pd.read_sql_query(f'SELECT * FROM "{table}"', src, chunksize=chunk_size):
68+
frame.to_sql(
69+
table,
70+
dst,
71+
if_exists="replace" if first else "append",
72+
index=False,
73+
method="multi",
74+
)
75+
total += len(frame)
76+
first = False
77+
if first:
78+
# Table was empty — still create it so the schema matches.
79+
pd.read_sql_query(f'SELECT * FROM "{table}" LIMIT 0', src).to_sql(
80+
table, dst, if_exists="replace", index=False
81+
)
82+
loaded[table] = total
83+
print(f" {table}: {total} rows", flush=True)
84+
_grant_readonly(dst)
85+
finally:
86+
src.close()
87+
dst.dispose()
88+
return loaded
89+
90+
91+
def _grant_readonly(engine: Engine) -> None:
92+
"""Best-effort: grant SELECT to nl_sql_ro if that role exists.
93+
94+
The role is created by scripts/sql/postgres_init.sql on first container boot.
95+
Newly created tables are not covered by the role's default privileges until
96+
granted, so re-grant after a load. Silently skipped if the role is absent.
97+
"""
98+
with engine.begin() as conn:
99+
exists = conn.execute(text("SELECT 1 FROM pg_roles WHERE rolname = 'nl_sql_ro'")).fetchone()
100+
if exists:
101+
conn.execute(text("GRANT SELECT ON ALL TABLES IN SCHEMA public TO nl_sql_ro"))
102+
103+
104+
def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
105+
parser = argparse.ArgumentParser(description="Load a SQLite DB into Postgres.")
106+
parser.add_argument("--sqlite", required=True, type=Path, help="path to the .sqlite file")
107+
parser.add_argument("--pg-dsn", required=True, help="owner DSN, e.g. postgresql://postgres:...")
108+
parser.add_argument("--table", action="append", dest="tables", help="limit to table (repeat)")
109+
parser.add_argument("--chunk", type=int, default=5_000, help="rows per insert batch")
110+
return parser.parse_args(argv)
111+
112+
113+
def main(argv: list[str] | None = None) -> int:
114+
args = _parse_args(argv)
115+
print(f"Loading {args.sqlite} -> Postgres", flush=True)
116+
counts = load_sqlite_into_postgres(
117+
args.sqlite, args.pg_dsn, only_tables=args.tables, chunk_size=args.chunk
118+
)
119+
print(f"Done: {len(counts)} tables, {sum(counts.values())} rows total", flush=True)
120+
return 0
121+
122+
123+
if __name__ == "__main__":
124+
sys.exit(main())

‎src/nl_sql/api/main.py‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,6 @@
2525
import uuid
2626
from collections import defaultdict, deque
2727
from functools import lru_cache
28-
from pathlib import Path
2928
from typing import Any, NamedTuple
3029

3130
from fastapi import Depends, FastAPI, Header, HTTPException, Request, status
@@ -44,6 +43,7 @@
4443
from nl_sql.llm.providers import build_provider
4544
from nl_sql.llm.providers.base import EmbeddingProvider, LLMProvider
4645
from nl_sql.llm.providers.mistral import MistralProvider
46+
from nl_sql.paths import under_root
4747
from nl_sql.schema_index.indexer import SchemaIndex
4848

4949
logger = logging.getLogger("nl_sql.api")
@@ -224,8 +224,12 @@ def _build_pipeline_components(
224224
embedder: EmbeddingProvider = CachingEmbeddingProvider(
225225
raw_embed, cache_dir=settings.llm_cache_dir
226226
)
227-
schema_index = SchemaIndex(persist_dir="chroma_data", embedder=embedder)
228-
registry = get_default_registry()
227+
schema_index = SchemaIndex(persist_dir=str(under_root("chroma_data")), embedder=embedder)
228+
registry = get_default_registry(
229+
pg_dsn=settings.pg_dsn,
230+
pg_db_id=settings.pg_db_id,
231+
pg_description=settings.pg_description,
232+
)
229233
return registry, schema_index, sql_provider, explain_provider
230234

231235

@@ -425,7 +429,7 @@ def eval_latest() -> EvalLatestResponse:
425429
"""Returns metadata of the latest hybrid eval report committed to repo."""
426430
import json
427431

428-
baseline = Path("eval/baselines/hybrid_n200_v0.json")
432+
baseline = under_root("eval", "baselines", "hybrid_n200_v0.json")
429433
if not baseline.exists():
430434
raise HTTPException(
431435
status_code=status.HTTP_404_NOT_FOUND,

‎src/nl_sql/config/settings.py‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,9 @@
44
from pydantic import Field
55
from pydantic_settings import BaseSettings, SettingsConfigDict
66

7-
ProviderName = Literal["mistral", "github_models", "groq", "ollama", "openrouter"]
7+
ProviderName = Literal[
8+
"mistral", "github_models", "groq", "ollama", "openrouter", "perplexity", "gracekelly"
9+
]
810

911

1012
class Settings(BaseSettings):
@@ -79,6 +81,14 @@ class Settings(BaseSettings):
7981
# value in .env is honoured, per the api/main module docstring.
8082
api_key: str = Field(default="", validation_alias="NL_SQL_API_KEY")
8183

84+
# Optional Postgres target. When `pg_dsn` is set, the registry registers a
85+
# Postgres-backed database under `pg_db_id`. Point it at the read-only role
86+
# (nl_sql_ro) for defence in depth — the engine also forces read-only
87+
# transactions on top (see db/connection.py). Empty = SQLite-only (default).
88+
pg_dsn: str = Field(default="", validation_alias="NL_SQL_PG_DSN")
89+
pg_db_id: str = "pg_codebase_community"
90+
pg_description: str = "StackExchange-derived BIRD codebase_community, loaded into Postgres 16."
91+
8292
# diskcache for LLM generate/embed responses (per docs/02_architecture_v2.md §6.5).
8393
# Two subdirs ("gen", "embed") are created under this root by `nl_sql.llm.cache`.
8494
llm_cache_dir: str = ".cache/llm"

‎src/nl_sql/db/connection.py‎

Lines changed: 41 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@
1919
from typing import Any, Literal
2020

2121
from sqlalchemy import Connection, Engine, create_engine, text
22+
from sqlalchemy.engine import make_url
2223

2324
Dialect = Literal["sqlite", "postgresql"]
2425

@@ -53,10 +54,44 @@ def _build_engine(spec: DatabaseSpec) -> Engine:
5354
if spec.dialect == "sqlite":
5455
return _build_sqlite_readonly_engine(Path(spec.url))
5556
if spec.dialect == "postgresql":
56-
return create_engine(spec.url, future=True, pool_pre_ping=True)
57+
return _build_postgres_readonly_engine(spec.url)
5758
raise ValueError(f"unsupported dialect: {spec.dialect}")
5859

5960

61+
def _normalise_pg_driver(url: str) -> str:
62+
"""Force the psycopg (v3) driver for bare ``postgresql://`` DSNs.
63+
64+
SQLAlchemy resolves an unqualified ``postgresql://`` to the psycopg2 driver,
65+
which is not installed here (only psycopg 3 is). Left unqualified, every
66+
Postgres connection would die with ModuleNotFoundError — which is exactly
67+
why the Postgres path had never actually run. Explicit ``+psycopg`` /
68+
``+psycopg2`` DSNs are respected as-is.
69+
"""
70+
parsed = make_url(url)
71+
if parsed.drivername == "postgresql":
72+
parsed = parsed.set(drivername="postgresql+psycopg")
73+
return parsed.render_as_string(hide_password=False)
74+
75+
76+
def _build_postgres_readonly_engine(url: str) -> Engine:
77+
"""Engine whose every transaction is genuinely READ ONLY.
78+
79+
The read-only guarantee comes from SQLAlchemy's ``postgresql_readonly``
80+
execution option, which sets the connection read-only at transaction start
81+
(the correct point). An earlier implementation issued
82+
``SET default_transaction_read_only = on`` from inside an already-open
83+
transaction — Postgres fixes a transaction's read-only status at BEGIN, so
84+
that statement was a no-op for the current transaction and the "layer 1"
85+
read-only defence never actually engaged.
86+
"""
87+
return create_engine(
88+
_normalise_pg_driver(url),
89+
future=True,
90+
pool_pre_ping=True,
91+
execution_options={"postgresql_readonly": True},
92+
)
93+
94+
6095
def _build_sqlite_readonly_engine(path: Path) -> Engine:
6196
if not path.is_absolute():
6297
path = path.resolve()
@@ -114,8 +149,12 @@ def execute_readonly(
114149
def _apply_runtime_limits(conn: Connection, statement_timeout_ms: int) -> None:
115150
dialect = conn.engine.dialect.name
116151
if dialect == "postgresql":
152+
# statement_timeout takes effect immediately for subsequent statements in
153+
# this session, so setting it here (inside the transaction) is correct.
154+
# Read-only is NOT set here — it is applied at transaction start by the
155+
# engine's postgresql_readonly execution option (see
156+
# _build_postgres_readonly_engine); issuing it here would be a no-op.
117157
conn.execute(text(f"SET statement_timeout = {int(statement_timeout_ms)}"))
118-
conn.execute(text("SET default_transaction_read_only = on"))
119158
elif dialect == "sqlite":
120159
# SQLite has no per-query timeout knob in the URL API; use connection
121160
# progress handler at the dbapi level.

‎src/nl_sql/db/registry.py‎

Lines changed: 26 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,11 @@
1212
from pathlib import Path
1313

1414
from nl_sql.db.connection import DatabaseSpec, sqlite_url_readonly
15+
from nl_sql.paths import under_root
1516

16-
DATA_ROOT = Path("data")
17+
# Anchored to the repo root (not CWD) so scanning finds the data/ tree no matter
18+
# where the process was launched from — Streamlit/uvicorn/pytest don't agree on CWD.
19+
DATA_ROOT = under_root("data")
1720

1821

1922
@dataclass(slots=True)
@@ -32,15 +35,36 @@ def ids(self) -> list[str]:
3235
return sorted(self.specs)
3336

3437

35-
def get_default_registry(data_root: Path = DATA_ROOT) -> DatabaseRegistry:
38+
def get_default_registry(
39+
data_root: Path = DATA_ROOT,
40+
*,
41+
pg_dsn: str = "",
42+
pg_db_id: str = "pg_codebase_community",
43+
pg_description: str = "",
44+
) -> DatabaseRegistry:
3645
"""Build a registry by scanning the data/ tree.
3746
3847
Resolution order:
3948
- data/chinook/Chinook.sqlite → id="chinook"
4049
- data/bird_mini_dev/MINIDEV/dev_databases/<db>/<db>.sqlite → id=f"bird_{db}"
50+
51+
When ``pg_dsn`` is non-empty, a Postgres-backed database is also registered
52+
under ``pg_db_id`` (load it first with scripts/load_postgres.py). The DSN
53+
should point at the read-only role; the engine additionally forces read-only
54+
transactions (see db/connection.py).
4155
"""
4256
registry = DatabaseRegistry()
4357

58+
if pg_dsn:
59+
registry.register(
60+
DatabaseSpec(
61+
id=pg_db_id,
62+
dialect="postgresql",
63+
url=pg_dsn,
64+
description=pg_description,
65+
)
66+
)
67+
4468
chinook_path = data_root / "chinook" / "Chinook.sqlite"
4569
if chinook_path.exists():
4670
registry.register(

‎src/nl_sql/paths.py‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
"""Repo-root-anchored filesystem paths.
2+
3+
Code that opens files with paths relative to the current working directory
4+
(e.g. ``Path("chroma_data")``) breaks silently when the process is launched
5+
from anywhere other than the repo root — Streamlit, uvicorn, pytest, and
6+
deployment entrypoints (Docker CMD, HF Spaces) don't all guarantee that CWD.
7+
This module resolves the repo root once, from this file's own location, so
8+
downstream code stays correct regardless of CWD.
9+
"""
10+
11+
from __future__ import annotations
12+
13+
from pathlib import Path
14+
15+
# src/nl_sql/paths.py -> parents[0]=src/nl_sql, [1]=src, [2]=repo root.
16+
PROJECT_ROOT: Path = Path(__file__).resolve().parents[2]
17+
18+
19+
def under_root(*parts: str) -> Path:
20+
"""Return ``PROJECT_ROOT`` joined with ``parts``, CWD-independent."""
21+
return PROJECT_ROOT.joinpath(*parts)

‎tests/test_db_registry.py‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,3 +59,18 @@ def test_default_registry_uses_readonly_urls(tmp_path: Path) -> None:
5959
spec = reg.get("chinook")
6060

6161
assert spec.url == sqlite_url_readonly(chinook)
62+
63+
64+
def test_default_registry_registers_postgres_when_dsn_given(tmp_path: Path) -> None:
65+
dsn = "postgresql://nl_sql_ro:pwd@localhost:5433/nl_sql_demo"
66+
reg = get_default_registry(data_root=tmp_path, pg_dsn=dsn, pg_db_id="pg_demo")
67+
68+
assert "pg_demo" in reg.ids()
69+
spec = reg.get("pg_demo")
70+
assert spec.dialect == "postgresql"
71+
assert spec.url == dsn
72+
73+
74+
def test_default_registry_skips_postgres_without_dsn(tmp_path: Path) -> None:
75+
reg = get_default_registry(data_root=tmp_path)
76+
assert not any(reg.get(db_id).dialect == "postgresql" for db_id in reg.ids())

0 commit comments

Comments
 (0)