Optimize PostgreSQL metadata passthrough

Author: benben

benchmark/postgres/.gitignore

@@ -0,0 +1,4 @@
+__pycache__/
+results/*.jsonl
+!results/release.jsonl
+!results/commit_stats.jsonl

benchmark/postgres/README.md

@@ -0,0 +1,27 @@
+# PostgreSQL metadata benchmarks
+
+`run.py` creates deterministic metadata-only fixtures and measures metadata operations through DuckDB. Fixture setup is excluded from the measured interval. Every trial uses a fresh PostgreSQL schema so mutation benchmarks are independent.
+
+The supported scales are `small` (100), `medium` (10,000), `large` (100,000), and `xlarge` (1,000,000). Depending on the operation, the scale controls snapshots, data files, or cleanup candidates.
+
+The runner expects a PostgreSQL-enabled DuckLake build at `/work/ducklake`, the PostgreSQL scanner extension beside that build, and standard `PGHOST`, `PGPORT`, `PGDATABASE`, `PGUSER`, `PGPASSWORD`, and `PGSSLMODE` environment variables. For example:
+
+```sh
+python3 benchmark/postgres/run.py \
+  --variant feature \
+  --scale xlarge \
+  --operation expire_snapshots \
+  --selectivity 0.5 \
+  --trials 3 \
+  --output /work/results/fast.jsonl
+```
+
+The benchmark compares the existing PostgreSQL passthrough manager before and after the change; there is no separate adapter mode. The Kubernetes manifest is intentionally credential-free. It expects a `metadata-postgres` Secret containing the PostgreSQL environment variables.
+
+Reported timings are medians of three trials from the same protected in-cluster pod. The release matrix contains pristine PostHog `posthog/v1.5.3` and the feature build, both using the existing PostgreSQL passthrough manager. Each timed trial launches a fresh DuckDB CLI, loads the PostgreSQL extension, attaches the catalog, runs the operation, and exits, so these are cold-process end-to-end measurements rather than PostgreSQL server execution time.
+
+`stats_lookup_micro` is an isolated query-shape benchmark. It uses `postgres_query` so filtering happens in PostgreSQL, matching the metadata manager. It is not a full commit benchmark.
+
+Each trial records actual PostgreSQL fixture counts before the timed operation in `fixture_row_counts`, their sum in `fixture_rows`, and the operation's logical output cardinality in `result_rows`. These are observed row counts, not estimates. They deliberately are not called “rows processed”: PostgreSQL may use indexes or other plans that avoid scanning every stored row.
+
+`max_rss_kib` is the Python process's cumulative `RUSAGE_CHILDREN` high-water mark. It can include fixture children and earlier trials, so it is retained for diagnostics but is not used for memory-improvement claims. The curated JSONL files contain the raw measurements used by [RESULTS.md](RESULTS.md); do not summarize the whole results directory because exploratory runs may contain duplicate trial identities.

benchmark/postgres/RESULTS.md

@@ -0,0 +1,28 @@
+# PostgreSQL metadata benchmark results
+
+These results compare pristine PostHog `posthog/v1.5.3` with the feature build. Both use the fork's existing PostgreSQL passthrough manager; the feature improves that path directly rather than introducing a parallel adapter. Measurements were collected on 2026-07-21 against PostgreSQL 18.3 from the protected `ducklake-bench-runner` pod.
+
+The baseline is PostHog commit `49ec0dc880a749e059594ee699c87ba3a9f88524`. Both builds use DuckDB `14eca11bd9d4a0de2ea0f078be588a9c1c5b279c`, extension-ci-tools `795096d04b009c0d087468439ebb526a5460dfac`, and the official DuckDB v1.5.3 PostgreSQL extension. Trials alternate baseline and feature. There are 9 trials at small, 7 at medium, 5 at large, and 3 at xlarge for each variant and operation.
+
+## Xlarge results
+
+| Operation | Fixture metadata rows | Result rows | v1.5.3 baseline | Feature | Change | Speedup |
+|---|---:|---:|---:|---:|---:|---:|
+| Expire snapshots | 2,000,000 | 500,000 | 14.160 s | 6.586 s | -53.49% | **2.15x** |
+| Stats lookup microbenchmark | 1,010,000 | 100 | 1.182 s | 0.378 s | -68.00% | **3.12x** |
+| Aggregate table information | 1,101,000 | 1,000 | 0.799 s | 0.557 s | -30.27% | **1.43x** |
+| List snapshots | 2,000,000 | 1,000,000 | 4.997 s | 4.959 s | -0.77% | 1.01x |
+| Current snapshot | 1,000,000 | 1 | 0.363 s | 0.359 s | -0.94% | 1.01x |
+| Cleanup candidates | 1,000,000 | 1,000,000 | 1.805 s | 1.838 s | +1.86% | 0.98x |
+| Find snapshots to expire (`dry_run`) | 2,000,000 | 500,000 | 2.150 s | 2.186 s | +1.66% | 0.98x |
+| File discovery | 1,100,001 | 1,000,000 | 4.461 s | 4.500 s | +0.87% | 0.99x |
+
+`Fixture metadata rows` is the sum of the actual relevant PostgreSQL table counts captured before timing; each JSON row also contains the per-table breakdown in `fixture_row_counts`. `Result rows` is the logical operation cardinality. These are observed counts, not planner-level “rows processed.”
+
+The expiration fixture contains 1,000,000 snapshots and 1,000,000 snapshot-change rows, with 500,000 snapshots selected. The table-information fixture contains 1,000 tables, 1,000,000 data files, and 100,000 delete files. File discovery has one table plus the same file counts. The stats fixture contains 10,000 table-stat rows and 1,000,000 column-stat rows; the feature fetches the 100 rows for the written table.
+
+Snapshot listing, current-snapshot lookup, cleanup-candidate listing, dry-run selection, and file discovery use the existing passthrough implementations. Their changes are within approximately 2% and are treated as noise, not claimed improvements.
+
+The authoritative artifact contains 384 unique rows across eight operations, four scales, and two variants: [`results/release.jsonl`](results/release.jsonl). The 48 stats-only rows are extracted to [`results/commit_stats.jsonl`](results/commit_stats.jsonl).
+
+`max_rss_kib` remains diagnostic only because Python's `RUSAGE_CHILDREN` high-water mark is cumulative.

benchmark/postgres/k8s/runner-pod.yaml

@@ -0,0 +1,37 @@
+apiVersion: v1
+kind: Pod
+metadata:
+  name: ducklake-bench-runner
+  namespace: ducklake-bench
+  annotations:
+    karpenter.sh/do-not-disrupt: "true"
+  labels:
+    app.kubernetes.io/name: ducklake-bench-runner
+spec:
+  restartPolicy: Never
+  containers:
+    - name: runner
+      image: ubuntu:24.04
+      command: ["sleep", "infinity"]
+      envFrom:
+        - secretRef:
+            name: metadata-postgres
+      env:
+        - name: DEBIAN_FRONTEND
+          value: noninteractive
+      resources:
+        requests:
+          cpu: "4"
+          memory: 16Gi
+          ephemeral-storage: 40Gi
+        limits:
+          cpu: "8"
+          memory: 32Gi
+          ephemeral-storage: 80Gi
+      volumeMounts:
+        - name: work
+          mountPath: /work
+  volumes:
+    - name: work
+      emptyDir:
+        sizeLimit: 80Gi

benchmark/postgres/results/.gitkeep

@@ -0,0 +1 @@
+