CockroachDB metadata-store support report + e2e test harness

experiments/crdb/REPORT.md

@@ -0,0 +1,155 @@
+# CockroachDB as a DuckLake Metadata Store — Support Report
+
+**Date:** 2026-06-12
+**Tested:** DuckDB v1.5.3 (Variegata) + ducklake extension (PostHog fork; patched-build verification on the `v1.5-variegata` branch, DuckDB v1.5.2) against CockroachDB v25.2.19, single node via docker compose (`experiments/crdb/docker-compose.yml`). Postgres 17 used as a control baseline.
+
+## TL;DR
+
+CockroachDB **mostly works** as a DuckLake metadata store today, but is **not production-ready without two fixes**:
+
+1. **Works out of the box: nothing.** The very first attach fails, because DuckDB's postgres scanner defaults to binary `COPY`, which CockroachDB has not implemented. Two global settings fix this (see Workarounds).
+2. **With workarounds applied:** the entire core feature surface works — DDL, DML, snapshots, time travel, schema evolution, transactions, views, partitioning, compaction, data inlining, change feeds, multi-process catalogs.
+3. **Two real blockers remain:**
+   - `ducklake_expire_snapshots()` and `ducklake_flush_inlined_data()` fail (ctid-based DELETE path) — **no workaround**; snapshot/inlined-data cleanup is impossible.
+   - Concurrent writers lose ~25% of commits as hard failures (CockroachDB's `40001 restart transaction` errors are not recognized as retryable by DuckLake). **A one-line DuckLake patch fixes this** (verified — see Concurrency).
+
+Verdict: **experimental / near-supportable**. The gaps are small, well-understood, and fixable upstream — none are architectural.
+
+## Architecture context
+
+DuckLake talks to a Postgres-class catalog two ways (this matters because the two paths fail differently):
+
+1. **Raw SQL passthrough** — `PostgresMetadataManager` wraps every metadata query in `CALL postgres_query(...)` / `postgres_execute(...)` and the text is executed verbatim by the server (`src/metadata_manager/postgres_metadata_manager.cpp:112`). This is ~all metadata reads and writes.
+2. **DuckDB-planned DML on the attached catalog** — a few maintenance operations (notably `DeleteSnapshots`, `src/storage/ducklake_metadata_manager.cpp:4329`) run `DELETE FROM {METADATA_CATALOG}.tbl` through `transaction.Query(...)`, so DuckDB plans the DELETE and the postgres scanner executes it via **ctid row addressing**.
+
+CockroachDB handles path 1 fine (its SQL dialect covered everything DuckLake sends, including `STRING_AGG`, `NULLS FIRST/LAST`, `::UUID` casts, `CREATE TABLE IF NOT EXISTS`, multi-statement batches). Path 2 is where it breaks.
+
+## Test results
+
+28/28 `ducklake_*` metadata tables created cleanly on first attach. Feature matrix (each test in its own process — also exercises re-attach):
+
+| Feature | Result |
+|---|---|
+| ATTACH / catalog bootstrap (all DDL + migrations) | ✅ (with COPY workaround) |
+| CREATE TABLE / INSERT / SELECT | ✅ |
+| UPDATE / DELETE (incl. deletion vectors) | ✅ |
+| `snapshots()`, time travel `AT (VERSION => n)` | ✅ |
+| Schema evolution (add/drop/rename column, type promotion) | ✅ |
+| Multi-schema, views | ✅ |
+| Multi-statement transactions, rollback | ✅ |
+| 20-type stress (incl. HUGEINT, UUID, INTERVAL, STRUCT, MAP, lists) | ✅ |
+| Data inlining (write + read back from CRDB tables) | ✅ |
+| Partitioning (`SET PARTITIONED BY`) | ✅ |
+| Compaction `merge_adjacent_files()` | ✅ |
+| `rewrite_data_files()` | ✅ |
+| `cleanup_old_files()` / `delete_orphaned_files()` | ✅ |
+| `table_changes()` change feed | ✅ |
+| `ducklake_table_info()` / `ducklake_list_files()` | ✅ |
+| Concurrent readers during writes (15 readers vs writer loop) | ✅ 0 failures |
+| **`ducklake_expire_snapshots()`** | ❌ ctid |
+| **`ducklake_flush_inlined_data()`** | ❌ ctid |
+| **Concurrent writers (3×15 commits)** | ❌ 24% hard-fail (fixed by patch below) |
+
+### Blocker 1: binary COPY (workaround exists)
+
+First attach fails with:
+
+```
+COPY (...) TO STDOUT (FORMAT "binary"): ERROR: at or near "binary":
+syntax error: unimplemented  (CockroachDB issue #96590)
+```
+
+The postgres scanner reads via binary COPY and parallelizes scans by `ctid`; CockroachDB implements neither. Both have scanner-level escape hatches, but they must be set **GLOBAL** — DuckLake issues metadata queries on its own internal connection, which doesn't see session-local `SET`:
+
+```sql
+LOAD postgres_scanner;
+SET GLOBAL pg_use_text_protocol = true;  -- slower, but compatible
+SET GLOBAL pg_use_ctid_scan = false;
+ATTACH 'ducklake:postgres:dbname=ducklakedb host=... port=26257 user=root' AS lake (DATA_PATH '...');
+```
+
+### Blocker 2: ctid-based metadata DELETEs (no workaround)
+
+`ducklake_expire_snapshots()` and `ducklake_flush_inlined_data()` fail:
+
+```
+Failed to delete snapshots in DuckLake: Failed to execute query
+"SELECT "snapshot_id", ctid FROM "public"."ducklake_snapshot" WHERE ...":
+ERROR: column "ctid" does not exist
+```
+
+These two operations route DELETEs through DuckDB's DML planner (path 2 above) instead of raw passthrough; the scanner identifies rows to delete by `ctid`, which CockroachDB does not expose. `pg_use_ctid_scan=false` does not affect the DML path.
+
+**Consequence:** snapshots and flushed inlined data can never be expired — the catalog grows forever. For an append-heavy production lake this rules CockroachDB out until fixed.
+
+**Fix difficulty: low.** `DeleteSnapshots` already builds plain `DELETE FROM ... WHERE snapshot_id IN (...)` strings (`src/storage/ducklake_metadata_manager.cpp:4341`); routing them through `PostgresMetadataManager::Execute()` (raw `postgres_execute`, like every other metadata write) instead of `transaction.Query()` would fix both functions on CockroachDB and would also be a minor efficiency win on vanilla Postgres (single server-side DELETE instead of scan-ctids-then-delete).
+
+### Blocker 3: concurrent-writer commits hard-fail (one-line fix, verified)
+
+3 writers × 15 sequential single-row commits each:
+
+| Configuration | Failed commits |
+|---|---|
+| CRDB default (serializable), unpatched DuckLake | 11/45 (24%), **rows lost** |
+| CRDB `read committed` cluster default | no change (scanner's explicit `BEGIN ... REPEATABLE READ` overrides it) |
+| CRDB `repeatable_read_isolation.enabled` | no change |
+| **CRDB default + 1-line DuckLake patch** | **0/45 — all rows landed, retries absorbed every collision** |
+
+Root cause: on commit contention, Postgres surfaces a duplicate-key error on `ducklake_snapshot`'s primary key — DuckLake's `RetryOnError` (`src/storage/ducklake_transaction.cpp:2545`) matches the substring "unique" and retries with backoff. CockroachDB instead aborts the transaction with:
+
+```
+ERROR: restart transaction: TransactionRetryWithProtoRefreshError: WriteTooOldError ...
+```
+
+(SQLSTATE 40001). No substring matches DuckLake's retry list ("primary key", "unique", "conflict", "concurrent"), so the commit fails permanently and the write is lost. CRDB-side isolation tuning can't help because the postgres scanner hardcodes `BEGIN TRANSACTION ISOLATION LEVEL REPEATABLE READ` per transaction.
+
+The fix:
+
+```cpp
+// in RetryOnError(), src/storage/ducklake_transaction.cpp
+// retry on serialization failures (SQLSTATE 40001), e.g. CockroachDB's
+// "restart transaction: TransactionRetryWithProtoRefreshError"
+if (StringUtil::Contains(message, "restart transaction")) {
+    return true;
+}
+```
+
+(A more principled version would match on SQLSTATE 40001 rather than message text — Postgres serialization failures say "could not serialize access", which is *also* unmatched today, so this gap technically exists for vanilla PG under repeatable read too.)
+
+Verified end to end: rebuilt the extension with this patch (worktree on `v1.5-variegata`) and reran the 3×15 concurrency test against default-config CockroachDB — **0/45 failures, every commit retried to success**.
+
+## Performance (directional only)
+
+10 sequential attach+commit cycles, single-node CRDB vs Postgres 17, same host:
+
+- CockroachDB: ~0.51 s/cycle
+- Postgres 17: ~0.27 s/cycle
+
+~2× slower per metadata commit, expected for a consensus-based engine even single-node. Reads of data files (parquet) are unaffected; only catalog round-trips pay the cost. The text-protocol fallback adds overhead on large metadata scans, but DuckLake metadata result sets are small.
+
+## What was NOT tested
+
+- Multi-node CockroachDB cluster (single node only; multi-node adds latency and more 40001 retries — the patch above becomes more important, not less)
+- The full DuckLake SQL test suite (`test/configs/postgres.json`) against CRDB
+- High-volume snapshot accumulation / catalog growth behavior
+- CockroachDB serverless / cloud offerings
+- DuckLake catalog migration between versions on CRDB
+
+## Recommendations
+
+To make CockroachDB a supported metadata store:
+
+1. **Ship the `RetryOnError` patch** (1 line; ideally match SQLSTATE 40001 / "could not serialize access" too, which also benefits vanilla Postgres).
+2. **Route `DeleteSnapshots` / inlined-data-flush DELETEs through raw `postgres_execute`** instead of DuckDB-planned DML (removes the ctid dependency; also fewer round trips on Postgres).
+3. **Auto-set text protocol** when the server is detected as CockroachDB (`SELECT version()` starts with "CockroachDB"), or document the two `SET GLOBAL`s.
+4. Optionally: add a CRDB job to CI mirroring the Postgres job (`cockroachdb/cockroach` single-node image, same test config + the two settings).
+
+With (1) and (2) upstreamed, CockroachDB support would be on par with Postgres for correctness, at ~2× metadata-commit latency.
+
+## Repro artifacts
+
+- `experiments/crdb/docker-compose.yml` — CRDB single node
+- `experiments/crdb/run_tests.sh`, `run_tests2.sh` — feature matrix
+- `experiments/crdb/concurrency_test.sh` — 3-writer concurrency test
+- `experiments/crdb/results/` — raw outputs
+- Patched build worktree: `/tmp/ducklake-crdb-patch` (branch `v1.5-variegata` + RetryOnError patch)

experiments/crdb/concurrency_test.sh

@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+# Two concurrent writers appending to the same DuckLake table backed by CockroachDB.
+cd "$(dirname "$0")/../.." || exit 1
+DUCKDB=build/release/duckdb
+ATTACH="LOAD postgres_scanner; SET GLOBAL pg_use_text_protocol = true; SET GLOBAL pg_use_ctid_scan = false; ATTACH 'ducklake:postgres:dbname=ducklakedb host=localhost port=26257 user=root' AS lake (DATA_PATH 'experiments/crdb/data/');"
+RESULTS=experiments/crdb/results
+N=${N:-15}
+
+$DUCKDB -unsigned -c "$ATTACH CREATE TABLE IF NOT EXISTS lake.conc (writer INTEGER, seq INTEGER);" >/dev/null 2>&1
+
+writer() {
+  local wid=$1
+  local fails=0
+  for i in $(seq 1 "$N"); do
+    if ! $DUCKDB -unsigned -c "$ATTACH INSERT INTO lake.conc VALUES ($wid, $i);" >>"$RESULTS/conc_w$wid.out" 2>&1; then
+      fails=$((fails+1))
+    fi
+  done
+  echo "writer $wid: $fails/$N failed"
+}
+
+writer 1 & writer 2 & writer 3 &
+wait
+$DUCKDB -unsigned -c "$ATTACH SELECT writer, count(*) FROM lake.conc GROUP BY writer ORDER BY writer; SELECT count(*) AS total FROM lake.conc;"

experiments/crdb/docker-compose.yml

@@ -0,0 +1,13 @@
+services:
+  cockroach:
+    image: cockroachdb/cockroach:latest-v25.2
+    container_name: ducklake-crdb
+    command: start-single-node --insecure
+    ports:
+      - "26257:26257"
+      - "8089:8080"
+    healthcheck:
+      test: ["CMD", "cockroach", "sql", "--insecure", "-e", "SELECT 1"]
+      interval: 3s
+      timeout: 5s
+      retries: 20

experiments/crdb/results/01_update.out

@@ -0,0 +1,7 @@
+┌───────┬─────────┐
+│  id   │  name   │
+│ int32 │ varchar │
+├───────┼─────────┤
+│     1 │ updated │
+│     2 │ world   │
+└───────┴─────────┘

experiments/crdb/results/02_delete.out

@@ -0,0 +1,6 @@
+┌──────────────┐
+│ count_star() │
+│    int64     │
+├──────────────┤
+│            1 │
+└──────────────┘

experiments/crdb/results/03_snapshots.out

@@ -0,0 +1,10 @@
+┌─────────────┬───────────────────────────────┬────────────────┬──────────────────────────────────────────┬─────────┬────────────────┬───────────────────┐
+│ snapshot_id │         snapshot_time         │ schema_version │                 changes                  │ author  │ commit_message │ commit_extra_info │
+│    int64    │   timestamp with time zone    │     int64      │         map(varchar, varchar[])          │ varchar │    varchar     │      varchar      │
+├─────────────┼───────────────────────────────┼────────────────┼──────────────────────────────────────────┼─────────┼────────────────┼───────────────────┤
+│           0 │ 2026-06-12 09:51:00.214338-07 │              0 │ {schemas_created=[main]}                 │ NULL    │ NULL           │ NULL              │
+│           1 │ 2026-06-12 09:51:58.59956-07  │              1 │ {tables_created=[main.t1]}               │ NULL    │ NULL           │ NULL              │
+│           2 │ 2026-06-12 09:51:58.688831-07 │              1 │ {inlined_insert=[1]}                     │ NULL    │ NULL           │ NULL              │
+│           3 │ 2026-06-12 09:52:49.269383-07 │              1 │ {inlined_insert=[1], inlined_delete=[1]} │ NULL    │ NULL           │ NULL              │
+│           4 │ 2026-06-12 09:52:49.484299-07 │              1 │ {inlined_delete=[1]}                     │ NULL    │ NULL           │ NULL              │
+└─────────────┴───────────────────────────────┴────────────────┴──────────────────────────────────────────┴─────────┴────────────────┴───────────────────┘

experiments/crdb/results/04_time_travel.out

@@ -0,0 +1,12 @@
+┌───────────┐
+│ now_count │
+│   int64   │
+├───────────┤
+│         2 │
+└───────────┘
+┌───────────┐
+│ old_count │
+│   int64   │
+├───────────┤
+│         2 │
+└───────────┘

experiments/crdb/results/05_schema_evolution.out

@@ -0,0 +1,16 @@
+┌───────┬────────────┬────────┐
+│  id   │   label    │ score  │
+│ int32 │  varchar   │ double │
+├───────┼────────────┼────────┤
+│     1 │ updated    │   NULL │
+│    10 │ v-new      │   NULL │
+│    20 │ with-score │    1.5 │
+└───────┴────────────┴────────┘
+┌───────┬────────────┐
+│  id   │   label    │
+│ int32 │  varchar   │
+├───────┼────────────┤
+│     1 │ updated    │
+│    10 │ v-new      │
+│    20 │ with-score │
+└───────┴────────────┘

experiments/crdb/results/06_type_promotion.out

@@ -0,0 +1,7 @@
+┌─────────────────────┐
+│          x          │
+│        int64        │
+├─────────────────────┤
+│                   1 │
+│ 9223372036854775807 │
+└─────────────────────┘

experiments/crdb/results/07_schemas_views.out

@@ -0,0 +1,6 @@
+┌─────────┬───────┐
+│  event  │   n   │
+│ varchar │ int64 │
+├─────────┼───────┤
+│ click   │     1 │
+└─────────┴───────┘

experiments/crdb/results/08_transactions.out

@@ -0,0 +1,6 @@
+┌─────────────┐
+│ should_be_3 │
+│    int64    │
+├─────────────┤
+│           3 │
+└─────────────┘

experiments/crdb/results/09_types.out

@@ -0,0 +1,6 @@
+┌─────────┬──────┬───────┬───────┬───────┬────────┬───────┬────────┬───────────────┬─────────┬──────────┬────────────┬──────────┬─────────────────────┬──────────────────────────┬──────────┬──────────────────────────────────────┬───────────┬──────────────────────────────┬───────────────────────┐
+│    a    │  b   │   c   │   d   │   e   │   f    │   g   │   h    │       i       │    j    │    k     │     l      │    m     │          n          │            o             │    p     │                  q                   │     r     │              s               │           t           │
+│ boolean │ int8 │ int16 │ int32 │ int64 │ int128 │ float │ double │ decimal(18,4) │ varchar │   blob   │    date    │   time   │      timestamp      │ timestamp with time zone │ interval │                 uuid                 │  int32[]  │ struct(x integer, y varchar) │ map(varchar, integer) │
+├─────────┼──────┼───────┼───────┼───────┼────────┼───────┼────────┼───────────────┼─────────┼──────────┼────────────┼──────────┼─────────────────────┼──────────────────────────┼──────────┼──────────────────────────────────────┼───────────┼──────────────────────────────┼───────────────────────┤
+│ true    │    1 │     2 │     3 │     4 │      5 │   1.5 │    2.5 │      123.4567 │ str     │ \xDE\xAD │ 2026-01-01 │ 12:34:56 │ 2026-01-01 12:34:56 │ 2026-01-01 04:34:56-08   │ 3 days   │ a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11 │ [1, 2, 3] │ {'x': 1, 'y': two}           │ {k=1}                 │
+└─────────┴──────┴───────┴───────┴───────┴────────┴───────┴────────┴───────────────┴─────────┴──────────┴────────────┴──────────┴─────────────────────┴──────────────────────────┴──────────┴──────────────────────────────────────┴───────────┴──────────────────────────────┴───────────────────────┘