[VibeSWE]

Stacked on #146, which this PR's base branch contains. Closes #144. Refs #141, #142, #143, #145.

Problem

All of Parseable's metadata (stream.json with its snapshot, manifests, schemas, RBAC, alerts, filters, dashboards, the node registry) lives as JSON objects in the object-store bucket. Two consequences:

• Every query pays two or three serial metadata reads before it touches data. With a bucket about 150 ms away that is roughly 0.3–0.5 s per query.
• Most metadata updates are unconditional read-modify-write, so concurrent nodes can lose each other's changes (Metadata updates can overwrite each other: stream.json, manifests, schemas, RBAC, alert state, filters and node registry are unconditional read-modify-write #141), and there is no transactional authority to build compaction or coordinated deletion on.

Change

An opt-in PostgreSQL implementation of the Metastore trait. Object storage remains the default metastore and needs no new configuration. With P_METASTORE_POSTGRES_URL set, metadata documents live in a parseable_documents table and Parquet data stays in the bucket. A marker object, .parseable/postgres-metastore-id, pairs a bucket with a database.

Backend

• A shared document layer serves both backends, so key mapping, tenant scoping and encodings cannot drift. The object-store path was diffed method by method against main and is identical apart from the wrapper.
• Every document has a monotonically increasing version. Conditional create, update and delete are single atomic statements.
• Embedded, forward-only migrations run under a transaction-scoped advisory lock. A node refuses a database with a newer schema, a different identity, or a missing pairing marker. There is no fallback to bucket metadata.
• Keys use COLLATE "C" with explicit range predicates, so prefix reads use the primary key. Listings are tenant-scoped and key-only. Manifests for a query are fetched in batched statements bounded by count and bytes, through the manifest cache.
• Every SQL call has a client-side deadline; a timed-out statement is cancelled where possible and its connection discarded (a write that timed out can still have committed; this is documented). Migrations have their own, longer, configurable budget. Readiness returns 503 when either the database or the bucket is unavailable, in bounded time even when packets are dropped.

Stream lifecycle in PostgreSQL mode

• Delete: commit a durable deleting marker, delete the bucket data, remove the stream's rows in one transaction. Repeating the request completes an interrupted deletion.
• Creating a stream, or publishing a manifest for it, while its deletion is pending is refused atomically with a 409. An uploader that hits this drops the staged files for that stream after settling the batch and removing its uploaded Parquet, instead of retrying forever. A publication delayed past the final row deletion is still not fenced; that needs a stream incarnation token (Stream deletion is not fenced against concurrent ingest, upload and running queries; the deleting guards are never armed #142).
• Retention finds and retries manifest rows left behind by an interrupted partition deletion.

Offline migration

parseable s3-store --metadata-migration import  --acknowledge-server-stopped [--dry-run]
parseable s3-store --metadata-migration verify  --acknowledge-server-stopped
parseable s3-store --metadata-migration export  --acknowledge-server-stopped

• Import copies every metadata document for every tenant with tenant, key and bytes preserved, validates before committing, and establishes pairing only after a verified import. The bucket's own metadata is left untouched as the rollback copy.
• Export validates every row before writing anything. Digests recorded at import let a normal rollback proceed without a force flag; overwriting objects changed since import, including ones deleted from the bucket since, needs --overwrite-bucket-changes-since-import, and every conflict is listed first. export --dry-run runs all of these checks read-only, so a rollback can be verified right after an import.
• --abort exists for both, refused once something irreversible has happened. A transfer and an abort share a session-level advisory lock, so a second invocation fails fast. Database and pairing checks run before any bucket inventory, and --dry-run performs them read-only.
• The inventory lists only where metadata lives and never descends into hour or minute directories. No object-store I/O happens inside the SQL transaction, the ordinary connection is released while documents are spooled, and reads run 16 at a time.
• Object-store mode refuses to start on a paired bucket unless P_ALLOW_PAIRED_BUCKET is set to the pairing id.

Settings

All prefixed P_METASTORE_POSTGRES_:

Setting	Default	Meaning
URL	unset	Setting it opts in. Unset or empty keeps the object-store metastore
POOL_SIZE	16	Connection pool size
CONNECT_TIMEOUT_MS	5000	Also bounds waits for a pooled connection
STATEMENT_TIMEOUT_MS	10000	Server-side limit; client deadlines derive from it
CA	none	PEM bundle. When set, only this CA is trusted. Required for managed databases whose CA is not publicly trusted
ID	none	Pins the expected backend UUID
MIGRATION_TIMEOUT_MS	1800000	Separate budget for schema migrations, which take an exclusive table lock
ALLOW_UNPAIRED	unset	Recovery override for a database that holds data but whose bucket has no pairing marker. Must be set to the backend UUID; see the operator document

TLS is required unless the URL says sslmode=disable.

Behaviour changes in default (object-store) mode

Four defects found by the new contract tests exist on main and are fixed here:

• Targets were written with a tenant field but read back expecting tenantId, so the tenant was lost on reload. They now write tenantId; documents with the old field still load.
• Alert listing included mttr.json.
• Local-filesystem listing consulted the global storage provider instead of the selected instance.
• Startup tenant cleanup could delete earlier invalid-tenant data before a later metadata read failed. It now deletes only after discovery and validation have fully succeeded.

Further default-mode changes from the second review round: tenant normalisation now covers metadata paths (a DEFAULT_TENANT alias and no tenant resolve to the same keys), default alert-state enumeration is corrected, and explicit-stream ingest propagates a lookup error instead of reporting 404.

Also in default mode: two extra GETs at startup (the pairing and migration marker probes), and the PostgreSQL client is compiled into every build (no cargo feature).

Effect on the query path

On local emulators, a repeated query makes one object-store request in PostgreSQL mode where the default mode makes four to nine, depending on topology: the metadata reads are gone and the Parquet read remains. This has been counted in requests, not yet measured as latency against a distant bucket.

Not in this PR

• Conditional writes for documents other than dashboards (Metadata updates can overwrite each other: stream.json, manifests, schemas, RBAC, alert state, filters and node registry are unconditional read-modify-write #141). Both backends still lose concurrent whole-document updates.
• Fencing of stream deletion against concurrent writers and readers (Stream deletion is not fenced against concurrent ingest, upload and running queries; the deleting guards are never armed #142), and automatic resumption of interrupted deletion (Interrupted stream deletion is not resumed; it completes only when the DELETE request is repeated #143).
• A semantic catalog API (atomic publication, reader pins), compaction, and a bucket export for upstream readers. The single documents table is interim for manifests.
• Older binaries ignore the pairing marker and would start on the stale bucket metadata after an import. This is documented as a hazard in the procedure.

Tests

• Library suite passes with and without a database (1,405 passed, 7 ignored).
• A contract suite holds both backends to the same expected values: distinct data per tenant, exact bytes, overwrite and read-back, absence after delete, a 10,000-entry compressed manifest, keys with unusual characters, concurrent conditional writes.
• Fault and restart tests against PostgreSQL 17 and MinIO: interrupted stream and partition deletion, a paused and a blackholed database, pool exhaustion, pairing failures, tenant discovery from both sources.
• Import and export: byte-exact round trip, an injected failure at each step with the documented recovery command, names with escaped characters through MinIO, unforced rollback after running in PostgreSQL mode.
• End-to-end scripts in object-store, PostgreSQL and PostgreSQL-with-TLS modes. docs/validation/ holds their generated evidence; machine-specific paths and host names in those files were redacted.
• Query plans for the hot statements on a 200,400-row table are in docs/validation/postgres-review-explain.txt.
• A new workflow, .github/workflows/postgres-metastore.yaml, runs the PostgreSQL tests and both scripts and fails if they were skipped. This PR is its first run.

Review status

Draft. The first backend commits were read by four independent reviewers and the import tool by one; their findings were addressed in the first commit of this PR. That commit was then reviewed again by two reviewers (three major and three minor findings on the backend; three major and six minor on the import and export tool; verdict that import is safe on a real deployment and export was not yet safe as a rollback). The six commits on top address all of those. They have not had an independent review, hosted CI has not yet produced a Rust result for this branch, and the tool's throughput at production scale is unmeasured.

Operator documentation, including the step-by-step migration and rollback procedure: docs/postgres-metastore.md.

🤖 Generated with Claude Code