Skip to content

Commit f3ccd7f

Browse files
Rusty Conoverclaude
andcommitted
feat(catalog): catalog_contents v2 - typed struct rows, etag revalidation, worker cache
Reshapes the unreleased 2.1.0 catalog_contents per CATALOG_CONTENTS_V2_SPEC: - Wire: CatalogContentsResponse{catalog_version, etag: utf8?, not_modified, schemas: list<struct SchemaContents>}; SchemaContents is an inline struct (path: list<utf8> first, then the SchemaInfo item and the eight list<binary> kinds), no longer a separately IPC-serialized blob. catalog_contents takes if_none_match: utf8 nullable. Protocol version stays 2.1.0. - Catalog API: catalog_contents(*, attach_opaque_data, if_none_match=None) -> CatalogContentsResult(schemas, etag=None, not_modified=False). Opt-ins: catalog_contents_etag = "content-hash" (framework SHA-256 etag, a match becomes not_modified) and catalog_contents_attach_independent (set by ReadOnlyCatalogInterface). - Worker: validates paths (unique, parent present), orders parents first, enforces not_modified semantics (etag == if_none_match, no schemas; no etag never yields not_modified). Version-frozen, attach-independent catalogs are built once per (catalog, catalog_version) and served as a pre-serialized response, so a cache hit skips the Arrow encode entirely (vgi-cloudflare: ~340 ms -> ~1.1 ms per warm call, 17.6 MB response). - Client: contents(attach_opaque_data, if_none_match=None) -> CatalogContents (version, etag, not_modified, schemas), checking path against SchemaInfo. - Fixtures: contents_memory now reports version 0 (exercises the client's version-0 rule); new contents_reval (generation-counter etag, short-circuits without building) and contents_hash (content-hash etag). - Codegen: go/rust/ts/java type emitters map a struct column to its record (SchemaContents); SchemaContents stays in EXTRA_RESPONSE_TYPES as a standalone schema for the per-language parity checks. Sibling SDK and C++ generated files are now stale until regenerated. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
1 parent bed86d1 commit f3ccd7f

19 files changed

Lines changed: 1095 additions & 89 deletions

‎CLAUDE.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -630,6 +630,21 @@ This allows workers to expose structured metadata (tables, views, functions) to
630630
The recommended pattern is to back tables with `TableFunctionGenerator` functions.
631631
The table schema is automatically derived from the function's `output_schema`:
632632

633+
### catalog_contents (whole-catalog load, protocol 2.1.0)
634+
635+
`Worker.catalog_contents(attach_opaque_data, if_none_match=None)` returns
636+
`CatalogContentsResponse{catalog_version, etag, not_modified, schemas: list<struct SchemaContents>}`
637+
(`path` first, then the `SchemaInfo` item and eight `list<binary>` kinds, items
638+
byte-identical to the per-schema RPCs). The catalog side is
639+
`CatalogInterface.catalog_contents(*, attach_opaque_data, if_none_match) -> CatalogContentsResult`.
640+
The worker validates paths, orders parents first and enforces the etag rules
641+
(`not_modified` only with an etag equal to `if_none_match`). Opt-ins:
642+
`catalog_contents_etag = "content-hash"` (framework SHA-256 etag) and
643+
`catalog_contents_attach_independent` (with `catalog_version_frozen`, the worker
644+
caches the serialized response per catalog version; `ReadOnlyCatalogInterface`
645+
sets it). Fixture catalogs: `vgi/_test_fixtures/catalog_contents.py`
646+
(`contents_probe/broken/legacy/memory/reval/hash`). See `docs/catalog-interface.md`.
647+
633648
## Additional Documentation
634649

635650
- **Access log**: See `vgi-rpc` docs site

‎docs/catalog-interface.md‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -243,6 +243,80 @@ class MyCatalog(CatalogInterface):
243243
| | `view_rename()` | `NotImplementedError` |
244244
| | `view_comment_set()` | `NotImplementedError` |
245245
| **Observability** | `loggable_attach_options()` | Returns `{}` (no options logged — see below) |
246+
| **Bulk load** | `catalog_contents()` | Composes `schemas()` + `schema_contents()` per kind; no etag (see below) |
247+
248+
---
249+
250+
## Loading the Whole Catalog: `catalog_contents`
251+
252+
A client normally learns a catalog lazily: `catalog_schemas`, then one
253+
`catalog_schema_contents_*` call per schema and object kind. When the attach
254+
result sets `supports_catalog_contents` (`ReadOnlyCatalogInterface` does), the
255+
client may instead load everything with one `catalog_contents` RPC (protocol
256+
2.1.0):
257+
258+
```text
259+
catalog_contents(attach_opaque_data: binary, if_none_match: utf8 nullable)
260+
-> CatalogContentsResponse {
261+
catalog_version: int64
262+
etag: utf8 nullable -- null: the worker does not revalidate
263+
not_modified: bool -- true: schemas is empty, keep what you have
264+
schemas: list<struct<path: list<utf8>, schema: binary,
265+
tables, views, scalar_functions, aggregate_functions,
266+
table_functions, scalar_macros, table_macros,
267+
indexes: list<binary>>> -- parents before children
268+
}
269+
```
270+
271+
Each struct row is one schema: `path` equals the `SchemaInfo.path` encoded in
272+
`schema`, and every item is byte-identical to what the matching per-schema RPC
273+
returns. There is no transaction parameter: it is the committed catalog at
274+
`catalog_version`.
275+
276+
`CatalogInterface.catalog_contents(*, attach_opaque_data, if_none_match=None)`
277+
returns a `CatalogContentsResult(schemas, etag=None, not_modified=False)`. The
278+
default composes `schemas()` and `schema_contents()` for every kind (skipping
279+
kinds whose `estimated_object_count` is exactly 0) and returns no etag.
280+
281+
**Revalidation.** Return an `etag` to let a client revalidate a non-frozen
282+
catalog with `if_none_match` instead of re-downloading it. Check it *before*
283+
building, so an unchanged catalog costs almost nothing:
284+
285+
```python test="lint"
286+
from vgi.catalog import CatalogContentsResult, CatalogInterface
287+
288+
289+
class GenerationCatalog(CatalogInterface): # other methods omitted
290+
generation = 1 # bumped by every DDL
291+
292+
def catalog_contents(self, *, attach_opaque_data, if_none_match=None):
293+
etag = f"gen-{self.generation}"
294+
if if_none_match == etag:
295+
return CatalogContentsResult(etag=etag, not_modified=True) # nothing built
296+
full = super().catalog_contents(attach_opaque_data=attach_opaque_data)
297+
return CatalogContentsResult(schemas=full.schemas, etag=etag)
298+
```
299+
300+
The worker enforces the rules: `not_modified` requires an etag equal to
301+
`if_none_match` and no schemas; a full answer whose etag equals
302+
`if_none_match` is turned into `not_modified`; a catalog with no etag always
303+
answers in full (it ignores `if_none_match`).
304+
305+
**Content-hash etag.** A catalog without a cheap validator can set
306+
`catalog_contents_etag = "content-hash"`: when it returns no etag, the worker
307+
uses the hex SHA-256 of the serialized snapshot. The worker still builds the
308+
snapshot on every call, but an unchanged catalog skips the transfer and the
309+
client's decode. It is off by default, because for a non-frozen catalog the
310+
client would revalidate with a full build on every transaction instead of a
311+
cheap `catalog_version` poll.
312+
313+
**Worker cache.** When the catalog's version is frozen
314+
(`catalog_version_frozen`) and its contents do not depend on the attach or the
315+
caller (`catalog_contents_attach_independent = True`, the
316+
`ReadOnlyCatalogInterface` default), the worker builds the response once per
317+
catalog version and serves the same pre-serialized bytes to every call. A
318+
`ReadOnlyCatalogInterface` subclass whose contents vary per caller must set
319+
`catalog_contents_attach_independent = False`.
246320

247321
---
248322

@@ -649,6 +723,7 @@ client.catalog_detach(attach_opaque_data=attach_opaque_data)
649723
| `schema_create()` | Create a schema |
650724
| `schema_drop()` | Drop a schema |
651725
| `schema_contents()` | List schema contents (optional `type` filter) |
726+
| `contents()` | Load every schema and its contents in one call (`if_none_match` revalidates) |
652727
| `table_get()` | Get table info |
653728
| `table_create()` | Create a table |
654729
| `table_drop()` | Drop a table |

0 commit comments

Comments
 (0)