Skip to content

Commit 6b3182b

Browse files
rustyconoverclaude
andcommitted
feat(client): load a whole catalog with catalog_contents
The client offered raw wrappers (schemas(), schema_contents(), contents()) but nothing enumerated a catalog, so catalog_contents was never used. Client.load_catalog(attach=, previous=, transaction_opaque_data=) returns a CatalogSnapshot (schemas, version, etag, source, not_modified, fallback_reason): - one catalog_contents call when the attach advertises it; - catalog_schemas + per-schema RPCs otherwise, skipping kinds whose estimated_object_count is 0; catalog_contents is never sent (nor used to revalidate) when not advertised; - falls back to the per-schema path on failure, including an unrequested not_modified or a not_modified carrying schemas; - `previous` revalidates with if_none_match; not_modified keeps the held contents, a full answer replaces them; an older snapshot is retried once, version 0 is "unknown" and accepted; - inside a transaction it always goes per-schema. Adds a `vgi-client catalog contents` CLI command and shows supports_catalog_contents in the attach output. The fixture docstring and CLAUDE.md now describe the six contents_* catalogs as the cross-SDK contract (the vgi tests run them against every SDK). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
1 parent a698bfa commit 6b3182b

9 files changed

Lines changed: 564 additions & 5 deletions

File tree

‎CLAUDE.md‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -643,7 +643,14 @@ The worker validates paths, orders parents first and enforces the etag rules
643643
`catalog_contents_attach_independent` (with `catalog_version_frozen`, the worker
644644
caches the serialized response per catalog version; `ReadOnlyCatalogInterface`
645645
sets it). Fixture catalogs: `vgi/_test_fixtures/catalog_contents.py`
646-
(`contents_probe/broken/legacy/memory/reval/hash`). See `docs/catalog-interface.md`.
646+
(`contents_probe/broken/legacy/memory/reval/hash`). Every SDK's fixture worker
647+
serves the same six catalogs: the extension's five `catalog_contents*.test` files
648+
run against all of them (no `VGI_CATALOG_CONTENTS_WORKER` gate any more), so these
649+
semantics are the cross-SDK contract. Client side, `Client.load_catalog(attach=,
650+
previous=)` is the whole-catalog enumeration: `catalog_contents` only when
651+
advertised, per-schema fallback on failure / in a transaction, `if_none_match`
652+
revalidation via `previous` (`tests/catalog/test_client_load_catalog.py`). See
653+
`docs/catalog-interface.md`.
647654

648655
## Additional Documentation
649656

‎docs/catalog-interface.md‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -697,6 +697,14 @@ for obj in client.schema_contents(attach_opaque_data=attach_opaque_data, path=["
697697
elif isinstance(obj, FunctionInfo):
698698
print(f"Function: {obj.name}")
699699

700+
# Or load the whole catalog at once. Uses one catalog_contents RPC when the
701+
# attach result advertises supports_catalog_contents, else the per-schema RPCs.
702+
snapshot = client.load_catalog(attach=result)
703+
for entry in snapshot.schemas:
704+
print(entry.schema.path, [t.name for t in entry.tables])
705+
# Later: revalidate. A not_modified answer returns the same content cheaply.
706+
snapshot = client.load_catalog(attach=result, previous=snapshot)
707+
700708
# Get only scalar functions using type filter
701709
from vgi.catalog import SchemaObjectType
702710
for obj in client.schema_contents(
@@ -723,7 +731,8 @@ client.catalog_detach(attach_opaque_data=attach_opaque_data)
723731
| `schema_create()` | Create a schema |
724732
| `schema_drop()` | Drop a schema |
725733
| `schema_contents()` | List schema contents (optional `type` filter) |
726-
| `contents()` | Load every schema and its contents in one call (`if_none_match` revalidates) |
734+
| `load_catalog()` | Load the whole catalog: one `catalog_contents` call when the attach advertises it, else (or on failure, or inside a transaction) `catalog_schemas` + the per-schema RPCs; pass `previous=` to revalidate with `if_none_match` |
735+
| `contents()` | The raw `catalog_contents` RPC (only valid when advertised; `if_none_match` revalidates) |
727736
| `table_get()` | Get table info |
728737
| `table_create()` | Create a table |
729738
| `table_drop()` | Drop a table |
Lines changed: 335 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,335 @@
1+
# Copyright 2025, 2026 Query Farm LLC - https://query.farm
2+
3+
"""``Client.load_catalog``: whole-catalog enumeration honours ``supports_catalog_contents``.
4+
5+
Runs the client against the ``contents_*`` fixture catalogs over every client
6+
transport and asserts which catalog RPCs it actually sent: one
7+
``catalog_contents`` when advertised, the per-schema RPCs when not advertised or
8+
when ``catalog_contents`` fails, and ``if_none_match`` revalidation.
9+
"""
10+
11+
from __future__ import annotations
12+
13+
from collections.abc import Iterator
14+
from contextlib import contextmanager
15+
from typing import Any
16+
17+
import pytest
18+
19+
from vgi._test_fixtures.catalog_contents import (
20+
BROKEN_MESSAGE,
21+
CATALOG_BROKEN,
22+
CATALOG_HASH,
23+
CATALOG_LEGACY,
24+
CATALOG_PROBE,
25+
CATALOG_REVAL,
26+
)
27+
from vgi.catalog import (
28+
AttachOpaqueData,
29+
CatalogAttachResult,
30+
SchemaContentsInfo,
31+
SchemaInfo,
32+
TransactionOpaqueData,
33+
)
34+
from vgi.client import CatalogClientMixin, CatalogContents, CatalogSnapshot
35+
from vgi.client.catalog_mixin import CatalogClientError
36+
37+
_KINDS = (
38+
"tables",
39+
"views",
40+
"scalar_functions",
41+
"aggregate_functions",
42+
"table_functions",
43+
"scalar_macros",
44+
"table_macros",
45+
"indexes",
46+
)
47+
48+
Call = tuple[str, str | None]
49+
50+
51+
class _RecordingProxy:
52+
"""Wraps a VgiProtocol proxy, recording each ``catalog_*`` method sent."""
53+
54+
def __init__(self, proxy: Any, calls: list[Call]) -> None:
55+
self._proxy = proxy
56+
self._calls = calls
57+
58+
def __getattr__(self, name: str) -> Any:
59+
attr = getattr(self._proxy, name)
60+
if not name.startswith("catalog_") or not callable(attr):
61+
return attr
62+
63+
def call(*args: Any, **kwargs: Any) -> Any:
64+
self._calls.append((name, kwargs.get("if_none_match")))
65+
return attr(*args, **kwargs)
66+
67+
return call
68+
69+
70+
def _record(client: Any) -> list[Call]:
71+
"""Make ``client`` record every catalog RPC it sends; returns the live list."""
72+
calls: list[Call] = []
73+
original = client._catalog_connect
74+
75+
@contextmanager
76+
def recording() -> Iterator[Any]:
77+
with original() as proxy:
78+
yield _RecordingProxy(proxy, calls)
79+
80+
client._catalog_connect = recording
81+
return calls
82+
83+
84+
def _attach(client: Any, name: str) -> CatalogAttachResult:
85+
return client.catalog_attach(name=name, data_version_spec=None, implementation_version=None) # type: ignore[no-any-return]
86+
87+
88+
def _names(entry: SchemaContentsInfo) -> dict[str, list[str]]:
89+
return {kind: sorted(item.name for item in getattr(entry, kind)) for kind in _KINDS}
90+
91+
92+
# ---------------------------------------------------------------------------
93+
# Against the fixture catalogs, over every client transport
94+
# ---------------------------------------------------------------------------
95+
96+
97+
def test_advertised_uses_one_catalog_contents_call(client_transport: Any) -> None:
98+
"""contents_probe: one ``catalog_contents``, every kind decoded, equal to the per-schema answer."""
99+
with client_transport() as client:
100+
attach = _attach(client, CATALOG_PROBE)
101+
assert attach.supports_catalog_contents
102+
calls = _record(client)
103+
snapshot = client.load_catalog(attach=attach)
104+
assert calls == [("catalog_contents", None)]
105+
assert snapshot.source == "catalog_contents"
106+
assert snapshot.fallback_reason is None
107+
assert snapshot.etag is None
108+
assert [list(s.schema.path) for s in snapshot.schemas] == [["main"], ["extra"]]
109+
main = _names(snapshot.schemas[0])
110+
assert main["tables"] == ["ten"]
111+
assert main["views"] == ["answer"]
112+
assert "double" in main["scalar_functions"]
113+
assert main["aggregate_functions"]
114+
assert main["table_functions"]
115+
assert main["scalar_macros"] == ["contents_triple"]
116+
assert main["table_macros"] == ["contents_range"]
117+
assert _names(snapshot.schemas[1])["tables"] == ["five"]
118+
119+
# The one-call answer decodes to exactly what the per-schema RPCs return.
120+
calls.clear()
121+
per_schema = client._load_catalog_per_schema(attach.attach_opaque_data, None, None)
122+
assert "catalog_contents" not in [name for name, _ in calls]
123+
assert snapshot.schemas == per_schema.schemas
124+
125+
126+
def test_failing_catalog_contents_falls_back_to_per_schema(client_transport: Any) -> None:
127+
"""contents_broken: ``catalog_contents`` fails, the per-schema RPCs answer instead."""
128+
with client_transport() as client:
129+
attach = _attach(client, CATALOG_BROKEN)
130+
assert attach.supports_catalog_contents
131+
calls = _record(client)
132+
snapshot = client.load_catalog(attach=attach)
133+
names = [name for name, _ in calls]
134+
assert names[:2] == ["catalog_contents", "catalog_schemas"]
135+
assert names.count("catalog_contents") == 1
136+
assert {n for n in names[2:]} <= {
137+
"catalog_schema_contents_tables",
138+
"catalog_schema_contents_views",
139+
"catalog_schema_contents_functions",
140+
"catalog_schema_contents_macros",
141+
"catalog_schema_contents_indexes",
142+
}
143+
assert "catalog_schema_contents_tables" in names
144+
assert snapshot.source == "per_schema"
145+
assert snapshot.fallback_reason is not None and BROKEN_MESSAGE in snapshot.fallback_reason
146+
assert [list(s.schema.path) for s in snapshot.schemas] == [["main"], ["extra"]]
147+
assert _names(snapshot.schemas[0])["views"] == ["answer"]
148+
149+
150+
def test_not_advertised_never_sends_catalog_contents(client_transport: Any) -> None:
151+
"""contents_legacy: no ``catalog_contents`` on the wire — not to load, not to revalidate."""
152+
with client_transport() as client:
153+
attach = _attach(client, CATALOG_LEGACY)
154+
assert not attach.supports_catalog_contents
155+
calls = _record(client)
156+
first = client.load_catalog(attach=attach)
157+
second = client.load_catalog(attach=attach, previous=first)
158+
names = [name for name, _ in calls]
159+
assert "catalog_contents" not in names
160+
assert names[0] == "catalog_schemas"
161+
assert first.source == second.source == "per_schema"
162+
assert first.fallback_reason is None
163+
assert [list(s.schema.path) for s in first.schemas] == [["main"], ["extra"]]
164+
assert first.schemas == second.schemas
165+
166+
167+
@pytest.mark.parametrize("catalog", [CATALOG_REVAL, CATALOG_HASH])
168+
def test_revalidation_with_if_none_match(client_transport: Any, catalog: str) -> None:
169+
"""contents_reval / contents_hash: ``if_none_match`` → ``not_modified`` keeps; a change replaces."""
170+
with client_transport() as client:
171+
attach = _attach(client, catalog)
172+
try:
173+
calls = _record(client)
174+
first = client.load_catalog(attach=attach)
175+
assert calls == [("catalog_contents", None)]
176+
assert first.etag is not None
177+
if catalog == CATALOG_REVAL:
178+
assert first.etag == "gen-1"
179+
assert [list(s.schema.path) for s in first.schemas] == [["main"]]
180+
181+
calls.clear()
182+
hit = client.load_catalog(attach=attach, previous=first)
183+
assert calls == [("catalog_contents", first.etag)]
184+
assert hit.not_modified
185+
assert hit.source == "catalog_contents"
186+
assert hit.schemas == first.schemas
187+
assert hit.etag == first.etag
188+
189+
client.schema_create(attach_opaque_data=attach.attach_opaque_data, path=["added"])
190+
calls.clear()
191+
changed = client.load_catalog(attach=attach, previous=hit)
192+
assert calls == [("catalog_contents", first.etag)]
193+
assert not changed.not_modified
194+
assert changed.etag not in (None, first.etag)
195+
if catalog == CATALOG_REVAL:
196+
assert changed.etag == "gen-2"
197+
assert sorted(list(s.schema.path) for s in changed.schemas) == [["added"], ["main"]]
198+
finally:
199+
client.catalog_detach(attach_opaque_data=attach.attach_opaque_data)
200+
201+
202+
# ---------------------------------------------------------------------------
203+
# Rules that need a misbehaving or transactional worker (stubbed)
204+
# ---------------------------------------------------------------------------
205+
206+
207+
_ATTACH = AttachOpaqueData(b"a" * 16)
208+
209+
210+
def _schema(name: str) -> SchemaContentsInfo:
211+
return SchemaContentsInfo(schema=SchemaInfo(attach_opaque_data=_ATTACH, path=[name], comment=None, tags={}))
212+
213+
214+
class _StubClient(CatalogClientMixin):
215+
"""Answers ``contents`` from a script and the per-schema RPCs with one empty schema."""
216+
217+
def __init__(self, answers: list[CatalogContents | Exception]) -> None:
218+
self.answers = answers
219+
self.calls: list[Call] = []
220+
221+
def contents(self, *, attach_opaque_data: AttachOpaqueData, if_none_match: str | None = None) -> CatalogContents:
222+
self.calls.append(("catalog_contents", if_none_match))
223+
answer = self.answers.pop(0)
224+
if isinstance(answer, Exception):
225+
raise answer
226+
return answer
227+
228+
def schemas(
229+
self, *, attach_opaque_data: AttachOpaqueData, transaction_opaque_data: TransactionOpaqueData | None = None
230+
) -> list[SchemaInfo]:
231+
self.calls.append(("catalog_schemas", None))
232+
return [
233+
SchemaInfo(
234+
attach_opaque_data=_ATTACH,
235+
path=["fallback"],
236+
comment=None,
237+
tags={},
238+
estimated_object_count={"table": 0},
239+
)
240+
]
241+
242+
def schema_contents(self, **kwargs: Any) -> list[Any]:
243+
self.calls.append((f"schema_contents:{kwargs['type'].value}", None))
244+
return []
245+
246+
247+
def _attach_result(supports: bool) -> CatalogAttachResult:
248+
return CatalogAttachResult(
249+
attach_opaque_data=_ATTACH,
250+
supports_transactions=True,
251+
supports_time_travel=False,
252+
catalog_version_frozen=False,
253+
catalog_version=1,
254+
attach_opaque_data_required=False,
255+
default_schema="main",
256+
supports_catalog_contents=supports,
257+
resolved_data_version=None,
258+
resolved_implementation_version=None,
259+
)
260+
261+
262+
def test_transactional_load_uses_per_schema_rpcs() -> None:
263+
"""``catalog_contents`` returns only the committed catalog, so a transaction never uses it."""
264+
client = _StubClient([])
265+
snapshot = client.load_catalog(attach=_attach_result(True), transaction_opaque_data=TransactionOpaqueData(b"t"))
266+
assert client.calls[0] == ("catalog_schemas", None)
267+
assert "catalog_contents" not in [name for name, _ in client.calls]
268+
assert snapshot.source == "per_schema" and snapshot.fallback_reason is None
269+
270+
271+
def test_per_schema_skips_kinds_counted_as_zero() -> None:
272+
"""``estimated_object_count`` 0 is a hard "none": that kind's RPC is skipped."""
273+
client = _StubClient([])
274+
client.load_catalog(attach=_attach_result(False))
275+
names = [name for name, _ in client.calls]
276+
assert "schema_contents:table" not in names
277+
assert "schema_contents:view" in names
278+
assert "schema_contents:index" in names
279+
280+
281+
def test_unsolicited_not_modified_falls_back() -> None:
282+
"""``not_modified`` to a request that sent no ``if_none_match`` is a protocol violation."""
283+
client = _StubClient([CatalogContents(catalog_version=1, etag="e", not_modified=True, schemas=[])])
284+
snapshot = client.load_catalog(attach=_attach_result(True))
285+
assert snapshot.source == "per_schema"
286+
assert snapshot.fallback_reason is not None and "not_modified" in snapshot.fallback_reason
287+
288+
289+
def test_older_snapshot_is_retried_once_then_falls_back() -> None:
290+
"""A snapshot older than the one held (a lagging replica) is retried, then per-schema."""
291+
previous = CatalogSnapshot(schemas=[_schema("main")], catalog_version=5, etag=None, source="catalog_contents")
292+
stale = CatalogContents(catalog_version=3, etag=None, not_modified=False, schemas=[_schema("old")])
293+
client = _StubClient([stale, stale])
294+
snapshot = client.load_catalog(attach=_attach_result(True), previous=previous)
295+
assert [name for name, _ in client.calls][:3] == ["catalog_contents", "catalog_contents", "catalog_schemas"]
296+
assert snapshot.source == "per_schema"
297+
assert snapshot.fallback_reason is not None and "older" in snapshot.fallback_reason
298+
299+
fresh = CatalogContents(catalog_version=6, etag=None, not_modified=False, schemas=[_schema("new")])
300+
client = _StubClient([stale, fresh])
301+
snapshot = client.load_catalog(attach=_attach_result(True), previous=previous)
302+
assert snapshot.source == "catalog_contents" and snapshot.catalog_version == 6
303+
304+
305+
def test_version_zero_is_never_older() -> None:
306+
"""Version 0 means "unknown" and is accepted as is."""
307+
previous = CatalogSnapshot(schemas=[], catalog_version=5, etag=None, source="catalog_contents")
308+
client = _StubClient([CatalogContents(catalog_version=0, etag=None, not_modified=False, schemas=[_schema("x")])])
309+
snapshot = client.load_catalog(attach=_attach_result(True), previous=previous)
310+
assert snapshot.source == "catalog_contents" and client.calls == [("catalog_contents", None)]
311+
312+
313+
def test_per_schema_snapshot_is_not_revalidated() -> None:
314+
"""Only a ``catalog_contents`` snapshot's etag is sent back."""
315+
previous = CatalogSnapshot(schemas=[], catalog_version=None, etag="x", source="per_schema")
316+
client = _StubClient([CatalogContents(catalog_version=1, etag="e", not_modified=False, schemas=[])])
317+
client.load_catalog(attach=_attach_result(True), previous=previous)
318+
assert client.calls == [("catalog_contents", None)]
319+
320+
321+
def test_error_on_revalidation_falls_back() -> None:
322+
"""A failing conditional call reloads through the per-schema RPCs."""
323+
previous = CatalogSnapshot(schemas=[], catalog_version=1, etag="e", source="catalog_contents")
324+
client = _StubClient([CatalogClientError("boom")])
325+
snapshot = client.load_catalog(attach=_attach_result(True), previous=previous)
326+
assert client.calls[0] == ("catalog_contents", "e")
327+
assert snapshot.source == "per_schema" and snapshot.fallback_reason == "boom"
328+
329+
330+
def test_absent_supports_catalog_contents_decodes_false() -> None:
331+
"""An attach result from a worker that predates the field reads as not advertised."""
332+
batch = _attach_result(True)._serialize()
333+
assert CatalogAttachResult.deserialize_from_batch(batch).supports_catalog_contents
334+
old = batch.drop_columns(["supports_catalog_contents"])
335+
assert not CatalogAttachResult.deserialize_from_batch(old).supports_catalog_contents

‎vgi/_test_fixtures/catalog_contents.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,9 @@
4242
table macros), split over two schemas (``main`` and ``extra``), so RPC
4343
accounting in ``vgi/test/sql/integration/catalog/catalog_contents*.test`` is
4444
deterministic. Served by ``vgi-fixture-worker`` and ``vgi-fixture-http``
45-
(MetaWorker), so the same tests run over every transport. Their functions
45+
(MetaWorker), so the same tests run over every transport. Every other SDK's
46+
fixture worker serves these six catalogs too, and the same ``.test`` files run
47+
against each of them, so the semantics here are the cross-SDK contract. Their functions
4648
re-use the example catalog's classes (``main.double`` ...), so every worker
4749
here sets ``route_unattached_calls = False``: they are reachable only through
4850
their own attach, and an unattached ``main.double`` still routes to the

0 commit comments

Comments
 (0)