Skip to content

Commit f360ed0

Browse files
rustyconoverclaude
andcommitted
feat: worker-declared global functions in system.main
Workers can now ask the client to publish selected functions into DuckDB's global function namespace, so they are callable unqualified without naming the attached catalog — the way ducklake_table_info is reachable after LOAD ducklake. - Catalog.global_functions / global_function_prefix, validated in __post_init__: each entry must be resident in exactly one schema (bind dispatch is keyed on (schema_name, name), so a global-only function would be published yet never dispatchable), no duplicate entries, and the prefix must be identifier-safe since it is concatenated as <prefix>_<name>. - CatalogAttachResult carries global_functions (IPC-serialized FunctionInfo) and global_function_prefix. Additive fields on the existing attach response rather than a new RPC — no extra round trip, and a new method would have required the same minor bump anyway. protocol_version 1.2.0 -> 1.3.0. - Four dedicated fixture probes, one per function type, in _test_fixtures/global_functions.py, published as vgi_example_global_*. Deliberately new functions rather than repurposed existing ones: the example catalog is a cross-language contract that Go/TS/Java workers mirror, so giving a shared fixture new semantics would force the same change on every implementation. Landing golden regenerated accordingly. - Conformance parity recorded via _EXEMPTIONS rather than a skip_area stub; the observable behaviour is covered cross-language by vgi/test/sql/integration/global_functions/. - docs/global-functions.md. The C++ extension side — registry, prefix application, first-attach-wins collision policy, post-DETACH liveness, vgi_global_functions() — is not implemented yet. Workers advertise globals; nothing consumes them so far. The regenerated C++/Go/TS protocol artifacts are intentionally left uncommitted in their own repos. Also included, developed in parallel: - BufferedFinalizeState.ns annotated as FrameworkNS so the namespace survives an HTTP round trip instead of returning as the enum's .name and scanning a state-log namespace nothing was written to. - vgi-rpc floor 0.31.0 -> 0.32.0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 2e27695 commit f360ed0

16 files changed

Lines changed: 859 additions & 14 deletions

‎CLAUDE.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -508,4 +508,5 @@ The table schema is automatically derived from the function's `output_schema`:
508508
- **Function lifecycle**: `docs/lifecycle.md`
509509
- **Generator API (advanced)**: `docs/generator-api.md`
510510
- **Catalog interface**: `docs/catalog-interface.md`
511+
- **Global functions (system.main)**: `docs/global-functions.md`
511512
- **Shared storage backends**: `docs/shared-storage.md`

‎docs/global-functions.md‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
# Global functions
2+
3+
A VGI catalog normally exposes its functions under its own name — `mycat.main.my_func()`.
4+
Some functions aren't *about* the attached catalog at all: diagnostics, converters, format
5+
helpers. For those, a worker can ask the client to also publish them into the **global**
6+
function namespace (DuckDB's `system.main`), so they can be called unqualified:
7+
8+
```sql
9+
ATTACH 'acme' AS acme (TYPE vgi, LOCATION 'acme-worker');
10+
SELECT * FROM acme_table_info('acme'); -- no catalog qualifier
11+
```
12+
13+
This mirrors how `ducklake_table_info` is reachable after `LOAD ducklake`, except that a VGI
14+
worker's globals appear at `ATTACH` time rather than at extension load.
15+
16+
## Declaring globals
17+
18+
Add the function classes to `Catalog.global_functions` and pick a prefix:
19+
20+
```python
21+
from vgi.catalog import Catalog, Schema
22+
23+
CATALOG = Catalog(
24+
name="acme",
25+
global_function_prefix="acme",
26+
global_functions=[TableInfoFunction, ChecksumFunction],
27+
schemas=[
28+
Schema(name="main", functions=[TableInfoFunction, ChecksumFunction, ...]),
29+
],
30+
)
31+
```
32+
33+
All four function types are supported: scalar, aggregate, table, and table-buffering.
34+
35+
**Every global must also appear in exactly one `Schema.functions`.** This is enforced in
36+
`Catalog.__post_init__`. Bind dispatch is keyed on `(schema_name, name)`, so a function listed
37+
only in `global_functions` would be registered by the client but never dispatchable. Keeping it
38+
schema-resident also means the qualified name (`acme.main.table_info()`) keeps working, which is
39+
the unambiguous fallback if the global name is unavailable.
40+
41+
### The prefix
42+
43+
`global_function_prefix` is applied by the client to form the globally visible name:
44+
45+
| prefix | function name | published as |
46+
|---|---|---|
47+
| `"acme"` | `table_info` | `acme_table_info` |
48+
| `None` | `table_info` | `table_info` |
49+
50+
The prefix must match `[a-z_][a-z0-9_]*` — it is concatenated into a SQL identifier.
51+
52+
**Prefer a prefix.** `system.main` is shared by every extension and every attached VGI worker in
53+
the process. An unprefixed name like `table_info` is very likely to be claimed by someone else,
54+
in which case yours is silently skipped (see below). A prefix that identifies your worker makes
55+
collisions essentially impossible.
56+
57+
## What the client does with them
58+
59+
Registration is **best-effort and advisory**. Do not build anything that requires a global name
60+
to resolve — treat it as an ergonomic alias for the schema-qualified name, which is the one with
61+
guarantees.
62+
63+
Specifically, on the DuckDB side:
64+
65+
- **First attach wins.** If the name is already owned by a different worker, yours is skipped and
66+
logged; your `ATTACH` still succeeds and the function stays reachable at its qualified path.
67+
- **Re-`ATTACH` is idempotent.** Attaching the same worker again reuses the existing registration
68+
and refreshes its connection state.
69+
- **The user can turn it off.** `ATTACH ... (TYPE vgi, global_functions false)` suppresses
70+
publishing entirely for that attach.
71+
- **Registration outlives `DETACH`.** DuckDB has no API to unregister a function, so the entry
72+
persists for the life of the process. Calling it after `DETACH` raises an error telling the user
73+
to re-`ATTACH` rather than silently reconnecting to a detached catalog.
74+
75+
Use `vgi_global_functions()` to see what is currently published, who owns it, and whether it is
76+
still live.
77+
78+
## Protocol
79+
80+
Globals ride on the existing attach response — there is no extra round trip and no separate RPC.
81+
`CatalogAttachResult` carries:
82+
83+
- `global_functions: list[bytes]` — IPC-serialized `FunctionInfo` records. `name` and
84+
`schema_name` are the real dispatch coordinates; the prefix is *not* baked into `name`.
85+
- `global_function_prefix: str` — empty string means publish bare names.
86+
87+
Both fields are additive and default to empty, so a worker that doesn't set them advertises
88+
nothing. Introduced in protocol version 1.3.0.

‎pyproject.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ dependencies = [
4040
"pyarrow",
4141
"typer>=0.9",
4242
"platformdirs",
43-
"vgi-rpc>=0.31.0",
43+
"vgi-rpc>=0.32.0",
4444
"httpx>=0.24",
4545
]
4646

‎tests/catalog/test_declarative.py‎

Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@
2626
Catalog,
2727
CatalogAttachResult,
2828
ForeignKeyDef,
29+
FunctionInfo,
2930
Macro,
3031
MacroInfo,
3132
MacroType,
@@ -41,6 +42,7 @@
4142
View,
4243
ViewInfo,
4344
)
45+
from vgi.catalog.catalog_interface import FunctionType as CatalogInterfaceFunctionType
4446
from vgi.invocation import BindResponse
4547
from vgi.schema_utils import schema
4648
from vgi.table_function import (
@@ -1394,6 +1396,152 @@ def test_catalog_rejects_duplicate_schema_names(self) -> None:
13941396
Catalog(name="myapp", schemas=[s1, s2])
13951397

13961398

1399+
class TestGlobalFunctionsValidation:
1400+
"""Validation for ``Catalog.global_functions`` / ``global_function_prefix``."""
1401+
1402+
def test_defaults_are_empty(self) -> None:
1403+
"""A catalog publishes nothing globally unless it asks to."""
1404+
catalog = Catalog(name="myapp", schemas=[Schema(name="main")])
1405+
assert catalog.global_functions == ()
1406+
assert catalog.global_function_prefix is None
1407+
1408+
def test_accepts_schema_resident_function(self) -> None:
1409+
"""The happy path: listed once globally, resident in exactly one schema."""
1410+
catalog = Catalog(
1411+
name="myapp",
1412+
schemas=[Schema(name="main", functions=[UsersFunction])],
1413+
global_functions=[UsersFunction],
1414+
global_function_prefix="myapp",
1415+
)
1416+
assert list(catalog.global_functions) == [UsersFunction]
1417+
assert catalog.global_function_prefix == "myapp"
1418+
1419+
def test_rejects_function_absent_from_every_schema(self) -> None:
1420+
"""A global function that isn't schema-resident would never dispatch."""
1421+
with pytest.raises(ValueError, match="does not appear in any Schema.functions"):
1422+
Catalog(
1423+
name="myapp",
1424+
schemas=[Schema(name="main")],
1425+
global_functions=[UsersFunction],
1426+
)
1427+
1428+
def test_rejects_function_resident_in_multiple_schemas(self) -> None:
1429+
"""Two homes make the global registration ambiguous about dispatch."""
1430+
with pytest.raises(ValueError, match="appears in multiple schemas"):
1431+
Catalog(
1432+
name="myapp",
1433+
schemas=[
1434+
Schema(name="main", functions=[UsersFunction]),
1435+
Schema(name="other", functions=[UsersFunction]),
1436+
],
1437+
global_functions=[UsersFunction],
1438+
)
1439+
1440+
def test_rejects_duplicate_global_entries(self) -> None:
1441+
"""Listing a class twice would publish the same name twice."""
1442+
with pytest.raises(ValueError, match="duplicate entry UsersFunction"):
1443+
Catalog(
1444+
name="myapp",
1445+
schemas=[Schema(name="main", functions=[UsersFunction])],
1446+
global_functions=[UsersFunction, UsersFunction],
1447+
)
1448+
1449+
@pytest.mark.parametrize("prefix", ["9bad", "Bad", "has-dash", "has space", "", "trailing;"])
1450+
def test_rejects_non_identifier_prefix(self, prefix: str) -> None:
1451+
"""The prefix is concatenated into a SQL identifier, so it must be safe."""
1452+
with pytest.raises(ValueError, match="not a valid SQL identifier prefix"):
1453+
Catalog(
1454+
name="myapp",
1455+
schemas=[Schema(name="main", functions=[UsersFunction])],
1456+
global_functions=[UsersFunction],
1457+
global_function_prefix=prefix,
1458+
)
1459+
1460+
def test_prefix_validated_even_without_global_functions(self) -> None:
1461+
"""A bad prefix is a bug worth reporting even if nothing uses it yet."""
1462+
with pytest.raises(ValueError, match="not a valid SQL identifier prefix"):
1463+
Catalog(name="myapp", schemas=[Schema(name="main")], global_function_prefix="Bad")
1464+
1465+
1466+
class TestGlobalFunctionsAttachResult:
1467+
"""``catalog_attach`` advertises globals so the client can publish them."""
1468+
1469+
@staticmethod
1470+
def _attach(catalog_obj: Catalog) -> CatalogAttachResult:
1471+
class _TestCatalog(ReadOnlyCatalogInterface):
1472+
catalog = catalog_obj
1473+
1474+
return _TestCatalog().catalog_attach(
1475+
name=catalog_obj.name,
1476+
options={},
1477+
data_version_spec=None,
1478+
implementation_version=None,
1479+
)
1480+
1481+
def test_absent_when_not_declared(self) -> None:
1482+
"""Catalogs that declare nothing advertise an empty list and no prefix."""
1483+
result = self._attach(Catalog(name="myapp", schemas=[Schema(name="main", functions=[UsersFunction])]))
1484+
assert result.global_functions == []
1485+
assert result.global_function_prefix == ""
1486+
1487+
def test_carries_serialized_function_info(self) -> None:
1488+
"""Each global arrives as a FunctionInfo with real dispatch coordinates.
1489+
1490+
``name``/``schema_name`` must stay the values bind dispatches on — the
1491+
prefix is applied client-side, not baked into the name here.
1492+
"""
1493+
result = self._attach(
1494+
Catalog(
1495+
name="myapp",
1496+
schemas=[Schema(name="analytics", functions=[UsersFunction])],
1497+
default_schema="analytics",
1498+
global_functions=[UsersFunction],
1499+
global_function_prefix="myapp",
1500+
)
1501+
)
1502+
assert result.global_function_prefix == "myapp"
1503+
assert len(result.global_functions) == 1
1504+
1505+
info = FunctionInfo.deserialize_from_bytes(result.global_functions[0])
1506+
assert info.name == "users"
1507+
assert info.schema_name == "analytics"
1508+
assert info.function_type == CatalogInterfaceFunctionType.TABLE
1509+
1510+
def test_prefix_empty_string_when_unset(self) -> None:
1511+
"""``None`` prefix serializes as empty string — the wire type is str."""
1512+
result = self._attach(
1513+
Catalog(
1514+
name="myapp",
1515+
schemas=[Schema(name="main", functions=[UsersFunction])],
1516+
global_functions=[UsersFunction],
1517+
)
1518+
)
1519+
assert result.global_function_prefix == ""
1520+
assert len(result.global_functions) == 1
1521+
1522+
def test_example_fixture_publishes_one_of_each_function_type(self) -> None:
1523+
"""The shipped fixture covers all four types for the C++ integration suite."""
1524+
from vgi._test_fixtures.worker import ExampleCatalog
1525+
1526+
result = ExampleCatalog().catalog_attach(
1527+
name="example",
1528+
options={},
1529+
data_version_spec=None,
1530+
implementation_version=None,
1531+
)
1532+
assert result.global_function_prefix == "vgi_example"
1533+
infos = [FunctionInfo.deserialize_from_bytes(b) for b in result.global_functions]
1534+
assert {i.function_type for i in infos} == {
1535+
CatalogInterfaceFunctionType.SCALAR,
1536+
CatalogInterfaceFunctionType.TABLE,
1537+
CatalogInterfaceFunctionType.AGGREGATE,
1538+
CatalogInterfaceFunctionType.TABLE_BUFFERING,
1539+
}
1540+
# Every advertised global must be schema-resident, or it could never
1541+
# be dispatched after the client registers it.
1542+
assert all(i.schema_name == "main" for i in infos)
1543+
1544+
13971545
# =============================================================================
13981546
# ReadOnlyCatalogInterface with Catalog Object Tests
13991547
# =============================================================================

‎tests/catalog/test_serialization.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1110,7 +1110,7 @@ class TestArrowSchemaCorrectness:
11101110
def test_catalog_attach_result_schema(self) -> None:
11111111
"""Verify CatalogAttachResult Arrow schema."""
11121112
schema = CatalogAttachResult.ARROW_SCHEMA
1113-
assert len(schema) == 15
1113+
assert len(schema) == 17
11141114
assert schema.field("attach_opaque_data").type == pa.binary()
11151115
assert schema.field("supports_transactions").type == pa.bool_()
11161116
assert schema.field("supports_time_travel").type == pa.bool_()
@@ -1121,6 +1121,8 @@ def test_catalog_attach_result_schema(self) -> None:
11211121
assert schema.field("settings").type == pa.list_(pa.binary())
11221122
assert schema.field("secret_types").type == pa.list_(pa.binary())
11231123
assert schema.field("attach_catalogs").type == pa.list_(pa.binary())
1124+
assert schema.field("global_functions").type == pa.list_(pa.binary())
1125+
assert schema.field("global_function_prefix").type == pa.string()
11241126
assert schema.field("resolved_data_version").type == pa.string()
11251127
assert schema.field("resolved_implementation_version").type == pa.string()
11261128

‎tests/conformance/test_directory_parity.py‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,19 @@
4242
"path, ProcessParams.if_none_match threading, and the revalidatable "
4343
"fixture's 304 behavior)."
4444
),
45+
"global_functions": (
46+
"Publishing worker-declared functions into DuckDB's system.main is "
47+
"entirely a C++-extension concern: the registry, prefix application, "
48+
"first-attach-wins collision policy, post-DETACH liveness, and the "
49+
"vgi_global_functions() introspection surface all live in the "
50+
"extension. The Python worker's only role is advertising "
51+
"global_functions/global_function_prefix on the attach response, which "
52+
"tests/catalog/test_declarative.py covers directly (descriptor "
53+
"validation + the serialized FunctionInfo payload). The observable "
54+
"behaviour is cross-language by construction and is covered by "
55+
"vgi/test/sql/integration/global_functions/, which any worker "
56+
"implementation runs via VGI_TEST_WORKER."
57+
),
4558
"filter_pushdown": (
4659
"Per-type filter-pushdown coverage is mirrored by tests/test_filter_pushdown.py "
4760
"and tests/test_filter_pushdown_extension.py, which drive the same predicate "

‎tests/conformance/test_protocol_version.py‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -32,14 +32,14 @@
3232

3333
def test_client_protocol_version_is_canonical() -> None:
3434
"""Sanity-check the constants this test reasons about haven't drifted apart."""
35-
assert CLIENT_PROTOCOL_VERSION == "1.2.0"
35+
assert CLIENT_PROTOCOL_VERSION == "1.3.0"
3636
# The fixture must be a genuine mismatch in the major/minor that the
3737
# framework compares; a patch-only difference would (correctly) pass.
3838
assert BAD_PROTOCOL_VERSION.split(".")[:2] != CLIENT_PROTOCOL_VERSION.split(".")[:2]
3939

4040

4141
def test_mismatched_worker_protocol_version_raises() -> None:
42-
"""A worker enforcing 99.0.0 must reject the 1.2.0 client at dispatch."""
42+
"""A worker enforcing 99.0.0 must reject the 1.3.0 client at dispatch."""
4343
with Client("vgi-fixture-bad-protocol-worker") as client, pytest.raises(ClientError) as exc_info:
4444
list(
4545
client.table_function(
@@ -54,5 +54,5 @@ def test_mismatched_worker_protocol_version_raises() -> None:
5454
assert "protocol_version mismatch" in message
5555
assert CLIENT_PROTOCOL_VERSION in message
5656
assert BAD_PROTOCOL_VERSION in message
57-
# Directional hint: the client (1.2.0) is older than the worker (99.0.0).
57+
# Directional hint: the client (1.3.0) is older than the worker (99.0.0).
5858
assert "upgrade the VGI extension/client" in message

0 commit comments

Comments
 (0)