Skip to content

Commit b7b728d

Browse files
rustyconoverclaude
andcommitted
feat(serve): one server builder, Worker.hosted_protocols, conformance.Secondary.v1 in the fixture
Five transports built their RpcServer separately and had drifted: only HTTP and Iroh hosted reflection, only HTTP hosted identity, and MetaWorker.serve (the path vgi-fixture-worker and the DuckDB launcher take) went through run_server's own construction. Every transport now builds through vgi.rpc_server.build_rpc_server(worker, *, transport, ...), so the hosted set is decided in one place. A structural test fails if RpcServer( appears outside the builder (secret_service and transactor serve other protocols and are exempt) and checks each entry point calls the builder for its transport. Worker.hosted_protocols() returns extra (protocol, implementation) pairs. It is called once per build and hosted on every transport after vgi.v2, in order. Duplicate names, vgi.v2 itself and the reserved vgi_rpc. prefix fail at startup with an error naming the hook. MetaWorker hosts its children's lists concatenated. Reflection is now hosted on stdio, unix and TCP too. vgi-rpc routes on vgi_rpc.protocol with no fallback, so vgi.v2 dispatch (all the DuckDB extension sends) is unchanged; the launcher and HTTP integration lanes pass. Identity stays opt-in via resolve_token/mint_grant and HTTP-only. A MetaWorker now hosts it for the one child that overrides each hook; two such children refuse to start. Docs that said identity "reaches every transport" were wrong (only create_app ever built it) and now say HTTP. The fixture: ExampleWorker hosts vgi-rpc's conformance.Secondary.v1, and vgi-fixture-http --identity (or VGI_FIXTURE_IDENTITY=1) opts into the IDENTITY_CONFORMANCE_FIXTURE.md policy using the reference hooks, not copies. scripts/hosted_protocols_conformance.sh runs vgi-rpc-test-hosted on stdio, unix and HTTP; a new CI job runs it. Requires vgi-rpc >= 0.48.0, also named directly by vgi-fixtures, which imports vgi_rpc.conformance. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
1 parent 84006d9 commit b7b728d

18 files changed

Lines changed: 1313 additions & 180 deletions

‎.github/workflows/ci.yml‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -126,6 +126,27 @@ jobs:
126126
args: "--no-progress --offline --root-dir ${{ github.workspace }}/docs docs/**/*.md *.md"
127127
fail: true
128128

129+
hosted-protocols:
130+
# vgi-rpc's cross-SDK hosted-protocols group: the fixture workers must host
131+
# vgi.v2 then conformance.Secondary.v1 on stdio, unix and HTTP, and pass the
132+
# Identity.v1 groups over HTTP (vgi-fixture-http --identity).
133+
name: Hosted-protocols conformance
134+
runs-on: ubuntu-latest
135+
steps:
136+
- uses: actions/checkout@v7
137+
138+
- name: Install uv
139+
uses: astral-sh/setup-uv@v10.2.0
140+
141+
- name: Set up Python 3.14
142+
run: uv python install 3.14
143+
144+
- name: Install dependencies
145+
run: uv sync --all-extras --python 3.14
146+
147+
- name: Run hosted-protocols group (stdio, unix, http)
148+
run: scripts/hosted_protocols_conformance.sh
149+
129150
s3-offload-localstack:
130151
name: S3 Offload Tests (LocalStack)
131152
runs-on: ubuntu-latest

‎CLAUDE.md‎

Lines changed: 28 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -428,9 +428,10 @@ caller's identity before it can authorize anything.
428428

429429
Through vgi-rpc 0.45.x this was an HTTP JSON route, `POST
430430
{prefix}/__introspect_token__`, which meant it existed on exactly one
431-
transport. As of **vgi-rpc 0.46.0** identity lives at the RPC layer, so it
432-
reaches every transport, and a client discovers it through ordinary reflection
433-
(`list_protocols`) rather than by calling and reading an error.
431+
transport. As of **vgi-rpc 0.46.0** identity lives at the RPC layer, and a
432+
client discovers it through ordinary reflection (`list_protocols`) rather than
433+
by calling and reading an error. VGI hosts it on HTTP only — the transport that
434+
authenticates callers (see `build_rpc_server` in `vgi/rpc_server.py`).
434435

435436
```python
436437
from vgi.auth import AuthUnavailableError, TokenIdentity
@@ -469,7 +470,10 @@ credentials. Throttle untrusted traffic at the proxy, per client.
469470

470471
Return `None` for "the store answered and this credential is unknown"; raise
471472
`AuthUnavailableError` for "the answer is not knowable". A caller that
472-
negative-caches the first must not cache the second. Never return claims — a
473+
negative-caches the first must not cache the second. Raising the same error
474+
an authenticator raises is correct here: vgi-rpc translates an
475+
`AuthUnavailableError` from either identity hook into `identity_unavailable`
476+
(code `UNAVAILABLE`) and carries its `retry_after` as `RetryInfo`. Never return claims — a
473477
pass-through claims field would let a worker choose its caller's tenant routing
474478
and policy branch.
475479

@@ -480,6 +484,26 @@ sidecar outage raised as one is read as "not my credential, try the next" and
480484
ends up a 401 from the end of the chain, restarting every session in the fleet
481485
over a thirty-second blip.
482486

487+
### Hosting Additional Protocols (`hosted_protocols`)
488+
489+
Every transport (stdio, unix/named pipe, TCP, Iroh, HTTP) builds its `RpcServer`
490+
through one helper, `build_rpc_server(worker, *, transport, ...)` in
491+
`vgi/rpc_server.py`. Never call `RpcServer(...)` for a worker anywhere else.
492+
`tests/test_hosted_protocols.py` enforces this structurally. Only
493+
`vgi/secret_service.py` and `vgi/transactor/server.py` are exempt, because they
494+
serve other protocols.
495+
496+
A worker hosts extra vgi-rpc protocols by overriding the classmethod
497+
`Worker.hosted_protocols() -> Sequence[tuple[type, object]]` (empty by
498+
default). It is called once per server build. Its pairs are hosted on every
499+
transport, after `vgi.v2` and before `vgi_rpc.Reflection.v1`, which is now
500+
hosted on every transport too. You cannot host part of a protocol: the
501+
protocol is the unit of optionality. Duplicate names, `vgi.v2` itself, and the
502+
reserved `vgi_rpc.` prefix all fail at startup with an error that names the
503+
hook. `MetaWorker` hosts its children's lists concatenated. Identity is not
504+
supplied through this hook. It stays opt-in via `resolve_token`/`mint_grant`
505+
and is hosted on HTTP only. See `docs/api/worker.md`.
506+
483507
### Access Log Sampling and Async Emission
484508

485509
`--access-log-sample 0.05` keeps 5% of *successful* calls. Errors are always

‎docs/api/auth.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,9 @@ require `pip install vgi-python[http]`; JWT authentication additionally requires
1919
re-exports it — a worker that implements `resolve_token` never has to name a private import path.
2020

2121
As of vgi-rpc 0.46.0 introspection is the `vgi_rpc.Identity.v1` protocol hosted on the RPC server,
22-
not an HTTP JSON route, so it reaches every transport rather than only HTTP. `TokenIdentity` and
23-
`TokenResolver` are correspondingly no longer gated on the `[http]` extra.
22+
not an HTTP JSON route, so clients discover it through reflection. VGI hosts it on HTTP only, the
23+
transport that authenticates callers: its allowlist names principals, which stdin/stdout, AF_UNIX
24+
and TCP do not have. `TokenIdentity` and `TokenResolver` are no longer gated on the `[http]` extra.
2425

2526
## Secret protocol
2627

‎docs/api/worker.md‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,86 @@ class MyWorker(Worker):
1414
# vgi-serve my_module:MyWorker --http # HTTP
1515
```
1616

17+
## Hosting additional protocols
18+
19+
A worker can serve other [vgi-rpc](https://vgi-rpc.query.farm) protocols from the same process and
20+
the same listener as the VGI protocol. Override
21+
[`Worker.hosted_protocols`][vgi.worker.Worker.hosted_protocols] to return
22+
`(protocol, implementation)` pairs:
23+
24+
```python
25+
from collections.abc import Sequence
26+
from typing import ClassVar, Protocol
27+
28+
from vgi import Worker
29+
30+
31+
class Audit(Protocol):
32+
protocol_name: ClassVar[str] = "acme.Audit.v1"
33+
34+
def record(self, event: str) -> int: ...
35+
36+
37+
class AuditImpl:
38+
def record(self, event: str) -> int:
39+
return len(event)
40+
41+
42+
class MyWorker(Worker):
43+
functions = []
44+
45+
@classmethod
46+
def hosted_protocols(cls) -> Sequence[tuple[type, object]]:
47+
return [(Audit, AuditImpl())]
48+
```
49+
50+
The rules:
51+
52+
- **Every transport hosts the same list.** stdin/stdout, AF_UNIX (or a Windows named pipe), TCP,
53+
the Iroh raw upstream and HTTP all build their server through one function,
54+
[`build_rpc_server`][vgi.rpc_server.build_rpc_server].
55+
- **Called once per server.** The hook may read configuration or the environment, but its answer
56+
is fixed for the life of the process, so reflection output and protocol hashes stay stable.
57+
- **The protocol is the unit of optionality.** There is no way to host part of a protocol. Make
58+
an optional capability its own protocol, and include it or leave it out.
59+
- **Names are routing keys.** Each protocol needs a distinct `protocol_name`. A repeated name, the
60+
worker's own `vgi.v2`, or the reserved `vgi_rpc.` prefix stops the worker at startup, and the
61+
error names `hosted_protocols()`.
62+
- **`vgi.v2` is unaffected.** vgi-rpc routes each request on its `vgi_rpc.protocol` key and never
63+
falls back to the primary protocol. A client that only speaks `vgi.v2`, such as the DuckDB
64+
extension, sees exactly the behaviour it saw before.
65+
66+
A `MetaWorker` (`vgi.meta_worker`) hosts the concatenation of its children's lists, in
67+
child order. Two children that list the same protocol name stop it at startup. Over HTTP it
68+
hosts Identity for the one child that overrides `resolve_token` or `mint_grant`; two children
69+
overriding the same hook stop it at startup.
70+
71+
### What each transport hosts
72+
73+
Reflection (`list_protocols`) reports protocols in this order:
74+
75+
| Transport | `vgi.v2` | `hosted_protocols()` | `vgi_rpc.Reflection.v1` | `vgi_rpc.Identity.v1` |
76+
| --- | --- | --- | --- | --- |
77+
| stdin/stdout | yes | yes | yes | no |
78+
| AF_UNIX / named pipe | yes | yes | yes | no |
79+
| TCP | yes | yes | yes | no |
80+
| Iroh raw upstream | yes | yes | yes | no |
81+
| HTTP | yes | yes | unless `--no-describe` | when `resolve_token` or `mint_grant` is overridden |
82+
83+
Identity stays framework-owned. You enable it by overriding
84+
[`resolve_token`][vgi.worker.Worker.resolve_token] or
85+
[`mint_grant`][vgi.worker.Worker.mint_grant], never through `hosted_protocols()`. It is hosted
86+
only on HTTP, which authenticates callers. Its introspection allowlist names principals, and the
87+
other transports have no principals to check.
88+
1789
## Worker
1890

1991
::: vgi.worker
2092

93+
## Server construction
94+
95+
::: vgi.rpc_server
96+
2197
## Serving
2298

2399
::: vgi.serve

‎packages/vgi-fixtures/pyproject.toml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,9 @@ classifiers = [
3434
# The fixture worker also builds statistics fixtures at import time, which
3535
# needs a DuckDB-compatible engine — vgi[duckdb] keeps the default dev env on
3636
# stock duckdb (the resolver prefers haybarn whenever it is installed too).
37-
dependencies = ["vgi-python[test-fixtures-writable]", "vgi-python[duckdb]"]
37+
# vgi-rpc is named directly because the fixture worker imports its conformance
38+
# fixtures (vgi_rpc.conformance.secondary, .identity_fixture), new in 0.48.0.
39+
dependencies = ["vgi-python[test-fixtures-writable]", "vgi-python[duckdb]", "vgi-rpc>=0.48.0"]
3840

3941
[project.scripts]
4042
vgi-fixture-worker = "vgi._test_fixtures.worker:main"

‎pyproject.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ dependencies = [
4040
"pyarrow",
4141
"typer>=0.9",
4242
"platformdirs",
43-
"vgi-rpc>=0.47.2",
43+
"vgi-rpc>=0.48.0",
4444
"httpx2>=2.12.0",
4545
]
4646

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
#!/usr/bin/env bash
2+
# Copyright 2025, 2026 Query Farm LLC - https://query.farm
3+
#
4+
# Run vgi-rpc's hosted-protocols conformance group (vgi-rpc-test-hosted)
5+
# against this repo's fixture workers on all three transports it covers:
6+
#
7+
# stdio vgi-fixture-worker (MetaWorker.serve, pipe)
8+
# unix vgi-fixture-worker --unix <sock> (the DuckDB launcher path)
9+
# http vgi-fixture-http --identity (+ the Identity.v1 groups)
10+
#
11+
# Every worker must host, in order, vgi.v2 then conformance.Secondary.v1.
12+
#
13+
# Usage: scripts/hosted_protocols_conformance.sh [stdio|unix|http ...]
14+
# Default: all three. Runs from the project env (`uv run --no-sync`), which
15+
# must have vgi-rpc with the [http,conformance] extras (the dev group does).
16+
set -euo pipefail
17+
18+
cd "$(dirname "$0")/.."
19+
20+
EXPECT="vgi.v2,conformance.Secondary.v1"
21+
RUN=(uv run --no-sync)
22+
BIN="$(dirname "$("${RUN[@]}" python -c 'import sys; print(sys.executable)')")"
23+
TRANSPORTS=("$@")
24+
[ ${#TRANSPORTS[@]} -eq 0 ] && TRANSPORTS=(stdio unix http)
25+
26+
# Short scratch dir: AF_UNIX paths are capped at ~104 bytes, and CI temp
27+
# dirs can be long.
28+
SCRATCH="$(mktemp -d /tmp/vgihp.XXXXXX)"
29+
PIDS=()
30+
cleanup() {
31+
for pid in "${PIDS[@]}"; do kill "$pid" 2>/dev/null || true; done
32+
rm -rf "$SCRATCH"
33+
}
34+
trap cleanup EXIT
35+
36+
wait_for() { # wait_for <description> <test command...>
37+
local what="$1"; shift
38+
for _ in $(seq 150); do "$@" && return 0; sleep 0.2; done
39+
echo "timed out waiting for $what" >&2
40+
return 1
41+
}
42+
43+
run_stdio() {
44+
echo "== hosted-protocols: stdio =="
45+
"${RUN[@]}" vgi-rpc-test-hosted --cmd "$BIN/vgi-fixture-worker" --expect "$EXPECT" -- -rs
46+
}
47+
48+
run_unix() {
49+
echo "== hosted-protocols: unix =="
50+
local sock="$SCRATCH/w.sock"
51+
"$BIN/vgi-fixture-worker" --unix "$sock" --idle-timeout 300 >"$SCRATCH/unix.out" 2>"$SCRATCH/unix.err" &
52+
PIDS+=($!)
53+
wait_for "unix socket $sock" test -S "$sock" || { cat "$SCRATCH/unix.err" >&2; return 1; }
54+
"${RUN[@]}" vgi-rpc-test-hosted --unix "$sock" --expect "$EXPECT" -- -rs
55+
}
56+
57+
run_http() {
58+
echo "== hosted-protocols: http (--identity) =="
59+
local port
60+
port="$("${RUN[@]}" python -c 'import socket; s = socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1])')"
61+
"$BIN/vgi-fixture-http" --identity --host 127.0.0.1 --port "$port" >"$SCRATCH/http.out" 2>"$SCRATCH/http.err" &
62+
PIDS+=($!)
63+
wait_for "HTTP worker on :$port" curl -sf -o /dev/null "http://127.0.0.1:$port/health" \
64+
|| { cat "$SCRATCH/http.err" >&2; return 1; }
65+
"${RUN[@]}" vgi-rpc-test-hosted --url "http://127.0.0.1:$port" --expect "$EXPECT" --identity -- -rs
66+
}
67+
68+
for t in "${TRANSPORTS[@]}"; do
69+
case "$t" in
70+
stdio) run_stdio ;;
71+
unix) run_unix ;;
72+
http) run_http ;;
73+
*) echo "unknown transport: $t (expected stdio, unix or http)" >&2; exit 2 ;;
74+
esac
75+
done

‎tests/_hosted_protocols_worker.py‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# Copyright 2025, 2026 Query Farm LLC - https://query.farm
2+
3+
"""A worker that hosts two extra protocols through ``Worker.hosted_protocols``.
4+
5+
Run as a script so each transport is exercised through its real entry point:
6+
7+
- no flags: ``Worker.main`` -> ``Worker.run`` (stdin/stdout)
8+
- ``--unix PATH`` / ``--tcp HOST:PORT``: ``Worker.main``'s launcher paths
9+
- ``--meta`` (stripped before the rest is parsed): ``MetaWorker.serve``
10+
"""
11+
12+
from __future__ import annotations
13+
14+
import sys
15+
from collections.abc import Sequence
16+
from typing import ClassVar, Protocol
17+
18+
from vgi.worker import Worker
19+
20+
21+
class EchoProtocol(Protocol):
22+
"""First extra protocol."""
23+
24+
protocol_name: ClassVar[str] = "vgi_test.Echo.v1"
25+
26+
def echo(self, text: str) -> str:
27+
"""Return *text*, tagged."""
28+
...
29+
30+
31+
class CounterProtocol(Protocol):
32+
"""Second extra protocol, to pin ordering."""
33+
34+
protocol_name: ClassVar[str] = "vgi_test.Counter.v1"
35+
36+
def count(self, text: str) -> int:
37+
"""Return the length of *text*."""
38+
...
39+
40+
41+
class EchoImpl:
42+
"""Implements `EchoProtocol`."""
43+
44+
def echo(self, text: str) -> str:
45+
"""Return *text*, tagged."""
46+
return f"echo:{text}"
47+
48+
49+
class CounterImpl:
50+
"""Implements `CounterProtocol`."""
51+
52+
def count(self, text: str) -> int:
53+
"""Return the length of *text*."""
54+
return len(text)
55+
56+
57+
class HostingWorker(Worker):
58+
"""Hosts `EchoProtocol` then `CounterProtocol` beside ``vgi.v2``."""
59+
60+
functions = []
61+
62+
@classmethod
63+
def hosted_protocols(cls) -> Sequence[tuple[type, object]]:
64+
"""Return the two extra protocols, Echo first."""
65+
return ((EchoProtocol, EchoImpl()), (CounterProtocol, CounterImpl()))
66+
67+
68+
if __name__ == "__main__":
69+
if "--meta" in sys.argv:
70+
sys.argv.remove("--meta")
71+
from vgi.meta_worker import MetaWorker
72+
73+
MetaWorker.serve(HostingWorker)
74+
else:
75+
HostingWorker.main()

0 commit comments

Comments
 (0)