Skip to content

Commit 6be6eec

Browse files
rustyconoverclaude
rustyconover
and
claude
committed
docs+test: opaque-value sealing spec, verified against the reference, and a cross-SDK conformance group
Checked docs/protocol/vgi-opaque-data-sealing.md against vgi/worker.py over HTTP. Corrections to the spec: - The uniform error was a plain ValueError, which reached the wire as error_code UNKNOWN with no error_kind. It is now a classified error, vgi.exceptions.OpaqueDataNotRecognizedError (a ValueError subclass): code INVALID_ARGUMENT, kind opaque_data_not_recognized, message exactly "<field> not recognized", and no details. The spec requires this shape. vgi-rpc's Python server still prefixes the log message with the class name, and the harness strips it. - The envelope version byte is 0x02 for both values. The attach plaintext is uuid(16) || catalog bytes, with any MetaWorker routing header inside the seal. The transaction AAD binds the *sealed* attach as presented. - Python seals on HTTP even with VGI_SIGNING_KEY unset: vgi-serve mints a key, and so does vgi-fixture-http. On stdio and unix it passes values through. - The short hash is SHA-256 over the value's lowercase hex text (matching vgi_rpc.sentry.short_hash), not over the bytes. A raw-hex prefix is not a hash. - Rule 5 binds the catalog implementation, because the framework never copies options into the value. - Known gap, noted in the spec: vgi-rpc's DEBUG access log (request_data) records full request payloads, catalog_attach secret options included. Python violated its own spec, now fixed: - MetaWorker answered an unroutable sealed value with "Cannot route ... (known catalogs: [...])", which distinguished the failure modes and listed the catalogs. vgi-fixture-http hit it on every bad value. It now raises the uniform rejection whenever the values are sealed. - MetaWorker's dispatch log id was the raw-hex prefix. The transactor logged 8-hex prefixes, and put full hex in its errors. The fixture catalog interpolated the value with repr(). All of these now use the short hash (vgi/_redact.py). Lifecycle call sites hand raw values to the redacting chokepoint. - _log_catalog_lifecycle already hashed its fields, so nothing changed there. New tests: - tests/sdk_conformance/test_opaque_sealing.py runs against any fixture worker over HTTP and skips when VGI_SDK_HTTP_URL is unset. It asserts the exact code, kind, message and empty details of every refusal. It covers replay by another principal, one flipped byte, a transaction under another attach, forged unsealed shapes, uniform errors, secret absence, and the optional worker log. - test_opaque_sealing_stdio.py checks rule 5 over stdio (VGI_SDK_STDIO_WORKER). - Unit tests cover MetaWorker uniformity and secret absence when unsealed. Results: - Python fixture: all pass. AttachOptionsWorker: all pass, secret case included. - Mutations turn the group red: removing the key fails 13, and an empty identity tail fails 2. - Go, Rust and C# origin/main fail 12 each, accepting everything. C# passes the secret case. Over stdio, Go, C# and Python pass rule 5. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
1 parent 2029f09 commit 6be6eec

13 files changed

Lines changed: 1044 additions & 37 deletions

‎docs/catalog-interface.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -351,6 +351,8 @@ ATTACH 'mydb' (TYPE vgi, LOCATION 'https://worker.example.com', api_key getenv('
351351

352352
On the wire, `required` and `secret` are nullable boolean columns appended after the four shared spec columns (`name`, `description`, `type`, `default_value`), in that order. Readers look columns up by name, so a spec from a peer that predates either column reads it as `false`, and older peers ignore columns they don't know.
353353

354+
**Never put a secret option's value in `attach_opaque_data`.** On HTTP the worker seals the value, but over stdio and a unix socket it reaches the client, and the DuckDB extension, unsealed. Keep the credential server-side, re-derive it per call, or store a digest of it, as the `ticket_probe` fixture does (see [Opaque values: sealing](protocol/vgi-opaque-data-sealing.md), rule 5).
355+
354356
`secret` does not change what the worker itself logs: attach options are never logged unless the catalog opts in through `loggable_attach_options()` (below), which must never return a secret option.
355357

356358
---

‎docs/protocol/vgi-opaque-data-sealing.md‎

Lines changed: 220 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,220 @@
1+
# Opaque values: sealing (normative)
2+
3+
Status: normative · 2026-10-07
4+
5+
`catalog_attach` returns `attach_opaque_data`, and `catalog_transaction_begin`
6+
returns `transaction_opaque_data`. The client stores these values and sends
7+
them back on every later call for that catalog or transaction. They are the
8+
worker's own state, held by the client.
9+
10+
No spec said how a worker must protect them, and SDKs diverged. Two SDKs carried
11+
every attach option, secret ones included, in plaintext. The extension logged
12+
the raw bytes, so those secrets reached `duckdb_logs`. Three SDKs let a client
13+
edit or replay the value to reach another catalog or session. This page states
14+
the rule every SDK follows.
15+
16+
## Scope
17+
18+
- **Covered values:** `attach_opaque_data`, `transaction_opaque_data`, and any
19+
other worker-issued value a client stores and returns.
20+
- **Authenticating transports** (HTTP, and any transport that carries a
21+
per-call identity): the rules below are **required**.
22+
- **OS-owned transports** (a subprocess over stdio, a unix socket under the
23+
same user): the operating system owns the trust boundary, so sealing is
24+
optional. Even there, the "no secrets" and "never log raw" rules still apply.
25+
26+
## Rules
27+
28+
1. **Seal.** On an authenticating transport, the worker MUST seal each value
29+
with an AEAD cipher keyed by the deployment's signing key: the same key that
30+
seals HTTP state tokens. That is `VGI_SIGNING_KEY` when configured, or a key
31+
the worker generates at startup when it is not. An unset `VGI_SIGNING_KEY`
32+
never means "don't seal".
33+
- XChaCha20-Poly1305 is recommended, with the envelope
34+
`version || nonce || ciphertext || tag`, as used for state tokens.
35+
- The value is only ever opened by the deployment that sealed it, so the
36+
exact bytes need not match between SDKs. The properties below must.
37+
- Reference: `vgi_rpc.crypto.seal_bytes` with version byte `0x02` for both
38+
values, so an envelope is `0x02 || nonce(24) || ciphertext || tag(16)`.
39+
The key is `VGI_SIGNING_KEY`'s UTF-8 bytes, any length, normalized by
40+
`vgi_rpc.crypto.normalize_key`. When it is unset, `vgi-serve` mints
41+
`secrets.token_urlsafe(32)` (and logs a warning that values won't survive a
42+
restart), and `vgi-fixture-http` mints 32 random bytes.
43+
- Reference attach plaintext: a fresh 16-byte UUID (the storage shard key)
44+
followed by the catalog's own bytes. Under a `MetaWorker`, a routing header
45+
naming the catalog wraps it inside the seal, never outside it.
46+
- Anonymous callers on an authenticating transport get sealed values too,
47+
bound to the anonymous identity below.
48+
2. **Bind to the caller.** The AAD MUST include the caller's identity. A value
49+
sealed for one caller MUST NOT open for another. The reference AAD is:
50+
51+
```text
52+
"vgi.attach_opaque_data.v1" 0x00 || identity
53+
identity = 0x01 || domain || 0x00 || principal (authenticated caller)
54+
| 0x00 || "anonymous" (anyone else)
55+
```
56+
57+
`domain` and `principal` are the UTF-8 strings of the caller's auth
58+
context. A missing domain is empty, so `(domain, principal)` both count:
59+
the same principal name under another auth domain does not open the value.
60+
3. **Bind transactions to their attach.** `transaction_opaque_data` MUST bind
61+
both the caller and the parent `attach_opaque_data`. The reference AAD is
62+
`"vgi.transaction_opaque_data.v1" 0x00 || identity || 0x00 ||
63+
attach_envelope`. `attach_envelope` is the *sealed* attach value exactly as
64+
that call presents it, not its plaintext. So a transaction minted under one
65+
attach can't be replayed under another, even by the same principal. That
66+
holds even for two attaches of the same catalog.
67+
4. **Reject uniformly; never fall back.**
68+
- A value that fails to open MUST be rejected. That covers a wrong caller, a
69+
wrong parent attach, tampering, a malformed value, or an unknown key.
70+
- Every one of those cases MUST give the same, classified error:
71+
72+
| Field | Value |
73+
|---|---|
74+
| `error_code` | `INVALID_ARGUMENT` |
75+
| `error_kind` | `opaque_data_not_recognized` |
76+
| message | exactly `<field> not recognized`, where `<field>` is `attach_opaque_data` or `transaction_opaque_data` |
77+
| `error_details` | none |
78+
79+
The two fields' errors differ only in the field name. A probing caller
80+
learns nothing about which check failed, and no refusal is `UNKNOWN`.
81+
- Routing is a check too. A worker that picks a handler from the value, such
82+
as the reference `MetaWorker`, MUST answer an unroutable value with this
83+
same error. It MUST NOT describe the failure or list what it does serve.
84+
- Reference: `vgi.exceptions.OpaqueDataNotRecognizedError`, a `ValueError`
85+
subclass that declares `error_code` and `error_kind` for vgi-rpc's error
86+
model. vgi-rpc's Python server puts the exception class name in front of
87+
every log message, so the reference's `vgi_rpc.log_message` reads
88+
`OpaqueDataNotRecognizedError: <field> not recognized`. The bare message
89+
travels as `exception_message` in `vgi_rpc.log_extra`. The conformance
90+
group strips a leading `"<error_type>: "` before comparing, so an SDK that
91+
sends the bare message passes too.
92+
- There is **no plaintext fallback**: no prefix, flag or missing call context
93+
may skip the open.
94+
5. **No secrets in plaintext.** Attach options declared `secret=True` MUST NOT
95+
appear in plaintext in any opaque value.
96+
- On an authenticating transport the seal satisfies this.
97+
- On an OS-owned transport that skips sealing, leave secret options out of
98+
the value. Re-derive them or keep them server-side.
99+
- Reference: the framework never copies attach options into the value. It
100+
wraps only the catalog's own bytes, so this rule binds the catalog
101+
implementation. A catalog that keeps options in its attach bytes MUST drop
102+
the secret ones, or keep only a digest. The `attach_options` fixture
103+
catalog echoes only its declared non-secret options. The `ticket_probe`
104+
fixture keeps `region` and `sha256(api_key)[:12]`, never the key.
105+
6. **Unpredictable identifiers.** Any id inside an opaque value (session id,
106+
attach uuid) MUST come from a cryptographically secure random source, never
107+
a seeded PRNG such as `mt19937`. The reference uses `uuid.uuid4()`.
108+
7. **Never log raw.** Workers and clients MUST NOT log, trace or report the raw
109+
bytes, the full hex, or a raw-hex *prefix* of an opaque value. That covers
110+
logs, OTel attributes, Sentry tags and breadcrumbs, and error messages.
111+
- Log a short hash instead: the first 12 hex characters of SHA-256 over the
112+
value's **lowercase hex text**, `sha256(hex(value).encode()).hexdigest()[:12]`.
113+
- Hashing the hex text rather than the bytes matches
114+
`vgi_rpc.sentry.short_hash`. The same value then carries the same token in
115+
the worker's logs and in vgi-rpc's Sentry tags, and an SDK that hashes the
116+
same way can be correlated with it. The reference helper is
117+
`vgi._redact.short_hash`.
118+
- See [Known gap](#known-gap) for the one place a secret still reaches a log.
119+
120+
## Transports without sealing (reference)
121+
122+
With no signing key, the reference stores and opens values unchanged. That
123+
covers a subprocess over stdio and a unix socket. `attach_opaque_data` is then
124+
`uuid(16) || catalog bytes`, and any caller on that transport can use any
125+
value. The operating system's process ownership is the boundary. Rules 5, 6
126+
and 7 still apply. A sealed value presented to an unsealed worker, or the
127+
reverse, is rejected or mis-read; values never cross transports.
128+
129+
## Known gap
130+
131+
vgi-rpc's access log, at `DEBUG`, records each request's full payload as
132+
base64 in `request_data`. A `catalog_attach` request carries its attach
133+
options, so a secret option reaches that log in plaintext. Other requests put
134+
sealed values in it. Rule 7 does not yet hold there, and the fix belongs in
135+
vgi-rpc, for example by redacting `secret=True` options and opaque fields from
136+
`request_data`. Until it lands, treat a `DEBUG` access log as secret-bearing
137+
and don't enable it in production. At `INFO` and above the payload is
138+
omitted.
139+
140+
## Rotation
141+
142+
Rotating the signing key invalidates every outstanding value. Clients then get
143+
`not recognized` and must re-attach. This is intended.
144+
145+
## Conformance
146+
147+
`tests/sdk_conformance/test_opaque_sealing.py` in vgi-python is the
148+
cross-SDK group. It runs against any SDK's fixture worker over HTTP and drives
149+
it with vgi-python's `Client`, so the test controls every opaque byte. It
150+
checks that:
151+
152+
- a value attached as one principal and replayed as another is rejected, for
153+
both attach and transaction values;
154+
- a value with one flipped byte (first, middle or last) is rejected, for both
155+
attach and transaction values;
156+
- a transaction value replayed under a different attach of the same catalog,
157+
by the same principal, is rejected;
158+
- forged values in every shape an SDK produced unsealed are rejected: a bare
159+
uuid, `uuid || catalog bytes`, a `writable:` prefix, an Arrow IPC options
160+
batch, JSON, and an envelope-shaped random value;
161+
- every refusal has `error_code` `INVALID_ARGUMENT`, `error_kind`
162+
`opaque_data_not_recognized`, the exact message `<field> not recognized`
163+
and no details. Within a field all refusals are identical, and the two
164+
fields differ only in the field name;
165+
- a `secret=True` attach option is absent from the attach and transaction
166+
values, as bytes, as hex, and as base64;
167+
- a `secret=True` attach option is absent from the *unsealed* values on
168+
stdio too. This is `test_opaque_sealing_stdio.py`, enabled by
169+
`VGI_SDK_STDIO_WORKER=<worker command>`;
170+
- optionally, the worker's log holds neither the secret nor any 12-byte window
171+
of a value's hex, nor its base64.
172+
173+
The owner's own use of each value is checked first, so a worker that rejects
174+
everything does not pass. Probes go through `catalog_version`, which every
175+
worker must answer by opening both values.
176+
177+
### Running it
178+
179+
The group skips entirely unless `VGI_SDK_HTTP_URL` is set, so a plain `pytest`
180+
stays green.
181+
182+
| Variable | Meaning |
183+
|---|---|
184+
| `VGI_SDK_HTTP_URL` | Base URL of the running fixture worker (required). |
185+
| `VGI_SDK_BEARER_A`, `VGI_SDK_BEARER_B` | Bearers for two distinct principals. The defaults are `vgi-test-alice` and `vgi-test-bob`, the fixture test bearers. |
186+
| `VGI_SDK_CATALOG` | Catalog for the replay, tamper and transaction cases. The default is the first that attaches with no options; `example` is tried first. |
187+
| `VGI_SDK_SECRET_CATALOG` | Catalog with a secret option. The default is `ticket_probe`, then any catalog with a secret option whose required options are strings. With none, that case skips. |
188+
| `VGI_SDK_PLAINTEXT_HEX` | Extra forged values, as comma-separated hex: the exact shape this SDK would produce unsealed. |
189+
| `VGI_SDK_STDIO_WORKER` | Worker command for the stdio rule-5 check, in its own module that skips when this is unset. Use the worker that serves the secret-option catalog. |
190+
| `VGI_SDK_WORKER_LOG` | Path of the worker's captured stdout and stderr. Set it to enable the log check. |
191+
192+
Each SDK's CI starts its fixture worker with `--http`, reads the `PORT:<n>`
193+
line, and runs the group from a vgi-python checkout:
194+
195+
```bash
196+
"$FIXTURE_WORKER" --http >worker.log 2>&1 &
197+
until grep -q '^PORT:' worker.log; do sleep 0.2; done
198+
port=$(sed -n 's/^PORT://p' worker.log | head -1)
199+
200+
git clone --depth 1 https://github.com/Query-farm/vgi-python.git vgi-python
201+
cd vgi-python && uv sync --all-extras
202+
VGI_SDK_HTTP_URL="http://127.0.0.1:$port" \
203+
VGI_SDK_WORKER_LOG="$OLDPWD/worker.log" \
204+
uv run pytest tests/sdk_conformance -q
205+
```
206+
207+
| SDK | `$FIXTURE_WORKER` |
208+
|---|---|
209+
| Python | `vgi-fixture-http --port 0`, which is HTTP already: drop `--http` |
210+
| Go | `vgi-example-worker-go` (`go build ./cmd/vgi-example-worker`) |
211+
| Rust | `target/debug/vgi-example-worker` |
212+
| C# | `fixtures/QueryFarm.Vgi.ExampleWorker/bin/Debug/net10.0/vgi-example-worker` |
213+
| C++ | the fixture worker, run with `VGI_FIXTURE_TEST_BEARERS=1` so it accepts the test bearers |
214+
| Java, TypeScript | the SDK's example fixture worker |
215+
216+
A worker that refuses correctly but with another code, kind or message
217+
fails with a list of exactly what differs. If the two bearers resolve to the
218+
same principal, or both to anonymous, the
219+
replay cases fail. The assertion says so. A worker without a secret-option
220+
catalog skips that one case and names the reason.

‎mkdocs.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -145,6 +145,7 @@ nav:
145145
- protocol/index.md
146146
- "Filter Encoding v2 (normative, proposed)": protocol/vgi-filter-encoding-v2-spec.md
147147
- "Runtime-filter artifacts (optional extension)": protocol/vgi-runtime-filter-artifacts.md
148+
- "Opaque values: sealing (normative)": protocol/vgi-opaque-data-sealing.md
148149
- "DuckDB filter adapter (informative)": protocol/vgi-duckdb-filter-adapter.md
149150
- "Catalog query pushdown (proposed design)": protocol/vgi-catalog-query-pushdown-design.md
150151
- "DuckDB 2.0 design audit (informative)": protocol/vgi-protocol-proposed-changes.md

‎tests/sdk_conformance/__init__.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# Copyright 2025, 2026 Query Farm LLC - https://query.farm
2+
3+
"""Cross-SDK conformance groups, run against any SDK's fixture worker over HTTP.
4+
5+
Each module skips entirely unless ``VGI_SDK_HTTP_URL`` names a running worker,
6+
so a plain ``pytest`` run stays green. See the module docstrings for the
7+
environment each group reads.
8+
"""

0 commit comments

Comments
 (0)