Skip to content

Commit 2ad2ad2

Browse files
rustyconoverclaude
andcommitted
test: pin hand-built wire records against their declared ARROW_SCHEMA
Most wire records here derive their schema from the dataclass, so the batch is built FROM ARROW_SCHEMA and cannot disagree with it. Six do not: ScanFunctionResult, ScanBranch, ScanBranchesResult, SettingSpec, AttachOptionSpec and SecretTypeSpec declare ARROW_SCHEMA by hand and build their row by hand, and pa.RecordBatch.from_pylist is forgiving in both directions — a key the schema does not declare is dropped without a word, and a column the row omits is written as null. Neither is visible in the bytes, and to a reader both look like a worker that had nothing to say. The sibling SDKs have paid for exactly this. TypeScript's ScanBranch builder supplied 7 of the schema's 9 columns after two format_* fields were added; the missing format_locations is a LIST and Arrow dereferences a list's children while writing, so every multi-branch table in that worker died with "Cannot read properties of undefined (reading 'slice')", including tables that predated those fields. Java marked two PlanResponse fields nullable where the protocol says non-null and the client rejected the whole response as an "out-of-date Apache Arrow schema". The new test builds each record with every field populated and checks the row's key set against the schema's columns, the serialized schema against ARROW_SCHEMA field-for-field, and that no column came back null. A fourth test walks the package for classes declaring their own ARROW_SCHEMA (read off __dict__, so the generated-descriptor classes are correctly ignored), so a record type added later is either covered or explicitly excused — never silently untested. _SpecBase and SecretTypeSpec built their rows inline inside serialize(), with nothing able to see the key list; both now go through to_row_dict(), matching the three records in catalog_interface.py and giving the parity check something to read. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent f393623 commit 2ad2ad2

3 files changed

Lines changed: 305 additions & 24 deletions

File tree

Lines changed: 264 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,264 @@
1+
# Copyright 2025, 2026 Query Farm LLC - https://query.farm
2+
3+
"""Parity between hand-built wire records and their declared ``ARROW_SCHEMA``.
4+
5+
Most wire records in this package derive their schema from the dataclass
6+
(``ArrowSerializableDataclass``), so the batch is built *from* ``ARROW_SCHEMA``
7+
and drift is structurally impossible. A handful do not: they declare
8+
``ARROW_SCHEMA`` by hand *and* build the row by hand, and ``from_pylist`` is
9+
forgiving in both directions — a key the schema does not declare is dropped
10+
without a word, and a column the row omits is written as null. Neither shows up
11+
in the bytes, and to a reader both look like a worker that had nothing to say.
12+
13+
That failure mode is not hypothetical; it has cost the sibling SDKs real
14+
outages:
15+
16+
* TypeScript's ``ScanBranch`` builder supplied 7 of the schema's 9 columns
17+
because it was never updated when the two ``format_*`` fields were added. The
18+
missing ``format_locations`` is a LIST, and Arrow dereferences a list's
19+
children while writing, so EVERY multi-branch table in that worker died with
20+
``TypeError: Cannot read properties of undefined (reading 'slice')`` —
21+
including tables that predated those fields entirely.
22+
* Java's ``PlanResponse`` marked two fields nullable where the protocol says
23+
non-null, and the client rejected the whole response as an "out-of-date
24+
Apache Arrow schema".
25+
26+
So for every such record this module builds one with every field populated and
27+
checks three things: the row's key set is exactly the schema's column set, the
28+
serialized batch's schema is exactly ``ARROW_SCHEMA``, and no column came back
29+
null. ``test_every_hand_declared_schema_is_covered`` then walks the package for
30+
hand-declared ``ARROW_SCHEMA`` class attributes, so a record type added later is
31+
either covered here or explicitly excused — never silently untested.
32+
"""
33+
34+
from __future__ import annotations
35+
36+
import importlib
37+
import pkgutil
38+
from collections.abc import Callable
39+
from dataclasses import dataclass
40+
from typing import Any
41+
42+
import pyarrow as pa
43+
import pytest
44+
from vgi_rpc.utils import deserialize_record_batch
45+
46+
import vgi
47+
from vgi.catalog.attach_option import AttachOptionSpec
48+
from vgi.catalog.catalog_interface import (
49+
ScanBranch,
50+
ScanBranchesResult,
51+
ScanFunctionResult,
52+
)
53+
from vgi.catalog.secret_type import SecretTypeSpec
54+
from vgi.catalog.setting import SettingSpec
55+
56+
57+
@dataclass(frozen=True)
58+
class WireRecordCase:
59+
"""One record that declares its own ``ARROW_SCHEMA`` and builds its own row.
60+
61+
``build`` returns an instance with EVERY field of the schema populated, so a
62+
null column in the result means the builder dropped the value rather than
63+
the fixture never supplying one.
64+
"""
65+
66+
name: str
67+
build: Callable[[], Any]
68+
69+
70+
def _scalar_args() -> tuple[list[pa.Scalar], dict[str, pa.Scalar]]:
71+
return (
72+
[pa.scalar("s3://bucket/x.parquet", pa.string())],
73+
{"hive_partitioning": pa.scalar(True, pa.bool_())},
74+
)
75+
76+
77+
def _scan_function_result() -> ScanFunctionResult:
78+
positional, named = _scalar_args()
79+
return ScanFunctionResult(
80+
function_name="read_parquet",
81+
positional_arguments=positional,
82+
named_arguments=named,
83+
required_extensions=["parquet"],
84+
)
85+
86+
87+
def _scan_branch() -> ScanBranch:
88+
positional, named = _scalar_args()
89+
return ScanBranch(
90+
function_name="read_parquet",
91+
positional_arguments=positional,
92+
named_arguments=named,
93+
branch_filter="ts >= '2026-01-01'",
94+
writable=True,
95+
source_catalog="lake",
96+
source_schema="main",
97+
source_table="events",
98+
format_name="parquet",
99+
format_locations=["s3://bucket/a.parquet"],
100+
)
101+
102+
103+
def _scan_branches_result() -> ScanBranchesResult:
104+
return ScanBranchesResult(branches=[_scan_branch()], required_extensions=["parquet"])
105+
106+
107+
CASES: list[WireRecordCase] = [
108+
WireRecordCase("ScanFunctionResult", _scan_function_result),
109+
WireRecordCase("ScanBranch", _scan_branch),
110+
WireRecordCase("ScanBranchesResult", _scan_branches_result),
111+
WireRecordCase(
112+
"SettingSpec",
113+
lambda: SettingSpec(name="greeting", desc="Greeting text", type=pa.string(), default="hello"),
114+
),
115+
WireRecordCase(
116+
# `required` and `default` are mutually exclusive by construction, so
117+
# this instance populates `default` and the `required` column carries
118+
# its meaningful False. The nullable-column check below skips it.
119+
"AttachOptionSpec",
120+
lambda: AttachOptionSpec(name="multiplier", desc="Row multiplier", type=pa.int64(), default=2, required=False),
121+
),
122+
WireRecordCase(
123+
"SecretTypeSpec",
124+
lambda: SecretTypeSpec(
125+
name="vgi_example",
126+
description="Example VGI secret",
127+
schema=pa.schema([pa.field("api_key", pa.string(), metadata={"redact": "true"})]),
128+
),
129+
),
130+
]
131+
132+
#: Hand-declared ``ARROW_SCHEMA`` holders that no case builds, and why. Listing
133+
#: them beats ignoring unknown classes: adding a record forces a deliberate
134+
#: choice between covering it and writing down why it needs no coverage.
135+
NOT_COVERED: dict[str, str] = {
136+
"_SpecBase": (
137+
"abstract base — SettingSpec and AttachOptionSpec are the concrete records on the wire and both are covered"
138+
),
139+
}
140+
141+
CASES_BY_NAME = {case.name: case for case in CASES}
142+
143+
144+
@pytest.mark.parametrize("case", CASES, ids=lambda c: c.name)
145+
def test_row_dict_covers_every_declared_column(case: WireRecordCase) -> None:
146+
"""The builder's row names exactly the columns the schema declares.
147+
148+
A missing key is the ScanBranch bug: silently null on the wire (or, for a
149+
list column, a crash inside Arrow's writer). An extra key is the opposite
150+
failure and just as quiet — ``from_pylist`` drops it, so a value the caller
151+
set never reaches the peer at all.
152+
"""
153+
record = case.build()
154+
schema: pa.Schema = type(record).ARROW_SCHEMA
155+
row_keys = set(record.to_row_dict())
156+
declared = set(schema.names)
157+
158+
assert row_keys == declared, (
159+
f"{case.name}.to_row_dict() does not match ARROW_SCHEMA: "
160+
f"missing {sorted(declared - row_keys)}, unexpected {sorted(row_keys - declared)}"
161+
)
162+
163+
164+
@pytest.mark.parametrize("case", CASES, ids=lambda c: c.name)
165+
def test_serialized_schema_is_the_declared_schema(case: WireRecordCase) -> None:
166+
"""The bytes on the wire carry ``ARROW_SCHEMA``, field for field.
167+
168+
Names, order, types and nullability all matter: the C++ client matches the
169+
response schema against its own generated copy and rejects the whole
170+
response on any difference, which is how Java's two over-nullable fields
171+
turned into "out-of-date Apache Arrow schema".
172+
"""
173+
record = case.build()
174+
schema: pa.Schema = type(record).ARROW_SCHEMA
175+
batch, _ = deserialize_record_batch(record.serialize())
176+
177+
assert batch.schema.names == schema.names, (
178+
f"{case.name}: serialized columns {batch.schema.names} != declared {schema.names}"
179+
)
180+
for declared_field in schema:
181+
actual = batch.schema.field(declared_field.name)
182+
assert actual.type == declared_field.type, (
183+
f"{case.name}.{declared_field.name}: serialized as {actual.type}, declared {declared_field.type}"
184+
)
185+
assert actual.nullable == declared_field.nullable, (
186+
f"{case.name}.{declared_field.name}: serialized nullable={actual.nullable}, "
187+
f"declared nullable={declared_field.nullable}"
188+
)
189+
190+
191+
@pytest.mark.parametrize("case", CASES, ids=lambda c: c.name)
192+
def test_every_column_is_populated(case: WireRecordCase) -> None:
193+
"""No column comes back null for a record whose every field is set.
194+
195+
A column that is present but never written is indistinguishable on the wire
196+
from "the worker has nothing to say", so the peer reads a default and the
197+
feature quietly does not exist.
198+
"""
199+
record = case.build()
200+
schema: pa.Schema = type(record).ARROW_SCHEMA
201+
batch, _ = deserialize_record_batch(record.serialize())
202+
203+
row = batch.to_pylist()[0]
204+
for name in schema.names:
205+
assert row[name] is not None, (
206+
f"{case.name}.{name} is null even though the fixture populates every field "
207+
f"— the builder is not wiring this column"
208+
)
209+
210+
211+
def _hand_declared_schema_classes() -> dict[str, type]:
212+
"""Classes in the ``vgi`` package that declare ``ARROW_SCHEMA`` themselves.
213+
214+
Deliberately reads ``__dict__`` rather than ``getattr``: an
215+
``ArrowSerializableDataclass`` subclass answers ``ARROW_SCHEMA`` from a
216+
descriptor that GENERATES it from the dataclass fields, so its batch cannot
217+
disagree with its schema. Only a class that spells the schema out by hand
218+
has two things that can drift apart, and only those belong here.
219+
"""
220+
found: dict[str, type] = {}
221+
for module_info in pkgutil.walk_packages(vgi.__path__, prefix="vgi."):
222+
try:
223+
module = importlib.import_module(module_info.name)
224+
except ImportError: # pragma: no cover - optional extras
225+
continue
226+
for obj in vars(module).values():
227+
if not isinstance(obj, type) or obj.__module__ != module_info.name:
228+
continue
229+
if isinstance(vars(obj).get("ARROW_SCHEMA"), pa.Schema):
230+
# Keyed by __qualname__, not the binding name, so a module-level
231+
# alias (WriteFunctionResult = ScanFunctionResult) does not read
232+
# as a second, uncovered record type.
233+
found[obj.__qualname__] = obj
234+
return found
235+
236+
237+
def test_every_hand_declared_schema_is_covered() -> None:
238+
"""No hand-built record type escapes the parity checks above.
239+
240+
A new record type FAILS here as unlisted rather than being picked up
241+
automatically. Auto-coverage would need a generic fully-populated builder
242+
per record, and there isn't one — the builders take dataclass fields, not
243+
schemas — so "automatic" would in practice mean "skipped", which recreates
244+
the gap this test exists to close.
245+
"""
246+
classes = _hand_declared_schema_classes()
247+
assert classes, (
248+
"found no classes declaring their own ARROW_SCHEMA — the discovery walk broke "
249+
"and this coverage guard is now silently vacuous"
250+
)
251+
252+
missing = sorted(set(classes) - set(CASES_BY_NAME) - set(NOT_COVERED))
253+
assert not missing, (
254+
f"{missing} declare their own ARROW_SCHEMA but no WireRecordCase builds them. "
255+
"Add a case, or add the class to NOT_COVERED with the reason it needs none."
256+
)
257+
258+
# The excuse list has to stay honest too: an entry for a class that no
259+
# longer exists reads as coverage that does not exist.
260+
stale = sorted(set(NOT_COVERED) - set(classes))
261+
assert not stale, f"NOT_COVERED lists {stale}, which no longer declare an ARROW_SCHEMA — drop them"
262+
263+
both = sorted(set(NOT_COVERED) & set(CASES_BY_NAME))
264+
assert not both, f"{both} are both covered by a case and excused in NOT_COVERED"

‎vgi/catalog/_descriptor_spec.py‎

Lines changed: 23 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -122,28 +122,38 @@ def _extra_kwargs(cls, row: dict[str, object]) -> dict[str, Any]:
122122
del row
123123
return {}
124124

125-
def serialize(self) -> bytes:
126-
"""Serialize to Arrow IPC bytes."""
127-
# Serialize type as a single-field schema
125+
def to_row_dict(self) -> dict[str, Any]:
126+
"""Build the wire row: one key per column of :data:`ARROW_SCHEMA`.
127+
128+
Kept separate from :meth:`serialize` because ``from_pylist`` is keyed by
129+
name and forgiving in both directions — a key the schema does not
130+
declare is dropped without a word, and a column this dict omits is
131+
written as null. Neither is visible in the bytes, so both look to a
132+
reader like a worker that had nothing to say. Exposing the row is what
133+
lets ``tests/test_wire_record_schema_parity.py`` compare its key set to
134+
the schema's.
135+
"""
128136
type_schema = schema(value=self.type)
129-
type_bytes = type_schema.serialize().to_pybytes()
130137

131138
# Serialize default value if present
132139
default_bytes: bytes | None = None
133140
if self.default is not None:
134141
default_batch = pa.RecordBatch.from_pydict({"value": [self.default]}, schema=type_schema)
135142
default_bytes = serialize_record_batch_bytes(default_batch)
136143

144+
return {
145+
"name": self.name,
146+
"description": self.desc,
147+
# The type travels as a single-field schema, not a type literal.
148+
"type": type_schema.serialize().to_pybytes(),
149+
"default_value": default_bytes,
150+
**self._extra_row(),
151+
}
152+
153+
def serialize(self) -> bytes:
154+
"""Serialize to Arrow IPC bytes."""
137155
batch = pa.RecordBatch.from_pylist(
138-
[
139-
{
140-
"name": self.name,
141-
"description": self.desc,
142-
"type": type_bytes,
143-
"default_value": default_bytes,
144-
**self._extra_row(),
145-
}
146-
],
156+
[self.to_row_dict()],
147157
schema=self.ARROW_SCHEMA,
148158
)
149159
return serialize_record_batch_bytes(batch)

‎vgi/catalog/secret_type.py‎

Lines changed: 18 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
"""
88

99
from dataclasses import dataclass
10-
from typing import ClassVar, Self, cast
10+
from typing import Any, ClassVar, Self, cast
1111

1212
import pyarrow as pa
1313
from vgi_rpc.utils import serialize_record_batch_bytes
@@ -60,19 +60,26 @@ class SecretTypeSpec:
6060
] # type: ignore[arg-type] # PyArrow field metadata typing limitation
6161
)
6262

63+
def to_row_dict(self) -> dict[str, Any]:
64+
"""Build the wire row: one key per column of :data:`ARROW_SCHEMA`.
65+
66+
Split out of :meth:`serialize` for the same reason as
67+
:meth:`vgi.catalog._descriptor_spec._SpecBase.to_row_dict` — a key the
68+
schema does not declare is silently dropped and a column this dict omits
69+
is silently null, so the row has to be inspectable for the parity test
70+
to have anything to check.
71+
"""
72+
return {
73+
"name": self.name,
74+
"description": self.description,
75+
# The parameters schema carries the per-field `redact` metadata.
76+
"parameters_schema": self.schema.serialize().to_pybytes(),
77+
}
78+
6379
def serialize(self) -> bytes:
6480
"""Serialize to Arrow IPC bytes."""
65-
# Serialize the parameters schema (with field metadata for redact keys)
66-
schema_bytes = self.schema.serialize().to_pybytes()
67-
6881
batch = pa.RecordBatch.from_pylist(
69-
[
70-
{
71-
"name": self.name,
72-
"description": self.description,
73-
"parameters_schema": schema_bytes,
74-
}
75-
],
82+
[self.to_row_dict()],
7683
schema=self.ARROW_SCHEMA,
7784
)
7885
return serialize_record_batch_bytes(batch)

0 commit comments

Comments
 (0)