Skip to content

Commit 0f593d5

Browse files
rustyconoverclaude
andcommitted
Hold every served table to the v3 ATTACH contract, relax nullability
The SDK now checks, before it serves a table and when the worker starts, everything the DuckDB extension refuses at ATTACH, and stops refusing what it can fix itself. - Nullability (RELAX): vgi::served_columns() serves every Arrow field no NOT_NULL/PRIMARY_KEY record names as nullable and every field a record names as non-nullable (ordinary STRUCTs descended, logical types and the is_row_id column untouched). Applied on the declarative, time-travel, catalog_contents (including custom-handler items) and in-memory paths. - validate_constraint_records() (new overload taking the owning schema path) mirrors DecodeNativeConstraints: owner identity, one-component schema path, duplicate ids, case-insensitive duplicate declared names, one PRIMARY KEY, NOT_NULL paths through ordinary STRUCTs, top-level key/FK columns, same-schema MATCH SIMPLE NO ACTION/RESTRICT foreign keys, CHECK profile/schema/context and ParseNumericCheck's grammar. Foreign keys are also checked against the target key the catalog declares. - Dispatcher::validate_catalogs() runs from install(): every table, time-travel version, view and macro of every catalog/data version fails at startup, naming catalog, table, version and constraint. - Time travel: a version without its own records serves the table-level constraints and definition_revision (an empty vector opts out); empty definition_revision is refused. - Generated columns: VIRTUAL-grammar, type, nullability, reference and cycle checks (src/sql_scalar.cpp); legacy generated_expression metadata is translated to the canonical pair when unambiguous. - CatalogTable::required_filters is vector<RequiredFilterGroup>; the flat {{"a","b"}} spelling no longer compiles. Paths are resolved against the served columns the way the extension does. - Legacy constraint fields are [[deprecated]]; non-"regular" view security is refused; vgi::macro_overload() builds MacroOverload records. - Public capability helpers: encode/decode_catalog_capabilities, client_catalog_offer, select_catalog_capabilities, CONSTRAINT_CREATE, CONSTRAINT_ENFORCE. - In-memory catalog: negotiates records read + create + enforce and supports CREATE TABLE ... NOT NULL as ENFORCED + ALL_ROWS records (it holds no rows and accepts no writes); PRIMARY KEY/UNIQUE/CHECK/FK and logical column types are refused by name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 54c045b commit 0f593d5

17 files changed

Lines changed: 2345 additions & 350 deletions

‎CMakeLists.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,7 @@ add_library(vgi
7676
src/attach_ticket_service.cpp
7777
src/catalog.cpp
7878
src/constraints.cpp
79+
src/sql_scalar.cpp
7980
src/catalog_contents.cpp
8081
src/memory_catalog.cpp
8182
src/arg_schema.cpp

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ of the [Rust](https://github.com/Query-farm/vgi-rust) and
1111
[Python](https://github.com/Query-farm/vgi-python) SDKs, and is built on the
1212
C++ [vgi-rpc](https://github.com/Query-farm/vgi-rpc-cpp) port.
1313

14-
Version **0.13.0** uses **`vgi.v3` / `3.0.0`** and requires a matching VGI
14+
Version **0.13.1** uses **`vgi.v3` / `3.0.0`** and requires a matching VGI
1515
extension. Keep 0.12.x for existing `vgi.v2` installations; upgrade the extension
1616
and worker SDKs together. The shared contract prepares for DuckDB 2.0 while
1717
remaining usable by the 1.5 adapter. Optional catalog and query capabilities

‎example-worker/table/rff.cpp‎

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -234,7 +234,7 @@ std::shared_ptr<arrow::Schema> hive_schema() {
234234
}
235235

236236
// The four bbox corners, each its own group: every one must carry a filter.
237-
std::vector<std::vector<std::vector<std::string>>> bbox_corners() {
237+
std::vector<vgi::RequiredFilterGroup> bbox_corners() {
238238
return {{{"bbox", "xmin"}}, {{"bbox", "xmax"}}, {{"bbox", "ymin"}}, {{"bbox", "ymax"}}};
239239
}
240240

@@ -266,8 +266,7 @@ void register_rff(vgi::Worker& worker) {
266266
// The `rff_*` tables, declared by the catalog.
267267
std::vector<vgi::CatalogTable> rff_tables() {
268268
const auto table = [](std::string name, std::shared_ptr<arrow::Schema> columns,
269-
std::string scan,
270-
std::vector<std::vector<std::vector<std::string>>> required) {
269+
std::string scan, std::vector<vgi::RequiredFilterGroup> required) {
271270
vgi::CatalogTable entry;
272271
entry.name = std::move(name);
273272
entry.columns = std::move(columns);
@@ -284,7 +283,7 @@ std::vector<vgi::CatalogTable> rff_tables() {
284283
// own column projection.
285284
const auto native = [](std::string name, std::shared_ptr<arrow::Schema> columns,
286285
std::string path, bool hive,
287-
std::vector<std::vector<std::vector<std::string>>> required) {
286+
std::vector<vgi::RequiredFilterGroup> required) {
288287
vgi::CatalogTable entry;
289288
entry.name = std::move(name);
290289
entry.columns = std::move(columns);
@@ -330,7 +329,7 @@ std::vector<vgi::CatalogTable> rff_tables() {
330329
// The mixed requirement: a top-level path alongside the corners, which
331330
// lands at a permuted slot once the native scan projects.
332331
auto mixed = bbox_corners();
333-
mixed.insert(mixed.begin(), {{"id"}});
332+
mixed.insert(mixed.begin(), vgi::RequiredFilterGroup{{"id"}});
334333
tables.push_back(native("rff_hive_mixed", hive_schema(), "rff_hive/*/*/*.parquet",
335334
/*hive=*/true, std::move(mixed)));
336335
return tables;

‎include/vgi/catalog.h‎

Lines changed: 115 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33

44
#include <cstdint>
55
#include <functional>
6+
#include <initializer_list>
67
#include <map>
78
#include <memory>
89
#include <optional>
@@ -23,6 +24,29 @@ namespace vgi {
2324
// Returns canonical base64 SqlExpression IPC; does not bind or execute SQL.
2425
std::string generated_expression_metadata(const std::string& sql, const SchemaPath& owner);
2526

27+
// One column path in a `required_filters` group: raw components from the
28+
// top-level column down through STRUCT children, e.g. {"ticker"} or
29+
// {"bbox", "xmin"}. Dots and quotes are literal.
30+
//
31+
// A type of its own, not a std::vector<std::string>, so that the flat spelling
32+
// `{{"ticker", "cik"}}` -- which against nested vectors picked vector's
33+
// iterator-pair constructor from the two string literals and compiled into
34+
// garbage paths or std::length_error -- is a compile error. Spell a group of
35+
// two alternative paths `{{{"ticker"}, {"cik"}}}`.
36+
struct RequiredFilterPath {
37+
std::vector<std::string> components;
38+
39+
RequiredFilterPath(std::initializer_list<std::string> path) : components(path) {}
40+
explicit RequiredFilterPath(std::vector<std::string> path) : components(std::move(path)) {}
41+
42+
bool operator==(const RequiredFilterPath& other) const {
43+
return components == other.components;
44+
}
45+
};
46+
47+
// An OR-group of required-filter paths: satisfied when any path is filtered.
48+
using RequiredFilterGroup = std::vector<RequiredFilterPath>;
49+
2650
// One source a multi-branch table reads from.
2751
//
2852
// A multi-branch table is one logical relation stitched from several scans —
@@ -58,10 +82,13 @@ struct TimeTravelVersion {
5882
std::string scan_arguments;
5983
// The calendar year this version became current, for `AT (TIMESTAMP => …)`.
6084
std::optional<int> timestamp_year;
61-
// Explicit canonical records for this version (vgi::constraint_record()).
62-
// Absence cannot inherit a constrained current table's definitions: its
63-
// columns may have changed.
85+
// Canonical records for this version (vgi::constraint_record()). Absent,
86+
// the version serves the table-level `CatalogTable::constraints` and
87+
// `definition_revision`; an empty vector states that this version has
88+
// none. Either way the records are checked against *this* version's
89+
// columns when the worker starts, naming the table and version.
6490
std::optional<std::vector<std::string>> constraints;
91+
// Nonempty when present.
6592
std::optional<std::string> definition_revision;
6693
};
6794

@@ -75,6 +102,12 @@ struct ForeignKey {
75102
std::optional<SchemaPath> referenced_schema_path;
76103
};
77104

105+
// The legacy constraint members below are deprecated so that code naming them
106+
// is warned. The struct's own implicitly defined members (copy, move,
107+
// destruction) touch them too; that use is not the caller's, so it is
108+
// silenced here and only here.
109+
#pragma GCC diagnostic push
110+
#pragma GCC diagnostic ignored "-Wdeprecated-declarations"
78111
// A table the catalog advertises.
79112
//
80113
// A VGI table is not storage: it is a *name* bound to a table function that
@@ -86,12 +119,15 @@ struct CatalogTable {
86119
// Canonical ConstraintInfo IPC records, each built with
87120
// vgi::constraint_record() (vgi/constraints.h). The provider attests every
88121
// one; identities are never synthesized and nothing is derived from the
89-
// legacy fields below. Serving a table checks that every record is
90-
// ENFORCED + ALL_ROWS and that `columns` is non-nullable exactly at the
91-
// NOT_NULL/PRIMARY_KEY paths -- an unconstrained table must therefore
92-
// declare every field nullable. Records make ATTACH offer the
93-
// constraint read profiles automatically.
122+
// legacy fields below or from Arrow nullability. The worker checks every
123+
// record against the extension's ATTACH rules when it starts (see
124+
// vgi::validate_constraint_records()), and serves `columns` with
125+
// nullability rewritten to match: non-nullable exactly at the
126+
// NOT_NULL/PRIMARY_KEY paths, nullable everywhere else. Records make ATTACH
127+
// offer the constraint read profiles automatically. A time-travel version
128+
// that states no records of its own serves these.
94129
std::vector<std::string> constraints;
130+
// An opaque token naming this definition; nonempty when present.
95131
std::optional<std::string> definition_revision;
96132
std::string name;
97133
std::shared_ptr<arrow::Schema> columns;
@@ -114,9 +150,11 @@ struct CatalogTable {
114150
//
115151
// A group is satisfied when *any* of its paths carries a filter, and every
116152
// group must be satisfied. A single-path group is a plain mandatory
117-
// filter; `{{"ticker"}, {"cik"}}` means "one of these two". Empty means no
118-
// requirement. The engine enforces it — the worker only declares it.
119-
std::vector<std::vector<std::vector<std::string>>> required_filters;
153+
// filter; `{{{"ticker"}, {"cik"}}}` means "one of these two" and
154+
// `{{{"bbox", "xmin"}}}` requires a filter on a STRUCT member. Empty means
155+
// no requirement. The engine enforces it — the worker only declares it,
156+
// and each path must resolve against `columns` through STRUCT parents.
157+
std::vector<RequiredFilterGroup> required_filters;
120158

121159
// Sources this table is stitched from.
122160
//
@@ -132,13 +170,23 @@ struct CatalogTable {
132170

133171
// Legacy (pre-vgi.v3) constraint descriptors. vgi.v3 has no wire form for
134172
// them and the SDK will not translate them into canonical records: a table
135-
// that sets any of these is refused when served. Declare
173+
// that sets any of these is refused when the worker starts. Declare
136174
// vgi::constraint_record() records in `constraints` instead.
137-
std::vector<int32_t> not_null;
138-
std::vector<std::vector<int32_t>> unique;
139-
std::vector<std::vector<int32_t>> primary_key;
140-
std::vector<std::string> check;
141-
std::vector<ForeignKey> foreign_keys;
175+
[[deprecated(
176+
"not sent under vgi.v3; declare vgi::constraint_record() records in "
177+
"CatalogTable::constraints")]] std::vector<int32_t> not_null;
178+
[[deprecated(
179+
"not sent under vgi.v3; declare vgi::constraint_record() records in "
180+
"CatalogTable::constraints")]] std::vector<std::vector<int32_t>> unique;
181+
[[deprecated(
182+
"not sent under vgi.v3; declare vgi::constraint_record() records in "
183+
"CatalogTable::constraints")]] std::vector<std::vector<int32_t>> primary_key;
184+
[[deprecated(
185+
"not sent under vgi.v3; declare vgi::constraint_record() records in "
186+
"CatalogTable::constraints")]] std::vector<std::string> check;
187+
[[deprecated(
188+
"not sent under vgi.v3; declare vgi::constraint_record() records in "
189+
"CatalogTable::constraints")]] std::vector<ForeignKey> foreign_keys;
142190

143191
// Per-column bounds for the optimizer. Empty means the table declines to
144192
// answer, which is different from answering with no columns: the engine
@@ -161,6 +209,7 @@ struct CatalogTable {
161209
// resolve the clause, so either setting travels correctly.
162210
bool inline_scan = true;
163211
};
212+
#pragma GCC diagnostic pop
164213

165214
// A view: a name bound to SQL the engine expands.
166215
struct CatalogView {
@@ -174,6 +223,8 @@ struct CatalogView {
174223
// SQL produces, and the engine matches these against them.
175224
std::vector<std::pair<std::string, std::string>> column_comments;
176225
std::vector<std::string> aliases;
226+
// DuckDB reads only "regular" views; any other value is refused when the
227+
// worker starts.
177228
std::string security = "regular";
178229
};
179230

@@ -191,15 +242,45 @@ struct CatalogMacro {
191242
bool table_macro = false;
192243
std::optional<std::string> comment;
193244
// Defaults for the simple descriptor, by name. This convenience form accepts integers;
194-
// canonical overload records can represent other typed values and declarations.
245+
// build overloads with vgi::macro_overload() for other typed values and declarations.
195246
std::vector<std::pair<std::string, int64_t>> defaults;
196-
// Per-parameter descriptions, surfaced by `vgi_function_arguments()`
197-
// through the same `vgi_doc` channel a function's arguments use.
247+
// Per-parameter descriptions, sent as each `MacroParameter.documentation`
248+
// and surfaced by `vgi_function_arguments()`.
198249
std::vector<std::pair<std::string, std::string>> parameter_docs;
199-
// Complete MacroOverload IPC records; simple-definition fields must be empty.
250+
// Complete MacroOverload IPC records, each built with vgi::macro_overload()
251+
// in this macro's schema and kind; the simple-definition fields above must
252+
// then be empty.
200253
std::optional<std::vector<std::string>> overloads;
201254
};
202255

256+
// One parameter of a macro overload.
257+
struct MacroParameterSpec {
258+
std::string name;
259+
// Null: the parameter takes any type.
260+
std::shared_ptr<arrow::DataType> declared_type;
261+
// A one-element array holding the literal default; null for none. A
262+
// parameter without a default may not follow one with a default.
263+
std::shared_ptr<arrow::Array> default_value;
264+
std::optional<std::string> documentation;
265+
};
266+
267+
// One overload of a macro family, as `macro_overload()` encodes it.
268+
struct MacroOverloadSpec {
269+
// The owning schema and kind: must equal the CatalogMacro's.
270+
SchemaPath schema_path = {"main"};
271+
bool table_macro = false;
272+
std::vector<MacroParameterSpec> parameters;
273+
// The macro body: a scalar expression, or a query for a table macro.
274+
std::string definition;
275+
};
276+
277+
// Encode one MacroOverload IPC record for `CatalogMacro::overloads`, under the
278+
// duckdb.macro.v1 profile and the default evaluation context. Throws
279+
// std::invalid_argument on an empty or NUL-bearing definition or name, a
280+
// duplicate (case-insensitive) parameter name, a default that is not a
281+
// one-element array, or a required parameter after a defaulted one.
282+
std::string macro_overload(const MacroOverloadSpec& spec);
283+
203284
// An option a caller may set in the ATTACH statement.
204285
//
205286
// Distinct from a setting: a setting is global and mutable, an attach option
@@ -316,10 +397,17 @@ std::string catalog_contents_digest(const std::vector<SchemaContents>& schemas);
316397

317398
// A DDL-capable in-memory catalog (Worker::register_memory_catalog).
318399
//
319-
// Accepts CREATE / DROP SCHEMA, TABLE (definitions only: it holds no rows) and
320-
// VIEW. Every ATTACH is private: it starts from an empty `main` schema and
321-
// never sees another attachment's objects, even in one worker process. Its
322-
// version starts at 1, is bumped by every DDL, and is not frozen.
400+
// Accepts CREATE / DROP SCHEMA, TABLE (definitions only: it holds no rows and
401+
// accepts no writes) and VIEW. CREATE TABLE may declare column NOT NULL: ATTACH
402+
// negotiates CONSTRAINT_RECORDS_READ, CONSTRAINT_CREATE and CONSTRAINT_ENFORCE,
403+
// and each NOT NULL becomes an ENFORCED + ALL_ROWS canonical record -- true
404+
// because no write path exists to store a NULL. PRIMARY KEY, UNIQUE, CHECK
405+
// and FOREIGN KEY are refused (UNIMPLEMENTED): nothing here could enforce
406+
// them. A non-nullable column no NOT NULL names is served nullable. Logical
407+
// (extension) column types such as UUID are refused: stored here they would
408+
// be read back as their storage type. Every ATTACH is private: it starts from an empty `main`
409+
// schema and never sees another attachment's objects, even in one worker process. Its version
410+
// starts at 1, is bumped by every DDL, and is not frozen.
323411
//
324412
// State lives in the worker process that served the ATTACH, which on every
325413
// transport this SDK offers is the one serving the rest of that session.
@@ -368,7 +456,8 @@ struct CatalogModel {
368456
// table this attachment serves carries `constraints`, ATTACH offers
369457
// CONSTRAINT_RECORDS_READ and SCALAR_EXPRESSION_READ without being asked.
370458
// Name them here only for tables the model cannot see (a custom
371-
// catalog_contents_handler). With nothing to offer, capability
459+
// catalog_contents_handler; its table items are still checked and served
460+
// with their nullability matched to their records). With nothing to offer, capability
372461
// negotiation stays absent; ATTACH selects only the client's offered
373462
// subset, and the scalar profile only alongside constraint records. The
374463
// provider remains responsible for the semantic guarantees of each

0 commit comments

Comments
 (0)