Skip to content

Commit 3c72132

Browse files
rustyconoverclaude
andcommitted
Fix vgi.v3 regressions: time-travel records, nullability, ATTACH rules
Time travel served a foo_v_N variant's constraint records under foo, still owned by foo_v_N, so ConstraintContract refused the table and AT (VERSION ..) on any versioned table with records failed (blocked 0.45.1). Variant records are now re-owned to the served name (self-referencing foreign keys too), and statistics / required filters carry through versions (the base's, restricted to the variant's columns, when the variant declares none). Serving (every path goes through TableInfoSerializer.served, plus provider catalog_contents tables via MetadataCollections.merge): - Nullability is rewritten, not checked: a field is served non-nullable exactly when a NOT_NULL/PRIMARY_KEY record names it (ordinary STRUCTs recursed, row id and logical-type storage untouched). - The extension's ATTACH rules are applied with table, constraint id and path in the message: one-component schema paths, duplicate declared names, a second PRIMARY KEY, unresolvable or nested non-NOT_NULL paths, FK shape (same schema, MATCH SIMPLE, NO_ACTION/RESTRICT), FK targets resolved across the catalog, the numeric-comparison CHECK grammar, size limits. - Generated columns: VIRTUAL, nullable, BOOLEAN/signed int/VARCHAR and the extension's scalar grammar; legacy generated_expression is translated. New public CatalogColumns.virtualGenerated(..). API: - CatalogProfiles: public encode/decode/select/negotiate/clientOffer helpers. - CatalogInterface#catalogCapabilities(attachId, request) and ExtraCatalog#withCapabilities; hosted CREATE TABLE forwards constraint definitions when create+enforce+read were selected. NOT NULL / logical-type refusals moved into InMemoryCatalog (which still enforces nothing). - viewCreate: the 9-arg default delegates to the 6-arg, which delegates to a restored @deprecated 5-arg (v0.44), so an override at any level is reached. - MacroInfo.of(Macro); registerMacro refuses expression defaults at startup. - Legacy CatalogTable constraint builders are @deprecated; ConstraintRecord builder refuses multi-component schemas, nested non-NOT_NULL paths and cross-schema FKs early; ConstraintRecord.decodeAll for clients. - examples/docs pin bumped to 0.45.1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent fccd6eb commit 3c72132

25 files changed

Lines changed: 2104 additions & 158 deletions

‎docs/catalog_publication.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,3 +23,17 @@ RPC 0.33.2 includes the scalar `DICT_INT16_UTF8` record fix required by macro an
2323
constraint metadata. Release qualification disables the composite source override
2424
and resolves that published dependency from Maven Central. No local repository
2525
fallback is part of the SDK build.
26+
27+
## Constraints, nullability, generated columns and macros (vgi.v3)
28+
29+
Constraints are canonical `ConstraintInfo` records the worker attests: build them with `ConstraintRecord.builder(schema, table, id)` (`notNull`, `primaryKey`, `unique`, `check`, `foreignKey`), state `ENFORCED` + `ALL_ROWS`, and attach them with `CatalogTable.Builder#constraints(records, revision)`. The pre-v3 `primaryKey`/`unique`/`check`/`foreignKeys` builders are `@Deprecated` and refused when the table is served. Every catalog path (main, auxiliary, hosted) advertises `vgi.constraint.records.read.v1` + `vgi.expression.duckdb.scalar.v1` and selects what the client offers automatically.
30+
31+
Before serving a table the framework applies the rules DuckDB applies at ATTACH and refuses, naming the table, constraint id and column path: records owned by another table or a multi-component schema path, duplicate ids or declared names (case-insensitive), a second PRIMARY KEY, column paths that do not resolve (nested paths only on NOT_NULL, through ordinary STRUCTs), foreign keys to another schema, with an action other than NO_ACTION/RESTRICT or whose referenced key the catalog does not declare, and CHECK expressions outside the one-numeric-comparison grammar (`budget >= 0`: unqualified signed-integer/float columns and numeric literals).
32+
33+
Nullability needs no author work: the served Arrow schema is rewritten so a field is non-nullable exactly when a NOT_NULL/PRIMARY_KEY record names it (descending through ordinary STRUCTs; the row-id column and logical-type storage are untouched). A non-nullable field no record names is served nullable.
34+
35+
Generated columns: `CatalogColumns.virtualGenerated(name, type, sql, schemaPath)` builds a VIRTUAL column (nullable; BOOLEAN, signed integer or VARCHAR; the extension's deterministic scalar grammar, no division or float literals). Legacy `generated_expression` metadata is translated to that form when served.
36+
37+
Capabilities: `CatalogProfiles` encodes, decodes and negotiates profiles (`negotiate`, `select`, `clientOffer`, `defaultClientOffer`). A hosted catalog that enforces constraints on its own write paths selects `vgi.constraint.create.v1` + `vgi.constraint.enforce.v1` from `CatalogInterface#catalogCapabilities`; only then does `CREATE TABLE` with constraint definitions reach its `tableCreate`. Auxiliary catalogs use `ExtraCatalog#withCapabilities`. `InMemoryCatalog` stores no rows and enforces nothing, so it refuses constrained, NOT NULL, logical-type and generated columns on `CREATE TABLE`.
38+
39+
Macros: hosted catalogs build `MacroInfo` from a `Macro` descriptor with `MacroInfo.of(macro)`. Parameter defaults are literal values only (`NULL`, booleans, numbers, `'strings'`); an expression default such as `current_date` is refused by `registerMacro` at startup.

‎examples/docs/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,5 +49,5 @@ These build against `project(":vgi")` so they track the SDK at HEAD. A reader
4949
does the same thing with one line:
5050

5151
```kotlin
52-
implementation("farm.query:vgi:0.41.1")
52+
implementation("farm.query:vgi:0.45.1")
5353
```

‎examples/docs/build.gradle.kts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
// the SDK at HEAD and a breaking change shows up here rather than in a reader's
1111
// editor. A reader does the same thing with one line instead:
1212
//
13-
// implementation("farm.query:vgi:0.44.0")
13+
// implementation("farm.query:vgi:0.45.1")
1414
//
1515
// which is what the documentation shows. The toolchain, `-parameters` and the
1616
// JVM opens all come from the root build, so nothing else is repeated here.

‎vgi-example-worker/src/main/java/farm/query/vgi/example/Main.java‎

Lines changed: 4 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -209,14 +209,8 @@ private static byte[] wkb(String hex) {
209209
}
210210

211211
/** Build a canonical VIRTUAL column owned by the data schema. */
212-
private static Field genCol(String name,
213-
ArrowType type,
214-
boolean nullable,
215-
String expr) {
216-
return new Field(name,
217-
new FieldType(nullable, type, null,
218-
Map.of("vgi.column.generated.v1", farm.query.vgi.internal.ColumnExpressions.generatedMetadata(expr, List.of("data")), "vgi.column.generation.v1", "virtual")),
219-
null);
212+
private static Field genCol(String name, ArrowType type, String expr) {
213+
return farm.query.vgi.catalog.CatalogColumns.virtualGenerated(name, type, expr, List.of("data"));
220214
}
221215

222216
/** STRUCT(xmin, ymin, xmax, ymax FLOAT) bbox column used by the native
@@ -1049,8 +1043,8 @@ private static void registerCatalogTables(Worker w) {
10491043
.registerCatalogTable(stubTable("data", "generated_sequence",
10501044
"Table with generated columns backed by sequence(10)",
10511045
col("n", Schemas.INT64, true),
1052-
genCol("doubled", Schemas.INT64, true, "n * 2"),
1053-
genCol("label", Schemas.UTF8, true, "'item_' || n::VARCHAR")))
1046+
genCol("doubled", Schemas.INT64, "n * 2"),
1047+
genCol("label", Schemas.UTF8, "'item_' || n::VARCHAR")))
10541048
.registerCatalogTable(stubTable("data", "projects",
10551049
"Projects with composite PK and FK to departments",
10561050
col("department_id", Schemas.INT64, false),

‎vgi/src/main/java/farm/query/vgi/CatalogInterface.java‎

Lines changed: 88 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,25 @@ default List<String> catalogReadProfiles(byte[] attachId) {
118118
return List.of();
119119
}
120120

121+
/**
122+
* This catalog's own capability advertisement and selection for an attach: build it with
123+
* {@link CatalogProfiles#negotiate}. The framework merges in {@link #catalogReadProfiles},
124+
* and gates the DDL it forwards on the selection: {@code CREATE TABLE} with constraint
125+
* definitions reaches {@link #tableCreate} only when {@link CatalogProfiles#CONSTRAINT_CREATE},
126+
* {@link CatalogProfiles#CONSTRAINT_ENFORCE} and {@link CatalogProfiles#CONSTRAINT_RECORDS_READ}
127+
* were selected (DuckDB sends none otherwise). A catalog that selects them enforces every
128+
* record it then serves, on every write path.
129+
*
130+
* <p>Default: no decision of its own.
131+
*
132+
* @param attachId the attach id
133+
* @param request the attach request (its {@code client_capabilities} is the client's offer)
134+
* @return the decision; either side may be null
135+
*/
136+
default Worker.CatalogCapabilitySelection catalogCapabilities(byte[] attachId, CatalogAttachRequest request) {
137+
return new Worker.CatalogCapabilitySelection(null, null);
138+
}
139+
121140
/**
122141
* Release an attach. Default: nothing to release.
123142
*
@@ -177,7 +196,8 @@ default Optional<SchemaInfo> schema(byte[] attachId, List<String> path) {
177196
default List<ViewInfo> views(byte[] attachId, List<String> path) { return List.of(); }
178197

179198
/**
180-
* The macros of a schema of one type. Default: none.
199+
* The macros of a schema of one type. Default: none. Build each family from a simple
200+
* descriptor with {@link MacroInfo#of(farm.query.vgi.catalog.Macro)}.
181201
*
182202
* @param attachId the attach id
183203
* @param path the schema path
@@ -280,7 +300,27 @@ default void tableDrop(byte[] attachId, List<String> path, String name, boolean
280300
}
281301

282302
/**
283-
* {@code CREATE VIEW}. Default: read-only.
303+
* {@code CREATE VIEW}, as vgi 0.44 declared it. Default: read-only.
304+
*
305+
* @param attachId the attach id
306+
* @param path the schema path
307+
* @param name the view name
308+
* @param definition the view's SQL
309+
* @param onConflict what to do when it exists
310+
* @deprecated override {@link #viewCreate(byte[], List, String, String, OnConflict, List)}
311+
* (column aliases) or the full form (comment, tags, column comments); an
312+
* override of this one is still reached for a view that uses none of them.
313+
*/
314+
@Deprecated
315+
default void viewCreate(byte[] attachId, List<String> path, String name, String definition,
316+
OnConflict onConflict) {
317+
throw readOnly("catalog_view_create");
318+
}
319+
320+
/**
321+
* {@code CREATE VIEW} with optional column aliases. Default: the 5-argument form when there
322+
* are no aliases; with aliases, refused as read-only (or, when only the 5-argument form is
323+
* overridden, as unsupported, rather than dropping them).
284324
*
285325
* @param attach the attach id
286326
* @param path the schema path
@@ -289,13 +329,48 @@ default void tableDrop(byte[] attachId, List<String> path, String name, boolean
289329
* @param conflict what to do when it exists
290330
* @param aliases optional view column aliases
291331
*/
292-
default void viewCreate(byte[] attach, List<String> path, String name, String definition, OnConflict conflict, List<String> aliases) {
293-
viewCreate(attach, path, name, definition, conflict, aliases, null, null, null);
332+
@SuppressWarnings("deprecation")
333+
default void viewCreate(byte[] attach, List<String> path, String name, String definition, OnConflict conflict,
334+
List<String> aliases) {
335+
if (aliases == null || aliases.isEmpty()) {
336+
viewCreate(attach, path, name, definition, conflict);
337+
return;
338+
}
339+
if (overrides("viewCreate", byte[].class, List.class, String.class, String.class, OnConflict.class)) {
340+
throw new UnsupportedOperationException("catalog " + name() + " does not support view column aliases:"
341+
+ " override viewCreate(attachId, path, name, definition, onConflict, aliases)");
342+
}
343+
throw readOnly("catalog_view_create");
294344
}
295345

346+
/**
347+
* {@code CREATE VIEW} with every option; the framework calls this one. Default: the
348+
* 6-argument form when there is no comment, tag or column comment; otherwise refused as
349+
* read-only (or, when a shorter form is overridden, as unsupported, rather than dropping them).
350+
*
351+
* @param attachId the attach id
352+
* @param path the schema path
353+
* @param name the view name
354+
* @param definition the view's SQL
355+
* @param onConflict what to do when it exists
356+
* @param aliases optional view column aliases
357+
* @param comment the view comment, or {@code null}
358+
* @param tags view tags, or {@code null}
359+
* @param columnComments per-output-column comments, or {@code null}
360+
*/
296361
default void viewCreate(byte[] attachId, List<String> path, String name, String definition,
297362
OnConflict onConflict, List<String> aliases, String comment,
298363
Map<String, String> tags, Map<String, String> columnComments) {
364+
if (comment == null && (tags == null || tags.isEmpty()) && (columnComments == null || columnComments.isEmpty())) {
365+
viewCreate(attachId, path, name, definition, onConflict, aliases == null ? List.of() : aliases);
366+
return;
367+
}
368+
if (overrides("viewCreate", byte[].class, List.class, String.class, String.class, OnConflict.class, List.class)
369+
|| overrides("viewCreate", byte[].class, List.class, String.class, String.class, OnConflict.class)) {
370+
throw new UnsupportedOperationException("catalog " + name() + " does not support view comments or tags:"
371+
+ " override viewCreate(attachId, path, name, definition, onConflict, aliases, comment, tags,"
372+
+ " columnComments)");
373+
}
299374
throw readOnly("catalog_view_create");
300375
}
301376

@@ -356,6 +431,15 @@ private UnsupportedOperationException readOnly(String method) {
356431
"catalog " + name() + " is read-only: " + method + " not supported");
357432
}
358433

434+
/** Whether this catalog's class overrides the named default method. */
435+
private boolean overrides(String method, Class<?>... parameters) {
436+
try {
437+
return getClass().getMethod(method, parameters).getDeclaringClass() != CatalogInterface.class;
438+
} catch (NoSuchMethodException e) {
439+
return false;
440+
}
441+
}
442+
359443
/**
360444
* Schema paths compare case-insensitively, as DuckDB identifiers do.
361445
*

0 commit comments

Comments
 (0)