Skip to content

Commit 5f30f00

Browse files
fix(spec)!: the published list-query schemas declare the page window the Data API enforces (#25300)
<!-- claude-projects-attribution: {"github_login":"os-bill"} --> _Requested by **Bill** · [project thread](https://claude.ai/code/project/chan_016iMXWSZDzqWRWvWzdNo8wg?thread=cmsg_016iMXWSZDzqWRWvWzdNo8wg6zwDuV2XvoDkqC7bXgqnpc)_ Fixes #25269 Clause-②: no (narrowing) Metadata-protocol: none ## What changes The Data API list doors refuse a page window that is not a non-negative integer (PR #25260, one normalizer in `findData`). The published request schemas for those doors still declared any number, so a client generated from them was told `-1`, `1.5` or a blank `?top=` was valid. Now they declare what the door enforces, from one declaration. - **One bound, exported by the spec** (`packages/spec/src/data/data-engine.zod.ts`): `QueryPageWindowSchema` (`z.number().int().nonnegative()`, which in zod 4 is exactly `Number.isSafeInteger(n) && n >= 0`) and `QueryPageWindowParamSchema` (that number, or a non-blank string whose `Number(...)` is one, parsed to the count). Type aliases per ADR-0122. - **The normalizer reads it** (`packages/metadata-protocol/src/protocol.ts`, `pageWindowValue`): the window is judged by `QueryPageWindowParamSchema.safeParse` instead of a restated predicate. `metadata-protocol` already imports the spec's transport tables from `@objectstack/spec/data`, so there is no layering inversion. Same accept set, same refusal text and code. - **`FindDataRequestSchema`** (the `POST /data/:object/query` body, through `QueryWithTransportSchema`): the canonical `limit` / `offset` / `top` members declare the page window; the transport spellings `$top` / `$skip` / `skip` (`QueryTransportParamsSchema`) declare it on their number arm, and the fold's string lowering judges the digits by `QueryPageWindowParamSchema` (it used to accept any finite number). `QuerySchema` itself is not touched (see Measured). - **`HttpFindQueryParamsSchema`** (the `GET /data/:object` querystring declaration): `top` / `skip` declare `QueryPageWindowParamSchema` instead of `z.coerce.number()`, which admitted `-1`, `1.5` and a blank value (as `0`). - **Generated artifacts**: reference docs, `api-surface`, `export-origins`, `declaration-map`, `json-schema.manifest`, and two new rows in `dropped-refinements.baseline.json` (the string arm's refine has no JSON Schema form; its `Number` semantics cannot be written as a pattern without approximating it, so it is declared there as the ledger asks). ## Measured: which schemas a door actually reads (base `8dfd1265f`, re-checked after merging `bda53bb0b`) The normalizer judges the top-level `limit` / `offset` of a `findData` call (`pageWindowValue`, two call sites). `findData` callers: `GET /data/:object` (querystring, no schema), `POST /data/:object/query` (validates `FindDataRequestSchema` first), the runtime dispatcher's twins (same schema on `POST`), declared `object_operation` endpoints, MCP `query`, action `callData('query')`, and server-built windows (import runner, connector pull, export loop: all integers). | Schema (card's list and found) | Read by a door the normalizer judges? | Disposition | |:--|:--|:--| | `FindDataRequestSchema` → `QueryWithTransportSchema` / `QueryTransportParamsSchema` (`limit`, `offset`, `top`, `$top`, `$skip`, `skip`) | yes, `POST /data/:object/query` and the dispatcher `POST` validate it | narrowed | | `HttpFindQueryParamsSchema` (`top`, `skip`) | describes `GET /data/:object` (no runtime parse; the SDK reads it as the declaration) | narrowed | | `EngineQueryOptionsSchema` (`data-engine.zod.ts:111/114/117`) | no: `engine.find` options; no door parses it, `ObjectQL.find` does not validate windows, and in-process callers reach it without the normalizer | out of scope | | `QuerySchema` (`limit` / `offset` / `top`) | only through the transport schema above, where the members are overridden; `QuerySchema` is also the driver contract and the recursive `expand` value, whose windows the normalizer never judges | not narrowed | | `get_record.limit` (`builtin-node-config.zod.ts:628`) | no: the executor calls the `data` service (`ObjectQL`, `IDataEngine.find` / `findOne`), not `findData` | out of scope | | `ListRecordResponseSchema.pagination` (`contract.zod.ts:445/446`) | no: a response, not a request window | out of scope | | `AuditMetaItemRequest` / `HistoryMetaItemRequest` / `ListNotificationsRequest` `limit` (`protocol.zod.ts:1823/1921/3281`) | no: metadata audit/history and notification routes, not the data list doors | out of scope | | `DataEngineFindRequestSchema` (RPC) | no reader outside the spec | out of scope | So `Metadata-protocol: none`: no authored metadata window is judged by the normalizer, and the two narrowed schemas are request schemas. No migration line is owed for an authored flow `query` window: `get_record` never reaches the normalizer, so its `limit` is not narrowed here. **Producer census** (fractional or negative windows): `examples/**` on this tree, no hit; HotCRM at `4166f2146bec`, 41 files name `get_record`, every literal `limit` is a positive integer (1, 500, 1000, and `BILLING_HANDOFF_LINE_ITEM_LIMIT = 200`), no fractional or negative window anywhere. ## One door answer moves: the code, not the verdict `POST /data/:object/query` validates its body against `FindDataRequestSchema` before `findData`, so now that the schema declares the window, a malformed one is refused there: `400 VALIDATION_FAILED` with `fields[0].field` = `query.limit` / `query.offset`, where it was `400 INVALID_REQUEST` from the normalizer. It is still a 400 before any read, and it is the same answer the door already gave `{"limit":"ten"}`. The runtime dispatcher's `POST` twin moves the same way, by reading: it asks the same schema and answers `requestSchemaRefusal`, a `VALIDATION_FAILED`. `GET /data/:object` has no body schema and is byte-unchanged. Nothing newly refused, nothing newly accepted. `packages/rest/src/data-page-window-door.test.ts`'s `POST` pin is updated to assert this. See the open question in the report. ## Tests - `packages/metadata-protocol/src/protocol.page-window-declared.test.ts` (new). The oracle is the door: each value goes through the real `findData` with a recording engine (accepted iff the engine is reached; the window it receives is the expected parse output). 13 schema rows (`FindDataRequest` × 6 spellings, `QueryWithTransport` × 3, `HttpFindQueryParams` × 2, the two exported schemas) × 17 numbers, 18 strings and 3 other values (`-0`, `MAX_SAFE_INTEGER` and one past it, `NaN`, `±Infinity`, `' 5 '`, `'1e2'`, `'0x10'`, blank, whitespace, `'-1'`, `'1.5'`, `true`, `{}`…). Every value the door refuses fails each schema's parse; every value it accepts passes with the same window. Number-only members (a JSON body's canonical keys) refuse every string, as before. 494 cases plus a non-vacuity check. - `packages/spec/src/data/query-transport.test.ts`: number and digit windows that are not a count are refused on every spelling (digits at the canonical path naming the spelling, one diagnostic); `0` stays a window. - `packages/spec/src/api/protocol.test.ts`: `HttpFindQueryParamsSchema` refuses `-1`, `1.5`, blank, whitespace, `abc`; keeps `'0'` and `' 5 '`. - Ablations (`scripts/ablation-replace.mjs`, run mode, restored by blob): `limit` member back to `QuerySchema`'s → `query-transport.test.ts` 2 failed / 70 passed; the fold's string lowering back to "any finite number" → 4 failed / 68; `HttpFindQueryParams.top` back to `z.coerce.number()` → `protocol.test.ts` 4 failed / 191. Each restored, blob equal to HEAD, `git diff HEAD` empty. - Full suites, on the tree with `origin/main` at `bda53bb0b` merged, after building the dependency closure of `@objectstack/runtime`, `@objectstack/rest` and `@objectstack/dogfood`: - `pnpm --filter @objectstack/spec test`: 709 files passed, 21545 tests passed, 1 todo (head `82fb363ac`). - `pnpm --filter @objectstack/metadata-protocol test`: 259 files passed, 3 skipped; 21410 tests passed, 19 skipped. - `pnpm --filter @objectstack/rest test`: 316 files, 6327 passed, 326 skipped. - `pnpm --filter @objectstack/runtime test`: 435 files, 6885 passed, 19 skipped (an importer of `FindDataRequestSchema`). - `typecheck` exit 0 on `@objectstack/spec`, `@objectstack/metadata-protocol` and `@objectstack/rest`. - Full dogfood (owed by the `(narrowing)` arm), as CI runs it, under the verify lock: VERDICT command-exit 0, 289 files passed and 3 skipped (292), 2491 tests passed and 18 skipped (2509). The packages were built at `685efa48` and the tree it ran on was `3fead31f3`. The commits after it add the `QueryPageWindowParsed` type alias (erased at run time), the dropped-refinements ledger header counts and regenerated JSON artifacts; none of them reaches a test at run time. ## Gates - `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` derived 114 families on this diff (the same 114 before and after merging main). All 114 ran, every final exit code is 0, recorded as `command :: exit code`, and `--ran` reconciled them: 114 run, 0 NOT-MEASURED, 0 UNRUN ("a DERIVED zero — all 114 recorded an exit code and none of them is 3"). - Five needed a second run. Four were local prerequisites: `check-plugin-teardown-shape --self-test` (the shallow clone lacked its pinned fixture commit, which was fetched), `check:skill-examples` and `check:dual-build-cjs-loads` (exit 3: packages with no `dist/` in this container, which were built), and `check:type-check-debt` (a 600 s local timeout while its closure build waited for the verify lock). One was a real finding: `check:spec-parsed-alias` (`QueryPageWindowSchema` had no `Parsed` alias, added in `3fead31f3`). - `pnpm --filter @objectstack/spec check:generated`: all 14 artifacts up to date on the final tree. `check-changeset-no-major` and `check-adr-0087-registration` exit 0 (`not-required (no-migration-prescription)`). ## Acceptance notes - `QueryTransportParamsSchema` on its own still accepts any string on `$top` / `$skip` / `skip`: its digits are judged by the fold inside `QueryWithTransportSchema` (at the canonical path, naming the spelling, one diagnostic, the shape `query-transport.test.ts` already pins). No door parses the fragment alone. - The JSON Schema for the string arm stays `string` (the refine is ledgered in `dropped-refinements.baseline.json`); the number arm publishes `integer`, `minimum 0`, `maximum 9007199254740991`. - `HttpFindQueryParamsSchema` still declares only `top` / `skip`, while the GET door also accepts `limit`, `offset`, `$top`, `$skip`. Declaring those would widen the schema, not this card. - Nested `expand` windows and `engine.find` windows are not judged by the normalizer and stay as declared; a direct `engine.find` with `offset: -1` still meets the driver (PR #25260's acceptance note 2). --- _Generated by [Claude Code](https://claude.ai/code/session_0153NL7xGym5Dpyb4sQbsArM)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 7db1697 commit 5f30f00

16 files changed

Lines changed: 415 additions & 52 deletions

File tree

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
---
2+
'@objectstack/spec': patch
3+
'@objectstack/metadata-protocol': patch
4+
---
5+
6+
fix(spec)!: the published list-query schemas declare the page window the Data API enforces — a non-negative integer — instead of any number
7+
8+
Clause-②: no (narrowing)
9+
Metadata-protocol: none
10+
11+
<!-- adr-0087: not-required (no-migration-prescription) No metadata moves: no metadata type, authorable metadata key, object definition or stored shape is added, removed, renamed or re-shaped. What narrows is two request schemas a Data API caller is validated against (`FindDataRequestSchema`, through `QueryWithTransportSchema` and `QueryTransportParamsSchema`, and `HttpFindQueryParamsSchema`), and every value they now refuse was already refused by the list doors since the page-window normalizer landed. There is nothing for `objectstack migrate meta` to rewrite; the remedy is in the request (send a whole count, or leave the parameter out). -->
12+
13+
**BREAKING** accept-set narrowing on two published request schemas. It changes what a schema-validated Data API request may carry, not what a metadata author can write, so it ships as `patch` with this banner.
14+
15+
- **Before.** `FindDataRequestSchema` (the `POST /api/v1/data/:object/query` body) declared `limit` / `offset` / `top` as any number and `$top` / `$skip` / `skip` as any number or string, and `HttpFindQueryParamsSchema` (the `GET /api/v1/data/:object` querystring) declared `top` / `skip` as `z.coerce.number()`. A client generated from them, or a request validated against them, was told `-1`, `1.5` or a blank `?top=` was valid, and the door then refused it with a `400`.
16+
- **Now.** Each of those parameters declares the page window the door enforces: a non-negative integer, or on the string-carrying spellings the digits a querystring carries it as (`'50'`). `-1`, `1.5`, `NaN`, `Infinity`, a blank or whitespace-only string, and a string that is not a count fail the parse. The bound is exported once, as `QueryPageWindowSchema` / `QueryPageWindowParamSchema` (`@objectstack/spec/data`), and the list-query normalizer in `@objectstack/metadata-protocol` now judges every window by that same schema, so the two accept sets are one declaration.
17+
- **One answer moves, and only its code.** On `POST /api/v1/data/:object/query` (and the runtime dispatcher's `POST` twin) a malformed window is now refused by the body schema, like every other malformed body: `400 VALIDATION_FAILED` with `fields[0].field` naming `query.limit` / `query.offset`, where it was `400 INVALID_REQUEST` from the normalizer. Still a `400`, still before any read. `GET /api/v1/data/:object` has no body schema and answers `400 INVALID_REQUEST` naming the parameter as before.
18+
- **Unchanged.** A well-formed window parses and pages exactly as before, `0` included (`$top=0` asks for the total alone). Windows inside `expand` and the engine and driver contracts (`QuerySchema`, `EngineQueryOptionsSchema`) are not narrowed: no door refuses those.
19+
20+
**The remedy.** Send the window as a whole count of zero or more, or leave the parameter out.

‎content/docs/references/api/protocol.mdx‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -904,9 +904,9 @@ Enable package response
904904
| **search** | `string \| { query: string; fields?: string[]; fuzzy?: boolean; operator?: Enum<'and' \| 'or'>; … }` | optional | Full-text search — the query text (canonical, ADR-0061 D1), or a structured FullTextSearch configuration |
905905
| **searchFields** | `string[]` | optional | Narrow the search to these fields (server-intersected with the allowed searchable set — can only narrow, never widen; ADR-0061 D1) |
906906
| **orderBy** | `{ field: string; order?: Enum<'asc' \| 'desc'> }[]` | optional | Sorting instructions (ORDER BY) |
907-
| **limit** | `number` | optional | Max records to return (LIMIT) |
908-
| **offset** | `number` | optional | Records to skip (OFFSET) |
909-
| **top** | `number` | optional | Alias for limit (OData compatibility) |
907+
| **limit** | `integer` | optional | Max records to return (LIMIT) |
908+
| **offset** | `integer` | optional | Records to skip (OFFSET) |
909+
| **top** | `integer` | optional | Alias for limit (OData compatibility) |
910910
| **cursor** | `never` | optional | [REMOVED] `query.cursor` was removed in @objectstack/spec 17 (ADR-0049) — no driver ever implemented keyset pagination, so the cursor was accepted and ignored and every page came back identical (a caller looping "until hasMore is false" never terminates). Delete the key; `QueryBuilder.cursor()` was removed with it. Express the keyset as an ordinary `where` predicate on your sort key — `where: { created_at: { $gt: last.created_at } }` with the matching `orderBy` — which every driver executes with canonicalised comparands. A first-class cursor, if ever built, will be a response-minted opaque token, not this caller-built record. |
911911
| **joins** | `never` | optional | [REMOVED] `query.joins` was removed in @objectstack/spec 17 (ADR-0049) — no engine or driver ever read it: a query carrying `joins` behaved exactly as if the key were absent, while its name squatted on the reserved REST parameter set. Delete the key. Related records are read through `expand` — `expand: { owner_id: { object: 'user', fields: ['name'] } }` — which the engine resolves via batch $in queries, and whose nested query selects the related record's own columns. Keep the foreign key in your own projection (`fields: ['title', 'owner_id']`): the relation is carried by that column, so projecting it away leaves expansion nothing to resolve. A dotted `fields` path is NOT a replacement — no driver ever resolved one and the ingress refuses it (`400 INVALID_FIELD`). |
912912
| **aggregations** | `{ function: Enum<'count' \| 'sum' \| 'avg' \| 'min' \| 'max' \| 'count_distinct'>; field?: string; alias: string; filter?: any }[]` | optional | Aggregation functions |
@@ -917,8 +917,8 @@ Enable package response
917917
| **expand** | `Record<string, { object: string; fields?: string[]; where?: any; search?: string \| object; … }>` | optional | Recursive relation loading map. Keys are lookup/master_detail field names; values are nested QueryAST objects that control select (`fields`) and filter (`where`, AND-merged with the batch $in), plus further expansion on the related object. The engine resolves expand via batch $in queries (driver-agnostic) with a default max depth of 3; per-parent `limit`/`offset`/`orderBy` are NOT applied on this path. |
918918
| **$filter** | `[string, string, any] \| [string, string] \| [string, object, ...object[]] \| object[] \| Record<string, any> \| any \| null` | optional | Transport spelling of `where` (OData `$filter`) |
919919
| **filters** | `[string, string, any] \| [string, string] \| [string, object, ...object[]] \| object[] \| Record<string, any> \| any \| null` | optional | Transport spelling of `where` (plural of `filter`) |
920-
| **$top** | `number \| string \| null` | optional | Transport spelling of `limit` (OData `$top`) — a number, or the digits a querystring carries it as |
921-
| **$skip** | `number \| string \| null` | optional | Transport spelling of `offset` (OData `$skip`) — a number, or the digits a querystring carries it as |
920+
| **$top** | `integer \| string \| null` | optional | Transport spelling of `limit` (OData `$top`) — a non-negative integer, or the digits a querystring carries it as |
921+
| **$skip** | `integer \| string \| null` | optional | Transport spelling of `offset` (OData `$skip`) — a non-negative integer, or the digits a querystring carries it as |
922922
| **$orderby** | `Record<string, Enum<'asc' \| 'desc'>> \| Record<string, 1 \| -1> \| { field: string; order?: Enum<'asc' \| 'desc'> }[] \| null` | optional | Transport spelling of `orderBy` (OData `$orderby`) |
923923
| **$select** | `string \| string[] \| null` | optional | Transport spelling of `fields` (OData `$select`) |
924924
| **$expand** | `string \| string[] \| Record<string, { object: string; fields?: string[]; where?: any; search?: string \| object; … }> \| null` | optional | Transport spelling of `expand` (OData `$expand`) |
@@ -929,7 +929,7 @@ Enable package response
929929
| **filter** | `[string, string, any] \| [string, string] \| [string, object, ...object[]] \| object[] \| Record<string, any> \| any \| null` | optional | Transport spelling of `where` |
930930
| **select** | `string \| string[] \| null` | optional | Transport spelling of `fields` |
931931
| **sort** | `Record<string, Enum<'asc' \| 'desc'>> \| Record<string, 1 \| -1> \| { field: string; order?: Enum<'asc' \| 'desc'> }[] \| null` | optional | Transport spelling of `orderBy` |
932-
| **skip** | `number \| string \| null` | optional | Transport spelling of `offset` |
932+
| **skip** | `integer \| string \| null` | optional | Transport spelling of `offset` — a non-negative integer, or the digits a querystring carries it as |
933933
| **populate** | `string \| string[] \| null` | optional | Transport spelling of `expand` |
934934

935935

@@ -2055,8 +2055,8 @@ The published metadata item body, opaque by ruling (1C). Shape is the item's own
20552055
| **select** | `string` | optional | Comma-separated list of fields to retrieve. |
20562056
| **sort** | `string` | optional | Sort expression (e.g. "name asc,created_at desc" or "-created_at"). |
20572057
| **orderBy** | `string` | optional | Alias for sort (OData compatibility). |
2058-
| **top** | `number` | optional | Max records to return (limit). |
2059-
| **skip** | `number` | optional | Records to skip (offset). |
2058+
| **top** | `integer \| string` | optional | Max records to return (limit) — a non-negative integer. |
2059+
| **skip** | `integer \| string` | optional | Records to skip (offset) — a non-negative integer. |
20602060
| **expand** | `string` | optional | Comma-separated list of lookup/master_detail field names to expand. Resolved to populate array and passed to the engine for batch $in expansion. |
20612061
| **search** | `string` | optional | Full-text search query. |
20622062
| **distinct** | `never` | optional | [REMOVED] `query.distinct` was removed in @objectstack/spec 17 (ADR-0049 / ADR-0078) — no driver ever rendered SELECT DISTINCT; the flag's only observable effect was MIS-WIRED: the REST list path treated a distinct query as not countable and silently degraded `total`/`hasMore` to a page-local estimate while still returning duplicate rows. Delete the key; `QueryBuilder.distinct()` was removed with it, and the count suppression is gone (`total` is truthful again). For unique values of one column use the SQL/memory drivers' `distinct(object, field)` door; for unique combinations, `groupBy`; for a deduplicated count, the `count_distinct` aggregation. |

0 commit comments

Comments
 (0)