Repository navigation
Commit b64980a
feat(data): host-configured default and maximum page size for list, report and analytics reads, and a per-query row budget, refused with a code naming the limit (#29796)
<!-- claude-projects-attribution: {"github_login":"os-zhuang"} -->
_Requested by **Sam** · [project
thread](https://claude.ai/code/project/chan_01GsBogozeZG5sdpAARrN7HP?thread=cmsg_01GsBogozeZG5sdpAARrN7HP1GdLsoH7z9V8sNW6YoLEMF)_
Part of #29783
This PR is the whole of that card. It says `Part of` rather than a
closing keyword because the card's claim names no `claude/issue-*`
branch and this PR's head branch cannot take that shape, so
`check:closing-target-claim` could never pass with a closing keyword.
The seat closes the card when this lands.
## What changes
A deployment can now bound how many rows one data API read returns and
walks, through three host variables. Each is a positive integer; all
three unset is today's behaviour exactly.
| Variable | Meaning | Unset |
|:--|:--|:--|
| `OS_DATA_DEFAULT_PAGE_SIZE` | the page a read gets when it names no
`limit` | the maximum when one is set, else no default |
| `OS_DATA_MAX_PAGE_SIZE` | the largest `limit` a read may name | no
maximum |
| `OS_DATA_QUERY_ROW_BUDGET` | the most rows one query may walk:
`offset` plus the rows it returns | no budget |
**The documented rule for a request over a limit is refuse, never
clamp.** A clamp answers a different page from the one asked for: a
caller paging by its own `limit` would skip the rows in between, and an
analytics answer has no `hasMore` to say a chart was cut. So:
- a `limit` over the maximum → `400 MAX_RECORDS_EXCEEDED`, the existing
standard code for a caller-side limit (the request asked for more rows
than a request may carry), naming the spelling the caller wrote (`$top`
/ `top` / `limit`) and the maximum;
- `offset + limit` over the budget → `400 QUERY_ROW_BUDGET_EXCEEDED`,
naming the budget. This is a new code because the budget bounds how deep
a query walks, not how big a page is: a page well within the maximum is
refused at a deep enough offset.
Both are refused before the driver is asked (a budget probe, below, runs
first and refuses on its answer). Each refusal writes one `warn` line:
`[data] read refused: code=… surface=list|analytics object=… rows=…
limit=…`. The line carries the object and counts, never a row value.
**A read that names no limit:**
- **List reads** (`GET /data/:object`, `POST /data/:object/query`,
grouped or flat, through `findData`) run with the default page size as
their `limit`. The answer already says when more rows exist (`hasMore`,
`total`), so this is an ordinary first page.
- **Analytics reads** (`AnalyticsService.query()` behind
`/analytics/query`, and `queryDataset()`, the dataset door reports and
dashboards read through) run with one row more than the default page. An
answer with more rows than the default is refused with
`MAX_RECORDS_EXCEEDED`, telling the caller to name a `limit` and page
with `offset`. The analytics doors judge the caller's own window; the
dataset executor's internal passes (totals, measure-scoped queries) are
not judged separately.
- **A budget with no page size** probes one row more than the budget
leaves after the `offset`. An answer over the budget is refused, and one
that fits is served whole (the probe limit is taken back off, so
`hasMore` / `total` read as an unpaged answer).
**Where it lives:**
- `@objectstack/types` `query-read-limits.ts` holds the rule:
`resolveQueryReadLimits` (strict: a value that is not a positive
integer, or a default above the maximum, throws naming the variable),
`planQueryReadWindow`, `assertQueryReadRows`, and the abstract
`QueryReadLimitError` (`limit`) with one subclass per code,
`QueryPageSizeExceededError` (`MAX_RECORDS_EXCEEDED`) and
`QueryRowBudgetExceededError` (`QUERY_ROW_BUDGET_EXCEEDED`), each
declaring its own `code` and `status 400` so
`check:error-status-conformance` pins the catalog's status.
`MAX_RECORDS_EXCEEDED` gains its first producer, so it leaves
`scripts/error-status-unpinned-baseline.json` (the shrink-only
direction).
- `metadata-protocol` `findData` plans every list read after the shape
gates and before the engine.
- `service-analytics` `query()` and `queryDataset()` plan the caller's
window at the public doors.
- Both constructors resolve the limits once, so a misconfigured variable
fails the boot instead of 500-ing every read.
- `QUERY_ROW_BUDGET_EXCEEDED` is registered in the error-code ledger
under `@objectstack/types`, with a line on why it is not
`MAX_RECORDS_EXCEEDED`; the references pages are regenerated;
`content/docs/api/data-api.mdx` documents the three variables and the
refusal rule; `content/docs/api/error-catalog.mdx` gains a
`QUERY_ROW_BUDGET_EXCEEDED` entry, names the page-size cause under
`MAX_RECORDS_EXCEEDED`, and counts 57 wire codes.
**What the budget does not count:** rows a database scans inside an
aggregate or behind an unindexed filter. The data API cannot see those,
so the budget counts the rows a query walks through its window. The
expand sub-reads, the export route and internal engine reads (backfills,
sweeps) are not data API list reads and are untouched.
## Why
The maintainer's 2026-11-03 04:26Z instruction on the cloud plan
decision, verbatim: 「同意:在代码里补两个保护再谈放宽:查询扫描行预算和强制分页;每周看板加「每个 Team
组织的数据库大小与行读量」。第 3 个月闸门前有这两组数,再决定存储包价格和数据边界的数字。」
The hosted data plane runs this data API, and on `main` there was no
general page-size cap in it or in the SQL / libSQL drivers (re-checked
at `946a173f`: `driver-odata`'s `pageSize` is only for its own upstream,
and MCP's `maxQueryLimit` only for its own tool). Cloud sets the numbers
in its own card (cloud#3510) once a pin carries this.
## Verification
- `packages/types/src/query-read-limits.test.ts` (18): configuration
(unset, all three, strict refusal of `0` / `-5` / `1.5` / `abc` / `10k`
/ `1e3`, default above maximum), no limits leave every read untouched,
refusal shape (status, code, `limit`, message, the one log line with no
row value), the default page, the analytics probe, the budget (including
the default page counting against it, and the budget-only probe), and
both codes resolving verbatim at 400 through `resolveThrownHttpError`.
- `packages/rest/src/data-query-read-limits.test.ts` (6, real
`SqlDriver` over SQLite, both list doors): defaults unchanged with
nothing set (every row, a 100,000 limit, a grouped read), the default
page with `hasMore: true` / `total: 5`, a limit within the maximum,
`MAX_RECORDS_EXCEEDED` on `$top`, `top`, `limit` and a grouped read with
the driver never called, `QUERY_ROW_BUDGET_EXCEEDED` with the driver
never called, and a budget-only read refused when it overruns and served
whole (`hasMore: false`, `total: 5`) when it fits.
-
`packages/services/service-analytics/src/__tests__/query-read-limits.test.ts`
(8): both analytics doors unchanged with nothing set (no `LIMIT`, 500
rows served), the probe reaching the query (`limit: 4`) and the SQL
(`LIMIT 4`), an answer past the default refused with the log line, a
caller's own limit run as asked, over-maximum and over-budget refused
with nothing run, and the budget-only probe.
- `data-page-window-door.test.ts` (the existing window pins) still
passes. Full local suites: `types` 860 tests, `metadata-protocol` 310
files, `service-analytics` 4,958 tests, `rest` 7,139 tests, all passing.
- `pnpm typecheck` for `types`, `metadata-protocol`, `service-analytics`
and `rest` (with its test layer); eslint on every touched file; `node
scripts/check-error-status-conformance.mjs` green; spec
`error-code-ledger.test.ts` green; `pnpm --filter @objectstack/spec
check:generated` named `content/docs/references/**` stale, regenerated
with `gen:docs`; `check-doc-anchors` resolves the new catalog links;
`check-error-status-conformance` (and its self-test) green.
- `node scripts/pm/dispatch-gates.mjs --tier`: no path-derived tier
mandate. Its Clause ② hint fires on
`packages/spec/src/api/error-code-ledger.zod.ts`: the new code widens
the published `ErrorCode` vocabulary, so a contract-tier review is owed
before the queue.
Clause-②: yes (widening)
Metadata-protocol: none
What Clause-② widens: the published error-code vocabulary. `ErrorCode`
(the union `ApiErrorSchema.code` parses against) gains one registered
member, `QUERY_ROW_BUDGET_EXCEEDED`. A page size over the maximum
answers the existing standard `MAX_RECORDS_EXCEEDED` (400), which gains
its first producer. No metadata an author writes changes, and no
existing code, status or response shape moves.
_Keep one value on each of the two lines above, each line starting with
its key.
[Human
Minimum](https://github.com/objectstack-ai/objectstack/blob/main/CONTRIBUTING.md#human-minimum)
item 5 says what they mean._
## Human Minimum
- [x] 1. Schemas start as Zod; `packages/spec` holds no runtime logic.
- [x] 2. Every name I added uses the casing for its kind.
- [x] 3. No formatter ran; `pnpm lint` passes.
- [x] 4. One issue, one branch off an up-to-date `main`, one pull
request.
- [x] 5. A change that publishes carries a changeset, and both
declaration lines are filled in here and in the changeset.
- [x] 6. `packages/spec` touched: its generated artifacts are
regenerated and committed.
- [x] 7. No hand edits to `content/docs/references/`,
`packages/*/CHANGELOG.md` or `content/docs/releases/`.
- [x] 8. Title, body and comments are in English.
- [x] The *Before Submitting* list in CONTRIBUTING.md is done.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01QUs3BtTosRanM94rtudkMk
---
_Generated by [Claude
Code](https://claude.ai/code/session_01QUs3BtTosRanM94rtudkMk)_
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 4b71a82 commit b64980a
15 files changed
Lines changed: 1081 additions & 9 deletions
File tree
- .changeset
- content/docs
- api
- references/api
- packages
- metadata-protocol/src
- rest/src
- services/service-analytics/src
- __tests__
- spec/src/api
- types/src
- scripts
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
212 | 212 | | |
213 | 213 | | |
214 | 214 | | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
215 | 241 | | |
216 | 242 | | |
217 | 243 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | | - | |
| 7 | + | |
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| |||
367 | 367 | | |
368 | 368 | | |
369 | 369 | | |
370 | | - | |
371 | | - | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
372 | 388 | | |
373 | 389 | | |
374 | 390 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
417 | 417 | | |
418 | 418 | | |
419 | 419 | | |
420 | | - | |
| 420 | + | |
421 | 421 | | |
422 | 422 | | |
423 | 423 | | |
| |||
693 | 693 | | |
694 | 694 | | |
695 | 695 | | |
| 696 | + | |
696 | 697 | | |
697 | 698 | | |
698 | 699 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
31 | | - | |
| 31 | + | |
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
| |||
311 | 311 | | |
312 | 312 | | |
313 | 313 | | |
| 314 | + | |
314 | 315 | | |
315 | 316 | | |
316 | 317 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
478 | 478 | | |
479 | 479 | | |
480 | 480 | | |
| 481 | + | |
481 | 482 | | |
482 | 483 | | |
483 | 484 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| 11 | + | |
11 | 12 | | |
12 | 13 | | |
13 | 14 | | |
| |||
1269 | 1270 | | |
1270 | 1271 | | |
1271 | 1272 | | |
| 1273 | + | |
| 1274 | + | |
| 1275 | + | |
| 1276 | + | |
1272 | 1277 | | |
1273 | 1278 | | |
1274 | 1279 | | |
| |||
6248 | 6253 | | |
6249 | 6254 | | |
6250 | 6255 | | |
| 6256 | + | |
| 6257 | + | |
| 6258 | + | |
| 6259 | + | |
| 6260 | + | |
| 6261 | + | |
6251 | 6262 | | |
| 6263 | + | |
| 6264 | + | |
| 6265 | + | |
| 6266 | + | |
| 6267 | + | |
| 6268 | + | |
| 6269 | + | |
| 6270 | + | |
| 6271 | + | |
| 6272 | + | |
6252 | 6273 | | |
6253 | 6274 | | |
6254 | 6275 | | |
| |||
6270 | 6291 | | |
6271 | 6292 | | |
6272 | 6293 | | |
| 6294 | + | |
6273 | 6295 | | |
6274 | 6296 | | |
6275 | 6297 | | |
| |||
6338 | 6360 | | |
6339 | 6361 | | |
6340 | 6362 | | |
| 6363 | + | |
| 6364 | + | |
| 6365 | + | |
| 6366 | + | |
| 6367 | + | |
| 6368 | + | |
| 6369 | + | |
| 6370 | + | |
| 6371 | + | |
| 6372 | + | |
| 6373 | + | |
| 6374 | + | |
| 6375 | + | |
| 6376 | + | |
| 6377 | + | |
6341 | 6378 | | |
6342 | 6379 | | |
6343 | 6380 | | |
| |||
6361 | 6398 | | |
6362 | 6399 | | |
6363 | 6400 | | |
| 6401 | + | |
| 6402 | + | |
| 6403 | + | |
| 6404 | + | |
| 6405 | + | |
6364 | 6406 | | |
6365 | 6407 | | |
6366 | 6408 | | |
6367 | 6409 | | |
6368 | | - | |
| 6410 | + | |
6369 | 6411 | | |
6370 | 6412 | | |
6371 | 6413 | | |
| |||
0 commit comments