You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 2ae8c89
Browse filesBrowse the repository at this point in the historyBrowse files
feat(objectql): the engine floor over a plugin driver reads the datasource's external.rowCap (#29876)
<!-- claude-projects-attribution: {"github_login":"os-warren"} -->
_Requested by **Warren** · [project
thread](https://claude.ai/code/project/chan_01U5RdnAAmkSQmZbqh6FtqZ8?thread=cmsg_01U5RdnAAmkSQmZbqh6FtqZ83WAcW3kdc2RHYZ2aeB5krB)_Fixes#29762
Clause-②: yes (widening) — the engine floor honours the per-datasource
cap from PR 1 (#29762 spec slice)
Metadata-protocol: none
Skills-docs: updated here —
content/docs/data-modeling/virtual-objects.mdx (whole-set views, both
ceilings) and odata-datasources.mdx
## 维护者速读
- 改了什么:插件驱动数据源的引擎地板改读 `external.rowCap`(上一个 PR #29854 已落 spec
键),每次读取都按当前定义取上限;指南新增「行数上限」与「整集视图」两节。
- 为什么改:客户的虚拟对象甘特图约 5,000 行,固定 1,000 行上限直接 422;Warren 裁定 (a)+(c),平台最大值
10,000。
- 风险与回滚:不写该键的数据源仍是 1,000,行为不变;引擎读到无法照写兑现的值(超 10,000
或非整数)时按上限/默认取值并在服务端日志警告一次;写了的数据源每页从远端读最多两倍上限行数。回滚即 revert 本 PR(spec
键留着,只是不生效)。
- 先看:`packages/objectql/src/external-driver-floor.ts`(`readCapped` 取
cap)→ `packages/objectql/src/engine.ts`(`applyExternalDriverFloor` 的
`rowCap` 回调)→ `content/docs/data-modeling/virtual-objects.mdx`(两节新文档)。
- 两分钟验证:`pnpm --filter @objectstack/objectql exec vitest run
--maxWorkers=2 src/external-driver-floor.test.ts -t 29762`,预期 7 个用例全绿。
- 需要你做的:无,席位落地。
## What changes
- **The floor reads the cap on every read.** `readCapped` calls
`resolveExternalRowCap(host.rowCap?.())`, sends `limit: cap + 1`, and
refuses past `cap` with `new ExternalRowCapExceededError(datasource,
object, cap)`. The engine's host callback reads
`this.datasourceDefs.get(name)?.external?.rowCap` at the time of the
read, because a redefinition that keeps the floor returns early in
`applyExternalDriverFloor` and never re-wraps the driver.
- **`resolveExternalRowCap`** (`external-driver-errors.ts`, not exported
from the package): a whole number from 1 to `EXTERNAL_ROW_CAP_CEILING`
is the cap; above the ceiling is held at the ceiling; anything else
(absent, 0, negative, fractional, a string) is the default. The schema
already refuses all of those at the authoring door; this is the engine's
reading of a def that reached it unparsed. When a written value is not
honoured as written, the floor says so in the server log (`logger.warn`,
meta `{ datasource, rowCapWritten, rowCapUsed }`), once per value
written to that datasource (contract-review should-fix 2 on 3f1cd43).
- **`EXTERNAL_ROW_CAP`** now reads `EXTERNAL_ROW_CAP_DEFAULT` from
`@objectstack/spec/data` (same value, 1000), and its docblock says it is
the default since #29762 rather than "not an authorable key".
- **The refusal names the cap.** `details.cap` and the message carry the
datasource's cap; below the ceiling the message adds that an
administrator may raise `external.rowCap` (at most 10,000).
- **Contracts.** `EngineDatasourceDef.external`
(`@objectstack/spec/contracts`) and objectql's `DatasourceDef.external`
declare `rowCap?: number`, the key the engine now reads.
- **Liveness.** `external.rowCap` flips from `planned` to `live`, citing
`external-driver-floor.ts#createExternalDriverFloor` and
`engine.ts#applyExternalDriverFloor` (with the connection service as
`producer`). The `external` parent's stale "5 of 7 live" note is
rewritten: 7 of 7, with the #4583 history of the two removed keys.
- **Docs.** `virtual-objects.mdx`: new sections "The row cap" (default,
ceiling, how to set it, what a higher cap costs the remote, what to do
past it) and "Views that read the whole set" (gantt, tree, calendar,
map: the server's row cap and the views' own 2,000-row ceiling in one
table, what each band of set size gets, and two habits for planners).
Every `1000` / `1001` the page pinned is restated against the cap,
including the driver-author rule "never hard-code 1001".
`odata-datasources.mdx` says the cap is the datasource's and that the
OData driver follows a raised cap unchanged.
## Tests
- `packages/objectql/src/external-driver-floor.test.ts`, describe
`[#29762] the datasource's external.rowCap`: a 2,500 cap is the limit
sent (2501) and enforced on count and a deep page; one row past it is
refused 422 with `details.cap` and the key in the message; a cap below
the default is honoured; a redefinition applies to the next read without
re-wrapping the driver (same held instance); `resolveExternalRowCap`
over undefined, in-range, the ceiling, ceiling + 1, and 0 / -5 / 1.5 /
NaN / '5000' / null; one warn for `50_000` (cap used 10,000) and one for
`'5000'` (cap used 1,000) across three reads each, none for `5000` or an
absent key; at the ceiling the message offers no higher cap.
- `packages/spec/src/data/datasource.test.ts`: the alias test now covers
`maxRowCount` beside `maxRows` and `rowLimit` (contract-review note on
#29854).
## Acceptance notes
- Measurement: issue comment 6595439927. Ruling: (a) + (c), ceiling
10,000. Spec slice: #29854 (merged). Contract review on #29854
(6598079463) listed six items for this PR; all six are here. Contract
review on this PR (6600064400) PASS at 3f1cd43 with two should-fixes,
both in 0fc5c2d: `Metadata-protocol: none` (PR 1 added the authorable
key; this diff changes nothing an author can write), both packages
`patch`; and the once-per-value warn above.
- **The 2,000-row view ceiling is unchanged.** `NON_GRID_ROW_CEILING`
(`ui/packages/core/src/utils/non-grid-row-ceiling.ts`, objectui#7210
ruling a′) was not ruled on. With a cap of 5,000, the customer's
5,000-row gantt draws its first 2,000 rows with the ceiling note instead
of failing with 422; drawing all of them needs its own ruling. The guide
says so.
- **No conformance-kit case.** The kit still sends `EXTERNAL_ROW_CAP +
1`. A driver that hard-codes 1001 would pass any small-fixture case and
come up short under a raised cap, so no case the kit can run detects it;
the guide tells driver authors to honour the limit they are sent
instead. `driver-odata` already does (`odata-driver.ts`, `$top` from
`query.limit`).
- The runtime spike test (`plugin-driver-external-object.spike.test.ts`)
pins `limit: EXTERNAL_ROW_CAP + 1` for a datasource with no key; it
stays valid as the default and is unchanged.
## Verification
Head `0ca2cf49`. Readings below are on `0fc5c2d3` (branch on
`origin/main` `2a1be29e` after #29854 merged; the review should-fixes
are the one commit on top of `3f1cd437`), except where named. `0ca2cf49`
changes only the `external.rowCap` liveness row's evidence text
(re-quoted against `capInForce`); on it, `pnpm --filter
@objectstack/spec run check:liveness` and `check:generated` both exit 0.
- `pnpm --filter @objectstack/objectql test`: Test Files 544 passed,
Tests 10777 passed (`VERDICT command-exit 0`). `pnpm --filter
@objectstack/objectql exec vitest run src/external-driver-floor.test.ts`
+ objectql typecheck: 64 passed, clean.
- On `3f1cd437` (the should-fix commit touches only objectql and the
changeset): `pnpm --filter @objectstack/spec typecheck && pnpm --filter
@objectstack/spec test`: Test Files 807 passed, Tests 24054 passed, 1
todo.
- `pnpm lint` (`eslint . --no-inline-config`): exit 0.
- `node scripts/pm/dispatch-gates.mjs --commands` derived 108 families
(the same 108 as on `3f1cd437`); `--ran` reconciliation: "108 derived
famil(ies) accounted for — 107 run, 1 NOT-MEASURED (1 DERIVED from a
recorded exit 3)".
<details><summary>Gate readings</summary>
- Every derived family exit 0 except one NOT MEASURED, exit 3
(prerequisite): `node scripts/check-plugin-teardown-shape.mjs
--self-test`, whose positive control is pinned to a commit outside this
shallow clone. It is checker health, not a verdict on this diff, and
runs in CI.
- `check:liveness` green with the row flipped to `live`;
`liveness/state-counts/datasource.md` regenerated on the new base
(`gen:liveness-counts`).
- Not run locally: full dogfood (the title carries no `!` and the
Clause-② arm is widening, not narrowing).
</details>
## Human Minimum
- [x] 1. Schemas start as Zod; `packages/spec` holds no runtime logic.
- [x] 2. Every name I added uses the casing for its kind.
- [x] 3. No formatter ran; `pnpm lint` passes.
- [x] 4. One issue, one branch off an up-to-date `main`, one pull
request.
- [x] 5. A change that publishes carries a changeset, and both
declaration lines are filled in here and in the changeset.
- [x] 6. `packages/spec` touched: its generated artifacts are
regenerated and committed.
- [x] 7. No hand edits to `content/docs/references/`,
`packages/*/CHANGELOG.md` or `content/docs/releases/`.
- [x] 8. Title, body and comments are in English. (The `维护者速读` block is
the os-dev maintainer brief, in Chinese by that file's rule.)
- [x] The *Before Submitting* list in CONTRIBUTING.md is done.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01JQwxbZdyXCxmKHk7PbwmfZ
---
_Generated by [Claude
Code](https://claude.ai/code/session_01JQwxbZdyXCxmKHk7PbwmfZ)_
<!-- signed: domain:server#2 · os-warren ·
session_01JQwxbZdyXCxmKHk7PbwmfZ -->
---------
Co-authored-by: Claude <noreply@anthropic.com>
feat(objectql): the engine's floor over a plugin driver reads the datasource's `external.rowCap`, so a whole-set view over a virtual object can match up to 10,000 rows (#29762)
7
+
8
+
Clause-②: yes (widening) — the engine floor honours the per-datasource cap from PR 1 (#29762 spec slice)
9
+
Metadata-protocol: none
10
+
11
+
A plugin-driver datasource (a non-managed datasource whose `driver` is not built in, OData included) reads at most `external.rowCap` rows from its driver for one request: the driver is sent `limit` of that cap plus one, and a read that matches more is refused with `EXTERNAL_ROW_CAP_EXCEEDED` (422). Without the key the cap stays 1,000, so no existing datasource changes.
12
+
13
+
-**Read on every request.** The floor asks the engine for the datasource's current definition on each read, so a datasource whose `external.rowCap` is edited takes the new cap on its next read. A definition that reached the engine without the spec's parse is held at `EXTERNAL_ROW_CAP_CEILING` (10,000) above it, and read as the default when it is not a whole number from 1; either way the server log says so once per value written, naming the datasource, the value and the cap in force.
14
+
-**The refusal names the cap.**`ExternalRowCapExceededError` carries the datasource's cap in its message and `details.cap`. Below the ceiling, its message adds that an administrator may raise `external.rowCap`.
15
+
-**Contract.**`EngineDatasourceDef.external` (`@objectstack/spec/contracts`) and objectql's `DatasourceDef.external` declare the optional `rowCap` the engine now reads. `EXTERNAL_ROW_CAP` (`@objectstack/objectql`) is now read from `EXTERNAL_ROW_CAP_DEFAULT` in `@objectstack/spec/data`, with the same value.
16
+
-**Docs.** The virtual-objects guide has a section on the row cap (what a higher cap costs the remote) and one on views that read the whole set (gantt, tree, calendar, map): the server's row cap and the views' own 2,000-row ceiling, and what a planner does past each. The OData page says the cap is the datasource's.
17
+
18
+
To raise the cap on a plugin-driver datasource, set `external: { rowCap: 5000 }` (at most 10,000). Each list page reads every match up to the cap from the remote twice, for the rows and for the total, so raise it only as far as the widest read needs.
translated, so the service never returns fewer rows than match. This is also why a
280
281
service that compares text case-insensitively still answers correctly.
281
282
-**`$orderby`** carries the sort, then the entity key, so pages partition the result.
282
-
-**Paging** follows `@odata.nextLink` until 1001 rows are read. A link to another
283
-
origin is refused, so the credential never goes to a host you did not configure.
283
+
-**Paging** follows `@odata.nextLink` until the cap plus one row is read (`$top=1001`
284
+
above, at the default cap). A link to another origin is refused, so the credential
285
+
never goes to a host you did not configure.
284
286
-**The page size and the total** are the engine's, applied to the rows read. A count
285
287
is computed the same way. The engine aggregates in memory.
286
288
287
-
**The 1000-row cap applies.** A read that matches more than 1000 rows answers
289
+
**The row cap applies.** A read that matches more rows than the cap (1000 unless the
290
+
datasource sets `external.rowCap`, at most 10,000) answers
288
291
`422 EXTERNAL_ROW_CAP_EXCEEDED` rather than a partial set
289
-
([the cap](/docs/data-modeling/virtual-objects#the-engine-re-applies-every-query)). Every
292
+
([the row cap](/docs/data-modeling/virtual-objects#the-row-cap)). Every
290
293
`$filter` the driver can push down makes the service return only matching rows, so the cap
291
294
counts matching rows, not the whole entity set. Give list views a filter the service
292
-
can run. For an entity set users must browse whole, mirror it into a real object
293
-
instead.
295
+
can run. The driver honours whatever `$top` it is sent, so a raised cap needs no
296
+
driver change. For an entity set users must browse whole past the ceiling, mirror it
297
+
into a real object instead.
294
298
295
299
## 5. Writes
296
300
@@ -382,7 +386,7 @@ The driver then speaks the public v2 protocol. Everything above holds, spelled t
382
386
| A read the service refuses for its own sake (`400`, `404`, `422`) |`400 INVALID_QUERY`, with a message naming the entity set, the status and the service's own text |
383
387
| A write the service refuses |`422 EXTERNAL_WRITE_REFUSED`, with the service's message |
384
388
| A `5xx`, a timeout, an unreachable service, a `401` after the retry, a `403` or a `429`|`503 ERR_DATASOURCE_UNAVAILABLE`; the cause goes to the server log |
385
-
| A read that matches more than 1000 rows|`422 EXTERNAL_ROW_CAP_EXCEEDED`|
389
+
| A read that matches more rows than the cap (1000, or `external.rowCap`)|`422 EXTERNAL_ROW_CAP_EXCEEDED`|
386
390
387
391
An outage is never answered as an empty list. A read refusal's message is withheld, and
388
392
goes to the server log, when the read carries a predicate the caller did not write, such
@@ -397,7 +401,8 @@ as a sharing scope. The full mapping is in
397
401
its own; an imported foreign key becomes a `lookup` onto it.
398
402
-**Actions and functions** of the service. `execute` is refused.
399
403
-**Transactions and `$batch`.** Each write is one request.
400
-
-**More than 1000 matching rows** in one read (the cap above).
404
+
-**More matching rows than the cap** in one read (the cap above: 1000, or the
405
+
datasource's `external.rowCap`, at most 10,000).
401
406
-**A read scoped by a parent's access** (an object `controlled_by_parent`): it is
402
407
refused, because `$filter` has no form of that join.
rowCap: 5000, // this remote's widest read matches about 5,000 rows
495
+
},
496
+
```
497
+
498
+
-**What a higher cap costs.** The engine's own share is small: decoding and re-applying
499
+
the query over 5,000 rows takes about 15 ms and 1 MiB, and over 10,000 about 35 ms and
500
+
2 MiB. The remote pays more. Every list page reads every matching row up to the cap,
501
+
twice (the rows, then the total), so a cap of 5,000 can mean 10,000 rows from the
502
+
remote for one page of 50. Raise it only as far as the widest read needs, and push
503
+
`where` down so the cap counts matching rows, not the resource.
504
+
-**A list cannot page past the cap.** When more rows match than the cap allows, every
505
+
page of that read is refused, the first page and the one that starts at the cap
506
+
included.
507
+
-**Past the cap**, narrow the filter so fewer rows match, raise `external.rowCap` up to
508
+
10,000, or, for data that size, mirror it into a real object with a
509
+
[scheduled pull](/docs/automation/jobs#pulling-a-mapping) and read that. The refusal's
510
+
message names the datasource and its cap.
511
+
512
+
### Views that read the whole set
513
+
514
+
A grid reads one page at a time. A gantt, a tree, a calendar and a map do not: each needs
515
+
the whole filtered set at once, because a gantt cannot draw its date range or roll up its
516
+
groups from one page, and a map cannot fit its camera to markers it never received. So
517
+
each of the four sends one read for every matching row, and two ceilings apply to it, in
518
+
this order:
519
+
520
+
| Ceiling | Where | Number | Past it |
521
+
| :--- | :--- | :--- | :--- |
522
+
| The row cap | the server, on a plugin-driver datasource | 1000, or `external.rowCap` up to 10,000 | the whole view fails with `422 EXTERNAL_ROW_CAP_EXCEEDED`|
523
+
| The view's row ceiling | the browser, on every datasource | 2,000 rows, not configurable | the view draws the first 2,000 rows in its sort order and says so: "Showing the first 2,000 of 5,000 records. Narrow the filter." |
524
+
525
+
Each of the four views asks for 2,001 rows, the ceiling plus one row to detect the cut.
526
+
On a plugin-driver datasource at the default cap, any filtered set past 1000 rows fails
527
+
the whole view, and raising the cap is what turns the failure into a drawn view:
528
+
529
+
-**Up to 1000 matching rows:** the view draws everything with the default cap.
530
+
-**1001 to 2,000:** set `external.rowCap` to at least the largest set the view's filter
531
+
admits, and the view draws everything.
532
+
-**More than 2,000:** a cap of at least that size lets the view draw its first 2,000 rows
533
+
with the note instead of failing. It never draws all of them. A cut gantt or tree is
534
+
missing whole branches, so a planner should not work from one. Give the view a filter
535
+
that keeps the set under 2,000 rows: a date window, a project, a status. Or split the
536
+
view, one per project or per line.
537
+
538
+
Two habits keep a whole-set view working as the data grows:
539
+
540
+
-**Put the narrowing in the view's `filter`**, not only in a quick filter a planner can
541
+
clear. Clearing a quick filter, or widening a date range, is how a view that worked in
542
+
testing reaches the cap in production.
543
+
-**Size the cap from the widest filter the view allows**, not from today's row count, and
544
+
keep it at or below 10,000. A set that will outgrow 10,000 matching rows belongs in a
545
+
real object fed by a [scheduled pull](/docs/automation/jobs#pulling-a-mapping).
546
+
483
547
### Credentials
484
548
485
549
⛔ Never put a credential in the stack — not in `config`, not in a hook body, not in an
@@ -556,7 +620,7 @@ Measured on the reference test, request by request, with the engine's floor
556
620
| The platform does | Your driver is called with |
557
621
| :--- | :--- |
558
622
| Boots |`connect()` once. No `syncSchema`, no `introspectSchema` (`checkOnBoot: false`). |
559
-
| Serves a list (`GET /data/berth_slot` with the view's filter, sort and page size) |`find(object, { where, orderBy, limit: 1001 })`, then `find(object, { where, limit: 1001 })` for the total. The page size is applied by the engine. |
623
+
| Serves a list (`GET /data/berth_slot` with the view's filter, sort and page size) |`find(object, { where, orderBy, limit: 1001 })`, then `find(object, { where, limit: 1001 })` for the total. The page size is applied by the engine. `1001` is the default [row cap](#the-row-cap) plus one; a datasource's `external.rowCap` moves it. |
560
624
| Serves a record (`GET /data/berth_slot/:id`) |`find(object, { where: { id }, limit: 1001 })`|
561
625
| Writes (`PATCH /data/berth_slot/:id`) | several by-id reads as `find` (the pre-image for hooks and checks), then `update(object, id, patch)` with the extra keys above |
562
626
| Refuses a caller with no grant on a list read | nothing |
0 commit comments