You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit a727ba2
Browse filesBrowse the repository at this point in the historyBrowse files
perf(service-messaging): an idle dispatcher tick reads the database twice, not 18 and 10 times (#28403)
<!-- claude-projects-attribution: {"github_login":"os-zhuang"} -->
_Requested by **Sam** · [project
thread](https://claude.ai/code/project/chan_01LjZwNRFiG2fkp8SNf6rHNx?thread=cmsg_01LjZwNRFiG2fkp8SNf6rHNxECLC1evcjsCNG7ex4pEc8Z)_Fixes#28383
Clause-②: no
Metadata-protocol: none
## 维护者速读
- 改了什么:通知调度器和 HTTP 调度器在 outbox 空闲时,每次轮询只读两次库(回收过期 claim
前先读一条、再用一条跨所有分区的"有没有到期行"读决定要不要逐分区探测),不再是 18 条和 10 条。
- 为什么改:cloud 实测一个空闲租户内核每分钟读库 56 次,全部是空转;这些读计入 Turso 的行读配额。
- 风险与回滚:有活时仍走原来的逐分区 claim/digest 路径,只多一次读;回收 UPDATE
的谓词不变,只在读到过期行时才执行。outbox 接口新增的 `hasDueWork` 是可选方法,自定义 outbox
不实现就保持旧行为。回滚即 revert 本 PR。
- 先看:`packages/services/service-messaging/src/dispatcher.ts`(`runTick`
里的门)、`src/sql-outbox.ts`(`reapExpired` 与
`hasDueWork`)、`src/dispatcher-idle-cost.integration.test.ts`(语句数钉子)。
- 两分钟验证:`pnpm --filter @objectstack/service-messaging exec vitest run
--maxWorkers=2 src/dispatcher-idle-cost.integration.test.ts
src/http-dispatcher-idle-cost.integration.test.ts`,预期 13 个用例全绿,空闲 tick
钉在 2 条 SQL。
- 需要你做的:无,席位落地。
## What changes
An idle tick of each delivery dispatcher now costs **two SQL
statements**, whatever `partitionCount` is. Measured at the client (knex
`query` events) on a harness with one global `afterUpdate` hook, the
kernel shape cloud measured:
| Dispatcher | main `ac09e7b85` | this PR |
|---|---|---|
| `NotificationDispatcher` | 18 | **2** |
| `HttpDispatcher` | 10 | **2** |
The card's bar is ≤ 6 per dispatcher.
- **Candidate 1: the reap writes only when there is something to reap.**
`reapExpired` in `sql-outbox.ts` and `sql-http-outbox.ts` first reads
one expired `in_flight` id (`fields: ['id'], limit: 1`). If it finds
none, it returns. If it finds one, it runs the predicate `UPDATE`, which
is unchanged, so every claim expired by then still goes back to
`pending`, not only the one the read saw. On a kernel the old
unconditional `UPDATE` was two statements (the engine reads the matched
rows for the per-row `afterUpdate` dispatch, then writes); it is now one
read.
- **Candidates 2 and 3: one due-work read gates every partition probe,
digest probes included.** `INotificationOutbox` and `IHttpOutbox` gain
an optional `hasDueWork({ now })`. It asks, in one `SELECT id … LIMIT 1`
across every partition and organization, whether any `pending` row is
due. The predicate is the claim's own with the partition term dropped,
and for notifications with the `digest_key` arm dropped, so it covers
immediate and batched rows alike. `runTick` asks it after the reap. If
nothing is due, the tick ends there: no partition lock is taken and no
claim or digest probe runs. If something is due, the tick runs the full
per-partition pass exactly as before, for one extra read. Both built-in
stores (SQL and memory) implement it. A store without it keeps probing
every partition. `DueWorkOptions` and `HttpDueWorkOptions` are exported.
- `partitionCount` and the stored `partition_key` are unchanged.
## Why
An idle hosted tenant kernel read its database 56 times a minute with
nothing queued: 2 ticks × (18 + 10) at the framework's 30 s idle
ceiling, plus a 140-statement backoff ramp in the first minute. Those
reads count against Turso's rows-read quota, and cloud#2177 once had a
staging account suspended for reads. With this PR an idle minute costs 8
statements at the 30 s ceiling.
## Tests
- `dispatcher-idle-cost.integration.test.ts` and
`http-dispatcher-idle-cost.integration.test.ts` now count SQL statements
on the delivery table at the client as well as engine calls, with a
global no-op `afterUpdate` hook registered so a predicate `UPDATE` is
the read-then-write pair it is on a kernel. On this harness, main
measures 18 and 10 statements per tick, the same figures cloud measured.
- The idle pin: `{ update: 0, find: 2 × TICKS }` engine calls, and
exactly `2 × TICKS` SQL statements, all `SELECT`. This replaces the old
`≤ TICKS × (1 + 2 × partitionCount)` and `≤ TICKS × (1 +
partitionCount)` ceilings.
- New: a row deferred into the future (notification `notBefore`; an HTTP
retry acked with `nextRetryAt`) keeps the tick at two reads and goes out
on the first tick after it comes due.
- New (notification): a due digest-only outbox is found by the due-work
read and collapsed on that tick.
- The crash-recovery legs now also assert the reap's `UPDATE` ran
exactly once when a claim had expired. This guards against a reap that
never writes.
- The legacy-outbox legs (no `reap`, and so no `hasDueWork`) are
unchanged and still see `2 × partitionCount` and `partitionCount` claims
a tick.
- Pins that assumed the old shape, reconciled:
- `sql-outbox-audit-columns.test.ts`: its fake engine's scripted `find`
queue now opens with a reap read that finds a stale row, so the reap
`UPDATE` payload it checks for `updated_at` is still produced.
- `system-context.pin.test.ts`: the population counts include the reap
reads and one `hasDueWork` call, so the new reads are also held to the
explicit system opt-in.
- `http-dispatcher-idle-backoff.test.ts`: it timestamped a tick at
`claim()`, which an idle tick no longer calls. It now timestamps at
`reap()`, which every tick calls once first, as the notification side's
file already did.
## Acceptance notes
- **Deviation from the dispatch's mechanism assumption (candidate 2).**
The dispatch described "one probe across the partitions this process
holds". The due-work read here spans all partitions and takes no lock,
and it gates the per-partition pass instead of replacing it. Lock state
is unknown until each `acquire`, and on the common single-node or memory
lock every partition is held anyway. A read that moves nothing needs no
lock, as with the reap. Keeping the per-partition claims under their
locks leaves the busy path, the batch size per partition and the
digest-under-lock rule exactly as they were. On a multi-node cluster, a
node whose partitions are all held elsewhere pays the same lock attempts
as before whenever something is due anywhere, and two reads when nothing
is.
- **Candidate 3 is folded into the one read**, which is the card's first
option ("fold into the claim probe"). A busy tick still runs both probes
in every held partition. Narrowing the busy path is not what this card
measured.
- **Out of scope, as dispatched:**
- Candidate 4: wake at the next due time instead of a blind ceiling
(`packages/core/src/dispatch-loop.ts`).
- Candidate 5: a persisted watermark for the objectql lifecycle sweep's
first run 60 s after every boot.
- The backoff ramp's first-minute price now falls with the per-tick
price (5 ramp ticks × 4 statements instead of × 28). cloud re-measures
under cloud#3413.
- The new reads add `ORDER BY id` (the engine's default when a `limit`
is set) over a `limit 1`. That is one row from the primary key ordering,
not a scan cost this card measured.
## Verification
All readings at head `95f4da30a` unless stated.
<details><summary>Statement counts (before / after)</summary>
A throwaway measurement on main `ac09e7b85` (not committed), with and
without a global `afterUpdate` hook, one tick each over an empty outbox:
- no hook: notification 17, HTTP 9; with the hook: notification 18, HTTP
10. The hook shape is `select * … where status = ? and claimed_at < ?`
then the `update`, followed by one or two `select id … partition_key =
?` per partition.
- After the change, both shapes: notification 2, HTTP 2. Each is `select
id … where status = ? and claimed_at < ? … limit ?`, then `select id …
where status = ? and ((next_… is null) or (next_… <= ?)) … limit ?`.
</details>
<details><summary>Package tests and typecheck</summary>
- `pnpm --filter @objectstack/service-messaging test`: Test Files 62
passed (62), Tests 759 passed (759). Lock VERDICT command-exit 0 at
`18e1c546b`. The later commit only adds the changeset and one `as
HttpClaimCredential` cast in a test.
- `pnpm --filter @objectstack/service-messaging typecheck`: exit 0 at
`95f4da30a`. At `18e1c546b` the first run caught one TS2345 in the new
HTTP retry leg (an `HttpDelivery` passed as the claim credential), fixed
with the cast the dispatcher itself uses.
- The two idle-cost files at `95f4da30a`: 13 passed (13).
</details>
<details><summary>Ablation (scripts/ablation-replace.mjs, restored, git
diff HEAD empty, marker grep 0)</summary>
Tests import `src/` relatively, so no dist rebuild sits between mutation
and run.
- A: the `hasDueWork` gate in `NotificationDispatcher.runTick` replaced
by a marker. Notification idle-cost file: 2 failed / 5 passed (the idle
pin: `find: 170` vs `20`, 17 statements vs 2; the deferred-row pin).
- B: `SqlHttpOutbox.reapExpired`'s early return replaced by a marker.
HTTP idle-cost file: 2 failed / 4 passed (the idle pin: `update: 10` vs
`0`; the retry pin: 4 statements vs 2).
</details>
<details><summary>Dispatch gates and lint</summary>
- `node scripts/pm/dispatch-gates.mjs --commands --repo
objectstack-ai/objectstack`: 62 families derived from 15 paths vs merge
base `ac09e7b85` (473 changed lines, under the 3000 threshold). Every
command was run with stdin closed, and each exit code was captured to
disk before any pipe.
- `--ran` reconciliation: 62 derived, 62 run, 0 NOT-MEASURED, 0 UNRUN;
it exits 0.
- Every derived command exited 0. Three first exited 3 (PREREQUISITE NOT
MET) and were re-run after their prerequisite:
- `pnpm check:dual-build-cjs-loads` and `pnpm check:i18n` after a full
`pnpm build` (lock VERDICT command-exit 0).
- `node scripts/check-plugin-teardown-shape.mjs --self-test` after `git
fetch --unshallow origin main` (48 cases pass).
- `pnpm lint` (repo-wide `eslint . --no-inline-config`): lock VERDICT
command-exit 0.
</details>
## Human Minimum
- [x] 1. Schemas start as Zod; `packages/spec` holds no runtime logic.
(Not touched.)
- [x] 2. Every name I added uses the casing for its kind.
- [x] 3. No formatter ran; `pnpm lint` passes. (See Verification.)
- [x] 4. One issue, one branch off an up-to-date `main`, one pull
request.
- [x] 5. A change that publishes carries a changeset, and both
declaration lines are filled in here and in the changeset.
- [x] 6. `packages/spec` touched: n/a, not touched.
- [x] 7. No hand edits to `content/docs/references/`,
`packages/*/CHANGELOG.md` or `content/docs/releases/`.
- [x] 8. Title, body and comments are in English.
- [x] The *Before Submitting* list in CONTRIBUTING.md is done.
---
_Generated by [Claude
Code](https://claude.ai/code/session_01QUs3Bvkfh86Jgvi7gewhBD)_
---------
Co-authored-by: Claude <noreply@anthropic.com>
perf(service-messaging): an idle notification or HTTP dispatcher tick reads the database twice, instead of 18 and 10 times
6
+
7
+
Clause-②: yes (widening)
8
+
Metadata-protocol: none
9
+
10
+
A kernel that mounts messaging runs two delivery dispatchers, and each one ticked against an empty outbox at a fixed price: one claim probe per partition (and, for notifications, one digest probe per partition), plus a visibility-timeout reap that issued its predicate `UPDATE` whether or not any claim had expired. With the default 8 partitions that was 18 statements per notification tick and 10 per HTTP tick, measured at the client on a kernel. A kernel's global `afterUpdate` hook makes the engine read the matched rows before it writes, so the reap cost two statements. At the framework's 30-second idle ceiling an idle tenant kernel read its database 56 times a minute.
11
+
12
+
An idle tick is now two reads per dispatcher, whatever `partitionCount` is:
13
+
14
+
-**The reap reads before it writes.** It reads one expired `in_flight` id. Only a hit runs the `UPDATE`, whose predicate is unchanged, so it still returns every claim expired by then to `pending`.
15
+
-**One due-work read gates the partition pass.** Before taking any partition lock, the dispatcher asks the outbox once, across every partition and organization, whether any `pending` row is due. For notifications this covers immediate and batched (digest) rows. If none is due, the tick ends: no lock is taken and no claim or digest probe runs. If one is due, the tick runs the full per-partition pass exactly as before, at the price of one extra read.
16
+
17
+
`INotificationOutbox` and `IHttpOutbox` gain an optional `hasDueWork({ now })`, with `DueWorkOptions` and `HttpDueWorkOptions` exported for it. Both built-in stores, SQL and memory, implement it. A custom outbox that does not implement it keeps the per-partition probes it had. `partitionCount` and the stored `partition_key` are unchanged.
18
+
19
+
The public surface widens, and that is all it does: the exported `INotificationOutbox` and `IHttpOutbox` interfaces gain one optional member each (`hasDueWork`), and the package exports two new option types (`DueWorkOptions`, `HttpDueWorkOptions`). No `packages/spec` path changes, and an existing outbox implementation compiles and behaves as before.
0 commit comments