Repository navigation
Commit 8b117c0
![objectstack-fleet[bot]](https://avatars.githubusercontent.com/u/254823548?v=4&size=40){kind=avatar}
fix(service-queue): reclaim a running job whose lease ran out, and renew a live handler's lease (#30277)
<!-- claude-projects-attribution: {"github_login":"os-bill"} -->
_Requested by **Bill** · [project
thread](https://claude.ai/code/project/chan_016iMXWSZDzqWRWvWzdNo8wg?thread=cmsg_016iMXWSZDzqWRWvWzdNo8wgC5DLD798CXXfMJSTLB6VJi)_
domain:server#5 · os-bill · session_01Utp6CYVV6fcAQrNTDQRvPb
## 维护者速读
-
改了什么：数据库队列在处理器还没跑完时进程挂了（崩溃、OOM、重新部署），那条任务现在会在租约到期后被下一次轮询收回：记一次失败尝试，按退避重排，次数用尽进死信；活着的慢处理器会定期续租，不会被别的节点抢走重跑。
- 为什么改：`leaseMs` 文档写着"到期后可由其他 worker 回收"，但认领只读 `pending`，挂掉的任务永远停在
`running`，既不重试也不进死信，运维看不到。
- 风险与回滚：多节点（未开 soleWriter）时每个租约窗口（默认 30 秒）多一条带索引的读，空闲一小时从 125 条变 246
条；单写者（soleWriter，云端租户内核）只在开机多读一次，空闲一小时不再多读。处理器若同步阻塞事件循环超过大半个租约，会被当成死掉而重跑。回滚即撤本
PR，无数据迁移。
- 先看：`packages/services/service-queue/src/db-queue-adapter.ts` 的
`reclaimRow`（CAS 谓词）、`renewLease`（续租）、`reclaimExpired`（读的节流界）与
`nextDueAt`（并入最早活租约）。
- 两分钟验证：`pnpm --filter @objectstack/service-queue exec vitest run
src/db-queue-idle-backoff.test.ts -t 30228`，预期 9 条全绿；其中"CRASHED-WORKER
FIXTURE"一条在修复前的代码上是红的。
- 需要你做的：无，席位落地。
Fixes #30228
Clause-②: no — restores the documented `leaseMs` reclaim; no accept set
or export widens
Metadata-protocol: none
## What changes
The card's acceptance, quoted:
> 1. A `running` row whose `locked_until` has passed is claimable again
(or moved to the retry path with its attempt counted), within one idle
tick of the lease running out.
> 2. A live worker that is still inside its lease is never
double-claimed.
> 3. Pins: a crashed-worker fixture (row left `running` with an expired
lease) is reclaimed; a row inside its lease is not. The `nextDueAt` read
#29919 adds (PR #30225) also counts the earliest lease expiry, so a
sole-writer loop wakes for it.
**Reclaim (item 1).** `pollOnce()` now starts with `reclaimExpired()`:
one read of the table's `status = 'running'` rows ordered by
`locked_until`, every queue's (reclaiming runs nothing, so a row of a
queue this node does not subscribe to just goes back to `pending` for
whoever does; the predicate is the claim read's plain `status` equality,
served by the declared `status` index, with no operator a stand-in
engine might not answer), and each row whose lease has passed (or that
has no lease at all) goes to the retry path. Of the card's two options
this takes the second, retry path with the attempt counted, because
`attempts` is written only after a handler settles: a worker that died
left the row one attempt short, so reclaiming without counting would
retry forever a message that kills its worker every time (an OOM on a
large payload). The reclaim writes that attempt down and takes the
branch a thrown handler takes: `pending` on its backoff, or `dlq` once
`max_attempts` is spent, with `last_error` naming the dead worker. A
reclaimed row counts in the tick's result, so the loop snaps back to
`pollIntervalMs` and claims it on its backoff rather than after the idle
ceiling.
**The reclaim write is a compare-and-set (item 2).** Its predicate
re-states the expiry: `id`, `status: 'running'`, the `locked_by` it
read, and `locked_until` null or at most now. A holder that renewed
between the read and the write matches nothing and keeps its run. It is
a `multi: true` write on purpose: the by-id path binds only the primary
key and drops the rest of the predicate (the
`outbox-dispatcher-scope.ts` precedent). Two workers reclaiming the same
row write the same values from the same read and neither runs it, so
that race costs nothing. `sys_job_queue` carries no tenant column
(ADR-0131 D7), so the predicate writes need no tenant-audit warrant.
**A live handler renews its lease (item 2).** This is the risk the
reclaim itself introduces. Before, nothing reclaimed, so a handler
slower than `leaseMs` was safe; with a reclaim and no renewal it would
be reclaimed mid-run and run twice. `dispatch()` now moves its row's
`locked_until` a full window ahead every third of `leaseMs` while the
handler runs: a `multi: true` compare-and-set on `locked_by = workerId`,
logged when it matches nothing (the lease was lost underneath). A lease
therefore passes only when its holder stopped renewing: the process
died, or its event loop was blocked for most of a window. That is how a
dead worker is told apart from a slow one, and the `leaseMs` doc now
says the one thing an author must know: keep synchronous stretches well
under the lease.
**The read is bounded, not per tick.** The read also returns the
earliest lease still running. No lease can run out before the earlier of
that lease and `now + leaseMs` (every running row was in the read, a row
claimed after it holds at least `leaseMs` from its claim, and renewal
only moves a lease later), so the next read waits until then. A full
page means more may be hidden, so the next tick reads again. Cost
without `soleWriter`: a busy 1 s poll pays one reclaim read per lease
window; an idle loop at the 30 s ceiling pays one per tick, and an idle
hour at the defaults goes from 125 to 246 reads (pinned).
**A sole writer reads once per boot.** With `soleWriter` no other
process claims rows, so the window term is dropped. The first read after
`start` finds what a crashed previous process left behind; after it, the
only leases that can run out are the live ones it saw and this process's
own claims whose settling write (completed, retry, dead letter, release)
failed and left the row `running`. The adapter keeps those unsettled
claims' leases (renewal moves them, a settling write drops them, the
next read replaces them all), reads again only when one runs out, and
folds them into `nextDueAt`. An idle sole writer's hour carries one
reclaim read, the boot's, against 120 for the same hour without the flag
(pinned); its fresh first minute goes from 2 to 3 reads.
**`nextDueAt` counts the earliest live lease (item 3).** Built on
#30225's `DbQueueAdapter.nextDueAt` after it merged (771b2db). The
reclaim read already returns the earliest lease still running; the
adapter keeps it, and `nextDueAt` answers the earliest of that, a sole
writer's unsettled claims and the earliest deferred `scheduled_for`, at
no read of its own. That read ran on the empty tick the answer follows,
or the bound it set has not passed. A `soleWriter` loop, which otherwise
sleeps to its ceiling, now wakes when a row its own previous process
left `running` across a restart runs out of lease. The window term (`now
+ leaseMs`) is deliberately not in the answer: it only bounds the next
read, and folding it in would wake a sole-writer loop every lease window
for nothing.
**Docs.** Read sentence by sentence, none made false by this PR, left
unchanged: `content/docs/plugins/packages.mdx:287` ("multi-node claim
via a lease on the `db` adapter", note 6648413565) is now true as
written; `content/docs/automation/webhooks.mdx`, the
`dispatchSoleWriter` / `db.soleWriter` paragraph and the idle-cost
sentences, describe the HTTP and notification dispatchers' reads and the
flag's meaning, both still exact (the queue's sole-writer loop still
sleeps to its next due time or the ceiling; a lease is one more due
time); `content/docs/kernel/runtime-services/audit-service.mdx:71`'s
`locked_until` is the account lock's field on an auth audit event, not
the queue's; the package README's `leaseMs` line is now true.
## Tests
`packages/services/service-queue/src/db-queue-idle-backoff.test.ts`, new
`#30228` block, on that file's counting engine (its `update` already
opens with `assertEngineUpdateDispatch`; it now applies a `multi`
dispatch to the rows its `where` matches and reports the count):
- **CRASHED-WORKER FIXTURE**: a row written `running`, `locked_by:
'worker_dead'`, lease 1 ms past: one tick returns it to `pending`,
`attempts: 1`, `scheduled_for` = now + 1 s, `last_error` naming the
worker; on its backoff it runs as attempt 2 and completes.
- A lease that runs out while the loop is deep in its idle backoff
(another node's crash, nothing tells this process) is reclaimed within
one idle interval of expiry and run.
- The lost run counts: a fixture at `attempts: 2` of 3 goes to `dlq`,
and `listFailed` returns it.
- A row inside its lease is not touched (no write at all).
- CAS: a renewal landing between the reclaim read and its write leaves
the row `running` under its holder.
- A live handler three leases long, with a second adapter polling the
same table: one run, `completed`, `attempts: 1`.
- In #29919's block: a sole-writer adapter booting over a row its
previous process left `running`, lease 40 s from expiry, ceiling 5 min:
the row runs at exactly 41 s (reclaimed at the lease's expiry, then its
1 s backoff).
- A sole writer's idle hour, with one job run at 10 min, makes exactly 1
reclaim read; the same hour without `soleWriter` makes 120 (the engine
counts the reclaim read on its own).
- A sole writer whose `completed` write fails once (the row stays
`running` under itself) reclaims it when its 3 s lease runs out and runs
it again: 2 runs, `completed`, `attempts: 2`, 2 reclaim reads (boot,
then the expired lease).
Moved pins in the same file, each with its reason in the comment: the
idle hour (125 → 125 + 121), the flat-poll negative control (3602 → 3602
+ 121), the per-tick cost (n claim reads + 1 reclaim read, then n on a
tick inside the same window), the sole-writer fresh minute (2 → 3) and
its ramping control (6 → 6 + 2).
<details><summary>Crashed-worker fixture, before / after</summary>
The six reclaim legs, run against the pre-fix adapter before the
`origin/main` merge (`db-queue-adapter.ts` restored from the branch base
`6f095b9c`, blob `67be673a`, then restored to HEAD and proved
byte-identical). The seventh leg (sole-writer restart) needs #30225's
loop; its before-reading is ablation A4 below.
```
before × CRASHED-WORKER FIXTURE: an expired lease goes back to the retry path … AssertionError: expected +0 to be 1
× a lease that runs out while the loop is deep in its idle backoff … row still { status: 'running', attempts: 0 }
× the lost run counts: … dead-lettered, not retried forever AssertionError: expected +0 to be 1
✓ a row INSIDE its lease is never reclaimed (trivially: nothing reclaimed anything)
✓ the reclaim is a compare-and-set … (trivially)
✓ a LIVE handler three leases long is never double-claimed … (trivially)
after ✓ all of them, and the whole package: Test Files 5 passed (5), Tests 91 passed (91)
```
</details>
<details><summary>Ablations (each through scripts/ablation-replace.mjs,
restored to HEAD blob, git diff HEAD empty)</summary>
| Leg | Mutation | Expected | Observed |
|---|---|---|---|
| A1 | `renewLease` replaced by a no-op | live handler reclaimed and run
twice | red: `expected [ 'a', 'b' ] to deeply equal [ 'a' ]` |
| A2 | the expired-lease term deleted from the reclaim predicate | CAS
leg red | red: `expected 1 to be +0` (a renewed lease was taken back) |
| A3 | the `now + leaseMs` window bound dropped | a lease another node
takes after the read is never re-read | red: idle-reclaim leg, plus both
cost pins (measured before the merge: 3602 and 125 reads against the
then pins 3722 and 245) |
| A4 | the live lease dropped from `nextDueAt` | sole-writer restart leg
red | red: `expected [] to have a length of 1` (no run inside the
minute) |
| A5 | a sole writer's claims not tracked as unsettled | failed-settle
leg red | red: `expected [ … ] to have a length of 2 but got 1` (the row
stays `running`) |
| A6 | the window term kept for a sole writer | sole-writer idle-hour
pin red | red: `expected 120 to be 1` |
A1–A6 were re-run on head `8d430e0c` (A3 now also reds the sole-writer
and ramp-control pins, 5 legs in all).
A0 (the whole pre-fix adapter) is the before/after above. Each leg's
on-disk landing and restore are printed by the tool (`anchor 1 -> 0`,
`blob == HEAD`, `git diff HEAD` empty).
</details>
## Acceptance notes
- The pending → running claim is still the by-id write it was (the
comment above it says so: concurrent claimers may both take a `pending`
row, and delivery is at-least-once). This PR does not change that; it
adds no new way to double-claim, and the reclaim never runs anything
itself.
- A worker whose lease was reclaimed while it was stalled and then
finishes still writes its terminal state by id. The renewal logs the
lost lease; making the terminal writes a compare-and-set on `locked_by`
too would close that last window and is a separate change.
- Deviation: `census +2 write sites (reclaimRow, renewLease), surface
widened by seat 03:38Z`. `check:tenant-audit-census` counts every engine
write call site, so `content/docs/permissions/tenant-audit-census.mdx`
and `docs/audits/2026-08-tenant-audit-write-call-sites.counts.md` are
regenerated (`node scripts/tenant-audit-census.mjs --write`) and the
page's hand-written figures moved with them (259 → 261 sites, 182 → 184
decidable, 107 → 109 undecidable, 66 → 68 const-named, 56% → 55%).
- Leases assume every node runs the same `leaseMs`, as they do when the
nodes share one `QueueServicePlugin` configuration. A node with a
shorter lease than this one's is still reclaimed, at most one lease
window late.
## Verification
All readings on head `8d430e0c` (code identical to `87fd3a2e`; the last
commit edits only the changeset).
- `pnpm --filter @objectstack/service-queue exec vitest run
--maxWorkers=2`: Test Files 5 passed (5), Tests 91 passed (91).
- `pnpm --filter @objectstack/plugin-email exec vitest run
--maxWorkers=2` (after building its closure): Test Files 48 passed (48),
Tests 785 passed (785). On `9a2d7712` its suites were red in CI (Test
Core 7/8, "fakeEngine: unsupported operator $in"): the reclaim read then
filtered `queue: { $in: … }`, which that package's stand-in engine
refuses; the read now carries only `status` equality.
- `pnpm --filter @objectstack/service-queue typecheck` (`tsc --noEmit`,
whose program lists `db-queue-idle-backoff.test.ts`): exit 0.
- `eslint --no-inline-config` on the two changed source files: exit 0.
`pnpm lint`: exit 0.
<details><summary>Derived gates: 90 families, 87 green, 3 NOT
MEASURED</summary>
`node scripts/pm/dispatch-gates.mjs --commands --repo
objectstack-ai/objectstack` against merge base `771b2db20`, each
command's exit code recorded before any pipe, reconciled with `--ran`:
`90 derived famil(ies) accounted for — 87 run, 3 NOT-MEASURED (3 DERIVED
from a recorded exit 3)`.
- NOT MEASURED, exit 3 (PREREQUISITE NOT MET), left to CI: `node
scripts/check-plugin-teardown-shape.mjs --self-test` (its positive
control is a pinned commit a shallow clone cannot reach), `pnpm --filter
@objectstack/spec run check:skill-examples` (needs the client packages
built), `pnpm check:dual-build-cjs-loads` (needs every package's
`dist/`; this worktree built only the queue and email closures).
- `pnpm check:tenant-audit-census` and its `--self-test`: exit 0 after
the census regeneration listed under Acceptance notes (red before it,
259 vs 261).
</details>
---
_Generated by [Claude
Code](https://claude.ai/code/session_01Utp6CaNicTDZwYQgDRwD7a)_
<!-- signed: domain:server#5 · os-bill ·
session_01Utp6CYVV6fcAQrNTDQRvPb -->
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent a26abbc commit 8b117c0
5 files changed
Lines changed: 630 additions & 56 deletions
File tree
- .changeset
- content/docs/permissions
- docs/audits
- packages/services/service-queue/src
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
132 | 132 | | |
133 | 133 | | |
134 | 134 | | |
135 | | - | |
| 135 | + | |
136 | 136 | | |
137 | 137 | | |
138 | 138 | | |
| |||
197 | 197 | | |
198 | 198 | | |
199 | 199 | | |
200 | | - | |
| 200 | + | |
201 | 201 | | |
202 | | - | |
203 | | - | |
| 202 | + | |
| 203 | + | |
204 | 204 | | |
205 | 205 | | |
206 | 206 | | |
| |||
211 | 211 | | |
212 | 212 | | |
213 | 213 | | |
214 | | - | |
| 214 | + | |
215 | 215 | | |
216 | 216 | | |
217 | 217 | | |
218 | 218 | | |
219 | 219 | | |
220 | | - | |
| 220 | + | |
221 | 221 | | |
222 | 222 | | |
223 | 223 | | |
224 | | - | |
| 224 | + | |
225 | 225 | | |
226 | 226 | | |
227 | 227 | | |
| |||
233 | 233 | | |
234 | 234 | | |
235 | 235 | | |
236 | | - | |
237 | | - | |
| 236 | + | |
| 237 | + | |
238 | 238 | | |
239 | | - | |
| 239 | + | |
240 | 240 | | |
241 | | - | |
| 241 | + | |
242 | 242 | | |
243 | 243 | | |
244 | 244 | | |
245 | 245 | | |
246 | 246 | | |
247 | 247 | | |
248 | | - | |
| 248 | + | |
249 | 249 | | |
250 | 250 | | |
251 | 251 | | |
252 | | - | |
| 252 | + | |
253 | 253 | | |
254 | 254 | | |
255 | 255 | | |
256 | 256 | | |
257 | 257 | | |
258 | | - | |
| 258 | + | |
259 | 259 | | |
260 | 260 | | |
261 | 261 | | |
| |||
307 | 307 | | |
308 | 308 | | |
309 | 309 | | |
310 | | - | |
| 310 | + | |
311 | 311 | | |
312 | 312 | | |
313 | 313 | | |
314 | | - | |
315 | | - | |
316 | | - | |
317 | | - | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
318 | 318 | | |
319 | 319 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
36 | | - | |
37 | | - | |
| 36 | + | |
| 37 | + | |
38 | 38 | | |
39 | | - | |
| 39 | + | |
40 | 40 | | |
41 | | - | |
| 41 | + | |
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
47 | 47 | | |
48 | | - | |
| 48 | + | |
49 | 49 | | |
50 | 50 | | |
51 | 51 | | |
| |||
90 | 90 | | |
91 | 91 | | |
92 | 92 | | |
93 | | - | |
| 93 | + | |
94 | 94 | | |
95 | 95 | | |
96 | 96 | | |
97 | | - | |
98 | | - | |
99 | | - | |
100 | | - | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
101 | 101 | | |
102 | 102 | | |
103 | 103 | | |
| |||
248 | 248 | | |
249 | 249 | | |
250 | 250 | | |
251 | | - | |
| 251 | + | |
252 | 252 | | |
253 | 253 | | |
254 | 254 | | |
| |||
0 commit comments