Repository navigation
Commit 9e1f1f9
fix(cluster): a Redis outage must not crash replicas; resync caches on reconnect (#31721)
<!-- claude-projects-attribution: {"github_login":"os-steve"} -->
_Requested by **Steve** · [project
thread](https://claude.ai/code/project/chan_01FdL1XeUyvVNxLreh3F4Qi9?thread=cmsg_01FdL1XeUyvVNxLreh3F4Qi9AS1kFKtVB2ZTW2g1kt4zuU)_
Fixes #31596
Clause-②: yes (widening)
Metadata-protocol: none
Measured on the built declarations (`dist/index.d.ts` of each touched
package): new exports in `@objectstack/service-cluster`
(`CLUSTER_RESYNC_EVENT`, `ClusterDriverLiveness`,
`offerClusterDriverLiveness`, `onClusterResync`,
`readClusterDriverHealth`, `describeConnectionError`,
`redactCredentials` and five types) and
`@objectstack/service-cluster-redis` (`REDIS_COMMAND_TIMEOUT_MS`,
`watchRedisConnection`, `describeRedisTarget`), optional members
`RedisDriverOptions.logger`, `RedisPubSubOptions.liveness`,
`DatasourceAdminServiceConfig.listConvergeNames` and the
connector-credential door's `resyncFromCluster`, and public methods
`MetadataManager.resyncClusterPubSub()`,
`DatasourceAdminService.resyncDatasourceMutationPubSub()`,
`AutomationEngine.resyncConnectorCredentialPubSub()`. Nothing is removed
or made required. The claim read `no`; the measurement says `yes
(widening)`. No metadata an author writes changes.
## 维护者速读
- 改了什么:多副本 Redis 集群下,Redis 宕机不再让副本进程崩溃;宕机只记一条告警,恢复后各缓存自动重新同步。
- 为什么改:此前定时任务拿不到集群锁时抛出未处理的拒绝,每个副本在下一次定时触发时退出;宕机期间丢失的缓存失效通知也要等到重启才会恢复。
- 风险与回滚:单节点(默认内存驱动)完全不变;Redis 驱动新增默认 2 秒命令超时,单条命令超过 2 秒会失败(驱动和共享缓存的命令都是
O(1))。回滚即撤销本 PR。
- 先看:`service-job` 的 `cron-job-adapter.ts`(不再崩溃)和 `service-cluster` 的新文件
`cluster-resync.ts`(`cluster.resync` 事件)。
- 两分钟验证:在装有 redis-server 的机器上跑 `service-cluster-redis` 的
`redis-outage.real-redis.test.ts`,它会在两个运行中的内核下停掉再启动 Redis。
- 需要你做的:无;标题里的 `!` 是否保留由席位复核(见 Acceptance notes)。
## What changes
- **No crash on a dead lock (`@objectstack/service-job`).**
`CronJobAdapter` leader-elects every scheduled fire on the cluster lock.
A rejected `acquire` used to escape the timer callback as an unhandled
rejection and end the process. It now skips that fire, and the next tick
retries. A failed `release` is absorbed, and the lease expires on its
TTL. The adapter logs one warning per lock outage and one info line when
the lock answers again. No process-wide `unhandledRejection` handler is
added. `runScheduled` is the only lock caller in the job adapters.
- **The driver watches its connections
(`@objectstack/service-cluster-redis`).** The command connection and the
pub/sub subscriber both get an `error` / `close` / `ready` listener
(`watchRedisConnection`). ioredis no longer prints `Unhandled error
event` per reconnect attempt. The driver logs one warning per outage,
naming `host:port` and never the URL or a password, and one info line
when the connection returns. New option `driverOptions.logger`.
- **Bounded commands.** `createRedisClient()` sets `commandTimeout:
REDIS_COMMAND_TIMEOUT_MS` (2000 ms). ioredis arms it before offline
queueing, so it bounds queued commands too. `client.duplicate()`
inherits it, so the shared cache's derived connection is bounded as
well.
- **Driver health and `cluster.resync`
(`@objectstack/service-cluster`).** `ClusterDriverLiveness` is the
connection state machine a remote driver feeds. Read health with
`readClusterDriverHealth(cluster)`, which returns `{ driver, nodeId,
state: 'unknown' | 'up' | 'down', since, outages, lastError? }`, or
`undefined` on the memory driver. This is the accessor #31601 reads.
When every failed connection is ready again, the liveness emits ONE
local event, `cluster.resync` (`CLUSTER_RESYNC_EVENT`). Subscribe with
`onClusterResync(cluster, handler)`, which returns a disposer. The event
is never published to peers. On the memory driver the subscription is an
inert no-op. #31597 joins here. The liveness is offered on the cluster
instance under a `Symbol.for` key, as `driver-connection.ts` already
does.
- **The four caches re-read on it.**
- `MetadataClusterBridgePlugin` subscribes once and calls each attached
lane's resync seam, with per-lane failure isolation:
- metadata list caches: `MetadataManager.resyncClusterPubSub()` drops
every list cache and the endpoint index; registry entries are kept.
- datasource pools:
`DatasourceAdminService.resyncDatasourceMutationPubSub()` re-converges
every datasource in the shared rows plus every pool this replica holds.
- connector credentials:
`AutomationEngine.resyncConnectorCredentialPubSub()` runs the reconcile
a peer's signal runs.
- `AuthzClusterBridgePlugin` advances the engine's write epoch with
reason `remote`, which retires cached grants exactly as a delivered
remote hint does.
R3: the single-node default path is unchanged (the memory driver offers
no liveness, so no listener, latch or resync runs on it). Nothing
switches on by itself: every new path needs the redis driver connected,
and the bridges still attach only when `isInProcessClusterDriver` is
false. The split-brain guard is untouched, and the deploy README's
"multi-node unsupported" line is untouched.
## Tests
- `service-cluster-redis/src/redis-outage.real-redis.test.ts` (new) uses
a real `redis-server` with a password. It is skipped where the binary is
absent. Two `LiteKernel` replicas, each with the redis cluster driver
and a cron job on a 100 ms interval, share one Redis. The Redis process
is SIGKILLed and restarted on the same port. Pinned:
- no unhandled rejection;
- exactly one "lost its connection" line and one "reconnected" line per
replica, with no password anywhere;
- at most one cron warning per replica;
- no fire during the outage, and the next fire after the return runs;
- a hint published during the outage is lost, and the peer converges on
its single `cluster.resync`;
- a hint published after the return reaches the peer.
The second case dials a black-holed TCP server: a GET on the cache's
`duplicate()`d connection rejects with `timed out` in about 2 s, not
about 30 s.
- `service-cluster/src/cluster-resync.test.ts` (new) covers:
- one warning per outage;
- resync once, only when every failed source is back;
- no resync on first connect;
- handler isolation, `close()` and redaction;
- the offer is non-enumerable;
- CONTROL: the memory driver gives no health and an inert subscription.
- Each lane got a new describe:
- `metadata-cluster-bridge-plugin.test.ts`: each lane is called once per
outage, a failing lane is isolated, nothing runs after shutdown, and
CONTROL memory.
- `authz-cluster-bridge-plugin.test.ts`: one `bump('remote')` while
bridged, nothing after shutdown, and CONTROL TTL 0.
- `metadata-manager-cluster.test.ts`,
`datasource-cluster-convergence.test.ts`,
`connector-credential-cluster.test.ts`: a lossy bus loses a peer write,
and the resync converges it (CREATE and DELETE for datasources, plus a
no-churn case). Each has a CONTROL with no transport.
- `cron-job-adapter.leader.test.ts`: a rejected acquire across 10 ticks
gives one warning and no rejection, recovery logs once, a release
rejection is absorbed, and a timer-driven fire raises no unhandled
rejection.
- Forced test edits: none. The authz bridge harness now hands back a
stable `objectql` service carrying a `writeEpoch` seam; no existing
assertion changed.
- Test resolution (from `pnpm check:test-source-alias`):
- `service-cluster-redis/vitest.config.ts` aliases `@objectstack/core`
and `@objectstack/service-job` to source. It previously re-exported the
shared preset, and it keeps the preset's one setting.
- The datasource convergence test loads `@objectstack/driver-sql` at
module top.
- `service-datasource/vitest.config.ts` aliases
`@objectstack/driver-memory` to source. A module-top import of it would
have been a new arrival in `pnpm check:driver-memory-census` (#6664).
The alias retires that package's shrink-only
`@objectstack/driver-memory` entry in
`scripts/check-test-source-alias.mjs`, as that gate requires. That is a
one-token edit outside the card's file surface.
- Ablations (each through `scripts/ablation-replace.mjs`; every one
landed and went red, and every restore left `git diff HEAD` empty):
| Ablation | Red tests |
| --- | --- |
| A1: acquire try/catch removed | 3 red, including the timer-driven
unhandled-rejection pin |
| A2: cron warn latch | 3 red, 10 warnings instead of 1 |
| A3: liveness warn latch | 2 red, 26 instead of 1 |
| A4: resync before every source is back | 1 red |
| A5: redaction | 2 red |
| A6: `commandTimeout` removed | the black-hole case times out at 10 s |
| A7: resync on first connect | 3 red |
| B1: metadata bridge shutdown detach | 1 red |
| B2: authz shutdown detach | 1 red |
| B3 / B4 / B5: metadata, datasource and automation no-transport guards
| 1 red each |
| B6: datasource connectivity check (no churn) | 1 red |
B5 first stayed green against a weaker control. That control was
strengthened in its own commit, and B5 then went red.
## Acceptance notes
- The `metadata.mutated` registry lane (lane 2) has no resync seam yet.
It still converges per item on delivered signals and fully at boot. I
recommend a follow-up card.
- Against a black-holed store the health reads `unknown` or `up` until
the socket errors. The command bound, not the health, is what stops a
caller hanging.
- A `SET NX` that timed out client-side may still land on the server, so
a lock can stay held until its TTL. The fire is skipped, never run
twice.
- Out of the file surface:
- The shared cache's derived connection
(`service-cache/src/cache-service-plugin.ts`, `base.duplicate()`) has no
`error` listener of its own, so it still prints ioredis' `Unhandled
error event` during an outage.
- The messaging dispatchers warn on each backed-off tick.
- `check:test-source-alias`'s source-reach remedy for a lazily loaded
`@objectstack/driver-memory` (a module-top import) is a binding that
`check:driver-memory-census` refuses. Any edited test that reaches the
default driver factory meets that conflict. This PR resolves it by
aliasing to source.
These are left for a follow-up.
- The `!` in the title: on a single node nothing changes. On the redis
driver the only new default is the 2 s command bound. The changeset is
`patch` with no BREAKING banner. The seat re-states whether the `!`
stands.
## Verification
- Per touched package, a full `vitest run`, `pnpm typecheck` and
package-scoped lint (root eslint on the directory, `--no-inline-config`)
all exit 0:
| Package | Test files | Tests |
| --- | --- | --- |
| service-cluster | 9 | 130 |
| service-cluster-redis | 4 | 41 (the real-redis file included) |
| service-job | 11 | 125 |
| metadata | 71 | 984 |
| service-datasource | 74 | 1299 |
| service-automation | 266 | 3458 |
The datasource suite was re-run on the final head. Lint on
`scripts/check-test-source-alias.mjs` exits 0.
- Full dogfood (`pnpm --filter @objectstack/dogfood test`, on
`2cb8ab79`): 440 test files passed and 3 skipped; 3652 tests passed and
41 skipped. The only commit after it touches test resolution and a gate
registry, not package source.
- `dispatch-gates.mjs --commands` was derived on the final head
`bf098837` and gave 105 commands. Each was run (all exit 0) and
reconciled with `--ran`: 105 derived, 105 run, 0 NOT-MEASURED.
---
_Generated by [Claude
Code](https://claude.ai/code/session_01ADHkmT4XhMdYmBfwHE4sQ7)_
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: objectstack-fleet[bot] <332303061+objectstack-fleet[bot]@users.noreply.github.com>1 parent 43fb482 commit 9e1f1f9
28 files changed
Lines changed: 1659 additions & 13 deletions
File tree
- .changeset
- content/docs/kernel
- packages
- metadata/src
- services
- service-automation/src
- service-cluster-redis
- src
- service-cluster/src
- service-datasource
- src
- __tests__
- service-job/src
- scripts
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
365 | 365 | | |
366 | 366 | | |
367 | 367 | | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
368 | 386 | | |
369 | 387 | | |
370 | 388 | | |
| |||
703 | 721 | | |
704 | 722 | | |
705 | 723 | | |
| 724 | + | |
| 725 | + | |
| 726 | + | |
| 727 | + | |
| 728 | + | |
| 729 | + | |
| 730 | + | |
| 731 | + | |
| 732 | + | |
| 733 | + | |
| 734 | + | |
| 735 | + | |
| 736 | + | |
| 737 | + | |
| 738 | + | |
| 739 | + | |
| 740 | + | |
| 741 | + | |
| 742 | + | |
| 743 | + | |
| 744 | + | |
| 745 | + | |
| 746 | + | |
| 747 | + | |
| 748 | + | |
| 749 | + | |
| 750 | + | |
| 751 | + | |
706 | 752 | | |
707 | 753 | | |
708 | 754 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
511 | 511 | | |
512 | 512 | | |
513 | 513 | | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3433 | 3433 | | |
3434 | 3434 | | |
3435 | 3435 | | |
| 3436 | + | |
| 3437 | + | |
| 3438 | + | |
| 3439 | + | |
| 3440 | + | |
| 3441 | + | |
| 3442 | + | |
| 3443 | + | |
| 3444 | + | |
| 3445 | + | |
| 3446 | + | |
| 3447 | + | |
| 3448 | + | |
| 3449 | + | |
| 3450 | + | |
| 3451 | + | |
| 3452 | + | |
| 3453 | + | |
| 3454 | + | |
| 3455 | + | |
| 3456 | + | |
| 3457 | + | |
| 3458 | + | |
| 3459 | + | |
| 3460 | + | |
| 3461 | + | |
| 3462 | + | |
| 3463 | + | |
| 3464 | + | |
| 3465 | + | |
3436 | 3466 | | |
3437 | 3467 | | |
3438 | 3468 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
301 | 301 | | |
302 | 302 | | |
303 | 303 | | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2616 | 2616 | | |
2617 | 2617 | | |
2618 | 2618 | | |
| 2619 | + | |
| 2620 | + | |
| 2621 | + | |
| 2622 | + | |
| 2623 | + | |
| 2624 | + | |
| 2625 | + | |
2619 | 2626 | | |
2620 | 2627 | | |
2621 | 2628 | | |
| |||
5025 | 5032 | | |
5026 | 5033 | | |
5027 | 5034 | | |
| 5035 | + | |
| 5036 | + | |
| 5037 | + | |
| 5038 | + | |
| 5039 | + | |
| 5040 | + | |
| 5041 | + | |
| 5042 | + | |
| 5043 | + | |
| 5044 | + | |
| 5045 | + | |
| 5046 | + | |
| 5047 | + | |
5028 | 5048 | | |
5029 | 5049 | | |
5030 | 5050 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1729 | 1729 | | |
1730 | 1730 | | |
1731 | 1731 | | |
| 1732 | + | |
1732 | 1733 | | |
1733 | 1734 | | |
1734 | 1735 | | |
| |||
4803 | 4804 | | |
4804 | 4805 | | |
4805 | 4806 | | |
| 4807 | + | |
| 4808 | + | |
| 4809 | + | |
| 4810 | + | |
| 4811 | + | |
| 4812 | + | |
| 4813 | + | |
| 4814 | + | |
| 4815 | + | |
| 4816 | + | |
| 4817 | + | |
| 4818 | + | |
| 4819 | + | |
| 4820 | + | |
| 4821 | + | |
| 4822 | + | |
4806 | 4823 | | |
4807 | 4824 | | |
4808 | 4825 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
| 36 | + | |
| 37 | + | |
36 | 38 | | |
37 | 39 | | |
38 | 40 | | |
| |||
0 commit comments