Skip to content

Commit 9e1f1f9

Browse files
fix(cluster): a Redis outage must not crash replicas; resync caches on reconnect (#31721)
<!-- claude-projects-attribution: {"github_login":"os-steve"} --> _Requested by **Steve** · [project thread](https://claude.ai/code/project/chan_01FdL1XeUyvVNxLreh3F4Qi9?thread=cmsg_01FdL1XeUyvVNxLreh3F4Qi9AS1kFKtVB2ZTW2g1kt4zuU)_ Fixes #31596 Clause-②: yes (widening) Metadata-protocol: none Measured on the built declarations (`dist/index.d.ts` of each touched package): new exports in `@objectstack/service-cluster` (`CLUSTER_RESYNC_EVENT`, `ClusterDriverLiveness`, `offerClusterDriverLiveness`, `onClusterResync`, `readClusterDriverHealth`, `describeConnectionError`, `redactCredentials` and five types) and `@objectstack/service-cluster-redis` (`REDIS_COMMAND_TIMEOUT_MS`, `watchRedisConnection`, `describeRedisTarget`), optional members `RedisDriverOptions.logger`, `RedisPubSubOptions.liveness`, `DatasourceAdminServiceConfig.listConvergeNames` and the connector-credential door's `resyncFromCluster`, and public methods `MetadataManager.resyncClusterPubSub()`, `DatasourceAdminService.resyncDatasourceMutationPubSub()`, `AutomationEngine.resyncConnectorCredentialPubSub()`. Nothing is removed or made required. The claim read `no`; the measurement says `yes (widening)`. No metadata an author writes changes. ## 维护者速读 - 改了什么:多副本 Redis 集群下,Redis 宕机不再让副本进程崩溃;宕机只记一条告警,恢复后各缓存自动重新同步。 - 为什么改:此前定时任务拿不到集群锁时抛出未处理的拒绝,每个副本在下一次定时触发时退出;宕机期间丢失的缓存失效通知也要等到重启才会恢复。 - 风险与回滚:单节点(默认内存驱动)完全不变;Redis 驱动新增默认 2 秒命令超时,单条命令超过 2 秒会失败(驱动和共享缓存的命令都是 O(1))。回滚即撤销本 PR。 - 先看:`service-job` 的 `cron-job-adapter.ts`(不再崩溃)和 `service-cluster` 的新文件 `cluster-resync.ts`(`cluster.resync` 事件)。 - 两分钟验证:在装有 redis-server 的机器上跑 `service-cluster-redis` 的 `redis-outage.real-redis.test.ts`,它会在两个运行中的内核下停掉再启动 Redis。 - 需要你做的:无;标题里的 `!` 是否保留由席位复核(见 Acceptance notes)。 ## What changes - **No crash on a dead lock (`@objectstack/service-job`).** `CronJobAdapter` leader-elects every scheduled fire on the cluster lock. A rejected `acquire` used to escape the timer callback as an unhandled rejection and end the process. It now skips that fire, and the next tick retries. A failed `release` is absorbed, and the lease expires on its TTL. The adapter logs one warning per lock outage and one info line when the lock answers again. No process-wide `unhandledRejection` handler is added. `runScheduled` is the only lock caller in the job adapters. - **The driver watches its connections (`@objectstack/service-cluster-redis`).** The command connection and the pub/sub subscriber both get an `error` / `close` / `ready` listener (`watchRedisConnection`). ioredis no longer prints `Unhandled error event` per reconnect attempt. The driver logs one warning per outage, naming `host:port` and never the URL or a password, and one info line when the connection returns. New option `driverOptions.logger`. - **Bounded commands.** `createRedisClient()` sets `commandTimeout: REDIS_COMMAND_TIMEOUT_MS` (2000 ms). ioredis arms it before offline queueing, so it bounds queued commands too. `client.duplicate()` inherits it, so the shared cache's derived connection is bounded as well. - **Driver health and `cluster.resync` (`@objectstack/service-cluster`).** `ClusterDriverLiveness` is the connection state machine a remote driver feeds. Read health with `readClusterDriverHealth(cluster)`, which returns `{ driver, nodeId, state: 'unknown' | 'up' | 'down', since, outages, lastError? }`, or `undefined` on the memory driver. This is the accessor #31601 reads. When every failed connection is ready again, the liveness emits ONE local event, `cluster.resync` (`CLUSTER_RESYNC_EVENT`). Subscribe with `onClusterResync(cluster, handler)`, which returns a disposer. The event is never published to peers. On the memory driver the subscription is an inert no-op. #31597 joins here. The liveness is offered on the cluster instance under a `Symbol.for` key, as `driver-connection.ts` already does. - **The four caches re-read on it.** - `MetadataClusterBridgePlugin` subscribes once and calls each attached lane's resync seam, with per-lane failure isolation: - metadata list caches: `MetadataManager.resyncClusterPubSub()` drops every list cache and the endpoint index; registry entries are kept. - datasource pools: `DatasourceAdminService.resyncDatasourceMutationPubSub()` re-converges every datasource in the shared rows plus every pool this replica holds. - connector credentials: `AutomationEngine.resyncConnectorCredentialPubSub()` runs the reconcile a peer's signal runs. - `AuthzClusterBridgePlugin` advances the engine's write epoch with reason `remote`, which retires cached grants exactly as a delivered remote hint does. R3: the single-node default path is unchanged (the memory driver offers no liveness, so no listener, latch or resync runs on it). Nothing switches on by itself: every new path needs the redis driver connected, and the bridges still attach only when `isInProcessClusterDriver` is false. The split-brain guard is untouched, and the deploy README's "multi-node unsupported" line is untouched. ## Tests - `service-cluster-redis/src/redis-outage.real-redis.test.ts` (new) uses a real `redis-server` with a password. It is skipped where the binary is absent. Two `LiteKernel` replicas, each with the redis cluster driver and a cron job on a 100 ms interval, share one Redis. The Redis process is SIGKILLed and restarted on the same port. Pinned: - no unhandled rejection; - exactly one "lost its connection" line and one "reconnected" line per replica, with no password anywhere; - at most one cron warning per replica; - no fire during the outage, and the next fire after the return runs; - a hint published during the outage is lost, and the peer converges on its single `cluster.resync`; - a hint published after the return reaches the peer. The second case dials a black-holed TCP server: a GET on the cache's `duplicate()`d connection rejects with `timed out` in about 2 s, not about 30 s. - `service-cluster/src/cluster-resync.test.ts` (new) covers: - one warning per outage; - resync once, only when every failed source is back; - no resync on first connect; - handler isolation, `close()` and redaction; - the offer is non-enumerable; - CONTROL: the memory driver gives no health and an inert subscription. - Each lane got a new describe: - `metadata-cluster-bridge-plugin.test.ts`: each lane is called once per outage, a failing lane is isolated, nothing runs after shutdown, and CONTROL memory. - `authz-cluster-bridge-plugin.test.ts`: one `bump('remote')` while bridged, nothing after shutdown, and CONTROL TTL 0. - `metadata-manager-cluster.test.ts`, `datasource-cluster-convergence.test.ts`, `connector-credential-cluster.test.ts`: a lossy bus loses a peer write, and the resync converges it (CREATE and DELETE for datasources, plus a no-churn case). Each has a CONTROL with no transport. - `cron-job-adapter.leader.test.ts`: a rejected acquire across 10 ticks gives one warning and no rejection, recovery logs once, a release rejection is absorbed, and a timer-driven fire raises no unhandled rejection. - Forced test edits: none. The authz bridge harness now hands back a stable `objectql` service carrying a `writeEpoch` seam; no existing assertion changed. - Test resolution (from `pnpm check:test-source-alias`): - `service-cluster-redis/vitest.config.ts` aliases `@objectstack/core` and `@objectstack/service-job` to source. It previously re-exported the shared preset, and it keeps the preset's one setting. - The datasource convergence test loads `@objectstack/driver-sql` at module top. - `service-datasource/vitest.config.ts` aliases `@objectstack/driver-memory` to source. A module-top import of it would have been a new arrival in `pnpm check:driver-memory-census` (#6664). The alias retires that package's shrink-only `@objectstack/driver-memory` entry in `scripts/check-test-source-alias.mjs`, as that gate requires. That is a one-token edit outside the card's file surface. - Ablations (each through `scripts/ablation-replace.mjs`; every one landed and went red, and every restore left `git diff HEAD` empty): | Ablation | Red tests | | --- | --- | | A1: acquire try/catch removed | 3 red, including the timer-driven unhandled-rejection pin | | A2: cron warn latch | 3 red, 10 warnings instead of 1 | | A3: liveness warn latch | 2 red, 26 instead of 1 | | A4: resync before every source is back | 1 red | | A5: redaction | 2 red | | A6: `commandTimeout` removed | the black-hole case times out at 10 s | | A7: resync on first connect | 3 red | | B1: metadata bridge shutdown detach | 1 red | | B2: authz shutdown detach | 1 red | | B3 / B4 / B5: metadata, datasource and automation no-transport guards | 1 red each | | B6: datasource connectivity check (no churn) | 1 red | B5 first stayed green against a weaker control. That control was strengthened in its own commit, and B5 then went red. ## Acceptance notes - The `metadata.mutated` registry lane (lane 2) has no resync seam yet. It still converges per item on delivered signals and fully at boot. I recommend a follow-up card. - Against a black-holed store the health reads `unknown` or `up` until the socket errors. The command bound, not the health, is what stops a caller hanging. - A `SET NX` that timed out client-side may still land on the server, so a lock can stay held until its TTL. The fire is skipped, never run twice. - Out of the file surface: - The shared cache's derived connection (`service-cache/src/cache-service-plugin.ts`, `base.duplicate()`) has no `error` listener of its own, so it still prints ioredis' `Unhandled error event` during an outage. - The messaging dispatchers warn on each backed-off tick. - `check:test-source-alias`'s source-reach remedy for a lazily loaded `@objectstack/driver-memory` (a module-top import) is a binding that `check:driver-memory-census` refuses. Any edited test that reaches the default driver factory meets that conflict. This PR resolves it by aliasing to source. These are left for a follow-up. - The `!` in the title: on a single node nothing changes. On the redis driver the only new default is the 2 s command bound. The changeset is `patch` with no BREAKING banner. The seat re-states whether the `!` stands. ## Verification - Per touched package, a full `vitest run`, `pnpm typecheck` and package-scoped lint (root eslint on the directory, `--no-inline-config`) all exit 0: | Package | Test files | Tests | | --- | --- | --- | | service-cluster | 9 | 130 | | service-cluster-redis | 4 | 41 (the real-redis file included) | | service-job | 11 | 125 | | metadata | 71 | 984 | | service-datasource | 74 | 1299 | | service-automation | 266 | 3458 | The datasource suite was re-run on the final head. Lint on `scripts/check-test-source-alias.mjs` exits 0. - Full dogfood (`pnpm --filter @objectstack/dogfood test`, on `2cb8ab79`): 440 test files passed and 3 skipped; 3652 tests passed and 41 skipped. The only commit after it touches test resolution and a gate registry, not package source. - `dispatch-gates.mjs --commands` was derived on the final head `bf098837` and gave 105 commands. Each was run (all exit 0) and reconciled with `--ran`: 105 derived, 105 run, 0 NOT-MEASURED. --- _Generated by [Claude Code](https://claude.ai/code/session_01ADHkmT4XhMdYmBfwHE4sQ7)_ --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: objectstack-fleet[bot] <332303061+objectstack-fleet[bot]@users.noreply.github.com>
1 parent 43fb482 commit 9e1f1f9

28 files changed

Lines changed: 1659 additions & 13 deletions
Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
---
2+
'@objectstack/service-cluster': patch
3+
'@objectstack/service-cluster-redis': patch
4+
'@objectstack/service-job': patch
5+
'@objectstack/metadata': patch
6+
'@objectstack/service-datasource': patch
7+
'@objectstack/service-automation': patch
8+
---
9+
10+
fix(cluster): a Redis outage no longer crashes replicas, is logged once, and caches re-read on reconnect
11+
12+
Clause-②: yes (widening)
13+
Metadata-protocol: none
14+
15+
Every API addition below is a new export or an optional member (a widening). The one new default is the bounded command time on the redis driver. Multi-replica deployments on the `redis` cluster driver only. The single-node default (the in-process `memory` driver, no declared replicas) behaves exactly as before: it never loses a connection, so none of the paths below runs on it.
16+
17+
- **No crash on a dead lock (`@objectstack/service-job`).** `CronJobAdapter` leader-elects every scheduled fire on the cluster lock. When the lock store was unreachable, `acquire` rejected inside a timer callback, the rejection went unhandled, and every replica exited at its next tick. A failed `acquire` now skips that fire (no node can prove it alone runs it) and the next tick asks again; a failed `release` after the fire ran is absorbed (the lease expires on its TTL). The adapter logs ONE warning per lock outage — not one per tick — and one info line when the lock answers again. No process-wide `unhandledRejection` handler is added.
18+
- **The driver watches its connections (`@objectstack/service-cluster-redis`).** Every ioredis client the driver holds (the command connection and the pub/sub subscriber) now has an `error` listener, so ioredis no longer prints `[ioredis] Unhandled error event` with a stack per reconnect attempt. The driver logs one warning when the store becomes unreachable (naming `host:port`, never the URL or a password) and one info line when it returns. New option: `driverOptions.logger` (`{ warn, info? }`, default `console`).
19+
- **Bounded commands.** `createRedisClient()` now sets `commandTimeout: REDIS_COMMAND_TIMEOUT_MS` (2000 ms; exported). A black-holed Redis (a socket that accepts and never answers) used to hold a caller about 30 s — a sign-in counting through the shared cache included, since the cache derives its connection with `client.duplicate()`, which inherits the bound. Every command the driver and the shared cache issue is O(1); a caller that needs longer passes its own `options.commandTimeout`, or its own client through `driverOptions.client`.
20+
- **`cluster.resync` (`@objectstack/service-cluster`).** Cross-node cache hints ride at-most-once pub/sub, so a hint published during an outage was lost and a peer kept serving what it had cached until restart. When the driver's connection returns after an outage it now emits ONE local `cluster.resync` event (never published to peers). New exports: `onClusterResync(cluster, handler)` to subscribe (a no-op on the memory driver), `readClusterDriverHealth(cluster)` (a passive reading of the driver's connection state, type `ClusterDriverLivenessHealth`: `{ driver, nodeId, state: 'unknown' | 'up' | 'down', since, outages, lastError? }`, `undefined` on the memory driver; the active store probe `/ready` uses is `probeClusterDriverHealth()`), and `ClusterDriverLiveness` / `offerClusterDriverLiveness` for driver authors, plus `CLUSTER_RESYNC_EVENT`, `describeConnectionError` and `redactCredentials`.
21+
- **The caches re-read on it.** `MetadataClusterBridgePlugin` calls each attached lane's resync seam: `MetadataManager.resyncClusterPubSub()` (`@objectstack/metadata`: drops every list cache and the endpoint index, so the next `list()` re-reads the loaders; registry entries are kept), `DatasourceAdminService.resyncDatasourceMutationPubSub()` (`@objectstack/service-datasource`: re-converges every stored datasource and every held pool from this replica's own read), and `AutomationEngine.resyncConnectorCredentialPubSub()` (`@objectstack/service-automation`: the connector reconcile a peer's signal runs; the connector-credential door gains an optional `resyncFromCluster()`). `AuthzClusterBridgePlugin` advances the engine's write epoch (reason `remote`), retiring cached grants. The `metadata.mutated` registry lane has no resync yet; it still converges per item on delivered signals and fully at boot.

‎content/docs/kernel/cluster.mdx‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -365,6 +365,24 @@ channel's own doc comment:
365365
exceeds at-most-once delivery (§4.2), so a lost message is not retried.
366366
Each lane narrows the staleness window this always had — "until the next
367367
full reload on restart" — to "one network hop"; it does not promise more.
368+
- **Resync after a driver outage.** A message lost while the store is
369+
unreachable is not retried, but a replica no longer waits for its restart
370+
to recover it. When a remote driver's connection returns after an outage,
371+
the driver emits ONE local `cluster.resync` event on that replica. The
372+
event is never published to peers; each replica emits its own when its own
373+
connection returns. `MetadataClusterBridgePlugin` re-reads every attached
374+
lane on it: lane 1 drops the metadata list caches
375+
(`MetadataManager.resyncClusterPubSub()`), lane 3 re-converges every stored
376+
datasource and every held pool
377+
(`DatasourceAdminService.resyncDatasourceMutationPubSub()`), and lane 4
378+
runs the connector reconcile
379+
(`AutomationEngine.resyncConnectorCredentialPubSub()`).
380+
`AuthzClusterBridgePlugin` advances the engine's write epoch, which retires
381+
cached grants. **Lane 2 has no resync yet**: it converges per item on the
382+
signals that are delivered, and fully at boot. A cache of your own
383+
subscribes with `onClusterResync(cluster, handler)` from
384+
`@objectstack/service-cluster`. On the `memory` driver that subscription is
385+
a no-op, because nothing there can lose a connection.
368386

369387
All four lanes skip their fan-out on the same condition — the in-process
370388
`memory` cluster driver, described for lane 1 immediately below, applies
@@ -703,6 +721,34 @@ membership view exists at boot. It is the right input for telling an operator
703721
about the configuration they wrote, and is deliberately not sufficient for
704722
enforcement.
705723

724+
The `redis` driver has two outage settings:
725+
726+
- `driverOptions.logger` (`{ warn, info? }`, default `console`): where the driver writes
727+
its outage lines. It logs ONE warning when the store becomes unreachable,
728+
naming `host:port` and never the URL or a password, and one info line when
729+
the store returns. It does not log a line per failed command.
730+
- Every connection the driver creates is built with `commandTimeout` set to
731+
`REDIS_COMMAND_TIMEOUT_MS` (2000 ms, exported by
732+
`@objectstack/service-cluster-redis`). A command, queued or sent, fails
733+
after 2 s instead of hanging on a black-holed store. The shared cache
734+
derives its connection with `client.duplicate()` and inherits the same
735+
bound. To run longer commands, pass your own client through
736+
`driverOptions.client`.
737+
738+
Two readings of the same driver answer different questions:
739+
740+
- `readClusterDriverHealth(cluster)` is **passive**. It reports what the
741+
driver's connections last said (`error`, `close`, `ready`), sends nothing to
742+
the store, and costs nothing. It returns a `ClusterDriverLivenessHealth`:
743+
`{ driver, nodeId, state: 'unknown' | 'up' | 'down', since, outages, lastError? }`.
744+
A black-holed store whose socket never errors still reads as `up`.
745+
- `probeClusterDriverHealth(cluster, driver)` is **active**. It sends a
746+
bounded `PING` to the store, and it is what `/ready` reports. It returns a
747+
`ClusterDriverHealth`: `{ driver, reachable, reason? }`. It does see a
748+
black-holed store, because the `PING` times out.
749+
750+
Both return `undefined` on the `memory` driver.
751+
706752
Every cluster primitive selects its implementation from `cluster.driver`.
707753
When the field is absent, `Runtime` auto-registers the in-memory, single-node
708754
driver (pass `cluster: false` to skip that auto-registration). No production

‎packages/metadata/src/metadata-manager-cluster.test.ts‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -511,3 +511,57 @@ describe('MetadataManager — default `memory` cluster driver does not cross OS
511511
}
512512
});
513513
});
514+
515+
describe('MetadataManager — resyncClusterPubSub() converges a hint lost in a cluster outage', () => {
516+
const view = (name: string, title: string) => ({ name, title, object: 'account' });
517+
518+
/** `makeCluster()` over a bus that drops every publish while `down` — the at-most-once loss. */
519+
function makeLossyCluster() {
520+
const bus = new TestPubSub();
521+
const state = { down: false };
522+
const publish = bus.publish.bind(bus);
523+
bus.publish = (async (channel: string, payload: unknown) => {
524+
if (state.down) return;
525+
await publish(channel, payload);
526+
}) as typeof bus.publish;
527+
const store = new SharedStoreLoader();
528+
const a = new MetadataManager({ formats: ['json'], loaders: [store] });
529+
const b = new MetadataManager({ formats: ['json'], loaders: [store] });
530+
a.attachClusterPubSub(bus, 'node-A');
531+
b.attachClusterPubSub(bus, 'node-B');
532+
return { store, a, b, state };
533+
}
534+
535+
it('a peer write whose hint was lost stays invisible until the resync, then list() re-reads it', async () => {
536+
const { store, a, b, state } = makeLossyCluster();
537+
await store.save('view', 'v_existing', view('v_existing', 'existing'));
538+
expect(viewNames(await b.list('view'))).toEqual(['v_existing']);
539+
540+
state.down = true;
541+
await a.register('view', 'v_new', view('v_new', 'new'));
542+
state.down = false;
543+
// The loss: B still serves its pre-write list (the 30s TTL has not lapsed).
544+
expect(viewNames(await b.list('view'))).toEqual(['v_existing']);
545+
546+
b.resyncClusterPubSub();
547+
expect(cachedTypes(b)).toEqual([]);
548+
expect(viewNames(await b.list('view'))).toEqual(['v_existing', 'v_new']);
549+
});
550+
551+
it('keeps registerInMemory() artefacts, which no loader can restore', async () => {
552+
const { b } = makeLossyCluster();
553+
b.registerInMemory('datasource', 'code_ds', { name: 'code_ds', origin: 'code' });
554+
await b.list('datasource');
555+
b.resyncClusterPubSub();
556+
expect(await b.get('datasource', 'code_ds')).toEqual({ name: 'code_ds', origin: 'code' });
557+
});
558+
559+
it('CONTROL — no transport attached (single node): resync drops nothing', async () => {
560+
const store = new SharedStoreLoader();
561+
const solo = new MetadataManager({ formats: ['json'], loaders: [store] });
562+
await store.save('view', 'v1', view('v1', 'one'));
563+
await solo.list('view');
564+
solo.resyncClusterPubSub();
565+
expect(cachedTypes(solo)).toEqual(['view']);
566+
});
567+
});

‎packages/metadata/src/metadata-manager.ts‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3433,6 +3433,36 @@ export class MetadataManager implements IMetadataService {
34333433
return () => this.detachClusterPubSub();
34343434
}
34353435

3436+
/**
3437+
* The `cluster.resync` answer for this lane: the cluster driver's connection
3438+
* came back after an outage, during which any `metadata.changed` a peer
3439+
* published was lost (the bus is at-most-once). Called by
3440+
* `MetadataClusterBridgePlugin` in `@objectstack/service-cluster` on the
3441+
* driver's local `cluster.resync` event; a no-op until a transport is
3442+
* attached.
3443+
*
3444+
* The receipt of a nameless event, for every type this manager holds a
3445+
* list for: {@link invalidateForForeignWrite} with no name drops the list
3446+
* cache (and the endpoint index), so the next `list()` re-reads the loaders
3447+
* — where a write the lost hint announced already is. Registry ENTRIES are
3448+
* kept, for the reason that method gives: which names a peer wrote is
3449+
* exactly what the lost hints carried, and dropping a whole type store would
3450+
* evict `registerInMemory()` artefacts no loader can restore.
3451+
*/
3452+
resyncClusterPubSub(): void {
3453+
if (!this.clusterPubSub) return;
3454+
const types = new Set<string>([
3455+
...this.listCache.keys(),
3456+
...this.inflightListReads.keys(),
3457+
MetadataManager.ENDPOINT_METADATA_TYPE,
3458+
]);
3459+
for (const type of types) this.invalidateForForeignWrite(type);
3460+
this.logger.info('MetadataManager resynced after a cluster outage: list caches dropped', {
3461+
nodeId: this.clusterNodeId,
3462+
types: types.size,
3463+
});
3464+
}
3465+
34363466
/** Tear down cluster wiring. Safe to call multiple times. */
34373467
detachClusterPubSub(): void {
34383468
if (this.clusterUnsubscribe) {

‎packages/services/service-automation/src/connector-credential-cluster.test.ts‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -301,3 +301,60 @@ describe('[#27428] the lane carries an address, and the publisher drops its own'
301301
expect(out.text()).toContain(`connector credential '${REF}' was written on node 'node-b'; reconciling the connectors on node 'node-a'`);
302302
});
303303
});
304+
305+
describe('resyncConnectorCredentialPubSub() — a credential write whose signal an outage lost converges on the resync', () => {
306+
/** `sharedBus()` that drops every publish while `down`: the at-most-once loss an outage causes. */
307+
function lossyBus() {
308+
const shared = sharedBus();
309+
const state = { down: false };
310+
const publish = shared.bus.publish.bind(shared.bus);
311+
shared.bus.publish = (async (channel: string, payload: unknown, opts?: never) => {
312+
if (state.down) return;
313+
await publish(channel, payload, opts);
314+
}) as typeof shared.bus.publish;
315+
return { ...shared, state };
316+
}
317+
318+
it('B holds its refused instance while A’s correction is lost, and re-materializes on the resync', async () => {
319+
const ql = await sharedDatabase();
320+
const { bus, state } = lossyBus();
321+
const a = await replica(ql, 'node-a', bus);
322+
const b = await replica(ql, 'node-b', bus);
323+
a.provider.state.reject = true;
324+
b.provider.state.reject = true;
325+
await a.engine.setConnectorCredential({ ref: REF, value: DUMMY_VALUE, context: ADMIN });
326+
await vi.waitFor(() => expect(b.held()).toContain('refused'));
327+
328+
a.provider.state.reject = false;
329+
b.provider.state.reject = false;
330+
state.down = true;
331+
await a.engine.setConnectorCredential({ ref: REF, value: DUMMY_VALUE_2, context: ADMIN });
332+
state.down = false;
333+
for (let i = 0; i < 5; i++) await flush();
334+
expect(b.held()).toContain('refused');
335+
336+
await b.engine.resyncConnectorCredentialPubSub();
337+
expect(b.held()).toBeUndefined();
338+
expect(b.provider.last()).toBe(DUMMY_VALUE_2);
339+
});
340+
341+
it('CONTROL — no transport attached (single node): the resync reconciles nothing', async () => {
342+
// The same lost-correction shape as above, with no transport on either
343+
// replica: B's resync is inert, so B keeps its refused instance (it
344+
// converges on its own reload or restart, as before this change).
345+
const ql = await sharedDatabase();
346+
const a = await replica(ql, 'node-a', undefined);
347+
const b = await replica(ql, 'node-b', undefined);
348+
b.provider.state.reject = true;
349+
await b.engine.setConnectorCredential({ ref: REF, value: DUMMY_VALUE, context: ADMIN });
350+
expect(b.held()).toContain('refused');
351+
352+
b.provider.state.reject = false;
353+
await a.engine.setConnectorCredential({ ref: REF, value: DUMMY_VALUE_2, context: ADMIN });
354+
const before = b.provider.state.materialized.length;
355+
await b.engine.resyncConnectorCredentialPubSub();
356+
for (let i = 0; i < 5; i++) await flush();
357+
expect(b.held()).toContain('refused');
358+
expect(b.provider.state.materialized).toHaveLength(before);
359+
});
360+
});

‎packages/services/service-automation/src/engine.ts‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2616,6 +2616,13 @@ export interface ConnectorCredentialDoor {
26162616
* {@link AutomationEngine.attachConnectorCredentialPubSub}.
26172617
*/
26182618
attachClusterPubSub(pubsub: IPubSub, nodeId: string): () => void;
2619+
/**
2620+
* The `cluster.resync` answer: reconcile from the shared rows after a
2621+
* cluster outage lost peer writes — see
2622+
* {@link AutomationEngine.resyncConnectorCredentialPubSub}. Optional, so a
2623+
* door written before it still attaches; without it a resync does nothing.
2624+
*/
2625+
resyncFromCluster?(): Promise<void>;
26192626
}
26202627

26212628
/**
@@ -5025,6 +5032,19 @@ export class AutomationEngine implements IAutomationService {
50255032
return this.requireConnectorCredentialDoor('attachConnectorCredentialPubSub()').attachClusterPubSub(pubsub, nodeId);
50265033
}
50275034

5035+
/**
5036+
* The `cluster.resync` answer for lane 4: the cluster driver's connection
5037+
* came back after an outage, during which any `connector-credential.mutated`
5038+
* a peer published was lost (the bus is at-most-once). Called by
5039+
* `MetadataClusterBridgePlugin` on the driver's local `cluster.resync`
5040+
* event. Runs the reconcile a received signal runs — it re-reads the shared
5041+
* credential rows, so every instance naming a reference that changed during
5042+
* the outage re-materializes. A no-op until a transport is attached.
5043+
*/
5044+
async resyncConnectorCredentialPubSub(): Promise<void> {
5045+
await this.requireConnectorCredentialDoor('resyncConnectorCredentialPubSub()').resyncFromCluster?.();
5046+
}
5047+
50285048
/** [#27237] `IAutomationService.listConnectorCredentialRefs` — references and states, never a value. */
50295049
async listConnectorCredentialRefs(): Promise<ConnectorCredentialRef[]> {
50305050
return this.requireConnectorCredentialDoor('listConnectorCredentialRefs()').list();

‎packages/services/service-automation/src/plugin.ts‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1729,6 +1729,7 @@ export class AutomationServicePlugin implements Plugin {
17291729
set: (request) => this.setConnectorCredential(ctx, request),
17301730
clear: (request) => this.clearConnectorCredential(ctx, request),
17311731
attachClusterPubSub: (pubsub, nodeId) => this.attachCredentialClusterPubSub(ctx, pubsub, nodeId),
1732+
resyncFromCluster: () => this.resyncCredentialClusterPubSub(ctx),
17321733
});
17331734

17341735
// [ADR-0015 D7] The connector credential source: a datasource that
@@ -4803,6 +4804,22 @@ export class AutomationServicePlugin implements Plugin {
48034804
return () => this.detachCredentialClusterPubSub();
48044805
}
48054806

4807+
/**
4808+
* The `cluster.resync` answer for the credential lane: the reconcile a
4809+
* peer's signal runs, once, because which references changed during the
4810+
* outage is what the lost signals carried. A no-op until a transport is
4811+
* attached. A failure is the caller's to log (the bridge does).
4812+
*/
4813+
private async resyncCredentialClusterPubSub(ctx: PluginContext): Promise<void> {
4814+
const cluster = this.credentialCluster;
4815+
if (!cluster) return;
4816+
ctx.logger.info(
4817+
`[Automation] the cluster connection returned after an outage; reconciling the connectors on node '${cluster.nodeId}' `
4818+
+ 'in case a peer wrote a connector credential meanwhile.',
4819+
);
4820+
await this.materializeDeclaredConnectors(ctx, { fatal: false });
4821+
}
4822+
48064823
/** [#27428] Tear down the credential cluster wiring. Safe to call multiple times. */
48074824
private detachCredentialClusterPubSub(): void {
48084825
const current = this.credentialCluster;

‎packages/services/service-cluster-redis/package.json‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,8 @@
3333
"ioredis": "^6.0.0"
3434
},
3535
"devDependencies": {
36+
"@objectstack/core": "workspace:*",
37+
"@objectstack/service-job": "workspace:*",
3638
"@types/node": "^26.6.5",
3739
"ioredis-mock": "^8.13.1",
3840
"typescript": "^6.0.3",

0 commit comments

Comments
 (0)