Skip to content

Commit 00205e1

Browse files
feat(runtime): /ready reports cluster driver health when multi-replica is declared (#31711)
<!-- claude-projects-attribution: {"github_login":"os-steve"} --> _Requested by **Steve** · [project thread](https://claude.ai/code/project/chan_01FdL1XeUyvVNxLreh3F4Qi9?thread=cmsg_01FdL1XeUyvVNxLreh3F4Qi9A6psgEaEgrJbD91eQmwK9B)_ Fixes #31601 `/ready` reports the cluster driver's health on a deployment that declares multi-replica and runs the redis cluster driver: Redis unreachable answers **200 with `status: 'degraded'`** and a `cluster` entry naming the driver unreachable (PM decision D4: never 503). Child of cloud epic #3814; Claim 6727414288 (`domain:server#4`, card-thread session_01ADHkmQwuHPnJPK768Mta39). Clause-②: yes (widening) — measured on base `1a20cbc8` (the merge base; its `http-dispatcher.ts` restored under this PR's pins) and head `ce30cfbf`: on a declared multi-node redis deployment with Redis down, base answers `200 {"status":"ready","state":"running"}` (`expected { status: 'ready', state: 'running' } to deeply equal { status: 'degraded', …(2) }`, the `200` assertion above it passing), head answers `200 {"status":"degraded","state":"running","cluster":{"driver":"redis","reachable":false,"reason":"ping-failed"}}` (also with a live redis-server stopped, below). Every request keeps the status code base answered; the `200` body gains a `cluster` key and the `degraded` value of `status` only behind the declaration plus the redis driver. Nothing base accepted is refused. Metadata-protocol: none ## 维护者速读 - 改了什么:声明多副本(`OS_EXPECT_MULTI_NODE=true` 或 `OS_CLUSTER_REPLICAS>1`)且集群驱动是 redis 时,`/ready` 的 200 响应多一个 `cluster` 条目;Redis 不可达时 `status` 变为 `degraded`,仍是 200。 - 为什么改:之前 Redis 停了,各副本的 `/ready` 一直是普通的 200,运维看不到集群已失去协调(锁、计数器、跨副本事件全失效)。不用 503 是 D4 的决定:所有副本同时失去 Redis,503 会把整个集群摘出负载均衡。 - 风险与回滚:单节点、memory 驱动、没有 cluster 服务时 body 与状态码逐字节不变,也不会去 ping Redis;启动中/停机(SIGTERM)的 503 和主数据源 503 在探测前返回,不变。探测是一次 500ms 上限的 PING,缓存 1 秒。回滚即 revert 本 PR。 - 先看:`packages/runtime/src/http-dispatcher.ts`(`clusterReadiness()` 与两个 200 分支)、`packages/services/service-cluster/src/driver-health.ts`、`packages/runtime/src/http-dispatcher.ready.test.ts` 新增的 describe。 - 两分钟验证:`pnpm --filter @objectstack/runtime exec vitest run src/http-dispatcher.ready.test.ts` —— 预期 41 passed。 - 需要你做的:无,席位落地。 ## What changes - **`packages/services/service-cluster/src/driver-health.ts`** (new) — `probeClusterDriverHealth(cluster, driver, { timeoutMs })`: reads the connection the driver offers (`readClusterDriverConnection`, the seam the shared cache already reads), sends one `PING` bounded by `DEFAULT_CLUSTER_PROBE_TIMEOUT_MS` (500 ms), and answers `{ driver, reachable: true }` on `PONG`, or `{ driver, reachable: false, reason }` with `reason` from a closed set: `timeout`, `ping-failed`, `unexpected-reply`, `no-probe`. Answers `undefined` when the cluster offers no connection for that driver (memory driver, no cluster). Never throws. The client's error text (it names the store's host and port) is never carried, because `/ready` answers anonymous callers. Exported from the package index with its types. - **`packages/runtime/src/http-dispatcher.ts`** — `clusterReadiness()`: applies only when `declaresMultiNode(process.env)` holds (the split-brain guard's predicate) AND the host kernel's `cluster` service offers a `redis` connection; memoized on the same 1 s TTL as the data-driver probe. It is consulted on the two `200` branches only, through `withClusterReading()`, which returns the body object as given when there is no reading. The booting/stopping `503` and the primary-datasource `503` return before it. - **`packages/runtime/src/http-dispatcher.ready.test.ts`** — new top-level describe in the existing `/ready` harness (no existing assertion edited): the card's four acceptance cases plus secondary-degraded beside Redis down, the primary `503` never probing Redis, the memo, and R3's "one half missing" cases (no declaration, `OS_CLUSTER_REPLICAS=1`, memory driver, no cluster service, a throwing lookup). - **`packages/services/service-cluster/src/driver-health.test.ts`** (new) — the probe's eight readings, including a late rejection after the timeout not leaking as unhandled. - **`.changeset/31601-ready-cluster-driver-health.md`** — `@objectstack/runtime` and `@objectstack/service-cluster` `patch`, both declaration lines, who sees it. No test was forced to change. ## Tests On head `04ca432f` (the merge of `origin/main`); the last commit `ce30cfbf` adds one collection-time import to the `/ready` test file (for `check:test-source-alias`), and that file was re-run on `ce30cfbf`: 41 passed. | Command | Result | |:--|:--| | `pnpm --filter @objectstack/runtime test` | 563 files, 9047 passed, 19 skipped | | `pnpm --filter @objectstack/runtime typecheck` | exit 0 (after building `@objectstack/driver-turso`, a devDependency outside the runtime closure: `check:test-typecheck` first read `TS2307` on `organization-ownership-boot-guard.turso-remote.kernel.test.ts`, a missing dist, not this diff) | | `pnpm --filter @objectstack/service-cluster test` / `typecheck` | 9 files, 123 passed / exit 0 | | `pnpm --filter @objectstack/service-cluster-redis test` | 3 files, 39 passed | | `pnpm exec eslint --no-inline-config packages/runtime` / `packages/services/service-cluster` | exit 0 / exit 0 (neither package has a `lint` script) | | `pnpm --filter @objectstack/dogfood test` (lock) | `04ca432f` · `VERDICT command-exit 0` · Test Files 440 passed, 3 skipped (443) · Tests 3652 passed, 41 skipped (3693). Not re-run on `ce30cfbf`: that commit touches only a runtime test file, which dogfood does not load. | <details><summary>Real Redis smoke: a live redis-server and the real ioredis driver behind /ready</summary> A one-off script (not committed): `defineCluster({ driver: 'redis', url })` from the built `@objectstack/service-cluster-redis`, the built `HttpDispatcher`, a fresh dispatcher per probe, a `redis-server` this run started and stopped. ```text OS_CLUSTER_REPLICAS=3 redis UP : 200 {"status":"ready","state":"running","cluster":{"driver":"redis","reachable":true}} (21 ms) redis-server stopped redis DOWN #1: 200 {"status":"degraded","state":"running","cluster":{"driver":"redis","reachable":false,"reason":"ping-failed"}} (188 ms) redis DOWN #2: 200 {"status":"degraded","state":"running","cluster":{"driver":"redis","reachable":false,"reason":"timeout"}} (501 ms) redis BACK : 200 {"status":"ready","state":"running","cluster":{"driver":"redis","reachable":true}} (2 ms) no declaration (single node), same sequence redis UP : 200 {"status":"ready","state":"running"} (14 ms) redis DOWN #1: 200 {"status":"ready","state":"running"} (1 ms) redis DOWN #2: 200 {"status":"ready","state":"running"} (1 ms) redis BACK : 200 {"status":"ready","state":"running"} (1 ms) ``` The first probe after the stop reads `ping-failed` (ioredis gives up after `maxRetriesPerRequest: 3`); later ones read `timeout` (the client queues while reconnecting) and are bounded at 500 ms. Recovery needs no restart. </details> <details><summary>Ablations (each through scripts/ablation-replace.mjs; restore proven by blob == HEAD and an empty git diff HEAD)</summary> The runtime test imports `./http-dispatcher.js` from source and the service-cluster test imports `./driver-health.js` from source, so no leg needed a rebuild or a dist preflight. | Leg | Mutation | Result | |:--|:--|:--| | A1 R3 declaration half | `if (declaresMultiNode(…))` → `if (true)` | red: 3 single-node pins (`no declaration … no ping`, `OS_CLUSTER_REPLICAS=1`, `#13408 body, no cluster key`) | | A2 D4 not 503 | unreachable store answers `503 Cluster unavailable` | red: `Redis down: 200 — not 503 …` | | A3 SIGTERM / booting 503 | probe the cluster before the kernel-state 503 and fold it into `details` | red: all four `kernel idle/initializing/stopping/stopped` pins | | A4 primary 503 | probe the cluster on the primary-datasource drain branch | red: `… byte-identical 503, and never probes Redis` (`expected "vi.fn()" to not be called at all, but actually been called 1 times`). The first attempt was refused by the tool (the replacement contained the anchor, anchor count 1 → 1) and ran no test; re-run with a different anchor. | | A5 no-probe never healthy | `no-probe` → `reachable: true` | red: `reads an offered client with no ping() as no-probe — never as healthy` | | A6 address never leaks | nested: keep the client error, serve its message as `reason` | red: both `ping-failed` pins (the closed-reason `toEqual` reds before the `not.toContain('10.0.0.7')` line is reached) | After the run: `git diff HEAD` empty, `git status --porcelain` clean apart from the then-uncommitted changeset. </details> ## Gates Final round on head `ce30cfbf`: `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` derived 71 commands (change set: 6 paths vs merge base `1a20cbc84`); every one ran once and exited 0. `dispatch-gates --ran`: `71 derived famil(ies) accounted for — 71 run, 0 NOT-MEASURED` (exit 0). An earlier round on `04ca432f` read three non-zero exits, all cleared: `check-plugin-teardown-shape --self-test` exit 3 (its pinned fixture commit was absent from the shallow clone; fetched, then exit 0), `check:dual-build-cjs-loads` exit 3 (`Run pnpm build first`: dists outside the runtime closure; exit 0 after `turbo run build --filter='./packages/**'`), and `check:test-source-alias` exit 1 (the edited `/ready` test reaches a lazy `@objectstack/metadata` import; fixed in `ce30cfbf` with the collection-time import the gate prescribes). <details><summary>The 71 commands and their exits on ce30cfb</summary> ```text node scripts/check-adr-0087-registration.mjs --base origin/main :: exit 0 node scripts/check-adr-0087-registration.mjs --self-test :: exit 0 node scripts/check-changeset-no-major.mjs --base origin/main :: exit 0 node scripts/check-closing-keyword-parity.mjs :: exit 0 node scripts/check-closing-keyword-parity.mjs --self-test :: exit 0 node scripts/check-comment-mask-adoption.mjs :: exit 0 node scripts/check-comment-mask-adoption.mjs --self-test :: exit 0 node scripts/check-comment-mask-corpus.mjs :: exit 0 node scripts/check-dts-emitted.mjs --self-test :: exit 0 node scripts/check-empty-changeset.mjs --base origin/main :: exit 0 node scripts/check-empty-changeset.mjs --self-test :: exit 0 node scripts/check-issue-citations.mjs :: exit 0 node scripts/check-plugin-teardown-shape.mjs :: exit 0 node scripts/check-plugin-teardown-shape.mjs --self-test :: exit 0 node scripts/check-tenant-audit-census.mjs :: exit 0 node scripts/check-tenant-audit-census.mjs --self-test :: exit 0 node scripts/ci/assert-turbo-remote-cache.mjs --self-test :: exit 0 node scripts/ci/plan-test-core-shards.mjs --self-test :: exit 0 node scripts/ci/prune-turbo-cache.mjs --self-test :: exit 0 node scripts/ci/snapshot-next.mjs --self-test :: exit 0 node scripts/docs-audit/check-affected-docs.mjs :: exit 0 node scripts/docs-audit/check-drift-comment.mjs :: exit 0 node scripts/pm/release-rehearsal-clone.mjs --self-test :: exit 0 node scripts/release-pending-publish.mjs --self-test :: exit 0 pnpm --filter @objectstack/spec run check:duration-unit-keys :: exit 0 pnpm --filter @objectstack/spec run check:error-code-provenance :: exit 0 pnpm check:adr-symbol-anchors :: exit 0 pnpm check:changeset-gate-self-tests :: exit 0 pnpm check:ci-filter-parity :: exit 0 pnpm check:closed-ceremony-text :: exit 0 pnpm check:cross-package-test-inputs :: exit 0 pnpm check:dispatcher-error-vocabulary :: exit 0 pnpm check:doc-authoring :: exit 0 pnpm check:driver-memory-census :: exit 0 pnpm check:dts-closure :: exit 0 pnpm check:dual-build-cjs-loads :: exit 0 pnpm check:engine-double-contract :: exit 0 pnpm check:error-status-conformance :: exit 0 pnpm check:gitlink-declared :: exit 0 pnpm check:init-service-contract :: exit 0 pnpm check:issue-citations :: exit 0 pnpm check:keyed-text-bounds :: exit 0 pnpm check:lean-entry-closure :: exit 0 pnpm check:logger-receiver-detach :: exit 0 pnpm check:nul-bytes :: exit 0 pnpm check:object-def-param-keys :: exit 0 pnpm check:objectql-double-limit :: exit 0 pnpm check:optional-error-sink :: exit 0 pnpm check:org-identifier :: exit 0 pnpm check:package-naming :: exit 0 pnpm check:page-declaration-shape :: exit 0 pnpm check:platform-object-tenancy-census :: exit 0 pnpm check:published-files :: exit 0 pnpm check:query-options-erasure :: exit 0 pnpm check:refd-timer-probe :: exit 0 pnpm check:registry-log-declared :: exit 0 pnpm check:rest-log-spy-declared :: exit 0 pnpm check:settings-bind-window :: exit 0 pnpm check:slot-lookup :: exit 0 pnpm check:sourcemap-no-sources-content :: exit 0 pnpm check:startup-registry-verdict :: exit 0 pnpm check:system-context-census :: exit 0 pnpm check:test-source-alias :: exit 0 pnpm check:tier-file-adoption :: exit 0 pnpm check:type-check-coverage :: exit 0 pnpm check:type-check-debt :: exit 0 pnpm check:undeclared-dep-imports :: exit 0 pnpm check:vouch-for-reference-callers :: exit 0 pnpm check:watch-hint-literal :: exit 0 pnpm check:where-matcher :: exit 0 pnpm check:wildcard-fallthrough :: exit 0 ``` </details> ## Acceptance notes - **F3a's seam.** #31596 (Redis disconnect without crashing, reconnect and resync; seat 1) was not merged and had no PR at the time of this card, so the probe pings the driver's offered connection itself, as the dispatch said. `probeClusterDriverHealth` lives in `@objectstack/service-cluster` beside `readClusterDriverConnection` so F3a can feed or replace it in one place; `/ready` only reads the `ClusterDriverHealth` shape. - **Observed, not changed (F3a's ground):** with Redis stopped, ioredis logs `[ioredis] Unhandled error event: Error: connect ECONNREFUSED …` on every reconnect attempt because the driver's client has no `error` listener. That is #31596's subject; this diff adds no listener. - **`status: 'degraded'` vs the #13408 secondary branch.** A degraded SECONDARY datasource still answers `status: 'ready'` with a `degraded` object (unchanged, single node included). The cluster reading uses `status: 'degraded'` as the card and D4 specify. When both hold on a declared deployment, `status` is `degraded` and both entries are present. Unifying the two was out of scope (it would change a single-node body). - **Docs not in the file surface.** `content/docs/protocol/kernel/lifecycle.mdx` (the `GET /ready` block) and `content/docs/deployment/self-hosting.mdx` (the probe table) do not yet mention the `cluster` entry; the claim's file surface excludes `content/docs/**`, so this is left to the seat. - **No published schema.** `/ready`'s body has no schema in `packages/spec` (searched before the first edit), so nothing in the spec moves. - The README "multi-node unsupported" line and the split-brain guard are untouched (R3). --- _Generated by [Claude Code](https://claude.ai/code/session_01ADHkmQwuHPnJPK768Mta39)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 1108509 commit 00205e1

8 files changed

Lines changed: 525 additions & 4 deletions

File tree

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
---
2+
'@objectstack/runtime': patch
3+
'@objectstack/service-cluster': patch
4+
---
5+
6+
feat(runtime): `/ready` reports the cluster driver's health on a declared multi-node deployment on the redis driver, answering `200` with `status: 'degraded'` while Redis is unreachable
7+
8+
Clause-②: yes (widening)
9+
Metadata-protocol: none
10+
11+
**Who sees this:** operators of a multi-replica deployment, one that declares itself multi-node (`OS_EXPECT_MULTI_NODE=true` or `OS_CLUSTER_REPLICAS` greater than 1) and runs the redis cluster driver (`OS_CLUSTER_DRIVER=redis`). Their readiness probe now shows when the cluster has lost coordination. Before this change a replica whose Redis had stopped kept answering `GET /ready` with a plain `200 { status: 'ready' }`, so nothing told the operator that locks, counters and cross-replica events had stopped working.
12+
13+
On such a deployment the `200` body gains a `cluster` entry: `{ driver: 'redis', reachable: true }` while the store answers, and `{ driver: 'redis', reachable: false, reason }` with `status: 'degraded'` while it does not. `reason` is one of `timeout`, `ping-failed`, `unexpected-reply` or `no-probe`. The store's address never appears in the body, because `/ready` answers anonymous callers. The status stays `200` on purpose: every replica loses a shared Redis at the same moment, so a `503` would take every replica out of the load balancer and turn a degraded cluster into a total outage while the database can still serve. Alert on `status: 'degraded'` or `cluster.reachable: false`.
14+
15+
The probe is one bounded `PING` (500 ms), cached for one second like the data-driver check, so frequent polling costs no extra round-trips.
16+
17+
Nothing else changes. A single-node deployment, one that runs the in-process `memory` driver, and one with no cluster service answer exactly the body they answered before, and their Redis is never probed. The booting and shutting-down `503` (graceful drain on SIGTERM) and the primary-datasource `503` are reached before the probe and are byte-identical.
18+
19+
`@objectstack/service-cluster` adds `probeClusterDriverHealth(cluster, driver, { timeoutMs })`, the reading `/ready` uses. It returns `undefined` when the cluster offers no connection for that driver and never throws.

‎content/docs/deployment/self-hosting.mdx‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -477,6 +477,24 @@ decrypt each other's secrets. All replicas must share the same
477477
`OS_SECRET_KEY`, `OS_AUTH_SECRET`, and database. See
478478
[Cluster](/docs/kernel/cluster).
479479

480+
**`/ready` reports the cluster driver's health once replicas are declared.**
481+
When the deployment declares itself multi-node — `OS_EXPECT_MULTI_NODE=true` or
482+
`OS_CLUSTER_REPLICAS` greater than 1; `OS_CLUSTER_DRIVER` alone is not a
483+
declaration — and runs the redis cluster driver, `/api/v1/ready` adds a
484+
`cluster` entry to its 200 body. While Redis answers it reads
485+
`{ driver: "redis", reachable: true }` and `status` stays `"ready"`. While it
486+
does not, the probe still answers **200**, with `status: "degraded"` and
487+
`cluster: { driver: "redis", reachable: false, reason }`, where `reason` is one
488+
of `timeout`, `ping-failed`, `unexpected-reply` or `no-probe`; the store's
489+
address never appears in the body. It is not a 503 on purpose: every replica
490+
loses a shared Redis at the same moment, so draining on it would take every
491+
replica out of the load balancer while the database can still serve. Wire your
492+
alerting to `status: "degraded"` (or `cluster.reachable: false`) rather than to
493+
the status code. The check is one `PING` bounded at 500 ms and cached ~1s, like
494+
the data-driver check, and recovery needs no restart. A single-node deployment
495+
sees no change: its body and status code are exactly what they were, and its
496+
Redis, if any, is never probed.
497+
480498
**Scheduled jobs' leader election has a window, not a deadline.** `cron` /
481499
`interval` / `once` schedules dedupe across replicas by holding a lock for
482500
the duration of each fire — the mutual-exclusion window is the handler's

‎content/docs/protocol/kernel/lifecycle.mdx‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -620,6 +620,12 @@ GET /ready
620620
→ 503 while still booting or shutting down
621621
→ 503 carrying { state, drivers: [...] } in error.details when a data driver
622622
stops answering
623+
→ 200 with a cluster entry when the deployment declares multi-node and runs
624+
the redis cluster driver:
625+
{ status: "ready", state, cluster: { driver: "redis", reachable: true } }
626+
{ status: "degraded", state,
627+
cluster: { driver: "redis", reachable: false, reason } }
628+
while Redis cannot be reached (still 200, never 503)
623629
```
624630

625631
The two probes answer deliberately different questions (#3756). `/health`
@@ -632,6 +638,23 @@ otherwise fail 100% of its requests. The readiness check fails **open**: a
632638
kernel with no data engine, or a probe that itself errors, still reads as ready
633639
rather than black-holing a working deployment.
634640

641+
On a deployment that declares itself multi-node (`OS_EXPECT_MULTI_NODE=true` or
642+
`OS_CLUSTER_REPLICAS` greater than 1) and runs the redis cluster driver, `/ready`
643+
also reports whether the replicas can still reach their shared Redis — the store
644+
their locks, counters and cross-replica events go through. The 200 body gains a
645+
`cluster` entry. While Redis answers it reads `reachable: true` and `status` stays
646+
`"ready"`; while it does not, `status` becomes `"degraded"`, `reachable` is
647+
`false`, and `reason` is one of `timeout`, `ping-failed`, `unexpected-reply` or
648+
`no-probe`. The store's address never appears in the body. The answer stays
649+
**200**, deliberately: every replica loses a shared Redis at the same moment, so
650+
a 503 would take all of them out of rotation and turn a degraded cluster into a
651+
total outage while the database can still serve. Alert on `status: "degraded"`
652+
instead. The check is one `PING` bounded at 500 ms and memoized ~1s, and it runs
653+
only on the 200 paths: the booting / shutting-down 503 and the data-driver 503
654+
are answered before it, unchanged. A single node, a deployment on the in-process
655+
`memory` cluster driver, and one with no cluster service see exactly the body
656+
they saw before, and their Redis, if any, is never probed.
657+
635658
<Callout type="info">
636659
`GET /health` returns a compact liveness body (`status`, `timestamp`, `version`,
637660
`uptime`) and `GET /ready` returns readiness. The per-plugin health report

‎packages/runtime/src/http-dispatcher.ready.test.ts‎

Lines changed: 187 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,8 @@
1-
import { describe, it, expect, vi } from 'vitest';
1+
import { afterEach, describe, it, expect, vi } from 'vitest';
2+
// Paid at collection, not inside a case's clocked window: the dispatcher's
3+
// reach loads `@objectstack/metadata` lazily (`check:test-source-alias`).
4+
import '@objectstack/metadata';
5+
import { offerClusterDriverConnection } from '@objectstack/service-cluster';
26
import { HttpDispatcher, type HttpDispatcherResult } from './http-dispatcher.js';
37

48
function kernel(state: string, dataService?: unknown): any {
@@ -313,6 +317,188 @@ describe('HttpDispatcher — GET /ready primary-vs-secondary drain (#13408)', ()
313317
});
314318
});
315319

320+
// #31601 (child of cloud epic #3814) — a declared multi-node deployment on the
321+
// redis cluster driver reports the cluster store's reachability on `/ready`.
322+
//
323+
// R3 「同意:每张卡都只在"声明多副本并接上 Redis 驱动"时才生效」
324+
// PM decision D4: an unreachable store answers 200 `status: degraded`, not
325+
// 503 — every replica loses the store together, so a 503 would turn a
326+
// degraded cluster into a total outage while the database can still serve.
327+
describe('HttpDispatcher — GET /ready cluster driver health (#31601)', () => {
328+
/** A `cluster` service on the redis driver, offering a client whose ping is `ping`. */
329+
function redisCluster(ping: () => Promise<unknown>) {
330+
const cluster = { driver: 'redis', nodeId: 'web-1' };
331+
offerClusterDriverConnection(cluster, { driver: 'redis', client: { ping }, keyPrefix: 'os:' });
332+
return cluster;
333+
}
334+
const redisDown = () => vi.fn(async () => { throw new Error('connect ECONNREFUSED 10.0.0.7:6379'); });
335+
const redisUp = () => vi.fn(async () => 'PONG');
336+
337+
/** A host kernel serving `data` and `cluster`. */
338+
function clusterKernel(state: string, cluster: unknown, dataService?: unknown): any {
339+
return {
340+
getState: () => state,
341+
getService: (name: string) => (name === 'data' ? dataService : name === 'cluster' ? cluster : undefined),
342+
getServiceAsync: async () => undefined,
343+
};
344+
}
345+
const ready = (k: any) => new HttpDispatcher(k).dispatch('GET', '/ready', undefined, undefined, ctx);
346+
347+
afterEach(() => { vi.unstubAllEnvs(); });
348+
349+
describe('declared multi-node on the redis driver', () => {
350+
it('Redis down: 200 — not 503 — with status degraded and a cluster entry naming the driver unreachable', async () => {
351+
vi.stubEnv('OS_CLUSTER_REPLICAS', '3');
352+
const ping = redisDown();
353+
const res = await ready(clusterKernel('running', redisCluster(ping)));
354+
355+
expect(responseOf(res).status).toBe(200);
356+
expect(responseOf(res).body.data).toEqual({
357+
status: 'degraded',
358+
state: 'running',
359+
cluster: { driver: 'redis', reachable: false, reason: 'ping-failed' },
360+
});
361+
// The reading is served anonymously: the store's address stays out.
362+
expect(JSON.stringify(responseOf(res).body)).not.toContain('10.0.0.7');
363+
expect(ping).toHaveBeenCalledTimes(1);
364+
});
365+
366+
it('Redis up: 200 with status ready and a reachable cluster entry', async () => {
367+
vi.stubEnv('OS_EXPECT_MULTI_NODE', 'true');
368+
const res = await ready(clusterKernel('running', redisCluster(redisUp())));
369+
370+
expect(responseOf(res).status).toBe(200);
371+
expect(responseOf(res).body.data).toEqual({
372+
status: 'ready',
373+
state: 'running',
374+
cluster: { driver: 'redis', reachable: true },
375+
});
376+
});
377+
378+
it('Redis down beside a degraded SECONDARY datasource: both are reported, still 200', async () => {
379+
vi.stubEnv('OS_CLUSTER_REPLICAS', '2');
380+
const res = await ready(clusterKernel(
381+
'running',
382+
redisCluster(redisDown()),
383+
engineWithPrimary([{ driverName: 'pg', healthy: true }, { driverName: 'tenant_mongo', healthy: false }], primary('pg')),
384+
));
385+
386+
expect(responseOf(res).status).toBe(200);
387+
expect(responseOf(res).body.data).toEqual({
388+
status: 'degraded',
389+
state: 'running',
390+
degraded: { drivers: ['tenant_mongo'], primaryDatasource: 'pg' },
391+
cluster: { driver: 'redis', reachable: false, reason: 'ping-failed' },
392+
});
393+
});
394+
395+
it('the primary datasource going down still answers the byte-identical 503, and never probes Redis', async () => {
396+
vi.stubEnv('OS_CLUSTER_REPLICAS', '3');
397+
const ping = redisDown();
398+
const res = await ready(clusterKernel(
399+
'running',
400+
redisCluster(ping),
401+
engineWithPrimary([{ driverName: 'pg', healthy: false }], primary('pg')),
402+
));
403+
404+
expect(responseOf(res).status).toBe(503);
405+
expect(responseOf(res).body.error.message).toBe('Data driver unavailable');
406+
expect(responseOf(res).body.error.details).toEqual({ state: 'running', drivers: ['pg'] });
407+
expect(ping).not.toHaveBeenCalled();
408+
});
409+
410+
it('the cluster probe is memoized with the same TTL — k8s polls must not each round-trip to Redis', async () => {
411+
vi.stubEnv('OS_CLUSTER_REPLICAS', '3');
412+
const ping = redisDown();
413+
const dispatcher = new HttpDispatcher(clusterKernel('running', redisCluster(ping)));
414+
415+
await dispatcher.dispatch('GET', '/ready', undefined, undefined, ctx);
416+
await dispatcher.dispatch('GET', '/ready', undefined, undefined, ctx);
417+
await dispatcher.dispatch('GET', '/ready', undefined, undefined, ctx);
418+
419+
expect(ping).toHaveBeenCalledTimes(1);
420+
});
421+
});
422+
423+
// ⛔ The stopping (SIGTERM) / booting 503 — the graceful-drain path — is
424+
// reached before any probe, so a declared deployment with Redis down answers
425+
// it exactly as a single node does.
426+
it.each(['idle', 'initializing', 'stopping', 'stopped'])(
427+
'kernel %s: the 503 envelope is identical to the single-node one, and Redis is never probed',
428+
async (state) => {
429+
const single = responseOf(await ready(kernel(state)));
430+
vi.stubEnv('OS_CLUSTER_REPLICAS', '3');
431+
const ping = redisDown();
432+
const declared = responseOf(await ready(clusterKernel(state, redisCluster(ping))));
433+
434+
expect(declared.status).toBe(503);
435+
expect(declared.body.error.message).toBe('Service not ready');
436+
expect(declared.body).toEqual(single.body);
437+
expect(declared.body.error.details).toEqual({ state });
438+
expect(ping).not.toHaveBeenCalled();
439+
},
440+
);
441+
442+
// ⛔ R3: nothing changes unless BOTH halves hold — the declaration AND the
443+
// redis driver. Each case below is missing one half; its body must be the
444+
// pre-#31601 body to the byte, and its cluster store is never pinged.
445+
describe('single node, or no redis driver: body and status unchanged', () => {
446+
it('single node (no declaration) with a redis cluster whose store is DOWN: the old body, no ping', async () => {
447+
const ping = redisDown();
448+
const res = await ready(clusterKernel('running', redisCluster(ping)));
449+
450+
expect(responseOf(res).status).toBe(200);
451+
expect(JSON.stringify(responseOf(res).body.data)).toBe(JSON.stringify({ status: 'ready', state: 'running' }));
452+
expect(ping).not.toHaveBeenCalled();
453+
});
454+
455+
it('single node: OS_CLUSTER_REPLICAS=1 is not a declaration', async () => {
456+
vi.stubEnv('OS_CLUSTER_REPLICAS', '1');
457+
const ping = redisDown();
458+
const res = await ready(clusterKernel('running', redisCluster(ping)));
459+
460+
expect(responseOf(res).body.data).toEqual({ status: 'ready', state: 'running' });
461+
expect(ping).not.toHaveBeenCalled();
462+
});
463+
464+
it('declared, but the cluster runs the in-process memory driver (guard escape hatch): the old body', async () => {
465+
vi.stubEnv('OS_CLUSTER_REPLICAS', '3');
466+
const res = await ready(clusterKernel('running', { driver: 'memory', nodeId: 'web-1' }));
467+
468+
expect(responseOf(res).status).toBe(200);
469+
expect(responseOf(res).body.data).toEqual({ status: 'ready', state: 'running' });
470+
});
471+
472+
it('declared, but no cluster service at all — or a lookup that throws: the old body', async () => {
473+
vi.stubEnv('OS_CLUSTER_REPLICAS', '3');
474+
const none = await ready(clusterKernel('running', undefined));
475+
expect(responseOf(none).body.data).toEqual({ status: 'ready', state: 'running' });
476+
477+
const throwing = await ready({
478+
getState: () => 'running',
479+
getService: (name: string) => { if (name === 'cluster') throw new Error('not registered'); return undefined; },
480+
getServiceAsync: async () => undefined,
481+
});
482+
expect(responseOf(throwing).status).toBe(200);
483+
expect(responseOf(throwing).body.data).toEqual({ status: 'ready', state: 'running' });
484+
});
485+
486+
it('single node with a degraded secondary: the #13408 body, no cluster key', async () => {
487+
const res = await ready(clusterKernel(
488+
'running',
489+
redisCluster(redisDown()),
490+
engineWithPrimary([{ driverName: 'pg', healthy: true }, { driverName: 'tenant_mongo', healthy: false }], primary('pg')),
491+
));
492+
493+
expect(responseOf(res).body.data).toEqual({
494+
status: 'ready',
495+
state: 'running',
496+
degraded: { drivers: ['tenant_mongo'], primaryDatasource: 'pg' },
497+
});
498+
});
499+
});
500+
});
501+
316502
describe('HttpDispatcher — GET /health liveness probe', () => {
317503
// framework#3756: liveness must NOT check the database. Its failure makes the
318504
// orchestrator restart the pod, which cannot fix an unreachable database but

0 commit comments

Comments
 (0)