Skip to content

Commit 8aaa69f

Browse files
feat(cloud-connection)!: refuse marketplace install-local when replicas are declared on the Redis driver (#31712)
<!-- claude-projects-attribution: {"github_login":"os-steve"} --> _Requested by **Steve** · [project thread](https://claude.ai/code/project/chan_01FdL1XeUyvVNxLreh3F4Qi9?thread=cmsg_01FdL1XeUyvVNxLreh3F4Qi968N7B64dDN7Noq8Fj3QSME)_ Fixes #31602 Maintainer ruling R6 on cloud#3814 (2026-11-08 14:16 UTC): 「多副本时拒绝」. Claim 6727409308 (`domain:server#4`, card-thread session_01ADHkmPnqZD15YnBkNbm1jJ). Clause-②: yes (narrowing) — narrowed, measured on head `6f4c42d9` and, for base, by the ablation that removes the new guard (equivalent to `main` `5d1382a5` on these doors): with `OS_CLUSTER_REPLICAS=3` (or `OS_EXPECT_MULTI_NODE=true`) and a composed `cluster` service on the `redis` driver, `POST /api/v1/marketplace/install-local` answered `200` with `hotLoaded: true` and wrote the ledger on base, and `DELETE /api/v1/marketplace/install-local/:manifestId` answered `200` and withdrew the package; on head both answer `409 MULTI_REPLICA_PACKAGE_CHANGE_REFUSED` and nothing is written, registered or withdrawn. Every other topology is unchanged. Widened: two published faces gain members, `ERROR_CODE_LEDGER` (`@objectstack/spec`) registers `MULTI_REPLICA_PACKAGE_CHANGE_REFUSED` under `@objectstack/cloud-connection` (the ledger header's "Registering a new code" rule), and the `@objectstack/service-cluster` root exports `declaresMultiReplicaOnRedis`, `composedClusterDriver`, `MULTI_REPLICA_CLUSTER_DRIVER` and `MULTI_REPLICA_PACKAGE_CHANGE_REFUSED`; nothing is removed or renamed. This PR takes a `CONTRACT_REVIEW_TIER` review (PASS, 6728523407). Metadata-protocol: none **Shared predicate (card item 4, for #31599):** `declaresMultiReplicaOnRedis(env, clusterDriver)` in `@objectstack/service-cluster` (`packages/services/service-cluster/src/multi-replica.ts`), with `composedClusterDriver(services)` to read the driver from a kernel context and the code constant `MULTI_REPLICA_PACKAGE_CHANGE_REFUSED`. `@objectstack/runtime` already depends on `@objectstack/service-cluster`, so the runtime package doors import it from there. ## 维护者速读 - 改了什么:部署声明了多副本(`OS_CLUSTER_REPLICAS` 大于 1 或 `OS_EXPECT_MULTI_NODE=true`)且集群驱动是 Redis 时,市场 install-local 的安装(POST)和卸载(DELETE)直接拒绝,返回 409 `MULTI_REPLICA_PACKAGE_CHANGE_REFUSED`,提示在维护窗口里跑安装脚本、再重启所有副本;不写账本、不热加载、不撤包。 - 为什么改:这两个门只改处理请求的那一个副本(它自己 `OS_HOME` 下的账本和它自己的内核),其他副本永远看不到,而且每个副本启动时只读自己的账本,所以分叉是永久的。维护者 R6 裁决「多副本时拒绝」。 - 风险与回滚:只在"声明多副本 + Redis 驱动"时生效;单节点、内存驱动、其他驱动、没有集群服务时字节不变,split-brain 守卫没动。回滚即 revert 本 PR。 - 先看:`packages/cloud-connection/src/marketplace-install-local-plugin.ts` 的 `refuseMultiReplicaPackageChange` 及两个门里的调用点;`packages/services/service-cluster/src/multi-replica.ts`(共享谓词);`error-code-ledger.zod.ts` 新增的一行和一条 provenance 豁免。 - 两分钟验证:`pnpm --filter @objectstack/cloud-connection exec vitest run src/marketplace-install-local-register-refusal.test.ts` —— 预期 27 passed,其中 R6 一组 8 个(4 个拒绝、4 个对照)。 - 需要你做的:无,席位落地(新错误码一行走 CONTRACT_REVIEW_TIER 复核)。 ## What changes - **`packages/services/service-cluster/src/multi-replica.ts`** (new) — the one condition every cloud#3814 card keys on: `declaresMultiReplicaOnRedis(env, clusterDriver)` = `declaresMultiNode(env)` (the split-brain guard's own reading, untouched) AND the driver is `MULTI_REPLICA_CLUSTER_DRIVER` (`'redis'`). `composedClusterDriver(services)` reads the composed `cluster` service's `driver` (the `IClusterService.driver` contract field) and never throws. `MULTI_REPLICA_PACKAGE_CHANGE_REFUSED` spells the code once, so every door refusing a package change under the condition answers the same code. Exported from the package root; unit pins in `multi-replica.test.ts` (new file: no harness exists for this module). - **`packages/cloud-connection/src/marketplace-install-local-plugin.ts`** — `refuseMultiReplicaPackageChange(c, ctx, subject)` answers `409` with the code and the recipe ("run the install script, then restart every replica"). Called in `handleInstall` right after `requireInstallCapability` (before the body is read, so before the cloud fetch, the ledger, `manifest.register` and `syncSchemas`) and in `handleUninstall` right after admission (before the ledger lookup, the guards and the withdrawal). The driver is read per request, where it is used. The grant path (`installForGrant`) calls `handleInstall`, so it is covered too. - **`packages/cloud-connection/src/connection-credential-ledger.ts`** — `sharesCloudBindingCredential` now delegates to `declaresMultiReplicaOnRedis` and `CLOUD_BINDING_SHARED_CLUSTER_DRIVER` aliases `MULTI_REPLICA_CLUSTER_DRIVER`: same answers (its suite is green unchanged), one predicate for the epic. - **`packages/spec/src/api/error-code-ledger.zod.ts`** — `MULTI_REPLICA_PACKAGE_CHANGE_REFUSED` registered under `@objectstack/cloud-connection` (the door), plus one `PROVENANCE_WAIVERS` entry for `@objectstack/service-cluster` (the shared constant one package over, the `UNIQUE_SCOPE_CONFIRMATION_REQUIRED` precedent). The claim's surface says "one row"; the waiver is the provenance gate's required companion for a constant defined one package over the door. - **`content/docs/references/api/{contract,automation-api,error-code-ledger}.mdx`** — regenerated (`gen:schema && gen:docs`) for the new enum member. - **`packages/cloud-connection/src/marketplace-install-local-register-refusal.test.ts`** — the shared real-engine install-local harness (AGENTS.md shared-harness rule): `bootPlugin` takes an optional `clusterDriver` that composes a `cluster` service; a new top-level `describe` pins the refusal. No existing test was changed. - **`.changeset/31602-install-local-multi-replica-refusal.md`** — `patch` for `@objectstack/cloud-connection`, `@objectstack/service-cluster`, `@objectstack/spec`; both declaration lines; `adr-0087: not-required (no-migration-prescription)`; who sees it. ## Tests The new describe, on a real `ObjectKernel` + `ObjectQLPlugin` and a real on-disk ledger: - for each declaration (`OS_CLUSTER_REPLICAS=3`, `OS_EXPECT_MULTI_NODE=true`) on the `redis` driver: `POST` (inline branch and cloud branch) answers `409 MULTI_REPLICA_PACKAGE_CHANGE_REFUSED` in the declared envelope, the ledger stays empty, the registry holds neither package, and the cloud leg is never fetched; `DELETE` of a package installed on a single node and rehydrated after a restart with replicas declared answers the same `409`, the ledger still holds it and the registry still serves it; - CONTROLs, install then uninstall both `200` with the ledger and registry moving as before: a single node on `redis`, `OS_CLUSTER_REPLICAS=1` on `redis`, declared replicas on the in-process driver (with the split-brain guard's escape hatch), declared replicas with no cluster service composed. ## Acceptance notes - **The recipe's wording.** The message names the ruling's recipe verbatim ("run the install script, then restart every replica"). This repository has no deployment install script to point at by name; if the seat wants a concrete command, the wording is one line in `refuseMultiReplicaPackageChange`. - **Dry run is refused too.** `POST` with `dryRun: true` writes nothing, but the guard sits before the body is read, so it is refused like any install: its plan reads the serving replica's own ledger, which is the per-replica state the refusal is about. - **Sibling doors not in this card.** `POST …/:manifestId/reseed` and `…/purge-sample-data` write rows to the shared database (every replica sees them) and are not refused. The runtime package doors (create/ensure customization, enable, disable, runtime install/uninstall) are #31599, which imports this predicate. - **A second reader of the same condition stays separate.** `@objectstack/service-cache` (`cache-service-plugin.ts`) reads `declaresMultiNode()` plus its own driver check for the shared Redis cache; it is not moved onto the new predicate here. ## Verification All on `6f4c42d9` unless stated (the feature commits were cut on `5d1382a5`; `origin/main` merged once at `03fed7a5`, 5 commits, none in these packages; `6f4c42d9` adds only the regenerated references; head `485e8f99` changes only the changeset's prose, after the contract review: the `Clause-②:` line now reads `yes (narrowing)` and names both halves, and the Now bullet says a dry run is refused too. On `485e8f99` the changeset gates were re-run, each exit 0: `check-adr-0087-registration`, `check-changeset-no-major`, `check-empty-changeset`, `check:changeset-gate-self-tests`, `check:nul-bytes`): <details><summary>Suites (os-verify-lock, every VERDICT command-exit 0)</summary> - `pnpm --filter @objectstack/service-cluster test` + `typecheck`: 9 files / 120 tests (`03fed7a5`). - `pnpm --filter @objectstack/cloud-connection test`: 80 files / 1116 tests; `typecheck` exit 0 (`03fed7a5`). - `pnpm --filter @objectstack/spec test`: 822 files / 26034 passed, 1 todo; `typecheck` exit 0 (`03fed7a5`). - `pnpm --filter @objectstack/dogfood test` (full battery, narrowing arm): 440 passed, 3 skipped files / 3652 passed, 41 skipped tests (`03fed7a5`). - Lint (`eslint --no-inline-config`) on `service-cluster`, `cloud-connection`, `spec` sources: exit 0 each. </details> <details><summary>Ablation (negative pins)</summary> `scripts/ablation-replace.mjs` on `51cac0fe` (fix committed first), marker `ABLATED_31602`: the guard's condition `|| 'ABLATED_31602'`, so `refuseMultiReplicaPackageChange` always returns `null` (the base behaviour). The test file imports the plugin from `src`, so no rebuild leg. Result: **4 of 27 red** — exactly the four refusal cases (POST and DELETE, both declarations), each `expected 200 to be 409` with the base answer in the message (`{"success":true,"data":{"manifestId":"com.acme.dispatch",…,"hotLoaded":true,…}}` for POST; the withdrawal answer for DELETE); the four CONTROLs and the 19 existing cases green. Restored: blob `4f8e111f` == HEAD, `git diff HEAD` empty. </details> <details><summary>Derived gates</summary> `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` on `6f4c42d9`: 120 families, each run and recorded with its exit code; `--ran` reconciliation: "120 derived famil(ies) accounted for — 120 run, 0 NOT-MEASURED". All 120 exit 0, among them `check:error-code-provenance`, `check:error-code-casing`, `check:dispatcher-error-vocabulary`, `check:error-status-conformance`, `check:route-envelope`, `check:startup-registry-verdict`, `check:api-surface`, `check:generated` and `check:docs`. A first pass on `03fed7a5` had `check:docs` red (the three references above were stale, now regenerated) and two exit-3 prerequisites (`check:plugin-teardown-shape --self-test` needed its pinned fixture commit fetched; `check:dual-build-cjs-loads` needed every package's dist, built with `turbo run build --filter='./packages/**'`); both are green on `6f4c42d9`. `pnpm --filter @objectstack/spec check:generated`: exit 0. </details> --- _Generated by [Claude Code](https://claude.ai/code/session_01ADHkmPnqZD15YnBkNbm1jJ)_ <!-- signed: domain:server#4 · os-steve · session_01ADHkmP8JJo4zF2sjTBRwnX --> --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent d655caf commit 8aaa69f

11 files changed

Lines changed: 329 additions & 10 deletions
Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
---
2+
'@objectstack/cloud-connection': patch
3+
'@objectstack/service-cluster': patch
4+
'@objectstack/spec': patch
5+
---
6+
7+
feat(cloud-connection)!: marketplace install-local refuses to install or uninstall a package when the deployment declares several replicas on the Redis cluster driver, instead of changing the one replica that served the request
8+
9+
Clause-②: yes (narrowing) — narrowed: `POST /api/v1/marketplace/install-local` and `DELETE /api/v1/marketplace/install-local/:manifestId` now answer `409 MULTI_REPLICA_PACKAGE_CHANGE_REFUSED`, right after the caller is admitted and before the body or the ledger is read, when the deployment declares several replicas (`OS_CLUSTER_REPLICAS` greater than 1, or `OS_EXPECT_MULTI_NODE=true`) and the composed cluster service runs the `redis` driver. Nothing is written, registered or withdrawn. On a single node, on the in-process driver or with no cluster service composed, both doors answer exactly as before. Widened: two published faces gain members, `ERROR_CODE_LEDGER` (`@objectstack/spec`) registers `MULTI_REPLICA_PACKAGE_CHANGE_REFUSED` under `@objectstack/cloud-connection`, and the `@objectstack/service-cluster` root exports `declaresMultiReplicaOnRedis`, `composedClusterDriver`, `MULTI_REPLICA_CLUSTER_DRIVER` and `MULTI_REPLICA_PACKAGE_CHANGE_REFUSED`; nothing is removed or renamed.
10+
Metadata-protocol: none
11+
12+
<!-- adr-0087: not-required (no-migration-prescription) No key of any schema is added, removed, renamed or re-shaped and no export is removed: a request that changed one replica's install ledger and kernel is now refused, with a new registered error code, only on a deployment that declares replicas on the Redis driver. There is nothing for a migration to rewrite and no tombstone. -->
13+
14+
**BREAKING** for a multi-replica deployment on the Redis cluster driver that installs or uninstalls packages through install-local (the Console's marketplace, `os package install`, or the HTTP door).
15+
16+
- **Before.** The install wrote the ledger under the serving process's own `OS_HOME` and hot-loaded the package into that process's kernel alone; the uninstall withdrew it from that one replica. The other replicas never saw the change, and each replica rehydrates its own ledger at boot, so the divergence was permanent.
17+
- **Now.** Both doors refuse with `MULTI_REPLICA_PACKAGE_CHANGE_REFUSED` (409) and name the maintenance-window recipe: run the install script, then restart every replica. A `POST` with `dryRun: true` is refused too: the refusal comes before the body is read, and the preview reads the serving replica's own ledger.
18+
- **Unchanged.** A single node, or replicas declared on any driver other than `redis`, install and uninstall as before. The split-brain guard is untouched: replicas declared on the in-process driver still refuse the boot unless `OS_ALLOW_MEMORY_CLUSTER_MULTINODE=true`.
19+
- **Who sees it.** An operator or administrator of a multi-replica deployment on the Redis cluster driver who installs or uninstalls a package at runtime.
20+
21+
`@objectstack/service-cluster` exports the condition, `declaresMultiReplicaOnRedis(env, clusterDriver)`, with `MULTI_REPLICA_CLUSTER_DRIVER` (`'redis'`), `composedClusterDriver(services)` and the code constant `MULTI_REPLICA_PACKAGE_CHANGE_REFUSED`, so every multi-replica refusal keys on one predicate; the cloud binding credential's `sharesCloudBindingCredential` now delegates to it, with the same answers. `@objectstack/spec` registers the code under `@objectstack/cloud-connection` in the error-code ledger.

‎content/docs/references/api/automation-api.mdx‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -417,7 +417,7 @@ const result = AutomationApiErrorCode.parse(data);
417417

418418
| Property | Type | Required | Description |
419419
| :--- | :--- | :--- | :--- |
420-
| **code** | `Enum<'VALIDATION_ERROR' \| 'INVALID_FIELD' \| 'MISSING_REQUIRED_FIELD' \| 'INVALID_FORMAT' \| 'VALUE_TOO_LONG' \| 'VALUE_TOO_SHORT' \| 'VALUE_OUT_OF_RANGE' \| … +364 more>` | ✅ | The inner refusal's code in its WIRE spelling - the ADR-0112 member a record save of the same write answers in `error.code` (e.g. `UNIQUE_VIOLATION`); the closed set `ApiErrorSchema.code` parses against |
420+
| **code** | `Enum<'VALIDATION_ERROR' \| 'INVALID_FIELD' \| 'MISSING_REQUIRED_FIELD' \| 'INVALID_FORMAT' \| 'VALUE_TOO_LONG' \| 'VALUE_TOO_SHORT' \| 'VALUE_OUT_OF_RANGE' \| … +365 more>` | ✅ | The inner refusal's code in its WIRE spelling - the ADR-0112 member a record save of the same write answers in `error.code` (e.g. `UNIQUE_VIOLATION`); the closed set `ApiErrorSchema.code` parses against |
421421
| **message** | `string` | ✅ | The inner refusal's own diagnostic text, verbatim, without the node and operation prefixes the run's failure text adds |
422422

423423
### Allowed Values: `FlowInnerRefusal.code`
@@ -640,6 +640,7 @@ const result = AutomationApiErrorCode.parse(data);
640640
* `METADATA_SCHEMA_INVALID`
641641
* `MIXED_ARTIFACT_COLLECTION_SHAPE`
642642
* `MONGODB_MULTI_TENANT_UNSUPPORTED`
643+
* `MULTI_REPLICA_PACKAGE_CHANGE_REFUSED`
643644
* `MULTI_UPDATE_HOOK_KEY_DIVERGENCE`
644645
* `NAMESPACE_CONFLICT`
645646
* `NAMESPACE_PREFIX`

‎content/docs/references/api/contract.mdx‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ const result = ApiErrorSchema.parse(data);
2828

2929
| Property | Type | Required | Description |
3030
| :--- | :--- | :--- | :--- |
31-
| **code** | `Enum<'VALIDATION_ERROR' \| 'INVALID_FIELD' \| 'MISSING_REQUIRED_FIELD' \| 'INVALID_FORMAT' \| 'VALUE_TOO_LONG' \| 'VALUE_TOO_SHORT' \| 'VALUE_OUT_OF_RANGE' \| … +364 more>` | ✅ | Error code (e.g. VALIDATION_ERROR; StandardErrorCode ∪ the ledger the serving side registers — ERROR_CODE_LEDGER for framework packages) |
31+
| **code** | `Enum<'VALIDATION_ERROR' \| 'INVALID_FIELD' \| 'MISSING_REQUIRED_FIELD' \| 'INVALID_FORMAT' \| 'VALUE_TOO_LONG' \| 'VALUE_TOO_SHORT' \| 'VALUE_OUT_OF_RANGE' \| … +365 more>` | ✅ | Error code (e.g. VALIDATION_ERROR; StandardErrorCode ∪ the ledger the serving side registers — ERROR_CODE_LEDGER for framework packages) |
3232
| **declaredCode** | `string` | optional | The producer-declared code, verbatim, when it is not a member of the closed `code` vocabulary — the open, author-authored channel (app-specific spellings; ADR-0112) |
3333
| **message** | `string` | ✅ | Readable error message |
3434
| **userMessage** | `string` | optional | Producer-marked user-facing refusal text, verbatim. Present exactly when the producer opted in at throw time; consumers render it to end users and keep their generic substitution for anything unmarked. Status-agnostic; never replaces `message`. |
@@ -258,6 +258,7 @@ const result = ApiErrorSchema.parse(data);
258258
* `METADATA_SCHEMA_INVALID`
259259
* `MIXED_ARTIFACT_COLLECTION_SHAPE`
260260
* `MONGODB_MULTI_TENANT_UNSUPPORTED`
261+
* `MULTI_REPLICA_PACKAGE_CHANGE_REFUSED`
261262
* `MULTI_UPDATE_HOOK_KEY_DIVERGENCE`
262263
* `NAMESPACE_CONFLICT`
263264
* `NAMESPACE_PREFIX`

‎content/docs/references/api/error-code-ledger.mdx‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -425,6 +425,7 @@ const result = ErrorCode.parse(data);
425425
* `METADATA_SCHEMA_INVALID`
426426
* `MIXED_ARTIFACT_COLLECTION_SHAPE`
427427
* `MONGODB_MULTI_TENANT_UNSUPPORTED`
428+
* `MULTI_REPLICA_PACKAGE_CHANGE_REFUSED`
428429
* `MULTI_UPDATE_HOOK_KEY_DIVERGENCE`
429430
* `NAMESPACE_CONFLICT`
430431
* `NAMESPACE_PREFIX`

‎packages/cloud-connection/src/connection-credential-ledger.ts‎

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -50,28 +50,31 @@
5050
*/
5151

5252
import { ConnectionCredentialStore, type StoredConnectionCredential } from './connection-credential-store.js';
53-
import { declaresMultiNode } from '@objectstack/service-cluster';
53+
import { declaresMultiReplicaOnRedis, MULTI_REPLICA_CLUSTER_DRIVER } from '@objectstack/service-cluster';
5454

5555
/**
5656
* The one cluster driver whose replicas share the binding credential. The
5757
* multi-replica cards take effect only over Redis (cloud#3814 R2/R3); the
58-
* entitlement half gates on the same driver.
58+
* entitlement half gates on the same driver. The cluster package's
59+
* {@link MULTI_REPLICA_CLUSTER_DRIVER}, the one every multi-replica card reads.
5960
*/
60-
export const CLOUD_BINDING_SHARED_CLUSTER_DRIVER = 'redis';
61+
export const CLOUD_BINDING_SHARED_CLUSTER_DRIVER = MULTI_REPLICA_CLUSTER_DRIVER;
6162

6263
/**
6364
* Does this deployment keep the binding credential in the shared row? Exactly
6465
* when replicas are declared (`declaresMultiNode`, the predicate the
6566
* entitlement half reads, so the two licence halves cannot drift) AND the
66-
* cluster driver is {@link CLOUD_BINDING_SHARED_CLUSTER_DRIVER}. The plugin
67-
* asks it with the cluster service's driver; the pre-boot read asks it with
68-
* `OS_CLUSTER_DRIVER`.
67+
* cluster driver is {@link CLOUD_BINDING_SHARED_CLUSTER_DRIVER} — the cluster
68+
* package's `declaresMultiReplicaOnRedis`, the one condition every
69+
* multi-replica card keys on, so this card and the package-change refusals
70+
* cannot disagree. The plugin asks it with the cluster service's driver; the
71+
* pre-boot read asks it with `OS_CLUSTER_DRIVER`.
6972
*/
7073
export function sharesCloudBindingCredential(
7174
env: Readonly<Record<string, string | undefined>>,
7275
clusterDriver: string | undefined,
7376
): boolean {
74-
return declaresMultiNode(env) && clusterDriver?.trim() === CLOUD_BINDING_SHARED_CLUSTER_DRIVER;
77+
return declaresMultiReplicaOnRedis(env, clusterDriver);
7578
}
7679

7780
/** The object a shared binding credential lives in (`sys-cloud-binding-credential.object.ts`). */

‎packages/cloud-connection/src/marketplace-install-local-plugin.ts‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -396,6 +396,7 @@ import {
396396
} from '@objectstack/metadata-protocol';
397397
import { resolveThrownHttpError, type ThrownHttpError } from '@objectstack/types';
398398
import { unknownQueryParamRefusalMessage } from '@objectstack/rest';
399+
import { composedClusterDriver, declaresMultiReplicaOnRedis, MULTI_REPLICA_PACKAGE_CHANGE_REFUSED } from '@objectstack/service-cluster';
399400

400401
/** What the auth service's in-process session read answers, as `AuthSessionApi` declares it. */
401402
type SessionRead = Awaited<ReturnType<NonNullable<AuthSessionApi['getSession']>>>;
@@ -1364,6 +1365,40 @@ function refuseHeldSession(c: any, principal: InstallPrincipal): Response | null
13641365
return c.json({ success: false, error: { code, message } }, 403);
13651366
}
13661367

1368+
/**
1369+
* [cloud#3814 R6] The refusal the install and uninstall doors give when this
1370+
* deployment declares several replicas over the Redis cluster driver
1371+
* (`declaresMultiReplicaOnRedis`, `@objectstack/service-cluster` — the one
1372+
* condition every multi-replica card keys on), right after the caller is
1373+
* admitted and before the body is read or anything is written.
1374+
*
1375+
* These doors change ONE replica: they write the install ledger under the
1376+
* serving process's own `OS_HOME` and load the package into (or withdraw it
1377+
* from) that process's kernel alone, and the other replicas never see it —
1378+
* a permanent divergence, since each replica rehydrates its own ledger at
1379+
* boot. So under declared replicas the change is refused, `409`, with
1380+
* `MULTI_REPLICA_PACKAGE_CHANGE_REFUSED` and the maintenance-window recipe;
1381+
* nothing is written, registered or withdrawn. The cluster service's driver
1382+
* is read here, per request, where the answer is used. `null` on a single
1383+
* node, the in-process driver or any other driver: the door proceeds exactly
1384+
* as before.
1385+
*/
1386+
function refuseMultiReplicaPackageChange(c: any, ctx: PluginContext, subject: 'Installing' | 'Uninstalling'): Response | null {
1387+
if (!declaresMultiReplicaOnRedis(process.env, composedClusterDriver(ctx))) return null;
1388+
const what = subject === 'Installing' ? 'the package' : 'the uninstall';
1389+
return c.json({
1390+
success: false,
1391+
error: {
1392+
code: MULTI_REPLICA_PACKAGE_CHANGE_REFUSED,
1393+
message: `${subject} a package through install-local was refused: this deployment declares several replicas `
1394+
+ '(OS_CLUSTER_REPLICAS greater than 1, or OS_EXPECT_MULTI_NODE=true) on the Redis cluster driver, and '
1395+
+ 'install-local changes only the replica that serves the request: its own install ledger and its own '
1396+
+ `kernel. The other replicas would never see ${what}. Make the change in a maintenance window instead: `
1397+
+ 'run the install script, then restart every replica. Nothing was installed, uninstalled or written.',
1398+
},
1399+
}, 409);
1400+
}
1401+
13671402
/**
13681403
* [#15204 W2] What an admitted mutating request carries past
13691404
* `requireInstallCapability`: the acting user, plus the two inputs ADR-0131
@@ -2094,6 +2129,13 @@ export class MarketplaceInstallLocalPlugin implements Plugin {
20942129
if (!admission.ok) return admission.response;
20952130
const userId = admission.userId;
20962131

2132+
// [cloud#3814 R6] Declared replicas over the Redis driver: this door
2133+
// would install into one replica only. Refused before the body is read.
2134+
{
2135+
const refused = refuseMultiReplicaPackageChange(c, ctx, 'Installing');
2136+
if (refused) return refused;
2137+
}
2138+
20972139
// [#23914] A declared JSON body that does not parse is refused 400,
20982140
// never read as no body (`./request-json-body.ts`).
20992141
const read = await readJsonRequestBody(c);
@@ -3433,6 +3475,13 @@ export class MarketplaceInstallLocalPlugin implements Plugin {
34333475
private handleUninstall = async (c: any, ctx: PluginContext): Promise<Response> => {
34343476
const admission = await this.requireInstallCapability(c, ctx, 'Uninstalling a package');
34353477
if (!admission.ok) return admission.response;
3478+
// [cloud#3814 R6] Declared replicas over the Redis driver: this door
3479+
// would withdraw the package from one replica only. Refused before
3480+
// the ledger is read.
3481+
{
3482+
const refused = refuseMultiReplicaPackageChange(c, ctx, 'Uninstalling');
3483+
if (refused) return refused;
3484+
}
34363485
const manifestId = String(c.req.param?.('manifestId') ?? c.req.params?.manifestId ?? '').trim();
34373486
if (!manifestId) {
34383487
return c.json({ success: false, error: { code: 'INVALID_REQUEST', message: 'manifestId path param required.' } }, 400);

‎packages/cloud-connection/src/marketplace-install-local-register-refusal.test.ts‎

Lines changed: 86 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,10 @@
4545
* - CONTROL, a register failure that declares no status (the runtime's own
4646
* fault): the lenient answer it had — `200`, `hotLoaded: false`, ledgered —
4747
* and the restart heals it: the next boot loads the package;
48+
* - (cloud#3814 R6) under declared replicas on the Redis cluster driver, the
49+
* install and uninstall doors refuse `409 MULTI_REPLICA_PACKAGE_CHANGE_REFUSED`
50+
* with nothing written, registered or withdrawn; on a single node, on the
51+
* in-process driver or with no cluster service they act as before;
4852
* - (#31566) a ledger entry an earlier build wrote for a register the engine
4953
* refuses: at boot nothing of it loads, and the listing marks it
5054
* `notLoaded: { code, message }` from the refusal; a healthy entry and a
@@ -185,7 +189,7 @@ function serveVersions(versions: Record<string, Bag>) {
185189
* restart. `failRegisterOf` makes the engine's register throw an undeclared
186190
* fault for that one package id.
187191
*/
188-
async function bootPlugin(failRegisterOf?: string, opts: { failSideEffectsOf?: string } = {}) {
192+
async function bootPlugin(failRegisterOf?: string, opts: { failSideEffectsOf?: string; clusterDriver?: string } = {}) {
189193
const kernel = new ObjectKernel({ logger: { level: 'silent' }, gracefulShutdown: false } as never);
190194
kernels.push(kernel);
191195
await kernel.use(new ObjectQLPlugin());
@@ -213,6 +217,8 @@ async function bootPlugin(failRegisterOf?: string, opts: { failSideEffectsOf?: s
213217
get manifests() { return (ql as unknown as { manifests: Map<string, unknown> }).manifests; },
214218
syncSchemas: async () => undefined,
215219
}),
220+
// [cloud#3814 R6] The composed cluster service, by its driver, when a case names one.
221+
...(opts.clusterDriver === undefined ? {} : { cluster: { driver: opts.clusterDriver } }),
216222
};
217223
const ctx = {
218224
hook: (e: string, h: () => Promise<void>) => hooks.set(e, h),
@@ -559,3 +565,82 @@ describe('a ledgered entry whose register the engine refuses at boot is listed n
559565
expect(item.withSampleData).toBe(false);
560566
});
561567
});
568+
569+
describe('install-local refuses a package change under declared replicas on the Redis driver (cloud#3814 R6)', () => {
570+
/** The two declarations the split-brain guard reads. */
571+
const DECLARATIONS = [
572+
{ name: 'OS_CLUSTER_REPLICAS=3', key: 'OS_CLUSTER_REPLICAS', value: '3' },
573+
{ name: 'OS_EXPECT_MULTI_NODE=true', key: 'OS_EXPECT_MULTI_NODE', value: 'true' },
574+
] as const;
575+
const HOLDER = `${HOLDER_ID}@1.0.0`;
576+
577+
function expectMultiReplicaRefusal(res: any, verb: 'Installing' | 'Uninstalling') {
578+
expect(res.status, JSON.stringify(res.payload)).toBe(409);
579+
expect(BaseResponseSchema.safeParse(res.payload).success).toBe(true);
580+
expect(envelopeViolations(res.payload)).toEqual([]);
581+
expect(ApiErrorSchema.safeParse(res.payload.error).success).toBe(true);
582+
expect(res.payload.success).toBe(false);
583+
expect(res.payload.error.code).toBe('MULTI_REPLICA_PACKAGE_CHANGE_REFUSED');
584+
// The first sentence names the door; the recipe is the remedy an operator acts on.
585+
expect(res.payload.error.message).toMatch(new RegExp(`^${verb} a package through install-local was refused: `));
586+
expect(res.payload.error.message).toContain('run the install script, then restart every replica');
587+
}
588+
589+
for (const d of DECLARATIONS) {
590+
it(`${d.name}: POST is refused with the code, nothing ledgered or registered (inline and cloud branches)`, async () => {
591+
vi.stubEnv(d.key, d.value);
592+
serveVersions(VERSIONS);
593+
const { ql, install, installFromCloud } = await bootPlugin(undefined, { clusterDriver: 'redis' });
594+
const fetchedAtBoot = vi.mocked(fetch).mock.calls.length;
595+
596+
expectMultiReplicaRefusal(await install({ manifest: holderVersion() }), 'Installing');
597+
expectMultiReplicaRefusal(await installFromCloud('v1-free'), 'Installing');
598+
599+
expect(ledger()).toEqual([]);
600+
expect(ql.registry.getPackage(HOLDER_ID)).toBeUndefined();
601+
expect(ql.registry.getPackage(INCOMING_ID)).toBeUndefined();
602+
// Refused before the body is read: the cloud leg was never asked.
603+
expect(vi.mocked(fetch).mock.calls.length).toBe(fetchedAtBoot);
604+
});
605+
606+
it(`${d.name}: DELETE is refused with the code, the package still ledgered and still served`, async () => {
607+
// Installed on a single node, then the deployment restarts with replicas declared.
608+
await (await bootPlugin(undefined, { clusterDriver: 'redis' })).installHolder();
609+
expect(ledger()).toEqual([HOLDER]);
610+
vi.stubEnv(d.key, d.value);
611+
const { ql, uninstall } = await bootPlugin(undefined, { clusterDriver: 'redis' });
612+
expect(ql.registry.getPackage(HOLDER_ID)?.manifest?.version).toBe('1.0.0');
613+
614+
expectMultiReplicaRefusal(await uninstall(HOLDER_ID), 'Uninstalling');
615+
616+
expect(ledger()).toEqual([HOLDER]);
617+
expect(ql.registry.getPackage(HOLDER_ID)?.manifest?.version).toBe('1.0.0');
618+
});
619+
}
620+
621+
const UNCHANGED = [
622+
{ title: 'a single node on the Redis driver', env: {}, clusterDriver: 'redis' },
623+
{ title: 'OS_CLUSTER_REPLICAS=1 on the Redis driver', env: { OS_CLUSTER_REPLICAS: '1' }, clusterDriver: 'redis' },
624+
{
625+
title: 'declared replicas on the in-process driver (the split-brain guard\'s escape hatch)',
626+
env: { OS_CLUSTER_REPLICAS: '3', OS_ALLOW_MEMORY_CLUSTER_MULTINODE: 'true' }, clusterDriver: 'memory',
627+
},
628+
{ title: 'declared replicas with no cluster service composed', env: { OS_CLUSTER_REPLICAS: '3' }, clusterDriver: undefined },
629+
] as const;
630+
631+
for (const u of UNCHANGED) {
632+
it(`CONTROL — ${u.title}: install and uninstall act as before`, async () => {
633+
for (const [k, v] of Object.entries(u.env)) vi.stubEnv(k, v);
634+
const { ql, installHolder, uninstall } = await bootPlugin(undefined, { clusterDriver: u.clusterDriver });
635+
636+
await installHolder();
637+
expect(ledger()).toEqual([HOLDER]);
638+
expect(ql.registry.getPackage(HOLDER_ID)?.manifest?.version).toBe('1.0.0');
639+
640+
const res = await uninstall(HOLDER_ID);
641+
expect(res.status, JSON.stringify(res.payload)).toBe(200);
642+
expect(ledger()).toEqual([]);
643+
expect(ql.registry.getPackage(HOLDER_ID)).toBeUndefined();
644+
});
645+
}
646+
});

0 commit comments

Comments
 (0)