fix(cloud-connection): every replica reads the cloud binding credential when several are declared (#31584)

<!-- claude-projects-attribution: {"github_login":"os-zhuang"} -->
_Requested by **Sam** · [project
thread](https://claude.ai/code/project/chan_01GsBogozeZG5sdpAARrN7HP?thread=cmsg_01GsBogozeZG5sdpAARrN7HPKXhgTmBFYjXgTdNcFMFqGb)_

Closes #31554

Clause-②: yes (widening)
Metadata-protocol: none

## 维护者速读

- **改了什么**：声明多副本（`OS_CLUSTER_REPLICAS` 大于 1 或
`OS_EXPECT_MULTI_NODE=true`）并且集群驱动是 Redis
时，云绑定凭据改存数据库里的一行加密记录（`sys_cloud_binding_credential`），所有副本读写同一行；另提供一个内核启动前的一次性读取，供启动前的多节点许可证检查使用。单机和其他驱动继续用原来的文件，路径、字节和权限都不变。
- **为什么改**：凭据原来只存在执行绑定那台副本的本地文件里，其余副本拿不到凭据，在线许可证检查报告没有控制平面，对云端的调用也不带凭据。
- **风险与回滚**：只在上面两个条件同时成立时生效，单机路径有逐字节回归测试；没有加密提供者时拒绝绑定，不会明文落库；启动前读取有 3
秒上限，失败时按未绑定启动，不会卡住启动。回滚即撤回本 PR，多副本部署会回到各自读文件（需要重新绑定一次）。
-
**先看**：`packages/cloud-connection/src/cloud-connection-plugin.ts`（`chooseLedger`、`door`
包装、bind/poll
写入失败分支）、`connection-credential-preboot.ts`（启动前读取的超时与关闭）、`connection-credential-ledger.ts`（读写、`adoptFrom`
迁移）。
- **两分钟验证**：`pnpm --filter @objectstack/cloud-connection exec vitest run
src/connection-credential-ledger.test.ts
src/connection-credential-store.test.ts
src/connection-credential-preboot.test.ts`，预期 38 个用例全绿，其中双进程用例显示 A 绑定后 B
立即带上凭据、A 解绑后 B 立即清空。
- **需要你做的**：无，席位落地。

## What changes

A self-hosted server kept its cloud binding credential only in the
working directory's `.objectstack/cloud-connection.json`. Under several
replicas, the bind ran on one replica and every other replica held no
credential.

- **New object `sys_cloud_binding_credential`**
(`packages/cloud-connection`): one row, keyed by `slot = 'runtime'`
under a global unique index. `runtime_token` is a `Field.secret`, so the
engine seals it through the host `ICryptoProvider` into `sys_secret`
(fail-closed with no provider), masks it on read and releases the handle
on update or delete. The object is engine-owned, tenant-free
(`systemFields.tenant: false`), `apiEnabled: false`, with `maxLength` on
every text column.
- **Where the credential lives is chosen once, in `init()`**, by
`sharesCloudBindingCredential(env, driver)`: the shared row exactly when
`declaresMultiNode(env)` holds (service-cluster's predicate, the one the
entitlement half in cloud#3880 reads) and the cluster driver is `redis`
(`CLOUD_BINDING_SHARED_CLUSTER_DRIVER`, per cloud#3814 R2/R3). Anything
else, including any other driver, keeps the file through
`fileCredentialLedger`, a pass-through over the unchanged
`ConnectionCredentialStore`. The object is registered with the
`manifest` service only on the shared path.
- **Reads**: on shared replicas every `/api/v1/cloud-connection/*` door
re-reads the row before it runs. The synchronous `cloud-binding` service
answers from a copy read at `kernel:ready` and re-read every 30 s
(`sharedCredentialRefreshMs`). A new `cloud-binding-credential` service
exposes an async `load()` that reads the row on demand, for readers that
can await (the EE licence check, after a pin).
- **Writes**: bind and unbind write that one row from any replica.
Unbind keeps the `runtimeId` residual, the same shape the file keeps. On
shared replicas a credential that cannot be sealed is refused to the
caller with `503 ENVIRONMENT_BIND_FAILED` and a message that names the
fix, instead of answering the bind as connected. The single node answers
as before.
- **Before the kernel**: `readCloudBindingCredentialBeforeBoot({
openDriver, cryptoProvider })` is for the host's config, ahead of the
multi-node licence gate, so a cold cluster can make its first-boot
exchange. It opens nothing unless the predicate above holds (reading
`OS_CLUSTER_DRIVER`, or a `clusterDriver` the host passes). Otherwise it
opens the one driver the host supplies and reads the row through a bare
ObjectQL engine with only this object registered. That is the same
`load()` the plugin uses, so the secret is dereferenced by the engine's
`resolveSecretField` and nothing else. It runs no schema sync and no
write. It initiates the disconnect on every path, including an open that
lands after the bound. Open, read and close share one deadline; the
close is waited for only with what is left (at least 50 ms), and on the
timeout path it is initiated, not awaited (`timeoutMs`, default
`PREBOOT_CREDENTIAL_TIMEOUT_MS` = 3000). Any failure answers `{ status:
'unavailable', reason }` and never throws, so the replica boots as an
unbound one does today.
- **Migration of an existing file**: the first shared boot moves the
file's credential into the row when the row is empty and logs the path.
The file is left in place and not read while replicas are declared.
Scaling back to one replica reads the file again; it does not follow
binds made meanwhile, so the docs say to disconnect and connect again
after scaling down.
- **Logging**: the credential is never logged, and the ledger scrubs it
from any engine error it re-throws. One `error` line, with consequence
and fix, when replicas are declared over Redis with no `manifest`
service (each replica keeps its own file), when the move fails, and when
a shared read fails (once per distinct failure).
- **Ledgers the new object needs**: `PLATFORM_OBJECTS_BY_PACKAGE` in
`@objectstack/spec`, the organization ownership inventory row in
`@objectstack/cli`, and the regenerated
`scripts/platform-object-tenancy-census.json`.
- **Docs**: `content/docs/deployment/cloud-connection.mdx` gains
"Several replicas".

## Why the database row and not the cluster KV

- The binding is a durable secret and the system of record. The `IKV`
contract says it is not a database, customer Redis may run without
persistence, and a flush must never disconnect the server.
- The row inherits the engine's secret-field seam: encryption at rest
with the platform crypto provider, masking on read, handle release, and
a fail-closed refusal without a provider. A KV entry would need its own
encryption and has none of that.
- No pubsub invalidation lane: the repo's rule is that the re-read
interval is the correctness bound and pubsub is only a hint. The doors
re-read on every request, so the only window is the synchronous
service's 30 s.
- The PM thread confirmed this route. The sibling cloud#3869 keeps its
entitlement cache in KV with a file fallback, by design.

## Tests

- **Two processes** (`connection-credential-ledger.test.ts`): two real
Node processes, each with its own working directory and credential file,
boot `CloudConnectionPlugin` from this checkout's source over one SQLite
database file and one crypto key, with `OS_CLUSTER_REPLICAS=2` and the
Redis cluster driver. A binds through its door. B's next door presents
the same bearer to a local control plane at once, and B's synchronous
service has it within the re-read interval. A unbinds, and B reads not
connected without calling the control plane, keeping the `runtimeId`
residual. Neither replica writes a credential file, the database file
never holds the bearer in plaintext, and neither replica's logs or
stderr contain it.
- **Single node byte-identical** (`connection-credential-store.test.ts`,
four topologies: no declaration over Redis, replicas declared over
`memory`, replicas declared over `postgres`, replicas declared with no
cluster service): bind/poll writes the exact bytes the store wrote
before this change, mode `0600`; nothing is registered with the manifest
service; the reader answers from the file; unbind leaves `{
runtimeToken: '', runtimeId }`.
- **Ledger over a real engine** (6 cases): round-trip with no plaintext
at rest; residual and rotation; a replica on another key refuses and
names `OS_SECRET_KEY`; no crypto provider refuses; no data engine
refuses; the file moves into the empty row once and never over a filled
one.
- **Plugin** (3 cases): the shared path registers the object, says so
once and moves the file before the first door; no manifest service logs
one `error` and keeps the file; a bind the row cannot seal answers 503
and stores nothing.
- **Pre-boot read** (`connection-credential-preboot.test.ts`, 16 cases,
including three that pin the one shared deadline: a hung open, a slow
open with a hung close, a hung read with a hung close): a real SQLite
file sealed by a replica's engine is read back through a fresh driver
and the same key, then disconnected; nothing bound reads as `null`; a
host-supplied `clusterDriver` is honoured. A database that never held
the row, another key, an open that throws, an open that lands after the
bound (answered at the bound, closed when it lands) and a hung read
(closed at the bound) are each `unavailable` without throwing, with no
credential in the reason. No declaration, `memory`, no driver,
`postgres` and an unknown driver each answer `not-shared` without
calling `openDriver`.
- **Ablations** (each restored by `scripts/ablation-replace.mjs`, blob
equal to HEAD, `git diff HEAD` empty): forcing `chooseLedger` to return
the file ledger turns the two-process case red (`expected '' to be
'oscc_dummy_31554_aaaa'`); removing the shared bind refusal turns its
case red (`expected 200 to be 503`); widening the predicate to any
non-`memory` driver turns the three non-Redis cases red; dropping the
close on the success path turns the four disconnect assertions red.

## Acceptance notes

- **File surface beyond the claim.** The claim names
`packages/cloud-connection/**` and the object that holds the row. The
diff also touches
`packages/spec/src/system/constants/platform-object-names.ts`,
`packages/cli/src/utils/organization-ownership-inventory.ts` and
`scripts/platform-object-tenancy-census.json`, because their conformance
tests fail for any new platform object not listed; plus the docs page,
the changeset and `pnpm-lock.yaml` for the new workspace dependencies.
- **Residual, not touched:** `marketplace-install-local-plugin.ts`
builds its own `ConnectionCredentialStore` and reads the file for
install-local's control-plane fetch. Install-local under several
replicas is ruled to refuse loudly on a separate card, so this PR leaves
it reading the file it read before.
- **Cloud half:** after the next pin bump, the host's
`objectstack.config.ts` calls `readCloudBindingCredentialBeforeBoot`
with its datasource driver and crypto provider ahead of the multi-node
gate, and the running licence check reads `cloud-binding-credential` or
the `cloud-binding` service.
- **Out of scope:** moving the single node's file under `OS_HOME` (today
it sits in the container layer) is a separate single-node bug that the
maintainer is being asked about.
- **Test budget:** the three credential suites together take about 7 s
of test time locally, under the 10 s line per file, so no budget header.

## Verification

All readings at head `786e5d7b`.

- `pnpm --filter @objectstack/cloud-connection exec vitest run`: Test
Files 79 passed (79), Tests 1081 passed (1081).
- `pnpm --filter @objectstack/cloud-connection typecheck` (`tsc
--noEmit` and `tsconfig.test.json`): clean, after a declaration build of
the dependency closure.
- `@objectstack/spec` `platform-object-names`,
`api-methods-batch-conformance` and `platform-tool-names` tests: 23/23.
`@objectstack/cli` `organization-ownership-inventory` test: 26/26.
- `eslint` on every changed source and test file: clean.

<details><summary>Derived gate union (`node
scripts/pm/dispatch-gates.mjs --commands`): 137 commands, 133 exit 0, 4
NOT MEASURED</summary>

- 125 exit 0 in one pass. The `@objectstack/spec` gates were then rerun
one at a time after `pnpm --filter @objectstack/spec build`
(declarations on), because this diff edits a spec source file: all 18
exit 0, including `check:api-surface`, `check:docs` and
`check:authorable-surface`.
- `check-plugin-teardown-shape --self-test` exited 3 on the shallow
checkout (its fixture commit was missing). It exits 0 after fetching
that one commit.
- Fixed during the run: `check:engine-double-contract` (the new
in-process engine fake now opens `update`/`delete` with the dispatch
asserts; ledger re-recorded with `--write`) and
`check:logger-receiver-detach` (the three `error` lines go through one
receiver-safe helper).
- NOT MEASURED, reason: prerequisite not met (exit 3), because they need
a whole-workspace build or the built CLI. These are
`check:dual-build-cjs-loads`, `check:i18n`, `check:i18n-coverage` and
`check:i18n-walk-parity`. CI runs all four.
- `node scripts/pm/dispatch-gates.mjs --ran`: 137 derived, 137 run.

</details>

## Human Minimum

- [x] 1. Schemas start as Zod; `packages/spec` holds no runtime logic.
- [x] 2. Every name I added uses the casing for its kind.
- [x] 3. No formatter ran; `eslint` passes on the changed files (the
repo-wide `pnpm lint` is CI's run).
- [x] 4. One issue, one branch off an up-to-date `main`, one pull
request.
- [x] 5. A change that publishes carries a changeset, and both
declaration lines are filled in here and in the changeset.
- [x] 6. `packages/spec` touched: its generated artifacts are
regenerated and committed.
- [x] 7. No hand edits to `content/docs/references/`,
`packages/*/CHANGELOG.md` or `content/docs/releases/`.
- [ ] 8. Title, body and comments are in English. (The 维护者速读 block is
Chinese by the dispatch contract.)
- [ ] The *Before Submitting* list in CONTRIBUTING.md is done.

---
_Generated by [Claude
Code](https://claude.ai/code/session_01QUs3BuscNGgAkKzuN3beuN)_


<!-- signed: epic:#3814#1 · os-zhuang · session_01QUs3BrVu5TqwtmuhsjdEvM
-->

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: objectstack-fleet[bot]

.changeset/31554-cloud-binding-credential-shared.md

@@ -0,0 +1,25 @@
+---
+'@objectstack/cloud-connection': patch
+'@objectstack/spec': patch
+'@objectstack/cli': patch
+---
+
+fix(cloud-connection): every replica reads the cloud binding credential when several are declared over a shared cluster driver
+
+Clause-②: yes (widening)
+Metadata-protocol: none
+
+A self-hosted server kept its cloud binding credential only in `<cwd>/.objectstack/cloud-connection.json`. With several replicas behind a load balancer, the bind ran on one replica and the others held no credential: their online licence checks reported no control plane, and their control-plane calls went without a bearer.
+
+**When several replicas are declared** (`OS_CLUSTER_REPLICAS` above 1, or `OS_EXPECT_MULTI_NODE=true`) **and the cluster driver is Redis** (`OS_CLUSTER_DRIVER=redis`), the credential now lives in the database, as the one row of the new `sys_cloud_binding_credential` object:
+
+- The row lives on the engine's default datasource, the same database as `sys_secret` (no datasource of its own, so a deployment that also registers a `cloud` datasource never splits the row from its secret). `runtime_token` is a secret field, sealed by the host crypto provider into `sys_secret`; the row holds only the handle. With no crypto provider the bind is refused with `503 ENVIRONMENT_BIND_FAILED` and a message naming the fix, never stored in clear text. A replica on another secret key refuses to read it and says to give every replica the same `OS_SECRET_KEY`.
+- Connect and disconnect work from any replica and write that one row. Every `/api/v1/cloud-connection/*` request re-reads it first; the synchronous `cloud-binding` service re-reads it every 30 seconds (`sharedCredentialRefreshMs`).
+- The row, not the cluster store, is the system of record: a cluster store flush, or one that does not persist, never disconnects the server.
+- **Before the kernel.** `readCloudBindingCredentialBeforeBoot({ openDriver, cryptoProvider })` reads the row once from the host's config, for the multi-node licence gate that runs before boot. It opens nothing unless replicas are declared over Redis; otherwise it opens the one driver the host supplies, reads the row through the engine's own secret-field path with no schema sync and no write, and initiates the disconnect on every path (also when the open lands late, the moment it lands). Open, read and close share one deadline, `timeoutMs` from the call (default 3000, `PREBOOT_CREDENTIAL_TIMEOUT_MS`): the close is waited for only with what is left of it, at least 50 ms, and on the timeout path the disconnect is initiated and not awaited. So the call returns within `timeoutMs` plus that 50 ms close floor, however long a disconnect hangs. Any failure answers `{ status: 'unavailable', reason }` and never throws, so the replica boots as an unbound one does today.
+- **Migration.** The first start with replicas declared moves an existing `cloud-connection.json` into the row when the row is empty, and logs the path it moved. The file is left in place and not read while replicas are declared. Scaling back to one replica reads the file again, which does not follow connects or disconnects made meanwhile: disconnect and connect again after scaling down.
+- Replicas declared over a shared driver with no data engine composed log one `error` at start (each replica keeps its own file; compose ObjectQL on every replica or declare one replica). A read of the row that fails later logs one `error` per distinct failure and keeps the binding last read.
+
+**A single node is unchanged**: no declaration, the `memory` driver or any driver other than Redis, or no cluster service keeps the same file, bytes and `0600` mode, and registers nothing.
+
+New exports from `@objectstack/cloud-connection`: `SysCloudBindingCredential`, `SharedConnectionCredentialLedger`, `SharedCredentialUnavailableError`, `fileCredentialLedger`, `CLOUD_BINDING_CREDENTIAL_OBJECT`, `CLOUD_BINDING_CREDENTIAL_SERVICE` (the `cloud-binding-credential` service, whose async `load()` reads the row on demand), `SHARED_CREDENTIAL_REFRESH_MS`, `CLOUD_BINDING_SHARED_CLUSTER_DRIVER`, `sharesCloudBindingCredential`, `readCloudBindingCredentialBeforeBoot`, `PREBOOT_CREDENTIAL_TIMEOUT_MS`, and the types `BindingCredentialEngine`, `CloudBindingCredentialReader`, `ConnectionCredentialLedger`, `CloudBindingCredentialBeforeBoot` and `CloudBindingCredentialBeforeBootOptions`. `@objectstack/spec` lists `sys_cloud_binding_credential` under `cloud-connection` in `PLATFORM_OBJECTS_BY_PACKAGE`, and `@objectstack/cli`'s organization ownership inventory records it as tenant-free (`systemFields.tenant: false`).

content/docs/deployment/cloud-connection.mdx

@@ -173,6 +173,55 @@ call, so a connection made while the server is running takes effect without a
 restart, and so does a disconnect. Keep the file on persistent storage and
 protect it like any other secret.
 
+### Several replicas [#several-replicas]
+
+A file on one replica's disk is invisible to the others. So when the server
+declares several replicas (`OS_CLUSTER_REPLICAS` above 1, or
+`OS_EXPECT_MULTI_NODE=true`) **and** its [cluster driver](/docs/kernel/cluster)
+is Redis (`OS_CLUSTER_DRIVER=redis`), the connection is kept in the database
+instead, as the one row of the `sys_cloud_binding_credential` object, which
+every replica reads:
+
+- **The credential is encrypted.** `runtime_token` is a secret field: the
+  platform crypto provider encrypts it before it is stored, and the row holds
+  only a reference to the encrypted value. With no crypto provider configured
+  the server refuses to connect rather than store it in clear text. Every
+  replica needs the same key to read it (`OS_SECRET_KEY` with the local
+  provider).
+- **Connect and disconnect work from any replica.** Both write that one row.
+  Every request to `/api/v1/cloud-connection/*` reads the row first, on
+  whichever replica serves it; anything else that presents the credential to
+  the cloud, such as an online licence check, sees a change within 30 seconds.
+- **The database, not the cluster store, holds it.** The cluster store is a
+  cache: flushing it, or running it without persistence, must never
+  disconnect the server. The row lives in the server's default database, the
+  same one that holds its encrypted secrets, even when another datasource
+  (such as one named `cloud`) is configured.
+- **It can be read before the server starts.** The multi-node licence check
+  runs before startup, so a host reads the row once at that point with
+  `readCloudBindingCredentialBeforeBoot` from `@objectstack/cloud-connection`.
+  The read opens one connection, starts closing it again before startup
+  continues, and never writes. The open, the read and the close share one
+  time limit, 3 seconds by default: if the database has not answered by then
+  the read gives up, and a close that is slow to finish is not waited for.
+  If the read fails, the replica starts as an unconnected one would.
+
+**Scaling out a server that is already connected.** The first start with
+several replicas declared moves the credential from
+`.objectstack/cloud-connection.json` into the row, when the row is still
+empty, and logs where it moved it from. The file stays where it was but is
+not read again while replicas are declared. Scaling back to a single replica
+reads the file again, and the file does not follow connects or disconnects
+made meanwhile: disconnect and connect again after scaling down.
+
+A single server, a server whose cluster driver is the in-process `memory`
+one or any driver other than Redis, and a server with no cluster service keep
+the file exactly as described above, and nothing is added to the database.
+Replicas declared over Redis without the data engine composed cannot share
+the row: the
+server logs an `error` at start that says so, and each replica keeps its own
+file.
+
 ## Rename [#rename]
 
 The name your server is known by is its **host name** at the moment you

packages/cli/src/utils/organization-ownership-inventory.ts

@@ -784,6 +784,14 @@ export const ORGANIZATION_OWNERSHIP_INVENTORY: readonly InventoryEntry[] = [
       `${CENSUS_MAIN}: systemFields.tenant: false; ${D7}: a declarative connector's stored credential is environment-level, ` +
       'read by the reference its instances name, never by organization',
   },
+  {
+    object: 'sys_cloud_binding_credential',
+    census: PMAIN,
+    fate: 'column-drop',
+    citation:
+      `${CENSUS_MAIN}: systemFields.tenant: false; ${D7}: the deployment's one cloud binding credential, read by its ` +
+      'fixed slot under the system context, never by organization',
+  },
   // #30959 (ruling record 6718345088, fork 1 (a)) — an identity-provider group
   // mapping is saved by the platform administrator for one SCIM connection and
   // always carries the organization that owns the connection's credential (the

packages/cloud-connection/package.json

@@ -32,12 +32,16 @@
     "@objectstack/observability": "workspace:*",
     "@objectstack/rest": "workspace:*",
     "@objectstack/runtime": "workspace:*",
+    "@objectstack/service-cluster": "workspace:*",
     "@objectstack/spec": "workspace:*",
     "@objectstack/types": "workspace:*"
   },
   "devDependencies": {
+    "@objectstack/driver-sql": "workspace:*",
     "@objectstack/lint": "workspace:*",
+    "@objectstack/platform-objects": "workspace:*",
     "@objectstack/plugin-security": "workspace:*",
+    "@objectstack/service-settings": "workspace:*",
     "@types/node": "^26.6.5",
     "typescript": "^6.0.3",
     "vitest": "^4.1.11"