Skip to content

Commit 4bb5eea

Browse files
docs(cluster, connectors): lane 2 also converges connector definitions on every replica (#28050)
domain:devx#2 · os-justin · session_01QSeKYmTYfaPbLQD4BJfomW Fixes #28030 Clause-②: no Metadata-protocol: none ## 维护者速读 - 改了什么:`cluster.mdx` §6.2 的 lane 2 表格行与「Lane 2」小节补写:同一次 `metadata.mutated` 回放还会让自动化服务在每个副本上重新物化连接器(非草稿的 `connector` 变更),直接的 active 保存也会让写入副本自己生效;`connectors.mdx` 故障表的 runtime reload 行补全触发路径(发布、直接 active 保存、删除,任一副本);§6.2 家族规则「`originNode` 回环抑制」一条改正:连接器 reconcile 在写入副本上是排队执行的,不是在门处同步跑完。 - 为什么改:PR #28028 合入后运行时已经这样工作,这两处文档仍写着 lane 2 只收敛 ObjectQL 对象注册表、reload 只来自 Studio 发布或 dev 重载,读文档的 AI 会得出错误结论。 - 风险与回滚:纯文档,两个 `.mdx` 文件、不发布任何包;回滚即还原本 PR。 - 先看:`content/docs/kernel/cluster.mdx`(Lane 2 小节新增的两段,逐句对照 `packages/services/service-automation/src/plugin.ts` 的 `subscribeFlowMutations` 与 `queueConnectorMutationReconcile`);`content/docs/automation/connectors.mdx` 第 458 行。 - 两分钟验证:`pnpm check:doc-authoring && pnpm check:doc-anchors`,预期两者均 exit 0;或打开文档站 /docs/kernel/cluster 的 6.2 节,看到 lane 2 小节新增「The same replay converges…」一段。 - 需要你做的:无,席位落地。 ## What changes Docs only. Every sentence is written from the code on `main` after PR #28028 (`cc5586da8c`). - `content/docs/kernel/cluster.mdx` §6.2, lanes table, lane 2 row, "Converges" column: after ObjectQL's OBJECT registry, the same replay also converges the automation service's armed flows and its materialized connectors (a `connector` mutation that is not a draft), each from the replica's own re-read. - `content/docs/kernel/cluster.mdx` §6.2, "Lane 2 — `metadata.mutated`" subsection: two new paragraphs. - On a peer, `applyRemoteMetadataMutation` replays the event into its local `onMetadataMutation` listeners after the write-through, and never re-publishes it. `AutomationServicePlugin`'s listener re-arms a `flow`. For a `connector` mutation that is not a draft (active or deleted), it queues the soft reconcile that `metadata:reloaded` runs. That reconcile re-reads the declaration through `getMetaItemsForExecution` (the registry with the `sys_metadata` overlay rows over it). One signal and no second channel; lane 4 stays the credential channel. A draft is dropped twice: the publisher does not publish it, and the listener returns on `state === 'draft'`. - On the writer, a direct active save (`PUT /api/v1/meta/connector/:name`) raises no `metadata:reloaded`, so the local half of the same signal is what re-materializes it. A publish raises both signals, and the second reconcile is a no-op on unchanged signatures. Mutation-queued reconciles coalesce, and a failure is logged at `warn`. - `content/docs/kernel/cluster.mdx` §6.2, family bullet "`originNode` loopback suppression" (file-surface extension the seat authorised at review): it said the connector reconcile "already ran, synchronously, at the door". It now says the write-through and the pool rebuild run synchronously at the door, and a connector reconcile is queued on the writer's reconcile queue. On lane 2 the local half of the same signal queues it (`emitMetadataMutation` notifies local listeners before it publishes), and the save does not wait for it. On lane 4 the credential door queues one job that writes the row, publishes, then runs the reconcile, and answers only after that reconcile finishes. - `content/docs/automation/connectors.mdx:458`, the failure-modes row "A bad entry arrives via runtime reload": its trigger list now names a Studio publish, a direct active save, or a delete, on any replica (linking §6.2), or a dev reload. ## Tests Prose only, so there is no package test. The docs gate families are under Verification. ## Acceptance notes - In-place fix, declared: the lane 2 row and subsection also name **flows**, not only connectors. The listener that carries connectors on the peer's replay is the flow listener of #21725. #28028 extended it, and its pin `connector-definition-cluster.test.ts` asserts "the flow save still arms the flow on both replicas". Without the flow clause, the edited row would list the listener's connector half as if it were the whole of it. Same row, same subsection, same claimed file surface. - The dispatch named #28016 as a possible edit to the same table. It had not landed on `main` at merge time; `main` was merged at `5aee206543`, and `origin/main` had not moved when the final gates ran. ## Verification Derived-family readings are at head `16f6a55a5f`, after the merge of `main` `5aee206543`. The follow-up commit `e3ce3a96cc` changes only the loopback bullet, and its readings are listed separately below. <details><summary>Derived docs gate families</summary> `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` derived 43 commands over the 2 changed paths. All 43 were run, with each exit code captured before any pipe. `--ran` reconciliation (exit 0): `✓ dispatch-gates --ran: 43 derived famil(ies) accounted for — 43 run, 0 NOT-MEASURED (a DERIVED zero — all 43 recorded an exit code and none of them is 3).` - `pnpm check:doc-authoring` exit 0, `pnpm check:doc-anchors` exit 0 (the new `#62-the-cluster-invalidation-family` link), `pnpm check:nul-bytes` exit 0. - All 43 exited 0 on the final run at `16f6a55a5f`. On an earlier pass, before any build, `check:doc-formula-expressions`, `check:doc-security-posture`, `check:docs-transcript-drift` and `check:skill-examples` refused with exit 3 (prerequisite not met, nothing measured), and `check:docs` exited 1 because `gen:schema` output was missing. After `@objectstack/lint...` and `@objectstack/client-react...` were built under `os-verify-lock.sh` (`VERDICT command-exit 0` both), every family measured and passed. </details> <details><summary>Follow-up commit e3ce3a9 (loopback bullet)</summary> - `pnpm check:doc-authoring` exit 0. - `pnpm check:doc-anchors` exit 0: `832 internal #fragment link(s) across 478 source file(s) all resolve to a real heading`. - `pnpm check:nul-bytes` exit 0. </details> --- _Generated by [Claude Code](https://claude.ai/code/session_01QSeKYmTYfaPbLQD4BJfomW)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent ba4ecf1 commit 4bb5eea

2 files changed

Lines changed: 35 additions & 5 deletions

File tree

‎content/docs/automation/connectors.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -456,7 +456,7 @@ retry):
456456
| MCP server unreachable / `tools/list` fails; remote spec URL unreachable or transiently failing (408/429/5xx) | **Not fatal**: the instance registers **degraded** (visible on `GET /api/v1/automation/connectors` with the reason), dispatches fail with that reason, and the platform retries on a 5s→5min backoff — recovery is automatic |
457457
| A retry of a degraded instance reaches the upstream and meets a configuration fault (an `include` naming a tool or operation the upstream does not offer) | **Not fatal**: the instance stays **degraded**, now with the configuration reason, and is **not retried** — dispatches fail saying to edit its declaration; an edit of the entry (or a restart) re-attempts it |
458458
| Wrong spec URL (other 4xx) or unparseable document | Configuration fault — **hard boot error** |
459-
| A bad entry arrives via **runtime reload** (Studio publish / dev reload) | Logged and **skipped**, never crashing the live server; a changed instance's old connector keeps serving until its replacement materializes |
459+
| A bad entry arrives via **runtime reload** (a Studio publish, a direct active save through `PUT /api/v1/meta/connector/:name`, or a delete, made on any replica — peers reconcile on [cluster lane 2](/docs/kernel/cluster#62-the-cluster-invalidation-family); or a dev reload) | Logged and **skipped**, never crashing the live server; a changed instance's old connector keeps serving until its replacement materializes |
460460

461461
## Catalog descriptors: entries without a `provider`
462462

‎content/docs/kernel/cluster.mdx‎

Lines changed: 34 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -333,7 +333,7 @@ goes stale when a shared record changes on a different node:
333333
| Lane | Channel | Payload type | Package | Converges |
334334
|---|---|---|---|---|
335335
| 1 — metadata service cache | `metadata.changed` | `ClusterMetadataChangedPayload` | `@objectstack/metadata` | `MetadataManager`'s registry/list caches |
336-
| 2 — ObjectQL object registry | `metadata.mutated` (`METADATA_MUTATION_CLUSTER_CHANNEL`, #13331) | `ClusterMetadataMutationPayload` | `@objectstack/metadata-protocol` | ObjectQL's OBJECT registry, from a `sys_metadata` re-read |
336+
| 2 — ObjectQL object registry | `metadata.mutated` (`METADATA_MUTATION_CLUSTER_CHANNEL`, #13331) | `ClusterMetadataMutationPayload` | `@objectstack/metadata-protocol` | ObjectQL's OBJECT registry, from a `sys_metadata` re-read; then, from the same replay, the automation service's armed flows and its materialized connectors (a `connector` mutation that is not a draft, #28002), each from the replica's own re-read |
337337
| 3 — ObjectQL driver registry | `datasource.mutated` (`DATASOURCE_MUTATION_CLUSTER_CHANNEL`, #13805) | `ClusterDatasourceMutationPayload` | `@objectstack/service-datasource` | ObjectQL's DRIVER registry, from a shared datasource-record re-read |
338338
| 4 — declarative connector instances | `connector-credential.mutated` (`CONNECTOR_CREDENTIAL_MUTATION_CLUSTER_CHANNEL`, #27428) | `ClusterConnectorCredentialMutationPayload` (`{ originNode, ref }`) | `@objectstack/service-automation` | the automation service's materialized connectors, from a reconcile that re-reads the shared `sys_connector_credential` rows; published by the credential door (`PUT` / `DELETE /api/v1/automation/connectors/credential-refs/:ref`) after the row is written or cleared |
339339

@@ -350,9 +350,15 @@ channel's own doc comment:
350350
from that read. A duplicate or re-ordered
351351
delivery converges to the same state as a single one.
352352
- **`originNode` loopback suppression.** Every payload carries the publishing
353-
node's id; a peer that receives its own `originNode` drops the message —
354-
the write-through, pool rebuild or connector reconcile already ran,
355-
synchronously, at the door that persisted the change.
353+
node's id; a peer that receives its own `originNode` drops the message,
354+
because the writer has already handled the change locally. The
355+
write-through and the pool rebuild ran synchronously at the door that
356+
persisted the change. A connector reconcile is queued on the writer's
357+
reconcile queue instead. On lane 2 the local half of the same signal queues
358+
it: the protocol notifies its own mutation listeners before it publishes,
359+
and the save does not wait for the reconcile. On lane 4 the credential door
360+
queues one job that writes the row, publishes, then runs the reconcile, and
361+
the door answers only after that reconcile finishes.
356362
- **At-most-once, bounded by boot rehydration.** No shipped `IPubSub` driver
357363
exceeds at-most-once delivery (§4.2), so a lost message is not retried.
358364
Each lane narrows the staleness window this always had — "until the next
@@ -445,6 +451,30 @@ content. Attached by `MetadataClusterBridgePlugin`'s protocol lane at
445451
`kernel:ready`, duck-typed against `protocol.attachMetadataMutationPubSub()`
446452
so this package takes no dependency on `@objectstack/metadata-protocol`.
447453

454+
**The same replay converges the automation service's flows and connectors.**
455+
After the write-through, the peer replays the event into its own
456+
`onMetadataMutation` listeners — locally, never re-published — and
457+
`AutomationServicePlugin`'s listener acts on two types. A `flow` mutation
458+
re-arms or disarms that flow from its stored row (#21725). A `connector`
459+
mutation that is not a draft (saved active, published or deleted) queues the
460+
soft connector reconcile that `metadata:reloaded` runs, on the reconcile queue
461+
that also serializes the credential door's writes (#28002). That reconcile re-reads the
462+
connector declaration from the replica's own protocol view
463+
(`getMetaItemsForExecution`, the registry with the `sys_metadata` overlay rows
464+
over it) and materializes, replaces or tears down instances from that read.
465+
One signal, no second channel: lane 2 carries connector-definition
466+
convergence on the replay it already delivers, and lane 4 remains the
467+
credential channel. A draft save reconciles nothing on any replica — the
468+
publisher does not publish drafts, and the listener drops them.
469+
470+
The listener runs on the writing replica too, on the local half of the same
471+
signal. A direct active save (`PUT /api/v1/meta/connector/:name`) raises no
472+
`metadata:reloaded` anywhere, so this listener is what re-materializes the
473+
writer. A publish raises both signals; the second reconcile finds every
474+
signature unchanged and rebuilds nothing. A reconcile queued and not yet
475+
started answers every connector mutation reported before it starts, and a
476+
failed one is logged at `warn` and left to the next reload or restart.
477+
448478
#### Lane 3 — `datasource.mutated` (the ObjectQL driver registry)
449479

450480
The datasource admin service (`@objectstack/service-datasource`) fans out on

0 commit comments

Comments
 (0)