Repository navigation
Commit cc5586d
fix(service-automation): converge a connector definition change on every replica (#28028)
<!-- claude-projects-attribution: {"github_login":"os-elon-musk"} -->
_Requested by **Elon** · [project
thread](https://claude.ai/code/project/chan_01WedDzV26nniwxv7jC88B9D?thread=cmsg_01WedDzV26nniwxv7jC88B9D2HoQfwCF75b2EvnX1Ze8Uc)_
Fixes #28002
Clause-②: no
Metadata-protocol: none
Skills-docs: none in skills (`git grep -n -i -E
"replica|cluster|metadata\.mutated|peer" -- skills/ | grep -i connector`
finds 0 lines); docs follow-up owed and outside this card's file
surface: `content/docs/kernel/cluster.mdx` §6.2 (the lane 2 row's
"Converges" column and the "Lane 2" subsection name only ObjectQL's
object registry, while the automation service's flows and, after this
PR, its materialized connectors converge on lane 2 as well) and
`content/docs/automation/connectors.mdx` (the fault table's "runtime
reload (Studio publish / dev reload)" row now also covers a direct
active save and a change made on another replica)
## 维护者速读
- 改了什么:在一个副本上保存、发布或删除连接器定义后,其它副本现在也会重新物化该连接器;直接 `PUT
/api/v1/meta/connector/:name`(不走发布门)也会让写入的那个副本自己生效。
- 为什么改:之前只有发布的那台副本通过本机的 `metadata:reloaded` 重建实例,其它副本虽然经 lane 2
收到了变更,但自动化插件的监听器只处理 `flow`,于是一直用旧定义直到重载或重启。
- 风险与回滚:只是让已有监听器对 `connector` 变更多排一次「软」对账(与 reload
同一路径),不新增集群通道;发布方会对账两次,第二次签名不变、不重建(已测)。回滚即还原本 PR 的 `plugin.ts`。
-
先看:`packages/services/service-automation/src/plugin.ts`(`subscribeFlowMutations`
里新加的 `connector` 分支与新方法
`queueConnectorMutationReconcile`);`connector-definition-cluster.test.ts`。
- 两分钟验证:`pnpm --filter @objectstack/service-automation exec vitest run
--maxWorkers=2 src/connector-definition-cluster.test.ts`,预期 7/7 通过。
- 需要你做的:无,席位落地。
## What changes
`AutomationServicePlugin` subscribes to the metadata protocol's
post-persistence signal (`onMetadataMutation`, #21725), and its listener
returned for every type but `flow`. A connector definition change
therefore re-materialized only where `metadata:reloaded` fired: on the
replica that published it (`ObjectQLPlugin.announcePerItemPublish`).
That hook is kernel-local. Lane 2 (`metadata.mutated`) did deliver
`connector/NAME` into a peer's mutation listeners, as #27977 measured,
and nothing there acted on it.
Now the listener also handles a `connector` mutation whose state is not
`draft` (`active` or `deleted`) by queuing
`queueConnectorMutationReconcile`: the soft reconcile
`metadata:reloaded` runs (`reconcileDeclaredConnectors(ctx, { fatal:
false })`), on the existing reconcile queue, so it never interleaves
with a credential write or another reconcile. The reconcile re-reads the
declaration from this replica's own protocol view, the same read the
reload path makes. No new lane, channel or payload: a peer converges on
the lane 2 replay it already receives (triage 6517791280, §6.2's
single-signal rule).
- Coalesced: while one mutation-queued reconcile is waiting to start,
further connector mutations add nothing. The flag clears when the run
starts, so a mutation reported during a run queues one more.
- A failure is logged (`warn`, cause in `meta` per #5575) and leaves the
next reload or restart to reconcile, as lane 4's receipt does.
- Flow mutations take the same path as before. Every other type is still
ignored.
## Tests
`packages/services/service-automation/src/connector-definition-cluster.test.ts`
(the #27977 two-kernel harness: real `ObjectKernel`, `ObjectQLPlugin`,
driver-sql on one shared better-sqlite3 file, lanes 2 and 4 attached as
the cluster bridge attaches them):
- The #27977 `it.fails` pin is now a plain `it`: the peer follows a
published definition with no reload on the peer.
- New: the publishing replica reconciles from both of its signals. For
two publishes it runs 4 reconciles (its own `metadata:reloaded` plus its
own mutation listener; lane 2's loopback guard drops the bus echo), and
the provider is called exactly `[V1, V2]`, so the second run is a no-op.
- New, the card's side finding: a direct active save (`saveMetaItem`
with no mode, the call behind `PUT /api/v1/meta/connector/:name`)
re-materializes the writing replica and converges the peer. The same
listener covers it, so it is pinned here and not widened elsewhere.
- New: a deleted connector is torn down on both replicas.
- New control: a draft connector save and a flow save run no connector
reconcile on either replica (counted through a spy on
`reconcileDeclaredConnectors`, attributed per plugin instance), the
provider is not called again, and the flow save still arms the flow on
both replicas.
- The existing reload control now runs the peer with lane 2 off, so a
`metadata:reloaded` is again its only trigger: the peer holds nothing
before the reload and converges from its own read after it.
## Acceptance notes
- The reconcile reads the protocol's execution view with no
organization, as the reload reconcile does, so an org-scoped connector
mutation converges exactly as far as a reload would.
- A held instance (a refused credential, or since #28014 a configuration
fault met on a retry) is skipped by an unchanged-signature reconcile, as
on a reload. An edit of its declaration re-attempts it, now on every
replica.
- The descriptor audit (`auditDeclaredConnectors`, the inert-descriptor
warning) still runs at `kernel:ready` and on `metadata:reloaded` only.
This PR adds convergence, not another warning per save.
## Verification
All readings at head `89267c8b` (branched off `main` at `066ff51e`,
after #28014 merged).
<details><summary>Package tests and typecheck</summary>
- `pnpm --filter @objectstack/service-automation typecheck` (under
`scripts/pm/os-verify-lock.sh`): exit 0, `check:test-typecheck: OK … 0
file(s) / 0 error(s)`. The test layer compiles under
`tsconfig.test.json`, which includes `src/**/*.test.ts`.
- `pnpm --filter @objectstack/service-automation test`: `Test Files 223
passed (223)`, `Tests 2846 passed (2846)`, lock `VERDICT command-exit
0`.
- `vitest run src/connector-definition-cluster.test.ts`: `Tests 7 passed
(7)`.
</details>
<details><summary>Ablation and restore</summary>
Through `scripts/ablation-replace.mjs` on `plugin.ts`: the anchor
`this.queueConnectorMutationReconcile(ctx);` plus the `return;` line
after it, replaced by `/* ABLATION-28002 */ return;` (a connector
mutation is then dropped, as on `main`). Anchor x1 to x0, marker x0 to
x1, blob `154b06fea8a2` to `9852d936898e`. The suite imports
`./plugin.js` from source, so no rebuild is in the path.
- Mutated run: `Tests 5 failed | 2 passed (7)`. Red: the flipped peer
pin, the publishing-replica count, the direct save, the delete, and the
draft/flow control (on its setup line, `expected undefined to be
'https://v1.billing.example.test'`: the peer never converges, so the
control's precondition cannot hold). Green: the lane 2 reach row and the
lane-2-off reload control, which do not depend on the listener.
- Restore: blob after restore equals the blob at HEAD (`154b06fea8a2`),
and `git diff HEAD` is empty.
</details>
<details><summary>Derived gate families and lint</summary>
`node scripts/pm/dispatch-gates.mjs --commands` derived 60 families at
`89267c8b`. All were run, plus `pnpm lint` (exit 0). `--ran`: `60
derived famil(ies) accounted for — 59 run, 1 NOT-MEASURED`.
- NOT MEASURED: `pnpm check:dual-build-cjs-loads`, exit 3, PREREQUISITE
NOT MET (it needs every package's `dist`; only the
`@objectstack/service-automation^...` closure was built). CI runs it.
- Every other derived family: exit 0.
</details>
---
_Generated by [Claude
Code](https://claude.ai/code/session_016WV4cxbvGGSLgjTAqnkkMK)_
Co-authored-by: Claude <noreply@anthropic.com>1 parent f296334 commit cc5586d
3 files changed
Lines changed: 196 additions & 19 deletions
File tree
- .changeset
- packages/services/service-automation/src
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
5 | | - | |
| 5 | + | |
| 6 | + | |
6 | 7 | | |
7 | 8 | | |
8 | 9 | | |
| |||
22 | 23 | | |
23 | 24 | | |
24 | 25 | | |
25 | | - | |
26 | | - | |
27 | | - | |
28 | | - | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
29 | 31 | | |
30 | 32 | | |
31 | 33 | | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
32 | 45 | | |
33 | 46 | | |
34 | 47 | | |
| |||
59 | 72 | | |
60 | 73 | | |
61 | 74 | | |
62 | | - | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
63 | 89 | | |
64 | 90 | | |
65 | 91 | | |
| 92 | + | |
66 | 93 | | |
67 | 94 | | |
68 | 95 | | |
| |||
105 | 132 | | |
106 | 133 | | |
107 | 134 | | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
108 | 141 | | |
109 | 142 | | |
110 | 143 | | |
111 | 144 | | |
112 | 145 | | |
113 | 146 | | |
114 | 147 | | |
115 | | - | |
| 148 | + | |
116 | 149 | | |
117 | 150 | | |
118 | 151 | | |
| |||
122 | 155 | | |
123 | 156 | | |
124 | 157 | | |
125 | | - | |
126 | | - | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
127 | 164 | | |
128 | 165 | | |
129 | 166 | | |
130 | 167 | | |
131 | 168 | | |
132 | | - | |
| 169 | + | |
| 170 | + | |
133 | 171 | | |
134 | 172 | | |
135 | 173 | | |
| |||
153 | 191 | | |
154 | 192 | | |
155 | 193 | | |
156 | | - | |
| 194 | + | |
157 | 195 | | |
158 | 196 | | |
159 | 197 | | |
| |||
164 | 202 | | |
165 | 203 | | |
166 | 204 | | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
167 | 210 | | |
168 | 211 | | |
169 | | - | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
170 | 215 | | |
171 | 216 | | |
172 | 217 | | |
| |||
185 | 230 | | |
186 | 231 | | |
187 | 232 | | |
188 | | - | |
189 | | - | |
| 233 | + | |
| 234 | + | |
190 | 235 | | |
191 | 236 | | |
192 | 237 | | |
| |||
198 | 243 | | |
199 | 244 | | |
200 | 245 | | |
201 | | - | |
| 246 | + | |
202 | 247 | | |
203 | 248 | | |
204 | 249 | | |
205 | 250 | | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
206 | 323 | | |
207 | 324 | | |
208 | 325 | | |
209 | | - | |
210 | 326 | | |
211 | | - | |
| 327 | + | |
| 328 | + | |
212 | 329 | | |
213 | 330 | | |
214 | 331 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
842 | 842 | | |
843 | 843 | | |
844 | 844 | | |
845 | | - | |
| 845 | + | |
| 846 | + | |
| 847 | + | |
| 848 | + | |
| 849 | + | |
| 850 | + | |
| 851 | + | |
846 | 852 | | |
847 | 853 | | |
848 | 854 | | |
| |||
3642 | 3648 | | |
3643 | 3649 | | |
3644 | 3650 | | |
| 3651 | + | |
| 3652 | + | |
| 3653 | + | |
3645 | 3654 | | |
3646 | 3655 | | |
3647 | 3656 | | |
| |||
3668 | 3677 | | |
3669 | 3678 | | |
3670 | 3679 | | |
3671 | | - | |
| 3680 | + | |
3672 | 3681 | | |
| 3682 | + | |
| 3683 | + | |
| 3684 | + | |
| 3685 | + | |
| 3686 | + | |
3673 | 3687 | | |
3674 | 3688 | | |
3675 | 3689 | | |
3676 | 3690 | | |
3677 | 3691 | | |
3678 | 3692 | | |
3679 | 3693 | | |
| 3694 | + | |
| 3695 | + | |
| 3696 | + | |
| 3697 | + | |
| 3698 | + | |
| 3699 | + | |
| 3700 | + | |
| 3701 | + | |
| 3702 | + | |
| 3703 | + | |
| 3704 | + | |
| 3705 | + | |
| 3706 | + | |
| 3707 | + | |
| 3708 | + | |
| 3709 | + | |
| 3710 | + | |
| 3711 | + | |
| 3712 | + | |
| 3713 | + | |
| 3714 | + | |
| 3715 | + | |
| 3716 | + | |
| 3717 | + | |
| 3718 | + | |
| 3719 | + | |
| 3720 | + | |
| 3721 | + | |
| 3722 | + | |
| 3723 | + | |
3680 | 3724 | | |
3681 | 3725 | | |
3682 | 3726 | | |
| |||
0 commit comments