Repository navigation
Commit 066ff51
fix(service-automation): end the backoff retry of a degraded connector on a configuration fault (#28014)
<!-- claude-projects-attribution: {"github_login":"os-elon-musk"} -->
_Requested by **Elon** · [project
thread](https://claude.ai/code/project/chan_01WedDzV26nniwxv7jC88B9D?thread=cmsg_01WedDzV26nniwxv7jC88B9DAgUzvVk8NtjaKQTR762kuz)_
Fixes #27992
Clause-②: no
Metadata-protocol: none
Skills-docs: follow-up owed — content/docs/automation/connectors.mdx
(its fault table has no row for a configuration fault met on a degraded
instance's retry; `git grep -n -i degraded -- skills` finds no retry
wording; `docs/adr/0097-declarative-connector-instances.md` §degrade is
governed and left as is)
## 维护者速读
- 改了什么:连接器在启动时连不上上游会进入「降级」并按退避定时重试;如果之后重试时上游可达、但配置本身有错(例如 `include`
写了上游没有的工具),现在会停止重试,并把原因换成配置错误的原文。
- 为什么改:之前这种情况只记一条日志,实例仍留在重试集合里,连接器列表一直显示过时的「连不上」原因,并每 5
分钟徒劳地重连一次;ADR-0097 规定配置错误不重试。
- 风险与回滚:只影响「已降级且重试时遇到配置错误」这一条路径;凭据被拒与上游仍不可达两条路径不变(有对照测试)。回滚即还原本 PR 的
`plugin.ts`。
- 先看:`packages/services/service-automation/src/plugin.ts`(工厂 catch
里新加的分支,与新方法
`failDegradedConnectorInstance`);`connector-retry-configuration-fault.test.ts`。
- 两分钟验证:`pnpm --filter @objectstack/service-automation exec vitest run
--maxWorkers=2 src/connector-retry-configuration-fault.test.ts`,预期 6/6
通过。
- 需要你做的:无,席位落地。
## What changes
`reconcileDeclaredConnectors` routes a provider factory's throw three
ways: a credential rejection holds, an upstream-unavailable fault
degrades and retries on the backoff, and anything else is a
configuration fault that `fail()`s (fatal at boot, logged and skipped on
a reload). The skip left an instance that was already degraded in
`degradedInstances`, with its husk still carrying the stale "could not
reach" reason, so the backoff timer kept reconnecting against a fault no
retry can clear.
Now, on a soft run, a configuration fault met by an instance in
`degradedInstances` goes to a new `failDegradedConnectorInstance`:
- The instance is held (the existing `held` flag the credential path
uses): `scheduleDeclarativeRetry` skips it, and a reconcile with the
same signature skips it. A change of its declaration or a restart
re-attempts it.
- With no live connector, the husk is registered again with the
provider's configuration message as its `degradedReason`, so `GET
/api/v1/automation/connectors` and the `connector_action` refusal name
the configuration fault.
- With a live connector (a changed entry whose new configuration was
pending a retry), the old connector keeps serving and the retry stops.
- One `error` record says the instance is no longer retried; the
provider's message rides `meta` (#5575).
Untouched: the boot path (still fatal), a configuration fault on an
instance that was not degraded (still logged and skipped), the
credential hold path, and the unavailable degrade path.
Reading chosen for "the failed state": `ConnectorState`
(`packages/spec/src/integration/connector-descriptor.ts`) has only
`ready` and `degraded`, so the descriptor keeps `state: 'degraded'` and
the failure is carried by the replaced reason plus the ended retry.
Adding a `failed` state would widen a published spec type and is outside
this card.
## Tests
New
`packages/services/service-automation/src/connector-retry-configuration-fault.test.ts`,
under fake timers, with fake `mcp` and `openapi` provider factories
shaped like the real ones (unavailable marker while down, a plain
`Error` naming `providerConfig.include` once up). The `openapi` pin
needed no provider-package edit: the path is provider-agnostic in
`plugin.ts`.
- `mcp` / `openapi`: degraded at boot, the backoff retry meets the
include fault, the descriptor's `degradedReason` is the configuration
message and no longer "could not reach", and 64 × the base delay later
the factory has not been called again.
- An unchanged reload does not re-attempt it; an edit of the declaration
does and recovers it to `ready`.
- A changed entry pending a retry: the old connector keeps serving and
the retry stops.
- Control: an upstream still unreachable on the retry keeps degrading
and is retried again.
- Control: a credential refused on the retry keeps its hold path.
## Acceptance notes
- The `connector_action` refusal (`builtin/connector-nodes.ts`) still
ends with "the platform retries automatically" for every degraded husk,
a held one included (a refused credential today, a configuration fault
after this PR); the engine's `Connector registered DEGRADED` warn
likewise says "the materializer retries with backoff". Both are outside
this card's file surface, so the reason text names the fault but the
sentence after it still promises a retry.
- The other per-instance `fail()` arms in `reconcileDeclaredConnectors`
(no provider factory, unparseable `retryConfig`, host credential
resolver throwing) can also be met by an instance that is already
degraded, after an edit; they keep the old skip-and-retry behaviour.
This PR routes only the factory's configuration fault, which is the
card's mechanism.
- Not touched, as ordered: a misspelt narrowing `include` on a hot
reload of a live connector still keeps the wider set serving.
## Verification
All readings at head `fbe1de19`.
<details><summary>Reproduction on main (red), fix (green), ablation,
restore</summary>
- On `main` (`46943657`'s `plugin.ts`, with the new test file): `Tests 4
failed | 2 passed (6)`. The `mcp` and `openapi` cases fail on the reason
(`expected 'connector-mcp provider: connector …' to match
/providerConfig\.include names 'nope'/`, the stale "could not reach"
text), and the changed-entry case fails with `expected 67 to be 3`: 67
factory calls in 64 base delays. The two controls pass.
- With the fix: `Tests 6 passed (6)`.
- Ablation through `scripts/ablation-replace.mjs`, anchor `if
(!opts.fatal && this.degradedInstances.has(name)) {` replaced by `if
(false /* ABLATION */ && this.degradedInstances.has(name)) {`: anchor 1
to 0, blob 95e8c37 to 577056522a42; run `Tests 4 failed | 2 passed
(6)` (the same four; both controls stay green); restore: blob after
restore equals the blob at HEAD (95e8c37), `git diff HEAD` empty;
re-run `Tests 6 passed (6)`. The test imports `./plugin.js` from source,
so no `dist/` leg applies.
</details>
<details><summary>Package suite and typecheck</summary>
- `pnpm --filter @objectstack/service-automation test`: `Test Files 222
passed (222)`, `Tests 2836 passed (2836)`.
- `pnpm --filter @objectstack/service-automation typecheck`: exit 0
(`check:test-typecheck: OK`).
</details>
<details><summary>dispatch-gates families (60 derived, 60 run, 0
NOT-MEASURED) and pnpm lint</summary>
Run after a whole-tree `turbo run build` (exit 0). `dispatch-gates
--ran`: 60 derived families accounted for, 60 run, 0 NOT-MEASURED.
```text
node scripts/check-adr-0087-registration.mjs --base origin/main :: exit 0
node scripts/check-adr-0087-registration.mjs --self-test :: exit 0
node scripts/check-changeset-no-major.mjs --base origin/main :: exit 0
node scripts/check-closing-keyword-parity.mjs :: exit 0
node scripts/check-closing-keyword-parity.mjs --self-test :: exit 0
node scripts/check-comment-mask-adoption.mjs :: exit 0
node scripts/check-comment-mask-adoption.mjs --self-test :: exit 0
node scripts/check-comment-mask-corpus.mjs :: exit 0
node scripts/check-dts-emitted.mjs --self-test :: exit 0
node scripts/check-empty-changeset.mjs --base origin/main :: exit 0
node scripts/check-empty-changeset.mjs --self-test :: exit 0
node scripts/check-issue-citations.mjs :: exit 0
node scripts/check-plugin-teardown-shape.mjs :: exit 0
node scripts/check-plugin-teardown-shape.mjs --self-test :: exit 0
node scripts/check-tenant-audit-census.mjs :: exit 0
node scripts/check-tenant-audit-census.mjs --self-test :: exit 0
node scripts/ci/assert-turbo-remote-cache.mjs --self-test :: exit 0
node scripts/ci/plan-test-core-shards.mjs --self-test :: exit 0
node scripts/ci/prune-turbo-cache.mjs --self-test :: exit 0
node scripts/docs-audit/check-affected-docs.mjs :: exit 0
node scripts/docs-audit/check-drift-comment.mjs :: exit 0
node scripts/pm/release-rehearsal-clone.mjs --self-test :: exit 0
node scripts/release-pending-publish.mjs --self-test :: exit 0
pnpm --filter @objectstack/spec run check:duration-unit-keys :: exit 0
pnpm check:changeset-gate-self-tests :: exit 0
pnpm check:ci-filter-parity :: exit 0
pnpm check:closed-ceremony-text :: exit 0
pnpm check:cross-package-test-inputs :: exit 0
pnpm check:doc-authoring :: exit 0
pnpm check:driver-memory-census :: exit 0
pnpm check:dts-closure :: exit 0
pnpm check:dual-build-cjs-loads :: exit 0
pnpm check:durability-log-level :: exit 0
pnpm check:engine-double-contract :: exit 0
pnpm check:gitlink-declared :: exit 0
pnpm check:issue-citations :: exit 0
pnpm check:keyed-text-bounds :: exit 0
pnpm check:lean-entry-closure :: exit 0
pnpm check:logger-receiver-detach :: exit 0
pnpm check:nul-bytes :: exit 0
pnpm check:objectql-double-limit :: exit 0
pnpm check:org-identifier :: exit 0
pnpm check:page-declaration-shape :: exit 0
pnpm check:platform-object-tenancy-census :: exit 0
pnpm check:pm-changeset-deadline-census :: exit 0
pnpm check:published-files :: exit 0
pnpm check:query-options-erasure :: exit 0
pnpm check:refd-timer-probe :: exit 0
pnpm check:registry-log-declared :: exit 0
pnpm check:rest-log-spy-declared :: exit 0
pnpm check:slot-lookup :: exit 0
pnpm check:sourcemap-no-sources-content :: exit 0
pnpm check:system-context-census :: exit 0
pnpm check:test-source-alias :: exit 0
pnpm check:tier-file-adoption :: exit 0
pnpm check:type-check-coverage :: exit 0
pnpm check:type-check-debt :: exit 0
pnpm check:undeclared-dep-imports :: exit 0
pnpm check:watch-hint-literal :: exit 0
pnpm check:where-matcher :: exit 0
pnpm lint :: exit 0
```
</details>
---
_Generated by [Claude
Code](https://claude.ai/code/session_016WV4czEP1ZwBWHusgCZugF)_
Co-authored-by: Claude <noreply@anthropic.com>1 parent 21b2a10 commit 066ff51
3 files changed
Lines changed: 292 additions & 0 deletions
File tree
- .changeset
- packages/services/service-automation/src
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
742 | 742 | | |
743 | 743 | | |
744 | 744 | | |
| 745 | + | |
| 746 | + | |
| 747 | + | |
| 748 | + | |
745 | 749 | | |
746 | 750 | | |
747 | 751 | | |
| |||
2355 | 2359 | | |
2356 | 2360 | | |
2357 | 2361 | | |
| 2362 | + | |
| 2363 | + | |
| 2364 | + | |
| 2365 | + | |
| 2366 | + | |
| 2367 | + | |
| 2368 | + | |
| 2369 | + | |
| 2370 | + | |
| 2371 | + | |
| 2372 | + | |
| 2373 | + | |
| 2374 | + | |
| 2375 | + | |
| 2376 | + | |
2358 | 2377 | | |
2359 | 2378 | | |
2360 | 2379 | | |
| |||
2517 | 2536 | | |
2518 | 2537 | | |
2519 | 2538 | | |
| 2539 | + | |
| 2540 | + | |
| 2541 | + | |
| 2542 | + | |
| 2543 | + | |
| 2544 | + | |
| 2545 | + | |
| 2546 | + | |
| 2547 | + | |
| 2548 | + | |
| 2549 | + | |
| 2550 | + | |
| 2551 | + | |
| 2552 | + | |
| 2553 | + | |
| 2554 | + | |
| 2555 | + | |
| 2556 | + | |
| 2557 | + | |
| 2558 | + | |
| 2559 | + | |
| 2560 | + | |
| 2561 | + | |
| 2562 | + | |
| 2563 | + | |
| 2564 | + | |
| 2565 | + | |
| 2566 | + | |
| 2567 | + | |
| 2568 | + | |
| 2569 | + | |
| 2570 | + | |
| 2571 | + | |
| 2572 | + | |
| 2573 | + | |
| 2574 | + | |
| 2575 | + | |
| 2576 | + | |
| 2577 | + | |
| 2578 | + | |
| 2579 | + | |
| 2580 | + | |
| 2581 | + | |
| 2582 | + | |
| 2583 | + | |
| 2584 | + | |
2520 | 2585 | | |
2521 | 2586 | | |
2522 | 2587 | | |
| |||
0 commit comments