Skip to content

fix(resilience): count resolved 5xx results against the provider breaker - #12360

Merged
diegosouzapw merged 4 commits into
diegosouzapw:release/v3.8.51from
pacocartones:fix/circuit-breaker-resolved-5xx
Sep 2, 2026
Merged

fix(resilience): count resolved 5xx results against the provider breaker#12360
diegosouzapw merged 4 commits into
diegosouzapw:release/v3.8.51from
pacocartones:fix/circuit-breaker-resolved-5xx

Conversation

@pacocartones

Copy link
Copy Markdown
Contributor

Summary

  • CircuitBreaker.execute() treated every resolved promise as a success, but handleChatCore() reports most upstream failures by resolving with { success: false, status: 5xx }. On the chat path that spurious _onSuccess() decayed failureCount right before the call site's _onFailure() for the same attempt, so a provider answering 503s indefinitely stayed CLOSED at failureCount: 1 and kept receiving traffic. Combo dispatches hit the same cancellation through recordProviderFailure() on the shared per-provider breaker.
  • execute() now accepts an optional per-call classifyResult (success / failure / ignore). Without it the "resolved = success" contract every throw-based caller relies on is unchanged.
  • executeChatWithBreaker() passes ignore; the chat path accounts for the outcome exactly once where the request context lives: chat.ts via the new classifyProviderBreakerResult() (built on the existing shouldTripProviderBreakerForResult() exclusions) at both its success and failure sites, combo.ts via recordProviderFailure() / recordProviderSuccess() (combo dispatches classify as ignore in chat.ts, so a combo success is no longer counted twice).

Related Issues

Validation

  • Change type: routing (provider circuit breaker on the chat path)
  • Focused tests and category gates from the golden path: tests/unit/circuit-breaker-resolved-5xx-12254.test.ts 5/5, tests/unit/breaker-network-error-guard.test.ts 15/15, plus chat-helpers, execute-chat-resource-pressure-breaker, circuit-breaker-client-abort, circuit-breaker-abort-provider-trip-7907, combo-provider-cooldown, chat-combo-live-test, provider-breaker-halfopen-recovery 56/56; npm run typecheck:core 0; git diff --check clean; check:changelog-integrity OK; check:file-size OK
  • npm run lint — repository-wide eslint exit 0 (run with --pass-on-unpruned-suppressions; the literal command reports only pre-existing unused global suppressions on this base)
  • Reconciled with the current active release base release/v3.8.51@158647618; focused checks rerun afterward
  • Production-code changes include a new or updated automated test in this PR
  • SonarQube is temporarily opt-in while the private project has no quota; it is not a PR gate.

TDD record: with the production hunks reverted, the end-to-end test shows five requests each reaching a 503 upstream while the breaker stays failureCount=1 state=CLOSED; with the fix it climbs 1→2→3, opens, and the next request is short-circuited without an upstream call.

Tests Added Or Updated

  • tests/unit/circuit-breaker-resolved-5xx-12254.test.ts (new): end-to-end single-model repro through chat.ts → executeChatWithBreaker → breaker.execute with a 503 upstream; execute() classifier contract (counts a resolved failure, ignore leaves state untouched, default unchanged, throwing classifier never wedges).
  • tests/unit/breaker-network-error-guard.test.ts: five classifyProviderBreakerResult() cases (resolved 503 → failure, success → success, excluded failure → ignore, combo → ignore for both outcomes, live combo test → ignore).

Coverage Notes

  • src/shared/utils/circuitBreaker.ts: _recordResolvedResult() and the execute() option are covered by the four contract tests above.
  • src/sse/handlers/chatPredicates.ts: classifyProviderBreakerResult() fully covered by the guard file.
  • src/sse/handlers/chatHelpers.ts / chat.ts: the end-to-end test exercises both execute() call sites and the chat.ts failure site; the success site is exercised by the existing chat-combo-live-test and combo-provider-cooldown suites.

Reviewer Notes

  • Behaviour change on the combo path: chat.ts no longer calls _onSuccess() for combo dispatches; combo.ts already calls recordProviderSuccess() at both of its success sites, so combo successes now count once instead of twice, and HALF_OPEN → CLOSED recovery from combo traffic goes through that call.
  • Unchanged and out of scope (pre-existing): execute()'s throw path still records via isFailure; the chat.ts all-rate-limited branch (breaker._onFailure() after handleNoCredentials) is not gated by isCombo; the duplicated PROVIDER_BREAKER_FAILURE_STATUSES in open-sse/services/combo/comboPredicates.ts is left in place because of the documented open-sse → src/sse import-cycle constraint.
  • Validation is test-first only (no live VPS run); the full unit suite, Vitest, coverage ratchet and build run in CI.

`CircuitBreaker.execute()` treated every resolved promise as a success, but
`handleChatCore()` reports most upstream failures by resolving with
`{ success: false, status: 5xx }` rather than throwing. On the chat path that
spurious `_onSuccess()` decayed `failureCount` by one right before the call
site's `_onFailure()` for the same attempt, so a provider answering 503s
indefinitely stayed `CLOSED` at `failureCount: 1` and kept receiving traffic.
Combo dispatches hit the same cancellation through `recordProviderFailure()`
on the shared per-provider breaker.

- `execute()` accepts an optional per-call `classifyResult` that decides how a
  resolved result is accounted (`success` / `failure` / `ignore`); without it
  the "resolved = success" contract every throw-based caller relies on is
  unchanged.
- `executeChatWithBreaker()` passes `ignore`: the chat path owns the
  accounting where the request context (combo, live test, probe isolation)
  lives.
- `classifyProviderBreakerResult()` in chatPredicates.ts is the single
  definition chat.ts now uses at both its success and failure sites, built on
  the existing `shouldTripProviderBreakerForResult()` exclusions. Combo
  dispatches classify as `ignore` there, so combo.ts's
  `recordProviderFailure()` / `recordProviderSuccess()` record each target
  outcome exactly once.

Closes diegosouzapw#12254

Co-authored-by: Leon Marcos <leonaniagomez@gmail.com>
pacocartones added a commit to pacocartones/OmniRoute that referenced this pull request Sep 1, 2026
@pacocartones
pacocartones force-pushed the fix/circuit-breaker-resolved-5xx branch from 9cc69d3 to 3c9d5fe Compare September 1, 2026 23:45
pacocartones and others added 2 commits September 2, 2026 00:40
The mutation-test-coverage gate (--strict) requires every unit test that
covers a mutated module to be listed in stryker.conf.json tap.testFiles.
tests/unit/circuit-breaker-resolved-5xx-12254.test.ts covers
src/shared/utils/circuitBreaker.ts, so its mutant kills now count.

Co-authored-by: Leon Marcos <leonaniagomez@gmail.com>
Co-authored-by: Leon Marcos <leonaniagomez@gmail.com>
@diegosouzapw
diegosouzapw merged commit 5a34111 into diegosouzapw:release/v3.8.51 Sep 2, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants