fix(resilience): count resolved 5xx results against the provider breaker - #12360
Merged
diegosouzapw merged 4 commits intoSep 2, 2026
Conversation
`CircuitBreaker.execute()` treated every resolved promise as a success, but
`handleChatCore()` reports most upstream failures by resolving with
`{ success: false, status: 5xx }` rather than throwing. On the chat path that
spurious `_onSuccess()` decayed `failureCount` by one right before the call
site's `_onFailure()` for the same attempt, so a provider answering 503s
indefinitely stayed `CLOSED` at `failureCount: 1` and kept receiving traffic.
Combo dispatches hit the same cancellation through `recordProviderFailure()`
on the shared per-provider breaker.
- `execute()` accepts an optional per-call `classifyResult` that decides how a
resolved result is accounted (`success` / `failure` / `ignore`); without it
the "resolved = success" contract every throw-based caller relies on is
unchanged.
- `executeChatWithBreaker()` passes `ignore`: the chat path owns the
accounting where the request context (combo, live test, probe isolation)
lives.
- `classifyProviderBreakerResult()` in chatPredicates.ts is the single
definition chat.ts now uses at both its success and failure sites, built on
the existing `shouldTripProviderBreakerForResult()` exclusions. Combo
dispatches classify as `ignore` there, so combo.ts's
`recordProviderFailure()` / `recordProviderSuccess()` record each target
outcome exactly once.
Closes diegosouzapw#12254
Co-authored-by: Leon Marcos <leonaniagomez@gmail.com>
…apw#12360 Co-authored-by: Leon Marcos <leonaniagomez@gmail.com>
pacocartones
added a commit
to pacocartones/OmniRoute
that referenced
this pull request
Sep 1, 2026
pacocartones
force-pushed
the
fix/circuit-breaker-resolved-5xx
branch
from
September 1, 2026 23:45
9cc69d3 to
3c9d5fe
Compare
The mutation-test-coverage gate (--strict) requires every unit test that covers a mutated module to be listed in stryker.conf.json tap.testFiles. tests/unit/circuit-breaker-resolved-5xx-12254.test.ts covers src/shared/utils/circuitBreaker.ts, so its mutant kills now count. Co-authored-by: Leon Marcos <leonaniagomez@gmail.com>
Co-authored-by: Leon Marcos <leonaniagomez@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CircuitBreaker.execute()treated every resolved promise as a success, buthandleChatCore()reports most upstream failures by resolving with{ success: false, status: 5xx }. On the chat path that spurious_onSuccess()decayedfailureCountright before the call site's_onFailure()for the same attempt, so a provider answering 503s indefinitely stayedCLOSEDatfailureCount: 1and kept receiving traffic. Combo dispatches hit the same cancellation throughrecordProviderFailure()on the shared per-provider breaker.execute()now accepts an optional per-callclassifyResult(success/failure/ignore). Without it the "resolved = success" contract every throw-based caller relies on is unchanged.executeChatWithBreaker()passesignore; the chat path accounts for the outcome exactly once where the request context lives:chat.tsvia the newclassifyProviderBreakerResult()(built on the existingshouldTripProviderBreakerForResult()exclusions) at both its success and failure sites,combo.tsviarecordProviderFailure()/recordProviderSuccess()(combo dispatches classify asignorein chat.ts, so a combo success is no longer counted twice).Related Issues
Validation
tests/unit/circuit-breaker-resolved-5xx-12254.test.ts5/5,tests/unit/breaker-network-error-guard.test.ts15/15, pluschat-helpers,execute-chat-resource-pressure-breaker,circuit-breaker-client-abort,circuit-breaker-abort-provider-trip-7907,combo-provider-cooldown,chat-combo-live-test,provider-breaker-halfopen-recovery56/56;npm run typecheck:core0;git diff --checkclean;check:changelog-integrityOK;check:file-sizeOKnpm run lint— repository-wide eslint exit 0 (run with--pass-on-unpruned-suppressions; the literal command reports only pre-existing unused global suppressions on this base)release/v3.8.51@158647618; focused checks rerun afterwardTDD record: with the production hunks reverted, the end-to-end test shows five requests each reaching a 503 upstream while the breaker stays
failureCount=1 state=CLOSED; with the fix it climbs 1→2→3, opens, and the next request is short-circuited without an upstream call.Tests Added Or Updated
tests/unit/circuit-breaker-resolved-5xx-12254.test.ts(new): end-to-end single-model repro throughchat.ts → executeChatWithBreaker → breaker.executewith a 503 upstream;execute()classifier contract (counts a resolved failure,ignoreleaves state untouched, default unchanged, throwing classifier never wedges).tests/unit/breaker-network-error-guard.test.ts: fiveclassifyProviderBreakerResult()cases (resolved 503 → failure, success → success, excluded failure → ignore, combo → ignore for both outcomes, live combo test → ignore).Coverage Notes
src/shared/utils/circuitBreaker.ts:_recordResolvedResult()and theexecute()option are covered by the four contract tests above.src/sse/handlers/chatPredicates.ts:classifyProviderBreakerResult()fully covered by the guard file.src/sse/handlers/chatHelpers.ts/chat.ts: the end-to-end test exercises bothexecute()call sites and thechat.tsfailure site; the success site is exercised by the existingchat-combo-live-testandcombo-provider-cooldownsuites.Reviewer Notes
chat.tsno longer calls_onSuccess()for combo dispatches;combo.tsalready callsrecordProviderSuccess()at both of its success sites, so combo successes now count once instead of twice, and HALF_OPEN → CLOSED recovery from combo traffic goes through that call.execute()'s throw path still records viaisFailure; thechat.tsall-rate-limited branch (breaker._onFailure()afterhandleNoCredentials) is not gated byisCombo; the duplicatedPROVIDER_BREAKER_FAILURE_STATUSESinopen-sse/services/combo/comboPredicates.tsis left in place because of the documented open-sse → src/sse import-cycle constraint.