Skip to content

Commit 80bee92

Browse files
authored
fix: circuit breaker ignores straggler outcomes, unwraps budget exhaustion, rate-limits circuit.rejected (#167)
* fix: circuit breaker ignores straggler outcomes, unwraps budget exhaustion, rate-limits circuit.rejected Outcomes now carry an epoch ticket from admission, so a request that finishes after its circuit changed state no longer reopens, closes or frees the probe slot. A RetryBudgetExhaustedError is classified by its last_exception. circuit.rejected is emitted outside the sync lock, at most once per second per circuit, with a suppressed count. * test: keep the async straggler harness covered on Python 3.11 On 3.11, awaiting a task that ended with an exception drops coverage line events in the awaiting frame until its next suspension. The harness tasks now swallow the breaker's error themselves. * fix: emit circuit breaker events outside the lock and count rejections per outage Breaker transitions queue their events and the drivers emit them after releasing the sync lock, so a logging handler that sends a request through the same breaker no longer deadlocks. Rejection accounting restarts with each outage: the first rejection is always emitted and circuit.closed reports the outage's total in a new rejected attribute.
1 parent 75f8544 commit 80bee92

9 files changed

Lines changed: 1153 additions & 115 deletions

‎CONTEXT.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -46,9 +46,9 @@ _Avoid_: adapter, which names the third-party object a decoder builds and memoiz
4646
**Counted failure**:
4747
The circuit breaker's unit of account: a `NetworkError`, an httpware `TimeoutError`, or a
4848
`StatusError` whose status is in the effective failure set (default: all 5xx). Every other 4xx —
49-
429 included — counts as a *success*, and any other exception type propagates without touching
50-
circuit state. "Failure" alone is ambiguous here: a failed request is very often not a counted
51-
failure.
49+
429 included — counts as a *success*. A `RetryBudgetExhaustedError` is classified by its
50+
`last_exception`, and any other exception type propagates without touching circuit state. "Failure"
51+
alone is ambiguous here: a failed request is very often not a counted failure.
5252

5353
**Circuit key**:
5454
The value a keyed circuit breaker computes from a request to pick its circuit; requests with equal

‎docs/observability.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ The logger and event names below are a stable public contract:
1111
| `httpware.circuit_breaker` | `circuit.opened` (WARNING), `circuit.rejected` (WARNING), `circuit.half_open` (INFO), `circuit.closed` (INFO) |
1212
| `httpware.timeout` | `timeout.exceeded` (WARNING) |
1313

14-
Each log record has an `event` field holding the event name, such as `event="circuit.opened"`, which you can filter on in your log aggregator. Events from `AsyncKeyedCircuitBreaker` and `KeyedCircuitBreaker` also carry `circuit_key`, the circuit they belong to. [Resilience](resilience.md) describes when each event fires.
14+
Each log record has an `event` field holding the event name, such as `event="circuit.opened"`, which you can filter on in your log aggregator. `circuit.rejected` is emitted at most once per second per circuit, so an open circuit under heavy traffic does not flood your logs. The first rejection of each outage is always emitted, `suppressed` counts the rejections left out since the previous one, and `circuit.closed` reports the outage's total in `rejected`. Events from `AsyncKeyedCircuitBreaker` and `KeyedCircuitBreaker` also carry `circuit_key`, the circuit they belong to. [Resilience](resilience.md) describes when each event fires.
1515

1616
Events that carry a `url` attribute get it redacted first: httpware strips `user:pass@` userinfo and replaces every query-parameter value with `REDACTED`, keeping the keys. A URL logged as `https://api.test/items?page=REDACTED&sort=REDACTED` came from `https://api.test/items?page=2&sort=name`.
1717

‎docs/resilience.md‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -160,6 +160,8 @@ A circuit breaker counts consecutive failures and, past a threshold, stops sendi
160160
- OPEN: until `reset_timeout` has passed, every request fails at once with `CircuitOpenError`, whose `retry_after` says how many seconds remain. The first request after that moves the circuit to HALF_OPEN and becomes the probe.
161161
- HALF_OPEN: one probe at a time is let through. After `success_threshold` successful probes in a row the circuit closes; one failed probe opens it again.
162162

163+
An outcome only counts against the state its request was admitted under. A request that is still in flight when the circuit changes state, such as a slow call sent before the circuit opened, finishes without touching the circuit. It cannot reopen a HALF_OPEN circuit, count toward `success_threshold`, or free the probe slot.
164+
163165
### Constructor
164166

165167
| Parameter | Default | Effect |
@@ -176,6 +178,8 @@ A circuit breaker counts consecutive failures and, past a threshold, stops sendi
176178

177179
A counted failure is a `NetworkError`, an httpware `TimeoutError`, or a `StatusError` whose status is in `failure_status_codes`. Other exceptions pass through without changing the circuit.
178180

181+
A `RetryBudgetExhaustedError` from a retry placed after the breaker is classified by its `last_exception`. If the last attempt failed with a counted failure, the breaker counts one; if it failed with a 4xx, that is a success.
182+
179183
4xx responses, 429 included, count as successes. A 429 means the service is up and throttling you; opening the circuit on it would add circuit-open rejections on top of the throttling.
180184

181185
### `CircuitOpenError`
@@ -189,9 +193,13 @@ Emitted on logger `httpware.circuit_breaker`:
189193
| Event | When |
190194
|---|---|
191195
| `circuit.opened` | The failure threshold was reached and the circuit went from CLOSED to OPEN. |
192-
| `circuit.rejected` | A request failed fast because the circuit was OPEN or a HALF_OPEN probe was already running. |
196+
| `circuit.rejected` | A request failed fast because the circuit was OPEN or a HALF_OPEN probe was already running. Emitted at most once per second per circuit, and always for the first rejection of an outage. Its `suppressed` attribute counts the rejections in the same outage that were not emitted since the previous `circuit.rejected`. |
193197
| `circuit.half_open` | The reset timeout passed and the circuit went from OPEN to HALF_OPEN. |
194-
| `circuit.closed` | The success threshold was reached and the circuit went from HALF_OPEN to CLOSED. |
198+
| `circuit.closed` | The success threshold was reached and the circuit went from HALF_OPEN to CLOSED. Its `rejected` attribute is the number of requests rejected during the outage, emitted or not. |
199+
200+
An outage runs from the moment the circuit opens until it closes again; a probe failure that reopens it stays in the same outage.
201+
202+
The breaker emits its events after releasing its internal lock, so a logging handler may send requests through a client that uses the same breaker.
195203

196204
### Time-based failure-rate mode
197205

0 commit comments

Comments
 (0)