Skip to content

Commit 7b41094

Browse files
authored
docs: put the circuit breaker before AsyncTimeout; document rate-mode and event-loop behaviour (#178)
* fix: explain per-event-loop construction in the async resilience cross-loop error * docs: recommend the circuit breaker before AsyncTimeout so timeouts count as failures * docs: explain that a success can open the circuit in rate mode * docs: describe the one-event-loop-per-instance rule for async resilience middleware * docs: align the full-stack demo with the breaker-before-timeout order
1 parent a45e0eb commit 7b41094

10 files changed

Lines changed: 141 additions & 54 deletions

File tree

‎docs/adr/0004-circuit-breaker-scope.md‎

Lines changed: 13 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,16 @@
22

33
Rate mode uses a rolling `window_seconds` gated by `minimum_calls`. A count-based "last N calls"
44
window was rejected because on low traffic it reaches back past deploys and recoveries, and
5-
`minimum_calls` already supplies the sample-size guarantee. A slow-call rate is not added because
6-
`AsyncTimeout` already converts a slow call into a counted `TimeoutError`. `force_open()` /
7-
`force_closed()` in the style of Polly's `ManualControl` were declined in
8-
[#118](https://github.com/modern-python/httpware/issues/118): breaker state is per process, so
9-
pinning one replica sheds nothing fleet-wide, and the fixes that do work, a config flag read by every
10-
replica or state in Redis, reduce to a check the caller makes before sending or to I/O inside a
11-
transition.
5+
`minimum_calls` already supplies the sample-size guarantee. A slow-call rate is not added because a
6+
deadline already turns a slow call into a counted failure, provided the breaker sits outside the
7+
deadline. That holds in the recommended order, `AsyncCircuitBreaker → AsyncTimeout → AsyncBulkhead
8+
→ AsyncRetry`: an expired `AsyncTimeout` raises httpware `TimeoutError` through the breaker, which
9+
counts it. With `AsyncTimeout` outside the breaker, the deadline cancels the breaker's call, the
10+
breaker sees `CancelledError`, which it does not count, and a hanging upstream never opens the
11+
circuit. httpware documents this order but does not enforce it, so the argument covers only callers
12+
who follow it. httpx2's per-request timeouts are raised inside the breaker in any order, and they
13+
are the only deadline on the sync side. `force_open()` / `force_closed()` in the style of Polly's
14+
`ManualControl` were declined in [#118](https://github.com/modern-python/httpware/issues/118):
15+
breaker state is per process, so pinning one replica sheds nothing fleet-wide, and the fixes that do
16+
work, a config flag read by every replica or state in Redis, reduce to a check the caller makes
17+
before sending or to I/O inside a transition.

‎docs/demos/engine.js‎

Lines changed: 12 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -793,11 +793,10 @@ window.HttpwareDemo = (function () {
793793
if (L.pend[i].land <= tick) {
794794
const p = L.pend[i]; L.pend.splice(i, 1);
795795
let retried = false;
796-
// AsyncTimeout bounds the WHOLE operation (outermost: timeout ->
797-
// circuitBreaker -> bulkhead -> retry -> terminal). An entry clamped
798-
// to its deadline (see clampToDeadline) lands as a TimeoutError
799-
// failure here, regardless of what the untimed outcome would have
800-
// been, and is never retried.
796+
// AsyncTimeout bounds the WHOLE operation (circuitBreaker -> timeout ->
797+
// bulkhead -> retry -> terminal). An entry clamped to its deadline (see
798+
// clampToDeadline) lands as a TimeoutError failure here, regardless of
799+
// what the untimed outcome would have been, and is never retried.
801800
if (L === B && p.timedOut) {
802801
timedOutCount++;
803802
// Retry decision happens at landing time: a failed lane-B attempt gets
@@ -811,7 +810,7 @@ window.HttpwareDemo = (function () {
811810
const backoffSec = rnd() * Math.min(retryCfg.maxDelay, retryCfg.baseDelay * Math.pow(2, p.attempt));
812811
const f2 = scenario.fault(now + backoffSec, rnd);
813812
const landTicks = Math.max(1, Math.round((backoffSec + f2.ms) * 1000 / TICK));
814-
// Composition order is timeout -> circuitBreaker -> bulkhead -> retry ->
813+
// Composition order is circuitBreaker -> timeout -> bulkhead -> retry ->
815814
// terminal: the bulkhead sits OUTSIDE retry, so one slot is acquired
816815
// once and held for the WHOLE operation (all attempts), released only
817816
// when the operation finally lands. Carry `bh` forward on every retry
@@ -833,14 +832,12 @@ window.HttpwareDemo = (function () {
833832
// so a null/absent bulkhead or a rejected (never-pushed) request is a no-op.
834833
// Breaker records ONCE per fully-exhausted retry sequence (not per attempt):
835834
// brk.res is called here, only on the terminal (non-retried) landing. A
836-
// timed-out entry is an outer-timeout cancellation: AsyncTimeout is
837-
// OUTERMOST, so the inner breaker's `except BaseException` clause does see
838-
// the CancelledError (it releases the in-flight probe slot) but does not
839-
// count it as a success/failure outcome — only the outer TimeoutError
840-
// surfaces to the caller.
835+
// timed-out entry counts as a failure: the breaker sits OUTSIDE
836+
// AsyncTimeout, so the TimeoutError raised on expiry passes through it
837+
// as a counted failure (a CancelledError would not be counted).
841838
if (!retried) {
842839
L.if--;
843-
if (L === B && p.timedOut) { L.bad++; }
840+
if (L === B && p.timedOut) { if (brk) brk.res(false, now); L.bad++; }
844841
else { if (L === B && brk) brk.res(p.ok, now); if (p.ok) L.ok++; else L.bad++; }
845842
if (L === B && bulk && p.bh) bulk.release();
846843
}
@@ -864,8 +861,9 @@ window.HttpwareDemo = (function () {
864861
B.rej++; spawnDot(dotsB, els.trackB, 'rej');
865862
} else {
866863
if (budget) budget.deposit(now);
867-
// Total deadline set once at the first attempt (Infinity when no
868-
// AsyncTimeout is configured, so the clamp below is always a no-op).
864+
// Total deadline set once at the first attempt, after the breaker admitted
865+
// the request, so an open circuit rejects before any deadline starts
866+
// (Infinity when no AsyncTimeout is configured: the clamp is a no-op).
869867
const deadline = tmoCfg ? now + tmoCfg.timeout : Infinity;
870868
const { land, timedOut } = clampToDeadline(tick, landTicks, deadline);
871869
B.if++; B.pend.push({ land, ok: f.ok, attempt: 0, bh: !!bulk, start: now, deadline, timedOut });

‎docs/demos/full-stack.md‎

Lines changed: 14 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Full stack: combining the patterns
22

33
Real clients combine these patterns, in the recommended order
4-
`AsyncTimeout -> AsyncCircuitBreaker -> AsyncBulkhead -> AsyncRetry -> terminal`.
4+
`AsyncCircuitBreaker -> AsyncTimeout -> AsyncBulkhead -> AsyncRetry -> terminal`.
55
Here both clients go through an incident in three phases, and each layer handles a
66
different phase.
77

@@ -13,30 +13,32 @@ document.addEventListener('DOMContentLoaded', function () {
1313
scenarios: [
1414
{ id: 'incident', label: 'Multi-phase incident', dur: 16.0,
1515
fault: (now, rnd) => {
16-
if (now >= 2.0 && now < 5.0) return { ok: true, ms: 4.0, label: 'latency spike' };
17-
if (now >= 5.0 && now < 8.0) return rnd() < 0.6
16+
if (now >= 2.0 && now < 5.0) return rnd() < 0.6
1817
? { ok: false, ms: 0.6, label: 'brownout' } : { ok: true, ms: 0.6 };
18+
if (now >= 5.0 && now < 8.0) return { ok: true, ms: 4.0, label: 'latency spike' };
1919
if (now >= 8.0 && now < 12.0) return { ok: false, ms: 0.3, label: 'hard down' };
2020
return { ok: true, ms: 0.05 };
2121
},
22-
chainB: { timeout: { timeout: 2.0 },
23-
circuitBreaker: { failureThreshold: 5, resetTimeout: 2.0, successThreshold: 1 },
22+
chainB: { circuitBreaker: { failureThreshold: 5, resetTimeout: 2.0, successThreshold: 1 },
23+
timeout: { timeout: 2.0 },
2424
bulkhead: { maxConcurrent: 10, acquireTimeout: 0 },
2525
retry: { maxAttempts: 3, baseDelay: 0.1, maxDelay: 5.0 },
2626
budget: { ttl: 10.0, minRetriesPerSec: 10.0, percentCanRetry: 0.2 } } },
2727
],
2828
macroStrip: true,
2929
stageLabel: (now) => now < 2 ? 'healthy'
30-
: now < 5 ? 'phase 1: latency spike'
31-
: now < 8 ? 'phase 2: brownout'
30+
: now < 5 ? 'phase 1: brownout'
31+
: now < 8 ? 'phase 2: latency spike'
3232
: now < 12 ? 'phase 3: hard down' : 'recovered',
3333
buildStops: () => [
34-
{ when: (s) => s.now >= 2.4, spot: ['poolB', 'elapsedB'], title: 'Phase 1: latency spike',
35-
body: 'The bulkhead caps concurrency so the slow phase cannot exhaust the client, and the timeout ends each operation after 2s.' },
36-
{ when: (s) => s.now >= 5.4, spot: ['ifB'], title: 'Phase 2: brownout',
34+
{ when: (s) => s.now >= 2.4, spot: ['ifB'], title: 'Phase 1: brownout',
3735
body: 'Retry recovers many of the transient errors, and the budget stops it from adding much load.' },
38-
{ when: (s) => s.mw.cb && s.mw.cb.state === 'OPEN', spot: ['brkB'], title: 'Phase 3: hard down',
39-
body: 'Repeated failures open the breaker. It sits before the retry layer, so it rejects whole retry sequences and counts one outcome per sequence, not per attempt.' },
36+
{ when: (s) => s.now >= 5.4, spot: ['poolB', 'elapsedB'], title: 'Phase 2: latency spike',
37+
body: 'The bulkhead caps concurrency so the slow upstream cannot exhaust the client, and the timeout ends each operation after 2s.' },
38+
{ when: (s) => s.mw.cb && s.mw.cb.state === 'OPEN', spot: ['brkB'], title: 'Timeouts open the breaker',
39+
body: 'The breaker sits before the timeout, so each expired deadline counts as a failure. After five in a row the circuit opens, and new calls fail at once instead of waiting 2s on the slow upstream.' },
40+
{ when: (s) => s.now >= 8.5, spot: ['brkB'], title: 'Phase 3: hard down',
41+
body: 'The circuit stays open through the outage, because each half-open probe fails and reopens it. It sits before the retry layer, so it rejects whole retry sequences and counts one outcome per sequence, not per attempt.' },
4042
{ when: (s) => s.now >= 13.5, spot: ['ifA', 'latA', 'ifB', 'latB'], title: 'Full stack vs none',
4143
body: 'The plain client fails through every phase. In the httpware client, each layer handles the phase it is built for.' },
4244
],

‎docs/resilience.md‎

Lines changed: 36 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -182,6 +182,8 @@ An outcome only counts against the state its request was admitted under. A reque
182182

183183
A counted failure is a `NetworkError`, an httpware `TimeoutError`, or a `StatusError` whose status is in `failure_status_codes`. Other exceptions pass through without changing the circuit.
184184

185+
An expired `AsyncTimeout` deadline counts only when the breaker comes before `AsyncTimeout`, as in the [composition order](#composition). With the breaker after it, the deadline cancels the breaker's call, and a cancellation is not a counted failure.
186+
185187
A `RetryBudgetExhaustedError` from a retry placed after the breaker is classified by its `last_exception`. If the last attempt failed with a counted failure, the breaker counts one; if it failed with a 4xx, that is a success.
186188

187189
4xx responses, 429 included, count as successes. A 429 means the service is up and throttling you; opening the circuit on it would add circuit-open rejections on top of the throttling.
@@ -222,6 +224,8 @@ breaker = AsyncCircuitBreaker(
222224
)
223225
```
224226

227+
Once the window holds at least `minimum_calls` outcomes, the rate is checked after every outcome, successes included. A success can therefore be the call that opens the circuit. With `minimum_calls=4` and a threshold of `0.5`, three failures leave the circuit closed because the window holds only three outcomes; a fourth call that succeeds brings it to four, the rate is 3/4, and the circuit opens. That call still returns its response to the caller, and the next request is rejected with `CircuitOpenError`.
228+
225229
In rate mode `failure_threshold` is ignored. Recovery through HALF_OPEN works the same in both modes, and the sync `CircuitBreaker` takes the same parameters.
226230

227231
### State introspection
@@ -311,7 +315,7 @@ In the [composition order](#composition), the keyed breaker takes the circuit br
311315
from httpware.middleware.resilience import AsyncTimeout
312316
```
313317

314-
Bounds the total time of everything after it in the chain. Put it first so a call, with all its retries and backoff, finishes within `timeout` seconds. When time runs out it raises `httpware.TimeoutError`.
318+
Bounds the total time of everything after it in the chain. Put it right after the circuit breaker and before `AsyncBulkhead` and `AsyncRetry`, so a call, with all its retries and backoff, finishes within `timeout` seconds and the breaker counts an expired deadline as a failure. When time runs out it raises `httpware.TimeoutError`.
315319

316320
| Parameter | Default | Effect |
317321
|---|---|---|
@@ -321,17 +325,18 @@ Bounds the total time of everything after it in the chain. Put it first so a cal
321325

322326
There is no sync `Timeout`, because sync Python cannot interrupt a blocking httpx2 call midway. In sync code, set `httpx2.Timeout` on the client or pass `timeout=` per request.
323327

324-
On expiry it emits `timeout.exceeded` on `httpware.timeout`. [Composition](#composition) has an example with `AsyncTimeout` first in the chain.
328+
On expiry it emits `timeout.exceeded` on `httpware.timeout`. [Composition](#composition) has an example with `AsyncTimeout` in the recommended place.
325329

326330
## Composition
327331

328332
httpware doesn't enforce an order, but this one is recommended:
329333

330334
```
331-
AsyncTimeout → AsyncCircuitBreaker → AsyncBulkhead → AsyncRetry → terminal
335+
AsyncCircuitBreaker → AsyncTimeout → AsyncBulkhead → AsyncRetry → terminal
332336
```
333337

334-
- `AsyncTimeout` comes first, so the deadline covers every retry and backoff.
338+
- `AsyncCircuitBreaker` comes before `AsyncTimeout`, so an expired deadline reaches the breaker as an httpware `TimeoutError`, which it counts as a failure, and an upstream that hangs opens the circuit. In the reverse order the deadline fires while the breaker is still waiting on its inner call: the breaker sees a `CancelledError`, which it doesn't count, and the circuit stays closed however long the upstream hangs. The breaker itself never waits, so the deadline loses nothing by starting after it. An open circuit still rejects at once, and the half-open probe runs under the deadline like any other call.
339+
- `AsyncTimeout` comes before `AsyncBulkhead` and `AsyncRetry`, so the deadline covers the wait for a slot, every retry and every backoff.
335340
- `AsyncCircuitBreaker` comes before `AsyncRetry`, so an open circuit rejects the call without starting the retry loop, and the breaker counts one outcome per retry sequence instead of one per attempt. Putting it before `AsyncBulkhead` too means a rejected call never takes a slot.
336341
- `AsyncBulkhead` comes before `AsyncRetry`, so one slot covers every attempt of a call. In the reverse order each retry takes a new slot, and under load the bulkhead admits more work than its cap intends.
337342

@@ -349,8 +354,8 @@ async def main() -> None:
349354
async with AsyncClient(
350355
base_url="https://api.example.com",
351356
middleware=[
352-
AsyncTimeout(timeout=30.0),
353357
AsyncCircuitBreaker(),
358+
AsyncTimeout(timeout=30.0),
354359
AsyncBulkhead(max_concurrent=10),
355360
AsyncRetry(),
356361
],
@@ -360,6 +365,32 @@ async def main() -> None:
360365

361366
Your own middleware that sets something per call, like the request-ID middleware in [Middleware](middleware.md), also belongs before `AsyncRetry`, so every attempt of a call shares one ID.
362367

368+
## One event loop per instance
369+
370+
`AsyncBulkhead`, `AsyncCircuitBreaker` and `AsyncKeyedCircuitBreaker` are bound to the event loop they first run on. Using the same instance from another loop raises `RuntimeError`, and so does a later `asyncio.run()` in the same thread, because each `asyncio.run()` starts a new loop. `httpx2.AsyncClient` can't be reused across `asyncio.run()` calls either. Create the async resilience middleware, and the `AsyncClient` holding it, once per event loop, for example inside the coroutine you pass to `asyncio.run()`:
371+
372+
```python
373+
import asyncio
374+
375+
from httpware import AsyncClient
376+
from httpware.middleware.resilience import AsyncCircuitBreaker, AsyncRetry
377+
378+
379+
async def main() -> None:
380+
async with AsyncClient(
381+
base_url="https://api.example.com",
382+
middleware=[AsyncCircuitBreaker(), AsyncRetry()],
383+
) as client:
384+
await client.get("/users/1")
385+
386+
387+
asyncio.run(main())
388+
```
389+
390+
The usual way to hit the error is in tests: pytest-asyncio gives each test function its own loop by default, so an instance created at module level or in a session-scoped fixture fails in the second test that uses it. Create it in a function-scoped fixture or in the test.
391+
392+
To share a cap or a circuit across threads, use the sync classes with `Client`.
393+
363394
## Sync Retry and Bulkhead
364395

365396
Use these with the sync `Client`.

‎src/httpware/middleware/resilience/bulkhead.py‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -31,9 +31,10 @@
3131
_MAX_CONCURRENT_INVALID = "max_concurrent must be >= 1"
3232
_ACQUIRE_TIMEOUT_INVALID = "acquire_timeout must be >= 0"
3333
_ASYNCBULKHEAD_CROSS_LOOP_MSG = (
34-
"AsyncBulkhead is bound to a single event loop. First seen on {first!r}; "
35-
"current request is on {current!r}. Use one AsyncBulkhead per loop; "
36-
"cross-thread sharing requires the sync Bulkhead primitive."
34+
"AsyncBulkhead is bound to the event loop it first ran on ({first!r}) and was called from {current!r}. "
35+
"Create async resilience middleware, and the AsyncClient holding it, once per event loop, "
36+
"for example inside the coroutine passed to asyncio.run(). "
37+
"Sharing one instance across pytest-asyncio's function-scoped loops is the usual way to hit this."
3738
)
3839

3940
_LOGGER = logging.getLogger("httpware.bulkhead")

‎src/httpware/middleware/resilience/circuit_breaker.py‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -75,14 +75,16 @@
7575
_WINDOW_SECONDS_INVALID = "window_seconds must be > 0"
7676
_MINIMUM_CALLS_INVALID = "minimum_calls must be >= 1"
7777
_CROSS_LOOP_MSG = (
78-
"AsyncCircuitBreaker is bound to a single event loop. First seen on {first!r}; "
79-
"current request is on {current!r}. Use one AsyncCircuitBreaker per loop; "
80-
"cross-thread sharing requires the sync CircuitBreaker primitive."
78+
"AsyncCircuitBreaker is bound to the event loop it first ran on ({first!r}) and was called from {current!r}. "
79+
"Create async resilience middleware, and the AsyncClient holding it, once per event loop, "
80+
"for example inside the coroutine passed to asyncio.run(). "
81+
"Sharing one instance across pytest-asyncio's function-scoped loops is the usual way to hit this."
8182
)
8283
_KEYED_CROSS_LOOP_MSG = (
83-
"AsyncKeyedCircuitBreaker is bound to a single event loop. First seen on {first!r}; "
84-
"current request is on {current!r}. Use one AsyncKeyedCircuitBreaker per loop; "
85-
"cross-thread sharing requires the sync KeyedCircuitBreaker primitive."
84+
"AsyncKeyedCircuitBreaker is bound to the event loop it first ran on ({first!r}) and was called from {current!r}. "
85+
"Create async resilience middleware, and the AsyncClient holding it, once per event loop, "
86+
"for example inside the coroutine passed to asyncio.run(). "
87+
"Sharing one instance across pytest-asyncio's function-scoped loops is the usual way to hit this."
8688
)
8789

8890
_DEFAULT_FAILURE_STATUS_CODES = frozenset(range(500, 600))

‎src/httpware/middleware/resilience/timeout.py‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,9 @@
44
right tool for bounding a single outbound call, and AsyncTimeout does not duplicate
55
them. What httpx2 cannot bound is the total wall-clock across the whole middleware
66
pipeline (most importantly across an AsyncRetry loop, whose attempts and backoff
7-
sleeps it knows nothing about). Place AsyncTimeout outermost to enforce
8-
"this whole operation must finish within `timeout` seconds, even across retries."
7+
sleeps it knows nothing about). Place AsyncTimeout right after the circuit breaker,
8+
before bulkhead and retry, to enforce "this whole operation must finish within
9+
`timeout` seconds, even across retries" and let the breaker count an expired deadline.
910
1011
Async-only by design: a sync total-deadline cannot interrupt a blocking httpx2 call
1112
mid-flight (sync Python has no cancellation), and httpx2 already covers sync per-call
@@ -38,8 +39,9 @@ class AsyncTimeout:
3839
including everything it wraps (retries, backoff sleeps, the call itself).
3940
Must be ``> 0``. On expiry the middleware raises ``httpware.TimeoutError``.
4041
41-
Place outermost in the chain for an overall-operation deadline. For bounding a
42-
single outbound call (connect/read/write/pool), configure ``httpx2`` instead.
42+
Place right after the circuit breaker, before bulkhead and retry, so the breaker
43+
counts an expired deadline. For bounding a single outbound call
44+
(connect/read/write/pool), configure ``httpx2`` instead.
4345
4446
"""
4547

0 commit comments

Comments
 (0)