|
6 | 6 | - **`RetryBudget`** — Finagle-style token bucket; safe to share across sync `Client` and `AsyncClient` in the same process. (Finagle-style bounds the global retry rate to prevent retry storms when downstreams degrade.) |
7 | 7 | - **`Bulkhead` / `AsyncBulkhead`** — concurrency limiter with bounded acquire-wait (`threading.Semaphore` and `asyncio.Semaphore` respectively) |
8 | 8 |
|
9 | | -The canonical composition is `middleware=[AsyncBulkhead(...), AsyncRetry()]` — `AsyncBulkhead` outside `AsyncRetry` so one slot covers all retry attempts of a single call. Reach for the [Middleware guide](middleware.md) when you want to write your own resilience policy. |
| 9 | +A key ordering constraint: `AsyncBulkhead` must sit outside `AsyncRetry` (before it in `middleware=`) so one slot covers all retry attempts of a single call. For the full recommended ordering across all four primitives, see [Composition](#composition). Reach for the [Middleware guide](middleware.md) when you want to write your own resilience policy. |
10 | 10 |
|
11 | 11 | ## `AsyncRetry` |
12 | 12 |
|
@@ -144,28 +144,155 @@ async with ( |
144 | 144 |
|
145 | 145 | When `acquire_timeout` elapses without a slot opening, `AsyncBulkhead` raises `BulkheadFullError` (carries the configured `max_concurrent` and `acquire_timeout` for caller logging). See the [Errors reference](errors.md). The `httpware.bulkhead` `bulkhead.rejected` observability event fires at the same site — see [Observability](index.md#observability). |
146 | 146 |
|
| 147 | +## `AsyncCircuitBreaker` / `CircuitBreaker` |
| 148 | + |
| 149 | +```python |
| 150 | +from httpware.middleware.resilience import AsyncCircuitBreaker # async |
| 151 | +from httpware.middleware.resilience import CircuitBreaker # sync |
| 152 | +``` |
| 153 | + |
| 154 | +Classic consecutive-failure circuit breaker. Counts failures and prevents requests from reaching a downstream that is known to be broken. |
| 155 | + |
| 156 | +### States |
| 157 | + |
| 158 | +- **CLOSED** — normal operation. Each counted failure increments the consecutive-failure counter. Once `failure_threshold` consecutive counted failures accumulate, the circuit opens. |
| 159 | +- **OPEN** — fast-fail. All requests are rejected immediately with `CircuitOpenError` (carrying `retry_after` seconds until the next probe window). After `reset_timeout` seconds the circuit moves to HALF_OPEN. |
| 160 | +- **HALF_OPEN** — exactly one probe is admitted. If `success_threshold` consecutive probe successes are observed, the circuit closes. A single probe failure re-opens the circuit. |
| 161 | + |
| 162 | +### Constructor |
| 163 | + |
| 164 | +| Parameter | Default | Effect | |
| 165 | +|---|---|---| |
| 166 | +| `failure_threshold` | `5` | Consecutive counted failures required to open. `<1` raises `ValueError`. | |
| 167 | +| `reset_timeout` | `30.0` (s) | Seconds to stay OPEN before admitting a probe. `<0` raises `ValueError`. | |
| 168 | +| `success_threshold` | `1` | Consecutive probe successes required to close. `<1` raises `ValueError`. | |
| 169 | +| `failure_status_codes` | `None` | Which status codes count as failures. `None` → all 5xx (`500`–`599`). | |
| 170 | + |
| 171 | +### Failure classification |
| 172 | + |
| 173 | +A **counted failure** is a `NetworkError`, an httpware `TimeoutError`, or a `StatusError` whose status code is in `failure_status_codes`. All other exceptions propagate without affecting circuit state. |
| 174 | + |
| 175 | +**4xx responses — including 429 — count as successes.** A 429 means the service is healthy but throttling; tripping the circuit on it would amplify an incident by adding circuit-open rejections on top of the throttle. |
| 176 | + |
| 177 | +### `CircuitOpenError` |
| 178 | + |
| 179 | +Raised when the circuit is OPEN (with a positive `retry_after: float`) or when HALF_OPEN with a probe already in flight (`retry_after=None`). Inherits `httpware.ClientError`. See the [Errors reference](errors.md). |
| 180 | + |
| 181 | +### Observability |
| 182 | + |
| 183 | +Emitted on logger `httpware.circuit_breaker`: |
| 184 | + |
| 185 | +| Event | When | |
| 186 | +|---|---| |
| 187 | +| `circuit.opened` | Failure threshold reached; circuit transitions CLOSED → OPEN | |
| 188 | +| `circuit.rejected` | Request fast-failed (OPEN or HALF_OPEN probe slot taken) | |
| 189 | +| `circuit.half_open` | Reset timeout elapsed; circuit transitions OPEN → HALF_OPEN | |
| 190 | +| `circuit.closed` | Success threshold reached; circuit transitions HALF_OPEN → CLOSED | |
| 191 | + |
| 192 | +### Sharing |
| 193 | + |
| 194 | +Pass the same instance to multiple clients to enforce one shared circuit across them. A `CircuitBreaker` (sync) cannot be shared with an `AsyncCircuitBreaker` — they use different concurrency primitives. |
| 195 | + |
| 196 | +### Async example |
| 197 | + |
| 198 | +```python |
| 199 | +from httpware import AsyncClient |
| 200 | +from httpware.middleware.resilience import AsyncCircuitBreaker |
| 201 | + |
| 202 | + |
| 203 | +breaker = AsyncCircuitBreaker(failure_threshold=3, reset_timeout=60.0) |
| 204 | + |
| 205 | +async with AsyncClient( |
| 206 | + base_url="https://api.example.com", |
| 207 | + middleware=[breaker], |
| 208 | +) as client: |
| 209 | + response = await client.get("/users/1") |
| 210 | +``` |
| 211 | + |
| 212 | +### Sync example |
| 213 | + |
| 214 | +```python |
| 215 | +from httpware import Client |
| 216 | +from httpware.middleware.resilience import CircuitBreaker |
| 217 | + |
| 218 | + |
| 219 | +breaker = CircuitBreaker(failure_threshold=3, reset_timeout=60.0) |
| 220 | + |
| 221 | +with Client( |
| 222 | + base_url="https://api.example.com", |
| 223 | + middleware=[breaker], |
| 224 | +) as client: |
| 225 | + client.get("/users/1") |
| 226 | +``` |
| 227 | + |
| 228 | +## `AsyncTimeout` |
| 229 | + |
| 230 | +```python |
| 231 | +from httpware.middleware.resilience import AsyncTimeout |
| 232 | +``` |
| 233 | + |
| 234 | +Bounds total wall-clock time across the entire inner pipeline. Place it outermost to enforce "this whole operation must finish within `timeout` seconds, even across retries and backoff sleeps." On expiry it raises `httpware.TimeoutError`. |
| 235 | + |
| 236 | +| Parameter | Default | Effect | |
| 237 | +|---|---|---| |
| 238 | +| `timeout` | **REQUIRED** | Overall deadline in seconds. Must be `> 0`; `≤0` raises `ValueError`. | |
| 239 | + |
| 240 | +**This is not a per-call timeout.** httpx2's connect/read/write/pool timeouts are the right tool for bounding a single outbound call; `AsyncTimeout` doesn't duplicate them. What httpx2 cannot bound is the total wall-clock across a whole retry sequence — `AsyncTimeout` fills that gap. |
| 241 | + |
| 242 | +**No sync `Timeout` exists.** Sync Python has no cancellation primitive that can interrupt a blocking httpx2 call mid-flight. For sync per-call bounds, configure `httpx2.Timeout` on the wrapped client or pass `timeout=` per request. |
| 243 | + |
| 244 | +Observability event: `timeout.exceeded` on logger `httpware.timeout`. |
| 245 | + |
| 246 | +```python |
| 247 | +from httpware import AsyncClient |
| 248 | +from httpware.middleware.resilience import AsyncCircuitBreaker, AsyncRetry, AsyncTimeout |
| 249 | + |
| 250 | + |
| 251 | +async with AsyncClient( |
| 252 | + base_url="https://api.example.com", |
| 253 | + middleware=[ |
| 254 | + AsyncTimeout(timeout=10.0), # overall deadline across the whole chain |
| 255 | + AsyncRetry(max_attempts=3), |
| 256 | + ], |
| 257 | +) as client: |
| 258 | + response = await client.get("/users/1") |
| 259 | +``` |
| 260 | + |
147 | 261 | ## Composition |
148 | 262 |
|
149 | | -The canonical ordering is `middleware=[AsyncBulkhead, AsyncRetry]` — `AsyncBulkhead` outermost so one slot covers all retry attempts of a single call: |
| 263 | +The recommended ordering (not enforced, but each position has a reason): |
| 264 | + |
| 265 | +``` |
| 266 | +AsyncTimeout → AsyncCircuitBreaker → AsyncBulkhead → AsyncRetry → terminal |
| 267 | +``` |
| 268 | + |
| 269 | +- `AsyncTimeout` outermost so the overall deadline covers the entire sequence including retries and backoff. |
| 270 | +- `AsyncCircuitBreaker` outside `AsyncRetry` so an open circuit short-circuits the whole retry loop without attempting any calls. This also means the breaker counts one outcome per fully-exhausted retry sequence rather than one per individual attempt. Placing it outside `AsyncBulkhead` too means a request the open circuit rejects never consumes a concurrency slot. |
| 271 | +- `AsyncBulkhead` outside `AsyncRetry` so one slot covers all retry attempts of a single call. Flip those two (`[AsyncRetry, AsyncBulkhead]`) and each retry grabs a fresh slot — defeating the bulkhead under load. |
150 | 272 |
|
151 | 273 | ```python |
152 | 274 | from httpware import AsyncClient |
153 | | -from httpware.middleware.resilience import AsyncBulkhead, AsyncRetry |
| 275 | +from httpware.middleware.resilience import ( |
| 276 | + AsyncBulkhead, |
| 277 | + AsyncCircuitBreaker, |
| 278 | + AsyncRetry, |
| 279 | + AsyncTimeout, |
| 280 | +) |
154 | 281 |
|
155 | 282 |
|
156 | 283 | async def main() -> None: |
157 | 284 | async with AsyncClient( |
158 | 285 | base_url="https://api.example.com", |
159 | 286 | middleware=[ |
| 287 | + AsyncTimeout(timeout=30.0), |
| 288 | + AsyncCircuitBreaker(), |
160 | 289 | AsyncBulkhead(max_concurrent=10), |
161 | 290 | AsyncRetry(), |
162 | 291 | ], |
163 | 292 | ) as client: |
164 | 293 | await client.get("/users/1") |
165 | 294 | ``` |
166 | 295 |
|
167 | | -Flipping the order (`[AsyncRetry, AsyncBulkhead]`) means each retry attempt grabs a fresh slot — defeating the bulkhead under load. Don't do that. |
168 | | - |
169 | 296 | Cross-cutting middleware that emit per-call state (e.g., the Request-ID middleware in the [Middleware guide](middleware.md)) should sit outside `AsyncRetry` for the same reason — so all attempts of one call share one ID rather than getting a fresh ID per attempt. |
170 | 297 |
|
171 | 298 | ## Sync Retry and Bulkhead |
@@ -227,6 +354,6 @@ with Client( |
227 | 354 | ## See also |
228 | 355 |
|
229 | 356 | - **[Middleware guide](middleware.md)** — write your own resilience middleware against the same protocol `AsyncRetry` and `AsyncBulkhead` use. |
230 | | -- **[Errors reference](errors.md)** — `RetryBudgetExhaustedError`, `BulkheadFullError`, and the broader exception tree. |
231 | | -- **[Observability](index.md#observability)** — the four operational events these middleware emit. |
| 357 | +- **[Errors reference](errors.md)** — `RetryBudgetExhaustedError`, `BulkheadFullError`, `CircuitOpenError`, and the broader exception tree. |
| 358 | +- **[Observability](index.md#observability)** — the operational events these middleware emit. |
232 | 359 | - **`planning/engineering.md` §3** — the formal Middleware/Seam-A contract. |
0 commit comments