Skip to content

Commit 2a2b541

Browse files
authored
Merge pull request #51 from modern-python/feat/circuit-breaker-timeout
Add CircuitBreaker + AsyncTimeout (0.10.0)
2 parents f201b31 + c52d343 commit 2a2b541

17 files changed

Lines changed: 3326 additions & 15 deletions

‎README.md‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77

88
**A Python HTTP client framework with sync and async clients for building resilient service clients.**
99

10-
`httpware` is a thin opinionated wrapper around `httpx2`. It re-exports `httpx2.Request`/`httpx2.Response`, adds a middleware chain composed at client construction, supports opt-in typed response decoding (pydantic and msgspec are both extras), and raises a status-keyed exception tree automatically on 4xx/5xx. It also ships a small resilience suite — `AsyncRetry`/`Retry` middleware with a Finagle-style `RetryBudget`, plus an `AsyncBulkhead`/`Bulkhead` concurrency limiter — under `httpware.middleware.resilience`.
10+
`httpware` is a thin opinionated wrapper around `httpx2`. It re-exports `httpx2.Request`/`httpx2.Response`, adds a middleware chain composed at client construction, supports opt-in typed response decoding (pydantic and msgspec are both extras), and raises a status-keyed exception tree automatically on 4xx/5xx. It also ships a resilience suite under `httpware.middleware.resilience` — `AsyncRetry`/`Retry` with a Finagle-style `RetryBudget`, `AsyncBulkhead`/`Bulkhead` concurrency limiter, `AsyncCircuitBreaker`/`CircuitBreaker` consecutive-failure breaker, and `AsyncTimeout` for overall-operation wall-clock bounds.
1111

1212
> **Status:** Pre-1.0. Public API is subject to change between minor releases until v1.0.
1313
@@ -116,16 +116,18 @@ All 4xx/5xx responses raise typed exceptions automatically: `NotFoundError`, `Se
116116

117117
## Observability
118118

119-
`AsyncRetry`/`Retry` and `AsyncBulkhead`/`Bulkhead` emit operational events via two channels — stdlib `logging` records (always on) and OpenTelemetry span events (when `opentelemetry-api` is installed). Event names and payloads are identical across sync and async; dashboards built against one class apply unchanged to the other.
119+
All resilience middleware emit operational events via two channels — stdlib `logging` records (always on) and OpenTelemetry span events (when `opentelemetry-api` is installed). Event names and payloads are identical across sync and async; dashboards built against one class apply unchanged to the other.
120120

121-
Logger names (`httpware.retry`, `httpware.bulkhead`) and event names (`retry.giving_up`, `retry.budget_refused`, `retry.streaming_refused`, `bulkhead.rejected`) are the stable public contract.
121+
Logger names and event names are the stable public contract: `httpware.retry` (`retry.giving_up`, `retry.budget_refused`, `retry.streaming_refused`), `httpware.bulkhead` (`bulkhead.rejected`), `httpware.circuit_breaker` (`circuit.opened`, `circuit.rejected`, `circuit.half_open`, `circuit.closed`), and `httpware.timeout` (`timeout.exceeded`).
122122

123123
```python
124124
import logging
125125

126-
# Enable visibility into retry / bulkhead operational events
126+
# Enable visibility into resilience operational events
127127
logging.getLogger("httpware.retry").setLevel(logging.WARNING)
128128
logging.getLogger("httpware.bulkhead").setLevel(logging.WARNING)
129+
logging.getLogger("httpware.circuit_breaker").setLevel(logging.WARNING)
130+
logging.getLogger("httpware.timeout").setLevel(logging.WARNING)
129131
```
130132

131133
For OTel attribute enrichment on the active span — install the extra:

‎docs/errors.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
`httpware` raises typed exceptions automatically — everything inherits `ClientError`, and HTTP responses with 4xx/5xx status raise status-keyed `StatusError` subclasses without you having to call `response.raise_for_status()`.
44

5-
For the resilience-specific errors (`RetryBudgetExhaustedError`, `BulkheadFullError`) see the [Resilience reference](resilience.md).
5+
For the resilience-specific errors (`RetryBudgetExhaustedError`, `BulkheadFullError`, `CircuitOpenError`) see the [Resilience reference](resilience.md).
66

77
The status-keyed exception tree is shared between `Client` and `AsyncClient`. Catching `NotFoundError` in sync code uses the same import as catching it in async code (`from httpware import NotFoundError`).
88

@@ -27,6 +27,7 @@ ClientError (catch-all for anything httpware raises)
2727
│ └── ServiceUnavailableError (503)
2828
├── RetryBudgetExhaustedError (a retry was needed but the budget refused)
2929
├── BulkheadFullError (acquire_timeout elapsed before a slot opened)
30+
├── CircuitOpenError (circuit is OPEN or HALF_OPEN probe slot taken; request not forwarded)
3031
├── DecodeError (response_model= decoder failed; HTTP call itself succeeded)
3132
└── MissingDecoderError (no registered decoder claims response_model=; fires before the HTTP call)
3233
```
@@ -119,6 +120,9 @@ exc.response.request.method # the HTTP method
119120
- `max_concurrent: int` — the configured cap
120121
- `acquire_timeout: float | None` — the configured timeout
121122

123+
`CircuitOpenError` carries:
124+
- `retry_after: float | None` — seconds until the circuit will next admit a probe; `None` when a concurrent probe is already in flight (HALF_OPEN slot taken).
125+
122126
Use these for caller-side logging / alerting:
123127

124128
```python

‎docs/resilience.md‎

Lines changed: 134 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
- **`RetryBudget`** — Finagle-style token bucket; safe to share across sync `Client` and `AsyncClient` in the same process. (Finagle-style bounds the global retry rate to prevent retry storms when downstreams degrade.)
77
- **`Bulkhead` / `AsyncBulkhead`** — concurrency limiter with bounded acquire-wait (`threading.Semaphore` and `asyncio.Semaphore` respectively)
88

9-
The canonical composition is `middleware=[AsyncBulkhead(...), AsyncRetry()]` — `AsyncBulkhead` outside `AsyncRetry` so one slot covers all retry attempts of a single call. Reach for the [Middleware guide](middleware.md) when you want to write your own resilience policy.
9+
A key ordering constraint: `AsyncBulkhead` must sit outside `AsyncRetry` (before it in `middleware=`) so one slot covers all retry attempts of a single call. For the full recommended ordering across all four primitives, see [Composition](#composition). Reach for the [Middleware guide](middleware.md) when you want to write your own resilience policy.
1010

1111
## `AsyncRetry`
1212

@@ -144,28 +144,155 @@ async with (
144144

145145
When `acquire_timeout` elapses without a slot opening, `AsyncBulkhead` raises `BulkheadFullError` (carries the configured `max_concurrent` and `acquire_timeout` for caller logging). See the [Errors reference](errors.md). The `httpware.bulkhead` `bulkhead.rejected` observability event fires at the same site — see [Observability](index.md#observability).
146146

147+
## `AsyncCircuitBreaker` / `CircuitBreaker`
148+
149+
```python
150+
from httpware.middleware.resilience import AsyncCircuitBreaker # async
151+
from httpware.middleware.resilience import CircuitBreaker # sync
152+
```
153+
154+
Classic consecutive-failure circuit breaker. Counts failures and prevents requests from reaching a downstream that is known to be broken.
155+
156+
### States
157+
158+
- **CLOSED** — normal operation. Each counted failure increments the consecutive-failure counter. Once `failure_threshold` consecutive counted failures accumulate, the circuit opens.
159+
- **OPEN** — fast-fail. All requests are rejected immediately with `CircuitOpenError` (carrying `retry_after` seconds until the next probe window). After `reset_timeout` seconds the circuit moves to HALF_OPEN.
160+
- **HALF_OPEN** — exactly one probe is admitted. If `success_threshold` consecutive probe successes are observed, the circuit closes. A single probe failure re-opens the circuit.
161+
162+
### Constructor
163+
164+
| Parameter | Default | Effect |
165+
|---|---|---|
166+
| `failure_threshold` | `5` | Consecutive counted failures required to open. `<1` raises `ValueError`. |
167+
| `reset_timeout` | `30.0` (s) | Seconds to stay OPEN before admitting a probe. `<0` raises `ValueError`. |
168+
| `success_threshold` | `1` | Consecutive probe successes required to close. `<1` raises `ValueError`. |
169+
| `failure_status_codes` | `None` | Which status codes count as failures. `None` → all 5xx (`500`–`599`). |
170+
171+
### Failure classification
172+
173+
A **counted failure** is a `NetworkError`, an httpware `TimeoutError`, or a `StatusError` whose status code is in `failure_status_codes`. All other exceptions propagate without affecting circuit state.
174+
175+
**4xx responses — including 429 — count as successes.** A 429 means the service is healthy but throttling; tripping the circuit on it would amplify an incident by adding circuit-open rejections on top of the throttle.
176+
177+
### `CircuitOpenError`
178+
179+
Raised when the circuit is OPEN (with a positive `retry_after: float`) or when HALF_OPEN with a probe already in flight (`retry_after=None`). Inherits `httpware.ClientError`. See the [Errors reference](errors.md).
180+
181+
### Observability
182+
183+
Emitted on logger `httpware.circuit_breaker`:
184+
185+
| Event | When |
186+
|---|---|
187+
| `circuit.opened` | Failure threshold reached; circuit transitions CLOSED → OPEN |
188+
| `circuit.rejected` | Request fast-failed (OPEN or HALF_OPEN probe slot taken) |
189+
| `circuit.half_open` | Reset timeout elapsed; circuit transitions OPEN → HALF_OPEN |
190+
| `circuit.closed` | Success threshold reached; circuit transitions HALF_OPEN → CLOSED |
191+
192+
### Sharing
193+
194+
Pass the same instance to multiple clients to enforce one shared circuit across them. A `CircuitBreaker` (sync) cannot be shared with an `AsyncCircuitBreaker` — they use different concurrency primitives.
195+
196+
### Async example
197+
198+
```python
199+
from httpware import AsyncClient
200+
from httpware.middleware.resilience import AsyncCircuitBreaker
201+
202+
203+
breaker = AsyncCircuitBreaker(failure_threshold=3, reset_timeout=60.0)
204+
205+
async with AsyncClient(
206+
base_url="https://api.example.com",
207+
middleware=[breaker],
208+
) as client:
209+
response = await client.get("/users/1")
210+
```
211+
212+
### Sync example
213+
214+
```python
215+
from httpware import Client
216+
from httpware.middleware.resilience import CircuitBreaker
217+
218+
219+
breaker = CircuitBreaker(failure_threshold=3, reset_timeout=60.0)
220+
221+
with Client(
222+
base_url="https://api.example.com",
223+
middleware=[breaker],
224+
) as client:
225+
client.get("/users/1")
226+
```
227+
228+
## `AsyncTimeout`
229+
230+
```python
231+
from httpware.middleware.resilience import AsyncTimeout
232+
```
233+
234+
Bounds total wall-clock time across the entire inner pipeline. Place it outermost to enforce "this whole operation must finish within `timeout` seconds, even across retries and backoff sleeps." On expiry it raises `httpware.TimeoutError`.
235+
236+
| Parameter | Default | Effect |
237+
|---|---|---|
238+
| `timeout` | **REQUIRED** | Overall deadline in seconds. Must be `> 0`; `≤0` raises `ValueError`. |
239+
240+
**This is not a per-call timeout.** httpx2's connect/read/write/pool timeouts are the right tool for bounding a single outbound call; `AsyncTimeout` doesn't duplicate them. What httpx2 cannot bound is the total wall-clock across a whole retry sequence — `AsyncTimeout` fills that gap.
241+
242+
**No sync `Timeout` exists.** Sync Python has no cancellation primitive that can interrupt a blocking httpx2 call mid-flight. For sync per-call bounds, configure `httpx2.Timeout` on the wrapped client or pass `timeout=` per request.
243+
244+
Observability event: `timeout.exceeded` on logger `httpware.timeout`.
245+
246+
```python
247+
from httpware import AsyncClient
248+
from httpware.middleware.resilience import AsyncCircuitBreaker, AsyncRetry, AsyncTimeout
249+
250+
251+
async with AsyncClient(
252+
base_url="https://api.example.com",
253+
middleware=[
254+
AsyncTimeout(timeout=10.0), # overall deadline across the whole chain
255+
AsyncRetry(max_attempts=3),
256+
],
257+
) as client:
258+
response = await client.get("/users/1")
259+
```
260+
147261
## Composition
148262

149-
The canonical ordering is `middleware=[AsyncBulkhead, AsyncRetry]` — `AsyncBulkhead` outermost so one slot covers all retry attempts of a single call:
263+
The recommended ordering (not enforced, but each position has a reason):
264+
265+
```
266+
AsyncTimeout → AsyncCircuitBreaker → AsyncBulkhead → AsyncRetry → terminal
267+
```
268+
269+
- `AsyncTimeout` outermost so the overall deadline covers the entire sequence including retries and backoff.
270+
- `AsyncCircuitBreaker` outside `AsyncRetry` so an open circuit short-circuits the whole retry loop without attempting any calls. This also means the breaker counts one outcome per fully-exhausted retry sequence rather than one per individual attempt. Placing it outside `AsyncBulkhead` too means a request the open circuit rejects never consumes a concurrency slot.
271+
- `AsyncBulkhead` outside `AsyncRetry` so one slot covers all retry attempts of a single call. Flip those two (`[AsyncRetry, AsyncBulkhead]`) and each retry grabs a fresh slot — defeating the bulkhead under load.
150272

151273
```python
152274
from httpware import AsyncClient
153-
from httpware.middleware.resilience import AsyncBulkhead, AsyncRetry
275+
from httpware.middleware.resilience import (
276+
AsyncBulkhead,
277+
AsyncCircuitBreaker,
278+
AsyncRetry,
279+
AsyncTimeout,
280+
)
154281

155282

156283
async def main() -> None:
157284
async with AsyncClient(
158285
base_url="https://api.example.com",
159286
middleware=[
287+
AsyncTimeout(timeout=30.0),
288+
AsyncCircuitBreaker(),
160289
AsyncBulkhead(max_concurrent=10),
161290
AsyncRetry(),
162291
],
163292
) as client:
164293
await client.get("/users/1")
165294
```
166295

167-
Flipping the order (`[AsyncRetry, AsyncBulkhead]`) means each retry attempt grabs a fresh slot — defeating the bulkhead under load. Don't do that.
168-
169296
Cross-cutting middleware that emit per-call state (e.g., the Request-ID middleware in the [Middleware guide](middleware.md)) should sit outside `AsyncRetry` for the same reason — so all attempts of one call share one ID rather than getting a fresh ID per attempt.
170297

171298
## Sync Retry and Bulkhead
@@ -227,6 +354,6 @@ with Client(
227354
## See also
228355

229356
- **[Middleware guide](middleware.md)** — write your own resilience middleware against the same protocol `AsyncRetry` and `AsyncBulkhead` use.
230-
- **[Errors reference](errors.md)** — `RetryBudgetExhaustedError`, `BulkheadFullError`, and the broader exception tree.
231-
- **[Observability](index.md#observability)** — the four operational events these middleware emit.
357+
- **[Errors reference](errors.md)** — `RetryBudgetExhaustedError`, `BulkheadFullError`, `CircuitOpenError`, and the broader exception tree.
358+
- **[Observability](index.md#observability)** — the operational events these middleware emit.
232359
- **`planning/engineering.md` §3** — the formal Middleware/Seam-A contract.

0 commit comments

Comments
 (0)