Skip to content

Commit 05c438f

Browse files
lesnik512claude
andcommitted
spec: retry-budget cluster (0.8.3) — close 7 audit findings
Pairs the 3 RetryBudget-cluster code fixes (deposit per-request hoist, math.ceil ceiling, Retry-After give-up when > max_delay) with the 2 chunk-3 test rewrites that should have caught them, plus the 2 adjacent retry nits (streaming-note scoping, RuntimeError → TransportError via is_closed). Decisions: deposit() hoists above the loop, math.ceil ceiling, Retry-After > max_delay raises the underlying StatusError with PEP 678 note rather than silently capping. All behavioral changes documented in 0.8.3 release notes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 00001c6 commit 05c438f

1 file changed

Lines changed: 288 additions & 0 deletions

File tree

Lines changed: 288 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,288 @@
1+
# Spec: Retry/Budget cluster — close 7 audit findings (0.8.3)
2+
3+
**Date:** 2026-06-08
4+
**Topic slug:** `retry-budget-cluster`
5+
**Branch:** `fix/retry-budget-cluster`
6+
**Target release:** `0.8.3` — patch (bug fixes, no API additions)
7+
**Status:** drafted, awaiting user review
8+
9+
## Purpose
10+
11+
Close the seven audit-flagged retry-and-budget findings the [deep audit](../audit/2026-06-07-deep-audit.md) identified as the second cross-cutting theme — five `RetryBudget` cluster findings (one of which already shipped: the docs/resilience.md "Single-thread assumption" High landed in `a801572`) plus two adjacent retry-surface nits the user opted in for this PR. The remaining six findings are:
12+
13+
| # | Severity | File | Headline |
14+
|---|---|---|---|
15+
| 1 | Low | `src/httpware/middleware/resilience/retry.py:105` (async) + `:236` (sync) | `deposit()` fires per attempt instead of per original request |
16+
| 2 | Low | `src/httpware/middleware/resilience/retry.py:189` (async) + `:320` (sync) | `Retry-After` silently capped to `max_delay` |
17+
| 3 | Low | `src/httpware/middleware/resilience/budget.py:67` | Ceiling truncates rather than rounds (`int(...)`) |
18+
| 4 | Low | `tests/test_budget_props.py:52` | Property test mirrors the same buggy formula |
19+
| 5 | Nit | `tests/test_retry_props.py:60` | Property tests use a budget that cannot be exhausted |
20+
| 6 | Nit | `src/httpware/middleware/resilience/retry.py:111` (async) + `:242`/`:249` (sync) | Streaming-body refusal note attached on wrong branch |
21+
| 7 | Nit | `src/httpware/client.py:135` (async) + `:852` (sync) | `RuntimeError → TransportError` uses brittle substring match on "closed" |
22+
23+
Items 1, 2, 3 reinforce each other — Chunk 4's High noted that the code defects and the doc/test weaknesses form one cluster. Item 4 is the test that should have caught item 3 but didn't (uses the same buggy `int(...)` formula as production). Item 5 leaves the budget-exhaustion path unexercised by property tests.
24+
25+
## Non-goals
26+
27+
- **No API additions.** No new public classes, methods, or kwargs. No `respect_retry_after_ceiling=` config. No new exception types.
28+
- **No semver minor bump.** This is a patch release; behavioral changes are bug fixes against documented contracts.
29+
- **No documentation rewrites beyond what the fixes require.** Resilience.md gets minimal touch-ups for the new behaviors; no full doc refresh.
30+
- **No follow-up audit findings deferred.** Every RetryBudget-flagged finding the user opted in for ships here. The remaining open Low/Nit items (sync `Bulkhead` Hypothesis-property gap, `test_expected_exports` one-directional, optional-extras edges, etc.) stay open and may ship in a later PR.
31+
- **No `Retry` constructor signature changes.** `respect_retry_after: bool = True` stays — the new give-up behavior is unconditional when the header value exceeds `max_delay`.
32+
- **No deprecation period.** Project is 0.x; user has stated no production deployments. Behavioral changes ship straight in.
33+
34+
## Architecture
35+
36+
### Branch + commit structure
37+
38+
Single feature branch `fix/retry-budget-cluster` off `main`. Sequenced commits, one per logical change:
39+
40+
1. `fix(budget): math.ceil for ceiling` — finding #3 + #4 paired (production fix + test rewrite that no longer mirrors the formula)
41+
2. `fix(retry): hoist deposit() above the attempt loop` — finding #1 (both async + sync)
42+
3. `feat(retry): give up when Retry-After exceeds max_delay` — finding #2 (both async + sync); behavioral change, includes docstring update + retry.md note
43+
4. `test(retry): property test for budget-exhaustion path` — finding #5
44+
5. `fix(retry): scope streaming-body refusal note correctly` — finding #6
45+
6. `fix(client): map RuntimeError → TransportError via is_closed, not substring` — finding #7
46+
7. `docs(release): draft 0.8.3 notes` — release notes file
47+
48+
Each commit includes the targeted tests that exercise the new behavior and references the audit finding in the message. After merge to `main`, the user tags `0.8.3` from the merge SHA following the project's bare-semver tag convention ([memory: release-0-1-0-shipped](../../.claude/projects/-Users-kevinsmith-src-pypi-httpware/memory/release_0_1_0_shipped.md)).
49+
50+
### PR + release notes
51+
52+
After all commits land on the branch:
53+
- Open PR against `main` with title `fix(retry-budget): close 7 audit findings (0.8.3)`.
54+
- PR body summarizes the 7 findings closed, behavioral impact, audit linkage.
55+
- Release notes drafted at `planning/releases/0.8.3.md` — mirrors the 0.8.1 file's structure (TL;DR + the change-by-change breakdown). Two-section: "Fixes" (mostly invisible) + "Behavioral changes you should know about" (the deposit-hoist semantics shift + Retry-After give-up).
56+
57+
## Per-finding change list
58+
59+
### Finding #1 — `deposit()` per-request (lines 105, 236)
60+
61+
**Current behavior:** `self.budget.deposit()` lives inside the per-attempt `for attempt in range(self.max_attempts):` loop. A request that retries twice deposits three tokens but withdraws two; the budget's deposits-to-withdrawals ratio is inflated by ~`(attempts-1)/attempts`, letting through more retries than `percent_can_retry` should permit.
62+
63+
**Change:** hoist `self.budget.deposit()` to immediately precede the loop:
64+
65+
```python
66+
self.budget.deposit()
67+
for attempt in range(self.max_attempts):
68+
is_last = attempt + 1 >= self.max_attempts
69+
try:
70+
return await next(request)
71+
except StatusError as exc:
72+
...
73+
```
74+
75+
Same hoist in the sync class around line 236.
76+
77+
**Behavioral impact:** users with retry traffic see the budget refuse retries earlier than before. The exact effect depends on `percent_can_retry` and average retry depth — for `percent_can_retry=0.2` and average 1 retry per request, withdrawals/deposits goes from `(1)/(2) = 0.5` to `(1)/(1) = 1.0` per request, so the budget hits ceiling sooner. **This is the documented contract**; the existing behavior was the bug.
78+
79+
### Finding #2 — Retry-After exceeds max_delay → give up (lines 189, 320)
80+
81+
**Current behavior:** `delay = min(retry_after, self.max_delay)` silently clamps. If server says wait 120s and `max_delay=5.0`, the client retries after 5s — almost certainly hitting the same 503/429 and burning an attempt.
82+
83+
**Change:** when `retry_after > self.max_delay`, do NOT sleep-and-retry; instead, raise the last observed `StatusError` with a PEP 678 note explaining the give-up. Same in sync class. The non-retry-after path (jittered backoff, line 192-195 / 322-326) is unchanged.
84+
85+
New retry.py code shape (async):
86+
87+
```python
88+
if retry_after is not None:
89+
if retry_after > self.max_delay:
90+
if last_exc is not None:
91+
last_exc.add_note(
92+
_RETRY_AFTER_EXCEEDS_MAX_DELAY_NOTE.format(
93+
retry_after=retry_after, max_delay=self.max_delay,
94+
),
95+
)
96+
raise last_exc
97+
# Should not reach here under normal flow (retry_after requires last_response)
98+
delay = self.max_delay
99+
else:
100+
delay = retry_after
101+
else:
102+
delay = full_jitter_delay(...)
103+
```
104+
105+
Add module constant near the existing constants:
106+
107+
```python
108+
_RETRY_AFTER_EXCEEDS_MAX_DELAY_NOTE = (
109+
"httpware: Retry-After ({retry_after}s) exceeded max_delay ({max_delay}s); giving up"
110+
)
111+
```
112+
113+
**Behavioral impact:** the rare case where Retry-After > max_delay now surfaces as the original StatusError (e.g., `ServiceUnavailableError`) with an explanatory PEP 678 note rather than triggering a likely-failing retry. Users who want the old behavior should set `max_delay` high enough to accommodate the server's hints, or set `respect_retry_after=False`. Doc this in the existing `respect_retry_after` description in resilience.md.
114+
115+
### Finding #3 — Ceiling math.ceil (budget.py:67)
116+
117+
**Current behavior:** `ceiling = int(len(self._deposits) * self._percent_can_retry) + floor` — truncates. For `len(_deposits)=4` and `percent_can_retry=0.2`, the term is `int(0.8) = 0`; combined with a `floor=0` (no `min_retries_per_sec` floor), zero retries are permitted even though the configured percentage says the first retry should be allowed at 5 deposits.
118+
119+
**Change:**
120+
121+
```python
122+
import math
123+
# ...
124+
ceiling = math.ceil(len(self._deposits) * self._percent_can_retry) + floor
125+
```
126+
127+
Add `import math` to budget.py imports (currently lacks it).
128+
129+
**Behavioral impact:** users with `min_retries_per_sec=0.0` and low traffic now reach the configured percentage threshold one deposit sooner. Combined with finding #1's deposit hoist (deposits per-request not per-attempt), users see fewer retries than before in low-traffic scenarios — but the new ratio MATCHES the documented `percent_can_retry`. The previous behavior was strictly under-permissive (rounding down).
130+
131+
### Finding #4 — Budget property test independent ceiling (test_budget_props.py:52)
132+
133+
**Current test:**
134+
135+
```python
136+
ceiling = int(deposits * percent) + floor
137+
# ...
138+
assert permitted <= ceiling
139+
```
140+
141+
The test computes its expected bound with the same `int(...)` truncation as production. The Finding #3 fix changes production to `math.ceil`; if the test still uses `int(...)`, the assertion `permitted <= ceiling` would become tighter than reality (production permits 1 more in some cases) — false failures.
142+
143+
**Change:** compute the new (fixed) expected ceiling using `math.ceil`. Tighten the assertion to a range so the test discriminates between the two roundings:
144+
145+
```python
146+
import math
147+
expected_ceiling = math.ceil(deposits * percent) + floor
148+
# permitted must equal the post-fix ceiling exactly (production now matches the test's math)
149+
assert permitted == expected_ceiling
150+
```
151+
152+
(`assert permitted == expected_ceiling` only works once production is also `math.ceil` — this commit pairs with finding #3 in commit 1 of the branch.)
153+
154+
### Finding #5 — Retry property suite that exhausts the budget (test_retry_props.py)
155+
156+
**Current state:** every `AsyncRetry` constructed in `test_retry_props.py` uses `RetryBudget(ttl=60.0, min_retries_per_sec=1000.0)` — floor of 60,000 retries permitted unconditionally. The budget can never be exhausted in the search space.
157+
158+
**Change:** add a new property test, **without removing the existing one**:
159+
160+
```python
161+
@given(
162+
max_attempts=st.integers(min_value=2, max_value=4),
163+
deposits=st.integers(min_value=1, max_value=20),
164+
percent=st.floats(min_value=0.0, max_value=0.5, allow_nan=False, allow_infinity=False),
165+
)
166+
@settings(max_examples=50, deadline=None)
167+
async def test_budget_exhaustion_raises_retry_budget_exhausted_error(
168+
max_attempts: int,
169+
deposits: int,
170+
percent: float,
171+
) -> None:
172+
"""When the budget is exhausted, the next retry raises RetryBudgetExhaustedError."""
173+
budget = RetryBudget(ttl=60.0, min_retries_per_sec=0.0, percent_can_retry=percent)
174+
# Pre-deposit (does NOT permit withdrawal — only deposits count toward ceiling)
175+
for _ in range(deposits):
176+
budget.deposit()
177+
# Drain the permitted ceiling by raw withdrawals (mock the budget caller path)
178+
import math
179+
ceiling = math.ceil(deposits * percent)
180+
for _ in range(ceiling):
181+
assert budget.try_withdraw()
182+
# The next withdrawal must fail
183+
assert not budget.try_withdraw()
184+
```
185+
186+
This is a property test on `RetryBudget` itself, not `AsyncRetry`. It complements the existing retry property tests without disturbing them.
187+
188+
### Finding #6 — Streaming-body refusal note (retry.py:111 + 242 + 249)
189+
190+
**Current:**
191+
192+
```python
193+
except StatusError as exc:
194+
retryable_status = exc.response.status_code in self.retry_status_codes
195+
if not method_eligible or not retryable_status:
196+
if retryable_status and request.extensions.get(STREAMING_BODY_MARKER):
197+
exc.add_note(_STREAMING_BODY_REFUSAL_NOTE)
198+
raise
199+
```
200+
201+
When `method_eligible=False` (e.g., POST not in `retry_methods`), the streaming-body note is added anyway — telling the user "stream cannot replay" when the real reason is method ineligibility.
202+
203+
**Change:** scope the note correctly. Only attach it when the streaming marker is the actual blocker (method IS eligible, status IS retryable, but body cannot replay). The cleanest version: drop the misleading branch entirely, and let the streaming-refusal happen at a single dedicated check earlier in the retry decision tree (the existing `STREAMING_BODY_REFUSAL_NOTE` site at the eligible-but-streaming path stays).
204+
205+
After investigation in the implementation phase: if no other site exists, add a check before the early-return that attaches the note ONLY when the streaming marker is the blocker. Concrete shape:
206+
207+
```python
208+
except StatusError as exc:
209+
retryable_status = exc.response.status_code in self.retry_status_codes
210+
if not method_eligible or not retryable_status:
211+
raise
212+
if request.extensions.get(STREAMING_BODY_MARKER):
213+
exc.add_note(_STREAMING_BODY_REFUSAL_NOTE)
214+
raise
215+
last_exc = exc
216+
last_response = exc.response
217+
```
218+
219+
Same pattern at the sync class lines 242 + 249 (the `NetworkError`/`TimeoutError` arm at 249 has the same shape).
220+
221+
### Finding #7 — `RuntimeError → TransportError` via `is_closed`
222+
223+
**Current (client.py:134-135 async, 852-853 sync):**
224+
225+
```python
226+
except RuntimeError as exc:
227+
if "closed" in str(exc):
228+
raise TransportError(str(exc)) from exc
229+
raise
230+
```
231+
232+
**Change:** check `self._httpx2_client.is_closed` instead of inspecting the exception message. `is_closed` is already used elsewhere in `client.py` (lines 774, 784, 870, 880 — proven public API).
233+
234+
```python
235+
except RuntimeError as exc:
236+
if self._httpx2_client.is_closed:
237+
raise TransportError(str(exc)) from exc
238+
raise
239+
```
240+
241+
**Behavioral impact:** any `RuntimeError` raised by httpx2 when the client is closed is now mapped to `TransportError` regardless of message wording. Any `RuntimeError` from other sources (caller mistakes, plugin bugs) propagates as-is — previously, if its message contained "closed", it would have been mis-classified.
242+
243+
## Behavioral impact summary
244+
245+
For the release-notes section:
246+
247+
- **Fewer retries permitted under load.** Finding #1 hoists `deposit()` to fire once per call, not once per attempt. Combined with finding #3's `math.ceil` ceiling, the budget now permits exactly the documented `percent_can_retry` rate. Users with finely-tuned percent settings should re-validate against their target retry rate.
248+
- **`Retry-After > max_delay` now raises instead of silently retrying.** A `ServiceUnavailableError` (or whatever the underlying status error is) with a PEP 678 note `httpware: Retry-After (Ns) exceeded max_delay (Ms); giving up` replaces the previous wrong-but-not-broken retry. Set `max_delay` higher or `respect_retry_after=False` to opt out.
249+
- **`RuntimeError → TransportError` mapping is now message-independent.** Driven by `httpx2.Client.is_closed`. Robust against httpx2 message rewording.
250+
- **Streaming-body refusal note is now precisely targeted.** Only attached when streaming IS the blocker — not when the method was simply ineligible.
251+
252+
## Documentation
253+
254+
- `docs/resilience.md`: extend the `respect_retry_after` description with one sentence on the new give-up behavior when the header exceeds `max_delay`. No other doc changes.
255+
- `planning/releases/0.8.3.md`: new file. Mirrors the 0.8.1 structure (title-line headline, "Patch release with..." TL;DR, per-finding sections, "Behavioral changes" callout).
256+
257+
## Verification
258+
259+
After each commit:
260+
261+
```bash
262+
just lint-ci
263+
uv run pytest -x --no-cov -q
264+
```
265+
266+
Full test suite must pass after every commit.
267+
268+
Behavioral verification per finding:
269+
270+
- Finding #1: existing `test_retry_budget_threadsafety` + `test_threading_with_shared_budget` should continue to pass (they count deposits exactly); a new unit test in `test_retry.py` asserts `budget.deposit()` is called exactly once per `dispatch` call regardless of attempts.
271+
- Finding #2: new tests in `test_retry.py` covering (a) `retry_after <= max_delay` → retries (existing behavior), (b) `retry_after > max_delay` → raises the underlying StatusError with the new PEP 678 note. Mirror in `test_retry_sync.py`.
272+
- Finding #3: the rewritten `test_try_withdraw_never_exceeds_theoretical_bound` (now using `math.ceil`) is the primary regression guard.
273+
- Finding #5: the new budget-exhaustion property test is the test for itself.
274+
- Finding #6: a new test_retry.py case for "POST with streaming body that gets 503" — should raise `MethodNotAllowedRetryError` (or whatever — the existing StatusError class) WITHOUT the streaming-body note. Plus a case for "GET with streaming body that gets 503" — should raise WITH the note.
275+
- Finding #7: a test injecting an `httpx2.MockTransport` that raises a custom `RuntimeError` whose message does NOT contain "closed" — should propagate as `RuntimeError`. Combined with one case where the underlying client is closed (forcing `is_closed=True`) — should map to `TransportError`.
276+
277+
## Open questions
278+
279+
None deferred. All behavioral choices are made (Retry-After: give up with PEP 678 note; deposit: hoist; ceiling: `math.ceil`).
280+
281+
## Acceptance criteria
282+
283+
1. Six fix commits + one release-notes commit land on branch `fix/retry-budget-cluster`.
284+
2. `just lint-ci` and `uv run pytest` are both green after every commit and after the last.
285+
3. New tests cover every finding's behavioral change.
286+
4. `planning/releases/0.8.3.md` exists and structurally mirrors `planning/releases/0.8.1.md`.
287+
5. PR opened against `main` with title `fix(retry-budget): close 7 audit findings (0.8.3)` and a body that summarizes each finding closed and the behavioral changes.
288+
6. After user approval + merge + tag, [memory: release_0_8_3_shipped](../../.claude/projects/-Users-kevinsmith-src-pypi-httpware/memory/) is added.

0 commit comments

Comments
 (0)