|
| 1 | +# Spec: Retry/Budget cluster — close 7 audit findings (0.8.3) |
| 2 | + |
| 3 | +**Date:** 2026-06-08 |
| 4 | +**Topic slug:** `retry-budget-cluster` |
| 5 | +**Branch:** `fix/retry-budget-cluster` |
| 6 | +**Target release:** `0.8.3` — patch (bug fixes, no API additions) |
| 7 | +**Status:** drafted, awaiting user review |
| 8 | + |
| 9 | +## Purpose |
| 10 | + |
| 11 | +Close the seven audit-flagged retry-and-budget findings the [deep audit](../audit/2026-06-07-deep-audit.md) identified as the second cross-cutting theme — five `RetryBudget` cluster findings (one of which already shipped: the docs/resilience.md "Single-thread assumption" High landed in `a801572`) plus two adjacent retry-surface nits the user opted in for this PR. The remaining six findings are: |
| 12 | + |
| 13 | +| # | Severity | File | Headline | |
| 14 | +|---|---|---|---| |
| 15 | +| 1 | Low | `src/httpware/middleware/resilience/retry.py:105` (async) + `:236` (sync) | `deposit()` fires per attempt instead of per original request | |
| 16 | +| 2 | Low | `src/httpware/middleware/resilience/retry.py:189` (async) + `:320` (sync) | `Retry-After` silently capped to `max_delay` | |
| 17 | +| 3 | Low | `src/httpware/middleware/resilience/budget.py:67` | Ceiling truncates rather than rounds (`int(...)`) | |
| 18 | +| 4 | Low | `tests/test_budget_props.py:52` | Property test mirrors the same buggy formula | |
| 19 | +| 5 | Nit | `tests/test_retry_props.py:60` | Property tests use a budget that cannot be exhausted | |
| 20 | +| 6 | Nit | `src/httpware/middleware/resilience/retry.py:111` (async) + `:242`/`:249` (sync) | Streaming-body refusal note attached on wrong branch | |
| 21 | +| 7 | Nit | `src/httpware/client.py:135` (async) + `:852` (sync) | `RuntimeError → TransportError` uses brittle substring match on "closed" | |
| 22 | + |
| 23 | +Items 1, 2, 3 reinforce each other — Chunk 4's High noted that the code defects and the doc/test weaknesses form one cluster. Item 4 is the test that should have caught item 3 but didn't (uses the same buggy `int(...)` formula as production). Item 5 leaves the budget-exhaustion path unexercised by property tests. |
| 24 | + |
| 25 | +## Non-goals |
| 26 | + |
| 27 | +- **No API additions.** No new public classes, methods, or kwargs. No `respect_retry_after_ceiling=` config. No new exception types. |
| 28 | +- **No semver minor bump.** This is a patch release; behavioral changes are bug fixes against documented contracts. |
| 29 | +- **No documentation rewrites beyond what the fixes require.** Resilience.md gets minimal touch-ups for the new behaviors; no full doc refresh. |
| 30 | +- **No follow-up audit findings deferred.** Every RetryBudget-flagged finding the user opted in for ships here. The remaining open Low/Nit items (sync `Bulkhead` Hypothesis-property gap, `test_expected_exports` one-directional, optional-extras edges, etc.) stay open and may ship in a later PR. |
| 31 | +- **No `Retry` constructor signature changes.** `respect_retry_after: bool = True` stays — the new give-up behavior is unconditional when the header value exceeds `max_delay`. |
| 32 | +- **No deprecation period.** Project is 0.x; user has stated no production deployments. Behavioral changes ship straight in. |
| 33 | + |
| 34 | +## Architecture |
| 35 | + |
| 36 | +### Branch + commit structure |
| 37 | + |
| 38 | +Single feature branch `fix/retry-budget-cluster` off `main`. Sequenced commits, one per logical change: |
| 39 | + |
| 40 | +1. `fix(budget): math.ceil for ceiling` — finding #3 + #4 paired (production fix + test rewrite that no longer mirrors the formula) |
| 41 | +2. `fix(retry): hoist deposit() above the attempt loop` — finding #1 (both async + sync) |
| 42 | +3. `feat(retry): give up when Retry-After exceeds max_delay` — finding #2 (both async + sync); behavioral change, includes docstring update + retry.md note |
| 43 | +4. `test(retry): property test for budget-exhaustion path` — finding #5 |
| 44 | +5. `fix(retry): scope streaming-body refusal note correctly` — finding #6 |
| 45 | +6. `fix(client): map RuntimeError → TransportError via is_closed, not substring` — finding #7 |
| 46 | +7. `docs(release): draft 0.8.3 notes` — release notes file |
| 47 | + |
| 48 | +Each commit includes the targeted tests that exercise the new behavior and references the audit finding in the message. After merge to `main`, the user tags `0.8.3` from the merge SHA following the project's bare-semver tag convention ([memory: release-0-1-0-shipped](../../.claude/projects/-Users-kevinsmith-src-pypi-httpware/memory/release_0_1_0_shipped.md)). |
| 49 | + |
| 50 | +### PR + release notes |
| 51 | + |
| 52 | +After all commits land on the branch: |
| 53 | +- Open PR against `main` with title `fix(retry-budget): close 7 audit findings (0.8.3)`. |
| 54 | +- PR body summarizes the 7 findings closed, behavioral impact, audit linkage. |
| 55 | +- Release notes drafted at `planning/releases/0.8.3.md` — mirrors the 0.8.1 file's structure (TL;DR + the change-by-change breakdown). Two-section: "Fixes" (mostly invisible) + "Behavioral changes you should know about" (the deposit-hoist semantics shift + Retry-After give-up). |
| 56 | + |
| 57 | +## Per-finding change list |
| 58 | + |
| 59 | +### Finding #1 — `deposit()` per-request (lines 105, 236) |
| 60 | + |
| 61 | +**Current behavior:** `self.budget.deposit()` lives inside the per-attempt `for attempt in range(self.max_attempts):` loop. A request that retries twice deposits three tokens but withdraws two; the budget's deposits-to-withdrawals ratio is inflated by ~`(attempts-1)/attempts`, letting through more retries than `percent_can_retry` should permit. |
| 62 | + |
| 63 | +**Change:** hoist `self.budget.deposit()` to immediately precede the loop: |
| 64 | + |
| 65 | +```python |
| 66 | +self.budget.deposit() |
| 67 | +for attempt in range(self.max_attempts): |
| 68 | + is_last = attempt + 1 >= self.max_attempts |
| 69 | + try: |
| 70 | + return await next(request) |
| 71 | + except StatusError as exc: |
| 72 | + ... |
| 73 | +``` |
| 74 | + |
| 75 | +Same hoist in the sync class around line 236. |
| 76 | + |
| 77 | +**Behavioral impact:** users with retry traffic see the budget refuse retries earlier than before. The exact effect depends on `percent_can_retry` and average retry depth — for `percent_can_retry=0.2` and average 1 retry per request, withdrawals/deposits goes from `(1)/(2) = 0.5` to `(1)/(1) = 1.0` per request, so the budget hits ceiling sooner. **This is the documented contract**; the existing behavior was the bug. |
| 78 | + |
| 79 | +### Finding #2 — Retry-After exceeds max_delay → give up (lines 189, 320) |
| 80 | + |
| 81 | +**Current behavior:** `delay = min(retry_after, self.max_delay)` silently clamps. If server says wait 120s and `max_delay=5.0`, the client retries after 5s — almost certainly hitting the same 503/429 and burning an attempt. |
| 82 | + |
| 83 | +**Change:** when `retry_after > self.max_delay`, do NOT sleep-and-retry; instead, raise the last observed `StatusError` with a PEP 678 note explaining the give-up. Same in sync class. The non-retry-after path (jittered backoff, line 192-195 / 322-326) is unchanged. |
| 84 | + |
| 85 | +New retry.py code shape (async): |
| 86 | + |
| 87 | +```python |
| 88 | +if retry_after is not None: |
| 89 | + if retry_after > self.max_delay: |
| 90 | + if last_exc is not None: |
| 91 | + last_exc.add_note( |
| 92 | + _RETRY_AFTER_EXCEEDS_MAX_DELAY_NOTE.format( |
| 93 | + retry_after=retry_after, max_delay=self.max_delay, |
| 94 | + ), |
| 95 | + ) |
| 96 | + raise last_exc |
| 97 | + # Should not reach here under normal flow (retry_after requires last_response) |
| 98 | + delay = self.max_delay |
| 99 | + else: |
| 100 | + delay = retry_after |
| 101 | +else: |
| 102 | + delay = full_jitter_delay(...) |
| 103 | +``` |
| 104 | + |
| 105 | +Add module constant near the existing constants: |
| 106 | + |
| 107 | +```python |
| 108 | +_RETRY_AFTER_EXCEEDS_MAX_DELAY_NOTE = ( |
| 109 | + "httpware: Retry-After ({retry_after}s) exceeded max_delay ({max_delay}s); giving up" |
| 110 | +) |
| 111 | +``` |
| 112 | + |
| 113 | +**Behavioral impact:** the rare case where Retry-After > max_delay now surfaces as the original StatusError (e.g., `ServiceUnavailableError`) with an explanatory PEP 678 note rather than triggering a likely-failing retry. Users who want the old behavior should set `max_delay` high enough to accommodate the server's hints, or set `respect_retry_after=False`. Doc this in the existing `respect_retry_after` description in resilience.md. |
| 114 | + |
| 115 | +### Finding #3 — Ceiling math.ceil (budget.py:67) |
| 116 | + |
| 117 | +**Current behavior:** `ceiling = int(len(self._deposits) * self._percent_can_retry) + floor` — truncates. For `len(_deposits)=4` and `percent_can_retry=0.2`, the term is `int(0.8) = 0`; combined with a `floor=0` (no `min_retries_per_sec` floor), zero retries are permitted even though the configured percentage says the first retry should be allowed at 5 deposits. |
| 118 | + |
| 119 | +**Change:** |
| 120 | + |
| 121 | +```python |
| 122 | +import math |
| 123 | +# ... |
| 124 | +ceiling = math.ceil(len(self._deposits) * self._percent_can_retry) + floor |
| 125 | +``` |
| 126 | + |
| 127 | +Add `import math` to budget.py imports (currently lacks it). |
| 128 | + |
| 129 | +**Behavioral impact:** users with `min_retries_per_sec=0.0` and low traffic now reach the configured percentage threshold one deposit sooner. Combined with finding #1's deposit hoist (deposits per-request not per-attempt), users see fewer retries than before in low-traffic scenarios — but the new ratio MATCHES the documented `percent_can_retry`. The previous behavior was strictly under-permissive (rounding down). |
| 130 | + |
| 131 | +### Finding #4 — Budget property test independent ceiling (test_budget_props.py:52) |
| 132 | + |
| 133 | +**Current test:** |
| 134 | + |
| 135 | +```python |
| 136 | +ceiling = int(deposits * percent) + floor |
| 137 | +# ... |
| 138 | +assert permitted <= ceiling |
| 139 | +``` |
| 140 | + |
| 141 | +The test computes its expected bound with the same `int(...)` truncation as production. The Finding #3 fix changes production to `math.ceil`; if the test still uses `int(...)`, the assertion `permitted <= ceiling` would become tighter than reality (production permits 1 more in some cases) — false failures. |
| 142 | + |
| 143 | +**Change:** compute the new (fixed) expected ceiling using `math.ceil`. Tighten the assertion to a range so the test discriminates between the two roundings: |
| 144 | + |
| 145 | +```python |
| 146 | +import math |
| 147 | +expected_ceiling = math.ceil(deposits * percent) + floor |
| 148 | +# permitted must equal the post-fix ceiling exactly (production now matches the test's math) |
| 149 | +assert permitted == expected_ceiling |
| 150 | +``` |
| 151 | + |
| 152 | +(`assert permitted == expected_ceiling` only works once production is also `math.ceil` — this commit pairs with finding #3 in commit 1 of the branch.) |
| 153 | + |
| 154 | +### Finding #5 — Retry property suite that exhausts the budget (test_retry_props.py) |
| 155 | + |
| 156 | +**Current state:** every `AsyncRetry` constructed in `test_retry_props.py` uses `RetryBudget(ttl=60.0, min_retries_per_sec=1000.0)` — floor of 60,000 retries permitted unconditionally. The budget can never be exhausted in the search space. |
| 157 | + |
| 158 | +**Change:** add a new property test, **without removing the existing one**: |
| 159 | + |
| 160 | +```python |
| 161 | +@given( |
| 162 | + max_attempts=st.integers(min_value=2, max_value=4), |
| 163 | + deposits=st.integers(min_value=1, max_value=20), |
| 164 | + percent=st.floats(min_value=0.0, max_value=0.5, allow_nan=False, allow_infinity=False), |
| 165 | +) |
| 166 | +@settings(max_examples=50, deadline=None) |
| 167 | +async def test_budget_exhaustion_raises_retry_budget_exhausted_error( |
| 168 | + max_attempts: int, |
| 169 | + deposits: int, |
| 170 | + percent: float, |
| 171 | +) -> None: |
| 172 | + """When the budget is exhausted, the next retry raises RetryBudgetExhaustedError.""" |
| 173 | + budget = RetryBudget(ttl=60.0, min_retries_per_sec=0.0, percent_can_retry=percent) |
| 174 | + # Pre-deposit (does NOT permit withdrawal — only deposits count toward ceiling) |
| 175 | + for _ in range(deposits): |
| 176 | + budget.deposit() |
| 177 | + # Drain the permitted ceiling by raw withdrawals (mock the budget caller path) |
| 178 | + import math |
| 179 | + ceiling = math.ceil(deposits * percent) |
| 180 | + for _ in range(ceiling): |
| 181 | + assert budget.try_withdraw() |
| 182 | + # The next withdrawal must fail |
| 183 | + assert not budget.try_withdraw() |
| 184 | +``` |
| 185 | + |
| 186 | +This is a property test on `RetryBudget` itself, not `AsyncRetry`. It complements the existing retry property tests without disturbing them. |
| 187 | + |
| 188 | +### Finding #6 — Streaming-body refusal note (retry.py:111 + 242 + 249) |
| 189 | + |
| 190 | +**Current:** |
| 191 | + |
| 192 | +```python |
| 193 | +except StatusError as exc: |
| 194 | + retryable_status = exc.response.status_code in self.retry_status_codes |
| 195 | + if not method_eligible or not retryable_status: |
| 196 | + if retryable_status and request.extensions.get(STREAMING_BODY_MARKER): |
| 197 | + exc.add_note(_STREAMING_BODY_REFUSAL_NOTE) |
| 198 | + raise |
| 199 | +``` |
| 200 | + |
| 201 | +When `method_eligible=False` (e.g., POST not in `retry_methods`), the streaming-body note is added anyway — telling the user "stream cannot replay" when the real reason is method ineligibility. |
| 202 | + |
| 203 | +**Change:** scope the note correctly. Only attach it when the streaming marker is the actual blocker (method IS eligible, status IS retryable, but body cannot replay). The cleanest version: drop the misleading branch entirely, and let the streaming-refusal happen at a single dedicated check earlier in the retry decision tree (the existing `STREAMING_BODY_REFUSAL_NOTE` site at the eligible-but-streaming path stays). |
| 204 | + |
| 205 | +After investigation in the implementation phase: if no other site exists, add a check before the early-return that attaches the note ONLY when the streaming marker is the blocker. Concrete shape: |
| 206 | + |
| 207 | +```python |
| 208 | +except StatusError as exc: |
| 209 | + retryable_status = exc.response.status_code in self.retry_status_codes |
| 210 | + if not method_eligible or not retryable_status: |
| 211 | + raise |
| 212 | + if request.extensions.get(STREAMING_BODY_MARKER): |
| 213 | + exc.add_note(_STREAMING_BODY_REFUSAL_NOTE) |
| 214 | + raise |
| 215 | + last_exc = exc |
| 216 | + last_response = exc.response |
| 217 | +``` |
| 218 | + |
| 219 | +Same pattern at the sync class lines 242 + 249 (the `NetworkError`/`TimeoutError` arm at 249 has the same shape). |
| 220 | + |
| 221 | +### Finding #7 — `RuntimeError → TransportError` via `is_closed` |
| 222 | + |
| 223 | +**Current (client.py:134-135 async, 852-853 sync):** |
| 224 | + |
| 225 | +```python |
| 226 | +except RuntimeError as exc: |
| 227 | + if "closed" in str(exc): |
| 228 | + raise TransportError(str(exc)) from exc |
| 229 | + raise |
| 230 | +``` |
| 231 | + |
| 232 | +**Change:** check `self._httpx2_client.is_closed` instead of inspecting the exception message. `is_closed` is already used elsewhere in `client.py` (lines 774, 784, 870, 880 — proven public API). |
| 233 | + |
| 234 | +```python |
| 235 | +except RuntimeError as exc: |
| 236 | + if self._httpx2_client.is_closed: |
| 237 | + raise TransportError(str(exc)) from exc |
| 238 | + raise |
| 239 | +``` |
| 240 | + |
| 241 | +**Behavioral impact:** any `RuntimeError` raised by httpx2 when the client is closed is now mapped to `TransportError` regardless of message wording. Any `RuntimeError` from other sources (caller mistakes, plugin bugs) propagates as-is — previously, if its message contained "closed", it would have been mis-classified. |
| 242 | + |
| 243 | +## Behavioral impact summary |
| 244 | + |
| 245 | +For the release-notes section: |
| 246 | + |
| 247 | +- **Fewer retries permitted under load.** Finding #1 hoists `deposit()` to fire once per call, not once per attempt. Combined with finding #3's `math.ceil` ceiling, the budget now permits exactly the documented `percent_can_retry` rate. Users with finely-tuned percent settings should re-validate against their target retry rate. |
| 248 | +- **`Retry-After > max_delay` now raises instead of silently retrying.** A `ServiceUnavailableError` (or whatever the underlying status error is) with a PEP 678 note `httpware: Retry-After (Ns) exceeded max_delay (Ms); giving up` replaces the previous wrong-but-not-broken retry. Set `max_delay` higher or `respect_retry_after=False` to opt out. |
| 249 | +- **`RuntimeError → TransportError` mapping is now message-independent.** Driven by `httpx2.Client.is_closed`. Robust against httpx2 message rewording. |
| 250 | +- **Streaming-body refusal note is now precisely targeted.** Only attached when streaming IS the blocker — not when the method was simply ineligible. |
| 251 | + |
| 252 | +## Documentation |
| 253 | + |
| 254 | +- `docs/resilience.md`: extend the `respect_retry_after` description with one sentence on the new give-up behavior when the header exceeds `max_delay`. No other doc changes. |
| 255 | +- `planning/releases/0.8.3.md`: new file. Mirrors the 0.8.1 structure (title-line headline, "Patch release with..." TL;DR, per-finding sections, "Behavioral changes" callout). |
| 256 | + |
| 257 | +## Verification |
| 258 | + |
| 259 | +After each commit: |
| 260 | + |
| 261 | +```bash |
| 262 | +just lint-ci |
| 263 | +uv run pytest -x --no-cov -q |
| 264 | +``` |
| 265 | + |
| 266 | +Full test suite must pass after every commit. |
| 267 | + |
| 268 | +Behavioral verification per finding: |
| 269 | + |
| 270 | +- Finding #1: existing `test_retry_budget_threadsafety` + `test_threading_with_shared_budget` should continue to pass (they count deposits exactly); a new unit test in `test_retry.py` asserts `budget.deposit()` is called exactly once per `dispatch` call regardless of attempts. |
| 271 | +- Finding #2: new tests in `test_retry.py` covering (a) `retry_after <= max_delay` → retries (existing behavior), (b) `retry_after > max_delay` → raises the underlying StatusError with the new PEP 678 note. Mirror in `test_retry_sync.py`. |
| 272 | +- Finding #3: the rewritten `test_try_withdraw_never_exceeds_theoretical_bound` (now using `math.ceil`) is the primary regression guard. |
| 273 | +- Finding #5: the new budget-exhaustion property test is the test for itself. |
| 274 | +- Finding #6: a new test_retry.py case for "POST with streaming body that gets 503" — should raise `MethodNotAllowedRetryError` (or whatever — the existing StatusError class) WITHOUT the streaming-body note. Plus a case for "GET with streaming body that gets 503" — should raise WITH the note. |
| 275 | +- Finding #7: a test injecting an `httpx2.MockTransport` that raises a custom `RuntimeError` whose message does NOT contain "closed" — should propagate as `RuntimeError`. Combined with one case where the underlying client is closed (forcing `is_closed=True`) — should map to `TransportError`. |
| 276 | + |
| 277 | +## Open questions |
| 278 | + |
| 279 | +None deferred. All behavioral choices are made (Retry-After: give up with PEP 678 note; deposit: hoist; ceiling: `math.ceil`). |
| 280 | + |
| 281 | +## Acceptance criteria |
| 282 | + |
| 283 | +1. Six fix commits + one release-notes commit land on branch `fix/retry-budget-cluster`. |
| 284 | +2. `just lint-ci` and `uv run pytest` are both green after every commit and after the last. |
| 285 | +3. New tests cover every finding's behavioral change. |
| 286 | +4. `planning/releases/0.8.3.md` exists and structurally mirrors `planning/releases/0.8.1.md`. |
| 287 | +5. PR opened against `main` with title `fix(retry-budget): close 7 audit findings (0.8.3)` and a body that summarizes each finding closed and the behavioral changes. |
| 288 | +6. After user approval + merge + tag, [memory: release_0_8_3_shipped](../../.claude/projects/-Users-kevinsmith-src-pypi-httpware/memory/) is added. |
0 commit comments