release(4.0.1): public redaction helpers, ReDoS validator latency, phantom ban, singleton resets, path-level gates, pytest pinned below 9

Author: rennf93

.mike.yml

@@ -2,6 +2,7 @@ version_selector: true
 title_switch: true
 versions_file: docs/versions/versions.json
 versions:
+  - 4.0.1
   - 4.0.0
   - 3.17.0
   - 3.16.0
@@ -41,4 +42,4 @@ versions:
   - 0.1.0
   - latest
 aliases:
-  latest: 4.0.0
+  latest: 4.0.1

CHANGELOG.md

@@ -10,6 +10,33 @@ Unreleased
 
 ___
 
+v4.0.1 (2026-09-04)
+-------------------
+
+Public redaction helpers exported from guard_core.utils, a v4.0.0 changelog correction for the excluded-header scope, and ReDoS arbiter and singleton reset fixes (v4.0.1)
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
+
+
+### Added
+
+- **`redact_blob_for_display` and `redact_url_for_display`** exported from `guard_core.utils` (and `guard_core.sync.utils`) so consumers that persist or render guard event text can apply the same grammar-based redaction with the default sensitive-name sets. Previously reachable only through the private `guard_core._utils.request_logging` module.
+
+### Changed
+
+- **v4.0.0 changelog correction: excluded-header scope (documentation).** The v4.0.0 "Proxy identity header false positives" line overstated the change; corrected in place below to state that the headers were added to the default excluded detection set, which in 4.0.0 skips only the `ssrf` category for IP-shaped values while every other category still scans them.
+- **Dev dependency `pytest` pinned below 9.** pytest 9.1.1 drops nested conftest fixtures when CLI file arguments interleave a nested test directory with its parent (pytest-dev/pytest#14971), so the dev extra now installs `pytest<9`; lift the pin once upstream ships the fix.
+- **The live-smoke driver clears only its own `smoke:` keys instead of flushing the whole Redis database.**
+
+### Fixed
+
+- **ReDoS reach-probe arbiter stalled 40 seconds per catastrophic custom pattern, and past 40 seconds once retries and multiple candidate builders were involved.** `_first_over_budget_reason` sent all four probe sizes to one child process whose timeout was `2.0s * 4 sizes * 5 samples = 40s`, so a catastrophic pattern burned the full 40 seconds before its structural rejection reason came back; each retry and each candidate builder got its own fresh 40s window, so the true worst case multiplied past the documented bound (measured at 40.8s wall-clock for `^[a-z]+[a-z]+$` alone). Reachable from `SecurityConfig` field validation, `decorators/content_filtering`, `compiler.add_pattern`, and `handlers/_dynamic_rule_application`. A pattern the structural prefilter already flags now measures one probe size per child process, ascending order, `2.0s` plus a measured interpreter-start allowance, stopping at the first child timeout or error and returning immediately instead of waiting on the remaining sizes; an unflagged pattern's reach-probe phase now shares one `2.0s * 4 sizes * 5 samples = 40s` wall-clock budget across every candidate builder and its retry, clamping each child process's own timeout to whatever remains of that shared budget and returning the timeout reason immediately once it is exhausted, so the whole phase is bounded by that single 40s ceiling (plus the same measured interpreter-start allowance) no matter how many builders or retries it takes. The unflagged, combined path still sent all four probe sizes to that shared child process even though `_reach_probe_verdict_from_samples` only ever read the 16000 and 32000 char results, so every builder's share of the 40s budget paid for two measurements the verdict discarded; the combined path now sends only the two sizes the verdict reads, derived from the tail of the same size ladder, while a pattern the structural prefilter already flagged still probes the full ascending ladder for its early-bailout behavior. Several candidate builders can also produce the exact same probe strings for a given pattern (`(?:foo|bar)+$` yields five builders but only two distinct probe sets), and each duplicate used to burn its own full measurement out of the same shared budget; the combined and ascending paths now measure each distinct probe set once per call and skip a builder whose probes were already timed, so duplicate builders cost nothing. A custom pattern whose honest measurement needs more than 40 seconds across all of its probe sets is refused with the timeout reason rather than accepted; the fix is to simplify the pattern. Sync mirror updated in lockstep.
+- **The reach-probe verdict flipped under concurrent CPU load.** `_reach_probe_verdict_from_samples` extrapolated cost from the median of five samples per probe size; under load, a near-budget built-in pattern's median rose enough to reject it even on the existing single retry. The ratio and the extrapolation now use the minimum of the samples, the noise-robust estimator for CPU-time acceptance, clamped to never read as sub-linear (a longer probe can only cost the same or more, so a noisy reading below that floor is measurement noise, not a real growth rate), while the human-readable reason text still reports both the minimum and the median. Sync mirror updated in lockstep.
+- **`reset_global_state()` left a closed `redis_handler` (and `agent_handler`) attached to the IP ban and security headers singletons, and `RateLimitManager.reset()` left a stale `agent_handler` too.** `IPBanManager()` and `SecurityHeadersManager()` return the same `_instance` on every call, and both handler references were only ever cleared on first construction, so a later middleware built with `enable_redis=False` could inherit a stale, closed handler from an earlier reset. `ipban_handler.reset_global_state()` now nulls both references directly; `SecurityHeadersCacheMixin.reset()` (used by `security_headers_handler.reset_global_state()`) now nulls them after flushing the Redis-side cache. `RateLimitManager.reset()` nulled `redis_handler` but never `agent_handler`; it now nulls both. `SusPatternsManager.reset()` already nulled both correctly on audit and was left unchanged. Sync mirrors updated in lockstep.
+- **An empty pair value immediately followed by an angle-bracket placeholder was not redacted.** `redact_blob_for_display("token=<PLANTED>", ...)` treated `<` as a hard separator and the empty span before it as the value, redacting only that empty string and leaving the bracketed placeholder in the output (`token=[REDACTED]<PLANTED>`). An empty value immediately followed by `<` now treats the whole run up to the next hard separator, excluding the angle brackets themselves, as the value, redacting that whole run in place of the brackets (`token=[REDACTED]`). Sync mirror updated in lockstep.
+- **A ban refused by the self-DoS guard (loopback or a configured trusted proxy) was logged and answered as if it had succeeded.** `_resolve_and_apply_threshold_ban` returned `(duration, reason, category)` whenever a threshold crossed, regardless of whether `ban_ip` actually applied the ban; `ban_ip`'s self-DoS refusal path returned `None` exactly like a successful ban, so `_try_threshold_ban` logged `"IP banned due to ..."` and `SuspiciousActivityCheck` answered 403 `"IP has been banned"` while `is_ip_banned` stayed `False` and the next request from that IP was served normally. `ban_ip` now returns `bool` (`True` applied, `False` refused); `_resolve_and_apply_threshold_ban` returns `None` on a refused ban so the request falls through to the normal 400 `"Suspicious activity detected"` response with no phantom ban log line and no penetration-attempt-banned event. The same blind spot existed in `DynamicRuleApplicationMixin._apply_ip_bans` (dynamic-rule IP bans logged `"Banned IP ..."` regardless of outcome) and `BehaviorActionDispatchMixin._execute_ban_action` (behavioral-rule bans logged the same false success and reported `action_taken="ban"` in the agent event even when refused); both now read `ban_ip`'s bool, `_apply_ip_bans` skipping the success log on a refusal and `_execute_ban_action` reporting `action_taken="tracked"` instead of `"ban"`. Sync mirrors updated in lockstep.
+
+___
+
 v4.0.0 (2026-09-04)
 -------------------
 
@@ -38,7 +65,7 @@ Grammar-based secret redaction across log lines and events, expanded detection c
 
 - **Sensitive header, parameter, and body secret leaks.** `log_activity`, `_log_detected_component`, and query URL parameters (`?access_token=...`, `#token=...`, `https://user:PASS@host/`) now mask sensitive values with `[REDACTED]` across all log lines, telemetry previews, and event payloads.
 - **`dump_last_known_rules_snapshot` validation exception.** Stripped unknown `DynamicRules` fields (`emergency_whitelist_only`, `message`) before snapshot validation, allowing local Redis/file fallback writes to succeed during backend outages.
-- **Proxy identity header false positives.** Added proxy identity headers (`forwarded`, `x-forwarded-for`, `x-forwarded-host`, `x-forwarded-proto`, `x-real-ip`, `x-client-ip`, `x-cluster-client-ip`, `cf-connecting-ip`, `true-client-ip`, `fly-client-ip`, `x-envoy-external-address`) to the default excluded detection set to prevent auto-banning proxy IPs.
+- **Proxy identity header false positives.** Added proxy identity headers (`forwarded`, `x-forwarded-for`, `x-forwarded-host`, `x-forwarded-proto`, `x-real-ip`, `x-client-ip`, `x-cluster-client-ip`, `cf-connecting-ip`, `true-client-ip`, `fly-client-ip`, `x-envoy-external-address`) to the default excluded detection set, which in 4.0.0 skips only the `ssrf` category for IP-shaped values while every other category still scans them.
 - **`require_referrer()` scheme parsing.** Reduced scheme-prefixed entries (e.g., `https://example.com:8443`) to host and port before comparison.
 - **`SecurityConfig` collection field revalidation.** Revalidated 13 collection fields (`exclude_paths`, `cors_allow_origins`, `custom_error_responses`, etc.) on attribute reassignment and `model_copy(update=...)`.
 - **SQLi and category detection context gaps.** Expanded SQLi, command injection, and traversal detection to `header` and `url_path` contexts. Kept noise-prone patterns (`ORDER BY n` bare) restricted to bodies/params.

docs/release-notes.md

@@ -10,6 +10,33 @@ Release Notes
 
 ___
 
+v4.0.1 (2026-09-04)
+-------------------
+
+Public redaction helpers exported from guard_core.utils, a v4.0.0 changelog correction for the excluded-header scope, and ReDoS arbiter and singleton reset fixes (v4.0.1)
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
+
+
+### Added
+
+- **`redact_blob_for_display` and `redact_url_for_display`** exported from `guard_core.utils` (and `guard_core.sync.utils`) so consumers that persist or render guard event text can apply the same grammar-based redaction with the default sensitive-name sets. Previously reachable only through the private `guard_core._utils.request_logging` module.
+
+### Changed
+
+- **v4.0.0 changelog correction: excluded-header scope (documentation).** The v4.0.0 "Proxy identity header false positives" line overstated the change; corrected in place below to state that the headers were added to the default excluded detection set, which in 4.0.0 skips only the `ssrf` category for IP-shaped values while every other category still scans them.
+- **Dev dependency `pytest` pinned below 9.** pytest 9.1.1 drops nested conftest fixtures when CLI file arguments interleave a nested test directory with its parent (pytest-dev/pytest#14971), so the dev extra now installs `pytest<9`; lift the pin once upstream ships the fix.
+- **The live-smoke driver clears only its own `smoke:` keys instead of flushing the whole Redis database.**
+
+### Fixed
+
+- **ReDoS reach-probe arbiter stalled 40 seconds per catastrophic custom pattern, and past 40 seconds once retries and multiple candidate builders were involved.** `_first_over_budget_reason` sent all four probe sizes to one child process whose timeout was `2.0s * 4 sizes * 5 samples = 40s`, so a catastrophic pattern burned the full 40 seconds before its structural rejection reason came back; each retry and each candidate builder got its own fresh 40s window, so the true worst case multiplied past the documented bound (measured at 40.8s wall-clock for `^[a-z]+[a-z]+$` alone). Reachable from `SecurityConfig` field validation, `decorators/content_filtering`, `compiler.add_pattern`, and `handlers/_dynamic_rule_application`. A pattern the structural prefilter already flags now measures one probe size per child process, ascending order, `2.0s` plus a measured interpreter-start allowance, stopping at the first child timeout or error and returning immediately instead of waiting on the remaining sizes; an unflagged pattern's reach-probe phase now shares one `2.0s * 4 sizes * 5 samples = 40s` wall-clock budget across every candidate builder and its retry, clamping each child process's own timeout to whatever remains of that shared budget and returning the timeout reason immediately once it is exhausted, so the whole phase is bounded by that single 40s ceiling (plus the same measured interpreter-start allowance) no matter how many builders or retries it takes. The unflagged, combined path still sent all four probe sizes to that shared child process even though `_reach_probe_verdict_from_samples` only ever read the 16000 and 32000 char results, so every builder's share of the 40s budget paid for two measurements the verdict discarded; the combined path now sends only the two sizes the verdict reads, derived from the tail of the same size ladder, while a pattern the structural prefilter already flagged still probes the full ascending ladder for its early-bailout behavior. Several candidate builders can also produce the exact same probe strings for a given pattern (`(?:foo|bar)+$` yields five builders but only two distinct probe sets), and each duplicate used to burn its own full measurement out of the same shared budget; the combined and ascending paths now measure each distinct probe set once per call and skip a builder whose probes were already timed, so duplicate builders cost nothing. A custom pattern whose honest measurement needs more than 40 seconds across all of its probe sets is refused with the timeout reason rather than accepted; the fix is to simplify the pattern. Sync mirror updated in lockstep.
+- **The reach-probe verdict flipped under concurrent CPU load.** `_reach_probe_verdict_from_samples` extrapolated cost from the median of five samples per probe size; under load, a near-budget built-in pattern's median rose enough to reject it even on the existing single retry. The ratio and the extrapolation now use the minimum of the samples, the noise-robust estimator for CPU-time acceptance, clamped to never read as sub-linear (a longer probe can only cost the same or more, so a noisy reading below that floor is measurement noise, not a real growth rate), while the human-readable reason text still reports both the minimum and the median. Sync mirror updated in lockstep.
+- **`reset_global_state()` left a closed `redis_handler` (and `agent_handler`) attached to the IP ban and security headers singletons, and `RateLimitManager.reset()` left a stale `agent_handler` too.** `IPBanManager()` and `SecurityHeadersManager()` return the same `_instance` on every call, and both handler references were only ever cleared on first construction, so a later middleware built with `enable_redis=False` could inherit a stale, closed handler from an earlier reset. `ipban_handler.reset_global_state()` now nulls both references directly; `SecurityHeadersCacheMixin.reset()` (used by `security_headers_handler.reset_global_state()`) now nulls them after flushing the Redis-side cache. `RateLimitManager.reset()` nulled `redis_handler` but never `agent_handler`; it now nulls both. `SusPatternsManager.reset()` already nulled both correctly on audit and was left unchanged. Sync mirrors updated in lockstep.
+- **An empty pair value immediately followed by an angle-bracket placeholder was not redacted.** `redact_blob_for_display("token=<PLANTED>", ...)` treated `<` as a hard separator and the empty span before it as the value, redacting only that empty string and leaving the bracketed placeholder in the output (`token=[REDACTED]<PLANTED>`). An empty value immediately followed by `<` now treats the whole run up to the next hard separator, excluding the angle brackets themselves, as the value, redacting that whole run in place of the brackets (`token=[REDACTED]`). Sync mirror updated in lockstep.
+- **A ban refused by the self-DoS guard (loopback or a configured trusted proxy) was logged and answered as if it had succeeded.** `_resolve_and_apply_threshold_ban` returned `(duration, reason, category)` whenever a threshold crossed, regardless of whether `ban_ip` actually applied the ban; `ban_ip`'s self-DoS refusal path returned `None` exactly like a successful ban, so `_try_threshold_ban` logged `"IP banned due to ..."` and `SuspiciousActivityCheck` answered 403 `"IP has been banned"` while `is_ip_banned` stayed `False` and the next request from that IP was served normally. `ban_ip` now returns `bool` (`True` applied, `False` refused); `_resolve_and_apply_threshold_ban` returns `None` on a refused ban so the request falls through to the normal 400 `"Suspicious activity detected"` response with no phantom ban log line and no penetration-attempt-banned event. The same blind spot existed in `DynamicRuleApplicationMixin._apply_ip_bans` (dynamic-rule IP bans logged `"Banned IP ..."` regardless of outcome) and `BehaviorActionDispatchMixin._execute_ban_action` (behavioral-rule bans logged the same false success and reported `action_taken="ban"` in the agent event even when refused); both now read `ban_ip`'s bool, `_apply_ip_bans` skipping the success log on a refusal and `_execute_ban_action` reporting `action_taken="tracked"` instead of `"ban"`. Sync mirrors updated in lockstep.
+
+___
+
 v4.0.0 (2026-09-04)
 -------------------
 
@@ -38,7 +65,7 @@ Grammar-based secret redaction across log lines and events, expanded detection c
 
 - **Sensitive header, parameter, and body secret leaks.** `log_activity`, `_log_detected_component`, and query URL parameters (`?access_token=...`, `#token=...`, `https://user:PASS@host/`) now mask sensitive values with `[REDACTED]` across all log lines, telemetry previews, and event payloads.
 - **`dump_last_known_rules_snapshot` validation exception.** Stripped unknown `DynamicRules` fields (`emergency_whitelist_only`, `message`) before snapshot validation, allowing local Redis/file fallback writes to succeed during backend outages.
-- **Proxy identity header false positives.** Added proxy identity headers (`forwarded`, `x-forwarded-for`, `x-forwarded-host`, `x-forwarded-proto`, `x-real-ip`, `x-client-ip`, `x-cluster-client-ip`, `cf-connecting-ip`, `true-client-ip`, `fly-client-ip`, `x-envoy-external-address`) to the default excluded detection set to prevent auto-banning proxy IPs.
+- **Proxy identity header false positives.** Added proxy identity headers (`forwarded`, `x-forwarded-for`, `x-forwarded-host`, `x-forwarded-proto`, `x-real-ip`, `x-client-ip`, `x-cluster-client-ip`, `cf-connecting-ip`, `true-client-ip`, `fly-client-ip`, `x-envoy-external-address`) to the default excluded detection set, which in 4.0.0 skips only the `ssrf` category for IP-shaped values while every other category still scans them.
 - **`require_referrer()` scheme parsing.** Reduced scheme-prefixed entries (e.g., `https://example.com:8443`) to host and port before comparison.
 - **`SecurityConfig` collection field revalidation.** Revalidated 13 collection fields (`exclude_paths`, `cors_allow_origins`, `custom_error_responses`, etc.) on attribute reassignment and `model_copy(update=...)`.
 - **SQLi and category detection context gaps.** Expanded SQLi, command injection, and traversal detection to `header` and `url_path` contexts. Kept noise-prone patterns (`ORDER BY n` bare) restricted to bodies/params.