release(v4.0.0): grammar-based secret redaction with a generated gate, detection measured per context with no silent gaps, one telemetry contract, gates that find instead of pass - #94
Merged
Conversation
…apshot persistence, proxy identity headers, decorator footguns, config revalidation, the Docker gate, and add live-smoke and ecosystem CI gates Secrets in logs and telemetry: - log_activity printed the raw request header dict and the full URL in every block, passive, rate-limit, banned-IP and request line; the detection engine's per-component line printed the raw header, query-parameter and body-field value that matched; the pattern_detected telemetry event carried the raw value as content_preview and a JSON key from inside a sensitive value in its reason. Values named in the hardcoded default sets or in the new log_sensitive_headers / log_sensitive_params / log_sensitive_body_fields fields are now [REDACTED] in every guard log line and in the telemetry preview and reason: headers, query strings (both separators, fragment, userinfo password), body fields at any depth including depth-capped subtrees, and JSON carried inside a header, parameter or path value, which now goes through the same recursive walker as a body. Dynamic rules (3.17.0 regression): - dump_last_known_rules_snapshot validated guard-agent's DynamicRules under the strict mirror and raised on every poll, so nothing was persisted, current_rules never updated, rules re-applied every cycle and emergency activation kept halving auto_ban_threshold. The dump keeps only mirrored fields, the read side stays strict, the snapshot build is log-and-continue, and regression tests use the real guard_agent model. Detection and decorators: - Proxy identity headers are excluded from pattern detection by default; behind nginx they tripped the SSRF private-address pattern on every request and auto-banned clients. - require_headers enforces exact values for non-sentinel entries (was a silent no-op); require_referrer accepts scheme-prefixed origins (was an always-block). Config validation: - Thirteen collection-typed SecurityConfig fields were not revalidated on reassignment or model_copy; all are now, exclude_paths rejects a bare string, the revalidation order no longer raises on a list of rule dicts, and a test walks every collection field. Quality gates: - The Docker test image now carries the Makefile, git, the repository .git (read-only) and an editable install, so make test collects and runs (broken since 3.13.0). - Two modules that had dropped to maintainability rank B are split back to rank A (body_json_scan, body_form_scan, embedded_json_scan, _security_config_typed_validators, _security_config_geo_validators) and a unit test enforces rank A on every module. - live-smoke.yml boots fastapi-guard's advanced example app with nginx, gunicorn, Redis, the real guard-agent against a stub SaaS, an OTLP stub, and guard-core from the PR wheel; 90 scenarios assert responses, log lines, Redis, files and stub traffic, and the job fails on any SecurityConfig field, decorator or check name without a live scenario. - ecosystem-gate.yml installs the PR wheel into fastapi-guard, flaskapi-guard, djapi-guard, guard-agent and guard-core-mcp at their default branches and runs their suites.
github-actions
Bot
added
documentation
This comment was marked as outdated.
This comment was marked as outdated.
…e URL entry point for log lines, detection line and on_block; sensitive-data invariant test; ecosystem gate compares base and candidate - Path segments: matrix parameters go through the same name check as query pairs; a whole path that decodes to JSON is handled before segment splitting; paths decode with unquote so a literal plus survives. - The log-line JSON redactor is iterative and capped at detection_max_json_depth; deeper subtrees become [REDACTED] and a RecursionError can no longer escape log_activity. - redact_url_for_display is the single URL redaction used by the log-line URL segment, the detection engine's URL-path line and telemetry preview, and the path field of the on_block payload. - tests/test_sensitive_data_invariant.py plants secrets on every surface, encoding and nesting and asserts absence in log lines, agent events, the on_block payload and OpenTelemetry spans. - ecosystem-gate.yml runs each consumer suite on the PyPI guard-core and on the PR wheel and fails only on candidate-only failures; baseline breakage is a warning.
This comment was marked as outdated.
This comment was marked as outdated.
…the redactors; blob bodies, multipart fallback, forwarded-for warnings, header whitespace; producer inventory gate; docs parity gate; docker-test job; gate hygiene - Every event, metric and hook field that carries a URL or path (endpoint, endpoint_id, excluded_path, redirect_url, CSP URIs) and every displayed header value (user agent, referrer, content type, decorator events) goes through redact_url_for_display or redact_blob_for_display; a failing check logs the exception type with a redacted message. - Bodies of other content types are displayed through name-based redaction (JSON, name=value pairs, XML); a multipart body with unparseable parts displays as [REDACTED]; the X-Forwarded-For warnings show only tokens that parse as IP addresses; header names match after stripping whitespace; the Headers segment redacts pairs and XML inside values, with whitespace as a pair separator. - tests/test_sensitive_data_invariant.py drives the secret matrix through every check and handler; tests/test_sensitive_output_producers.py inventories every log, event, hook and span site in guard_core and fails on an unregistered producer. - tests/test_config_docs_parity.py keeps the config reference and the in-package skill in step with SecurityConfig; 15 missing rows and 10 missing mentions added. - ci.yml gains a docker-test job that runs the suite inside the Docker image; ecosystem-gate.yml gains an aggregating job and treats an identical abnormal exit on base and candidate as baseline breakage; detection-gate.yml minimizes its earlier PR reports and posts a summary built by .github/scripts/detection_gate_comment.py; docs, release and scheduled-lint workflows are actionlint clean.
…ide values under non-sensitive names; Docker image builds without a lockfile - The detection engine's per-component line and its content_preview for a header, query parameter, form field, multipart text part or JSON string value whose name is not sensitive now go through redact_blob_for_display, so a password=SECRET pair inside such a value prints as password=[REDACTED]; a value with nothing sensitive inside is still shown as before, and the embedded-JSON walker keeps its per-key redaction. - sensitive_params is threaded through the header and body scan chain (body_content_scan, body_form_scan, body_json_scan, penetration_detection) and the generated sync mirrors. - tests/test_sensitive_data_invariant.py gains six plain-pair cases under non-sensitive names: header, cookie-style header, query parameter, form field, multipart text part and JSON string value. - The Dockerfile resolves dependencies without --frozen: uv.lock is not committed, so the docker-test job could never build the image from a clean checkout.
- uv.lock is not committed, so from a clean checkout the mount source did not exist, Docker created a directory in its place and could not mount it over the lockfile the image build now writes; docker compose run, the docker-test CI command, failed before pytest started. - Proven by docker compose run from a fresh clone with no lockfile: 13814 passed, 100% line and branch.
This comment has been minimized.
This comment has been minimized.
- On the CI runner the checkout is owned by a different user than the container, so git status inside the image failed with "detected dubious ownership" (exit 128) and both unasync gate tests failed; reproduced in the image with a foreign-owned copy of the repository and proven fixed by the safe.directory entry.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
…redaction gate in the suite and in CI, state paths in the invariant matrix - guard_core/_utils/pair_redaction.py, pair_value_scan.py and pair_hidden_assign.py replace every per-caller separator regex: a name is a run of name characters, percent-encoded bytes and plus signs, optionally quoted; an assignment is a run of literal or encoded equals or colon tokens with whitespace around and between; any other character is a boundary; a sensitive value runs to the next hard separator, closing quote or the next pair; non-sensitive values are rescanned; JSON string leaves and XML text and attributes go through the same rule; a name still percent-encoded after three decode rounds is treated as sensitive; pair names match the union of the header, parameter and body-field sets on every surface. - The previous redactor was quadratic on repeated "a=" input (200 KB took over two minutes on the log path); the new one is a single pass held under 100 ms per 200 KB by a CPU-time test. - The detection engine's per-component line and content preview redact pairs under non-sensitive names and inside headers excluded from detection, the raw query-parameter name no longer reaches trigger text, a custom pattern source is redacted before it is displayed in trigger text, reason, pattern_matched, metadata.pattern and the on_block payload, an encoded outer assignment wrapping a quoted inner pair no longer leaves the quoted value after [REDACTED], the endpoint rate-limit reason and its on_block and telemetry endpoint are redacted, and JSON inside a form field or multipart part gets the embedded-JSON walk it never had. The CSP violation event carried the report's source-file and line-number raw; every field is now redacted or coerced. A custom pattern source that holds a secret-shaped literal is redacted at every display: trigger and reason text, pattern_matched and metadata.pattern, the on_block payload, the pattern-timeout and pattern-error log lines, the anomaly event and pattern report, the add_pattern rejection and dynamic-rule log lines, and the detect_pattern_match return value. Every header of a multipart part is scanned, since a bare carriage return inside a filename smuggled a second header past the scanner. The behavior handler's endpoint hashing helper moves to guard_core/_utils/identity_hash.py so the rate limiter can share it. The per-category excluded-header table in guard_core/_utils/detection_config.py lands in this commit because the body scanner imports it; the detection commit describes its measurement. - tests/redaction_gate/ generates the grammar (every sensitive name, casing, assignment, whitespace, separator, quoting, wrapper, encoding round, trigger, and a state axis for rate-limited, banned, emergency, cloud, country, time-window and per-request-log paths) across twenty request surfaces, drives async and sync detection and the pipeline, and fails on a secret in any log line, event, metric, hook payload or span; .github/workflows/redaction-gate.yml runs it on every PR and nightly with rotating seeds. The invariant matrix carries the same shapes, the state scenarios, and a request builder that splits path and query like the adapters do.
…and paths, excluded headers scanned per category, no silent gaps in the benchmark, legacy path deprecated - The per-category request-context matrix was reviewed pair by pair with a new per-context benchmark (recall floor per category and context, false-positive ceiling per context, in the gated suite): nineteen pairs gained coverage, the command-injection additive view and the glued-shell patterns follow the category set, and the matrix is documented in docs/configuration/detection-tuning.md. - The SQLi category never ran on header values or URL paths; the 3.13.0 exclusion of paths, measured on that day's corpus, is reversed, three noise-prone rows stay narrow at pattern level, a quote- or digit-prefixed ORDER BY and a terminator-prefixed EXEC run everywhere, and WAITFOR DELAY and WAITFOR TIME rows are added. - Excluded headers are scanned by every category except ssrf on twelve address-carrying identity headers, each justified by a named value in a 459-value realistic header corpus; a 60-value malicious corpus is detected in full. Breaking: a user-added excluded header is fully scanned unless its value is an address chain. - The benchmark shipped 34 malicious items pinned as documented gaps and 25 benign items pinned as false positives; 33 gaps are closed structurally, the 34th is reclassified benign, 12 false positives are closed, the 13 byte-identical to attacks stay detected and disclosed by name, the eight body-context glued-shell items carry their measured reason, and all three benchmark gates now fail on any undetected item or firing benign item without a written reason. Four bare private-address values are disclosed as SSRF false positives by name with the exclusion knobs as the remedy, and the end-to-end benchmark's detected-total pins (311 and 305 against a measured 362) now pin the measured value. Recall 89.0% to 97.9% (378 of 386), disclosed false positives 25 to 23 of 370, baselines tightened. - A multipart part whose filename parameter is malformed was truncated by the parser before scanning; the raw Content-Disposition value is now scanned alongside the parsed filename. - Legacy (unconfigured) detection was blind to encoded path traversal; the decoded view now runs there too, and running detection without a SecurityConfig warns once per process as deprecated. - Five timing-ratio test files corroborate before failing, the ReDoS arbiter's retry bound is tested, and a latent catastrophic template regex shape is restructured (16 s to 2 ms on 200 KB of adversarial fill).
…ry on pattern_detected, handler names everywhere, decorator events through the bus - pattern_detected carries pattern_matched (redacted), handler_name, metadata.threat_categories and metadata.category with one vocabulary for regex and semantic hits; decorator_type and rule_type passed by the checks land in the typed top-level fields; every emitter sets handler_name. - The behaviour event's constant and enricher weight said behavior_violation while the wire carried behavioral_violation; the wire spelling wins, five event types that left guard-core as literals are registered as constants, every emitter uses the constants (an AST test forbids literals and f-strings), the enricher covers every constant, and a parity test compares the list with guard-agent's KNOWN_EVENT_TYPES in both directions. - The six SecurityDecorator event methods are wired into the checks that own their conditions and emit through the event bus (gate, GeoIP country, redaction); route-level cloud and country blocks emit their own decorator events; the enricher no longer imports SecurityConfig at module level, guarded by a fresh-interpreter import test.
…leak, rate limiter and Redis events use the calling config, header groups cleared on reconfigure, loopback exempt from the country allowlist - Reassigning a SecurityConfig field could raise TypeError when the private revision counter resolved to its descriptor; the counter is now read from the instance dictionaries directly. - The rate limiter is a process-wide singleton whose config is whichever middleware constructed it last; enforcement, display and its telemetry event now take the calling middleware's own config, and the Redis connect events display the connecting config's URL. - SecurityHeadersManager only set groups when given a value and never cleared them or its per-path cache on reconfigure; every configure clears what it does not set. - whitelist_countries blocked loopback and any explicitly whitelisted IP whose country could not be resolved; loopback and a whitelist match pass, every other unresolvable address still fails closed. - Breaking: the endpoint rate-limit Redis key and in-memory bucket hashed the client address but embedded the raw request path, so a client-chosen path segment became a Redis key name; the path segment is now hashed like the behavior handler's endpoint identity (endpoint counters reset once on upgrade), and check_rate_limit_by_ip's rejection of a path containing ':' no longer echoes the raw path. - The test Redis prefix always appends the process id and a random suffix, so two suites given the same REDIS_PREFIX cannot delete each other's keys. - The test fixture that resets the rate-limit singleton now resets both runtimes' singletons before and after every test.
…, detection measured per context with no silent gaps, one telemetry contract, gates that find instead of pass Version bumped through make bump-version. The changelog and docs/release-notes.md carry the full account of this cycle; the documentation gains the per-category context matrix, the redaction grammar and verification gate, the telemetry field list, and drops every em and en dash.
github-actions
Bot
added
the
area: detection-engine
rennf93
changed the title
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
This comment has been minimized.
This comment has been minimized.
…n, pattern source in six more displays, hidden input in config errors
- A sensitive value whose opening quote closes beyond the end of a URL path segment (the quote-aware splitter cut the segment at a slash inside the nested quoted value) was read as empty, so the secret after the quote printed raw in the log line, the event endpoint, the on_block path and both spans; found by the CI redaction gate on seeds 0 and 1. A quote preceded by nothing but literal or percent-encoded whitespace now opens the value and runs to the segment end when unclosed; a stray quote after real value content still terminates the value.
- The escape gap between a name and its assignment covers the union of the JSON, JavaScript, C and Python escape grammars (short escapes, octal, \xHH and \XHH, \uHHHH, \u{H}, \UHHHHHHHH, \N{NAME}) behind any run of backslashes, when the escape decodes to whitespace, a control character or a quote; the generated gate carries five more escaped whitespace values.
- A custom pattern source is redacted in six more displays: the SecurityConfig validators for blocked_user_agents and return_pattern rules, the block_user_agents decorator rejection, the behavior handler's body-unavailable warning, the ReDoS arbiter's structural-disagreement warning and the pattern-timeout warning. SecurityConfig sets hide_input_in_errors, because pydantic appended the raw input value to every validation error and echoed the pattern, a Redis URL with its password or a token anyway. tests/test_pattern_source_interpolation.py walks the package and fails on any pattern interpolated into a log line, exception or warning outside the redactor.
- The redactor CPU budget test allows 300 ms per 100000 pairs instead of 100 ms; the two-vCPU CI runner measured 106 to 110 ms and the test exists to catch the quadratic version (over 120 s), with the linearity ratio assertion unchanged.
…n leaves of embedded JSON
- A JSON field whose whole value is a source-code path ({"path": "/opt/app/worker.py"}) tripped sensitive_file once delivered as a form field, multipart part or query parameter: the form and multipart embedded-JSON walk and the query-parameter widening met the whole-value source-extension row, which reads a bare path ending in .py, .ts, .go and similar as a source-disclosure probe. That reading is sound for a bare value (a URL path, a field or parameter whose whole value is the path, a JSON body leaf) and not for a leaf isolated from JSON embedded inside another value. The embedded-JSON scanner marks its leaves in the context it passes down, the scan loop keeps that marker on the context the acceptance validators see, and the row's validator rejects a hit on such a leaf; the glued-shell validators normalise the context themselves. Measured with the per-mechanism corpus gate against the previous release: no new miss and no new false positive. The 3.13.0 end-to-end pin that recorded this item as a known false positive on the URL path is retired: the item no longer fires there either, and the pin gate rejects a vacuous pin.
- CodeQL analysed the tests together with the package and reported fifteen clear-text-logging alerts whose taint source is a test constant named _SECRET_PATTERN flowing into product lines that already redact, plus two ReDoS alerts on an intentionally catastrophic test regex and one on the redaction harness printing its own planted secret; the analysis is now scoped to guard_core through .github/codeql/codeql-config.yml. - CHANGELOG.md and docs/release-notes.md carry the follow-up fixes and no longer list known gaps.
…daction
- The escape-gap scan rescanned to the end of the text from every name character when a value was a run of escape sequences with no assignment anywhere (51 KB of backslash-t units cost 30 s of CPU through log_activity on any warning-level log line, no sensitive name needed). The scan now caches the end of every escape run it crosses, so each position is crossed once; a linearity test pins 200 KB runs of backslash-t, N{TAB} and u{9} units and of password-tab-colon-quote fragments under the CPU budget with a sub-3 growth ratio.
- A custom pattern source that separates name and value with regex syntax (\s*=\s*, [:=], (?:=|:), an escaped space, a quantifier on the assignment) printed its secret-shaped literal raw in pattern_matched, metadata.pattern, the reason, the on_block payload and the log line. The pattern-source redactor now reads the source as a regex before the pair grammar runs, and the pattern_detected event builds pattern_matched and metadata.pattern through that redactor; built-in sources display unchanged.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
…redaction; operator log handlers kept; coarse pattern-source rule - The global rate-limit check called the shared limiter without its own config, limit or window, so a middleware with rate_limit=1 was never enforced once another middleware existed in the process; it now passes its own values like the endpoint path does. - SecurityHeadersManager held one CSP, HSTS, CORS, custom-header and enabled state for the process, so app A served app B's headers after B was constructed; get_headers and get_cors_headers take the response context's config and derive the headers from it, the per-path cache is keyed by config, and a caller without a config keeps the previous behaviour. - The pattern-source redactor read the detection singleton's config, which flips on every request between two middlewares; the singleton now accumulates the union of every configured config's sensitive sets and the redactor uses the union. - The regex normaliser for pattern sources had seven bypasses; a source whose letters, digits, underscores and hyphens contain any sensitive name now displays as [REDACTED] whole (two built-in sources, http_split and recon, display that way; 155 unchanged), and pattern_source_display.py is removed. - setup_custom_logging removed every handler on the guard_core logger when a middleware was constructed, including handlers the application attached; it now replaces only handlers it installed itself. - The escape-run linearity test gets a 1.0 s absolute ceiling (the CI runner measured 0.34 s against 0.3 s); the growth-ratio assertion is unchanged.
This comment has been minimized.
This comment has been minimized.
…e decodes escapes; ban reset clears state - The endpoint, route and geo rate-limit paths called the shared limiter without their own config, so their enable gate read the last-constructed middleware's enable_rate_limiting and a later middleware with rate limiting off silently switched off an earlier one's endpoint limits; the shared call site now passes the calling config like the global path does, both trees, with a two-middleware test that fails without it. - The coarse pattern-source rule did not decode hex, octal or unicode escapes inside a sensitive name, so a source spelled PASS\x57ORD=... displayed raw; escapes are decoded before the letters-digits normalisation. - reset_global_state() in the ban handler rebound the module name to the same populated singleton, so banned addresses and networks survived a reset unless the caller cleared the instance first; it now clears them in place.
This comment has been minimized.
This comment has been minimized.
…ixtures need - The driver step piped pytest into tee under the default shell, which has no pipefail, so the job reported success on every run while all 90 scenarios errored at setup; the step now runs under bash, whose pipefail propagates pytest's exit code. - The live-smoke tests inherit the root conftest's autouse Redis fixture, which connects to localhost:6379, and the workflow started no Redis service; the job now runs the same Redis service the CI workflow uses.
…enario waits for a detection event - make live-smoke left every previously built guard-core wheel in tests/live_smoke/stack/wheels, and the app image install then saw two URLs for one package (3.17.1 next to 4.0.0) and failed; the target deletes old guard_core wheels before building. - sensitive_data_redacted_across_log_lines read the agent stub as soon as any event arrived; with decorator events now emitted through the bus an earlier event lands before the pattern_detected batch, so the check ran on an empty preview list. The wait now requires a pattern_detected event; on the real stack the scenario passes with every secret redacted in log lines and telemetry.
This comment has been minimized.
This comment has been minimized.
…esolve the same IP on every platform - The stack forwarded Docker's NAT address as the client IP; it differs between Docker Desktop and the GitHub runner and was not stable across requests there, so the blacklist, whitelist, geo, ban and redaction scenarios failed on CI only. nginx now injects a fixed RFC1918 client address and the scenarios use that constant; 90 passed locally on the real stack.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Contributor
Detection gate reportGate green: no category regressed against its committed baseline and no benign string is newly flagged. Total recall: 378/386 (97.9%). Total false-positive rate: 23/370 (6.2%).
Full per-case output is in the job log. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #93
Five commits: redaction, detection, telemetry, config, release. Full account in CHANGELOG.md (v4.0.0).
Fixed
Breaking
Gates
Consumer releases waiting on this