Add Google SecOps (Chronicle) support: events.import output and UDM parser

[seanthegeek]

Summary

Adds two delivery paths for getting parsedmarc data into Google Security Operations (Chronicle), sharing one UDM mapping:

1. [gsecops] output (new) — sends reports directly to the GA v1 Chronicle API events.import method as pre-normalized UDM events. No collector, no custom parser to install; standard Google Cloud IAM auth (service-account key file or Application Default Credentials, Chronicle API Editor role in the project linked to the SecOps instance). Batches per the documented best practices (1,000 events / 60 s timeout) and bisects rejected batches, since events.import is all-or-nothing, so one invalid event can't discard a run. Because additional is a protobuf Struct, counts/ASNs stay numbers and alignment flags stay booleans — range-queryable with no coercion. Config: project_id, instance_id, region, optional credentials_file, wired through INI schema, env vars (PARSEDMARC_GSECOPS_*), docs, and SIGHUP reload. Fully unit-tested against real sample reports with payload-level assertions on a mocked AuthorizedSession.

2. CBN parser (google_secops_parser/) — the collector-based alternative for deployments that want raw-log retention or already run a Bindplane pipeline: parsedmarc's [syslog] output → Bindplane → custom log type → this parser. Both paths emit the same UDM event shape and the same additional keys, so searches and dashboards port between them.

Google's ingestion docs recommend sending UDM events when possible; the README cross-links both paths and explains the trade-off (API path = no tenant-side install, CI-tested mapping; parser path = raw logs retained in SecOps).

One paired library fix ships with it: parsed_smtp_tls_reports_to_csv_rows now includes policy_domain / policy_type on SMTP TLS failure-detail rows (RFC 8460 §4.3 nests each failure detail inside a policy). Without this, failure-detail rows carried none of the shape-identifying fields — the CBN parser could not detect them at all, and the policy_domain / policy_type CSV columns were always empty on those rows. The same commit fixes a cross-policy leak where policy_strings / mx_host_patterns from an earlier policy bled into a later policy that didn't define them. Both regression tests fail on the previous serializer.

What it does

parsedmarc report	Detected by	UDM metadata.event_type
DMARC aggregate	xml_schema	EMAIL_TRANSACTION
DMARC failure	feedback_type or arrival_date_utc	EMAIL_TRANSACTION
SMTP TLS (RFC 8460)	policy_type or result_type	GENERIC_EVENT

The or fallbacks matter: text-format failure reports have no Feedback-Type field (parsedmarc emits no feedback_type key for them, but always computes arrival_date_utc), and SMTP TLS failure-detail rows from parsedmarc versions predating this PR lack policy_type (every RFC 8460 failure detail carries result_type).

What actually identifies an aggregate report is its <feedback> XML root element — parse_aggregate_report_xml anchors on it. The raw XML never reaches SecOps, so the CBN parser tests xml_schema instead: the field parsedmarc synthesizes on every row that came from a <feedback> document (guaranteed non-empty as of this PR — "draft" whenever <version> is missing, empty, or whitespace). Testing xml_schema is therefore one-to-one equivalent to testing for the <feedback> root.

• Sets the required metadata.event_timestamp from the report's own window (via date{}) for all three types — aggregate/TLS use the report window, failure uses arrival time — so events land on the right day rather than at ingest time.
• Maps disposition (aggregate) and delivery_result (failure) to security_result.action using valid enum values, and sets security_result.category = AUTH_VIOLATION when DMARC alignment fails.
• Derives the required noun for SMTP TLS from policy_domain → target.hostname — present on every row as of the paired serializer fix — with a receiving_mx_hostname fallback for rows from older parsedmarc versions.
• Uses real network.email field names. to and subject are repeated UDM fields per the field list, so they are merged (like principal.ip), not replaced.
• Strips syslog framing (the <PRI> prefix) before JSON parsing.
• Handles Chronicle's type-preserving json{} filter: booleans (*_aligned, normalized_timespan) are converted to strings for comparison (stored as string_value), numbers (count, *_session_count, source_asn) are stored as number_value so they stay range-queryable, and every if-tested field is initialized and guarded with != "" so an absent field neither triggers _failed_parsing_ nor misfires detection. Note testing is not a boolean — parsedmarc emits the RFC 9990 t= flag as the string "y"/"n" — so it is guarded with != "".

Every UDM field and enum value used is grounded in the official docs, cited inline in parsedmarc.conf (UDM overview, parsing overview, parser-syntax reference, UDM field list, SecurityResult reference), and cross-checked against Google's content-hub CBN parsers.

Validation status

Built strictly to the documentation, but not yet run against a live SecOps tenant. The README ships six single-line sample events, all byte-identical to real [syslog] serializer output generated from this repository's samples/ files (each event names its source sample).

How to validate (in order):

1. The failure sample first. It is the only shape that emits JSON null values, and Google documents only absent-field behavior (the pre-json "" initialization), not whether json{} overwrites an initialized field with a null. This is the least-grounded behavior the parser depends on.

2. The two aggregate samples — confirm the dmarc_aligned=false one yields security_result.category = AUTH_VIOLATION, that count / source_asn land as number_value, and the boolean→string handling holds.

3. The three SMTP TLS rows — success (noun + counts, no security_result), failure with MTA IPs, and failure without MTA IPs (noun from policy_domain only).

4. Count-weighted dashboards. The per-record message count is what the existing OpenSearch/Splunk/Grafana DMARC dashboards sum, stored here as number_value in additional.fields. Confirm a UDM-search aggregation sums it correctly — e.g. "messages by disposition":

   metadata.product_name = "parsedmarc"
   metadata.event_type = "EMAIL_TRANSACTION"
   $disposition = additional.fields["disposition"].value.string_value
   $messages    = additional.fields["count"].value.number_value
   match:   $disposition
   outcome: $total = sum($messages)
   order:   $total desc
   

   $total must equal the true sum of the count values — verify it is not multiplied by the number of additional.fields entries per event. YARA-L's sum() has a documented over-count on repeated fields, but it should not apply to this key-isolated map-sum (count lives on a single entry); this test confirms it.

Notes

• A CBN parser can't be exercised in this repo's CI; validation happens in a SecOps tenant. The paired serializer fix is covered by regression tests.
• Caveats (boolean coercion, JSON nulls in failure rows, aggregate count representing N messages, GENERIC_EVENT only surfacing in raw/UDM search) are documented in the README.
• Needs review — numeric number_value idiom. The numeric additional.fields land a typed number_value via the content-hub string-hop: build string_value from %{field} → convert to uinteger → rename to number_value. This matches the only number_value example in chronicle/content-hub (Azure Cosmos DB's number_of_rows_returned), which interpolates a raw numeric field the same way. Each interpolation now carries on_error so a tenant where %{} rejects non-string fields degrades to a missing field instead of _failed_parsing_. During tenant validation, confirm the number_value lands correctly.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 96.15385% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.25%. Comparing base (e48521b) to head (579626d).
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
parsedmarc/cli.py	65.00%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #790      +/-   ##
==========================================
+ Coverage   93.15%   93.25%   +0.09%     
==========================================
  Files          19       20       +1     
  Lines        5216     5394     +178     
==========================================
+ Hits         4859     5030     +171     
- Misses        357      364       +7     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

[seanthegeek]

Review follow-up: verified the draft's claims empirically against the parsedmarc serializers, the UDM field list, Google's parser troubleshooting doc, and the content-hub CBN parsers, then pushed two commits (branch also rebased onto current master):

0e3f5f3 — library fix. SMTP TLS failure-detail rows carried none of feedback_type / policy_type / xml_schema, so the parser dropped every one of them as TAG_UNSUPPORTED — the PR's original "policy_domain is present on every row" claim was false, and its own failure-detail sample disproved it. parsed_smtp_tls_reports_to_csv_rows now attaches policy_domain / policy_type to failure rows (RFC 8460 §4.3 nests failure details inside a policy), and rebuilds the row template per policy so policy_strings / mx_host_patterns no longer leak across policies. Both regression tests fail on the previous serializer with KeyError: 'policy_domain'.

95e881b — parser/README fixes.

• Detection fallbacks: arrival_date_utc for failure reports (text-format failure reports have no Feedback-Type, and parse_failure_report never defaults it) and result_type for SMTP TLS rows from pre-fix parsedmarc versions, with a receiving_mx_hostname noun fallback.
• network.email.to and network.email.subject are repeated UDM fields — switched from scalar replace (a type mismatch; the unguarded subject one could have failed parsing on every failure event) to merge, matching the principal.ip idiom.
• testing is the RFC 9990 t= flag emitted as "y"/"n", not a boolean — the == "true"/"false" guard could never fire, so dmarc_testing was never emitted. Now guarded with != "".
• Hardening per Google's parser tips doc: on_error on the four numeric %{} interpolations, TAG_MALFORMED_MESSAGE for non-JSON input, dropped the concatenated security_result.description strings (multiple values in one UDM field is a documented anti-pattern; each value already lands in additional.fields).
• The SMTP TLS success sample was not generated from this repo's samples (it looked like real third-party report data: dengage.com / inboxsys.net / Synametrics) — replaced with rfc8460.json output. All six README samples are now byte-identical to real [syslog] serializer output, verified by exact string comparison.
• Validation plan reordered to start with the failure sample (the only shape emitting JSON nulls — the one json{} behavior the docs don't cover), doc links updated to docs.cloud.google.com, parser linked from the syslog section of the usage docs, CHANGELOG entry added under ## Unreleased.

pytest (685 passed), ruff check / ruff format --check, and pyright (0 errors, 0 warnings) are all clean. The PR description has been updated to match reality. Tenant validation is still pending — the README's validation order is the checklist.

[seanthegeek]

Added the direct API output discussed in review (commit fba083f): a new [gsecops] config section that sends reports straight to the GA v1 Chronicle API events.import method as pre-normalized UDM events — no collector or tenant-side parser required.

Key points, all grounded in the v1 reference docs:

• Endpoint/auth: POST https://chronicle.{region}.rep.googleapis.com/v1/projects/{project}/locations/{region}/instances/{instance}/events:import, standard Google Cloud IAM (Chronicle API Editor role). Uses a service-account key file when credentials_file is set, Application Default Credentials otherwise. google-auth was already installed transitively via mailsuite[gmail]; it's now a declared dependency.
• Batching/robustness: 1,000 events per request and 60 s timeout per the documented best practices. events.import is all-or-nothing, so a rejected batch is bisected to isolate invalid events — valid events still deliver, and the drop count surfaces as an output error (compatible with fail_on_output_error).
• Same mapping as the CBN parser: both paths emit identical UDM shapes and additional keys, so searches/dashboards port between them. Because additional is a protobuf Struct on this path, counts and ASNs stay numbers and alignment flags stay booleans — none of the CBN string-coercion caveats apply.
• Tested: 21 new tests build events from real sample reports and assert on the payloads sent through a mocked AuthorizedSession (event shapes, repeated-field handling, batching, the 400-bisect path, config parsing incl. PARSEDMARC_GSECOPS_* env vars). A CLI smoke run confirmed the wiring end-to-end up to the credential boundary. Full suite: 706 passed; ruff and pyright clean.

The parser README now cross-links the two paths and explains the trade-off (API = no tenant install + CI-tested mapping; syslog+parser = raw-log retention). Unlike the CBN parser, the API output needs no parser-validation-tool pass — but its first live run against a real tenant is still the remaining validation step for both.

[seanthegeek]

Follow-up on a review question about why the CBN parser stored booleans as string_value when UDM supports booleans: resolved from Google's docs, and the questioner was right — commit pushed switching the four boolean fields (dmarc_aligned, spf_aligned, dkim_aligned, normalized_timespan) to typed bool_value.

Documentation trail:

• Parser extension examples document the exact idiom: build a string → convert => "boolean" → rename into a boolean UDM field (their example targets network.tls.established).
• The UDM search docs explicitly state additional fields can be matched with value.bool_value (and value.number_value).
• The same string→convert→rename chain into a typed value.* slot is production-proven for number_value in the content-hub Azure Cosmos DB parser.

The earlier "content-hub parsers never use bool_value" observation was accurate but was absence of precedent, not prohibition — the documented boolean example is the stronger authority. The string conversion in step 1b stays, because CBN conditionals compare against the preserved JSON type, so the == "true"/"false" guards still need strings; only the stored value type changed.

Net effect: the CBN parser and the [gsecops] API output now emit identical value types for every additional.fields key (booleans as bool_value, counts/ASNs as number_value), so UDM searches and dashboards port between the two delivery paths unchanged. The validation checklist now includes confirming the booleans land as bool_value in the tenant.

[seanthegeek]

Review follow-up on aggregate-report detection: the <feedback> XML root element is the true identifier of an aggregate report — parse_aggregate_report_xml already anchors on it — while <version> is optional metadata (RFC 7489 Appendix C). But the xml_schema value derived from <version> didn't honor that: the "draft" fallback only covered a fully absent element, so an empty <version/> (parsed as None by xmltodict) or whitespace-only value produced an empty xml_schema in the flat rows — and the CBN parser's report-type cascade would silently drop such aggregate rows as TAG_UNSUPPORTED.

Fixed in the library so the guarantee holds at the source: xml_schema now falls back to "draft" unless <version> carries non-empty text (attribute-wrapped values unwrapped via _text). Regression tests fail on the previous code; the CBN parser comment and README now state the invariant (xml_schema is guaranteed non-empty on every aggregate row, so the detection cascade needs no fallback for aggregates). 708 tests passing, ruff/pyright clean.

[Copilot]

Pull request overview

Adds first-class Google SecOps (Chronicle) integrations to parsedmarc, enabling both direct UDM event ingestion via the Chronicle events.import API and an alternative collector-based path via a SecOps CBN parser, while fixing SMTP TLS CSV row identity fields and hardening aggregate xml_schema defaults for downstream detection.

Changes:

• Add a new [gsecops] output path that converts parsedmarc results into UDM events and uploads them to Chronicle events.import, including batching and 400-response bisection.
• Add a SecOps-side CBN parser (google_secops_parser/) for mapping parsedmarc [syslog] JSON into the same UDM shape as the API path.
• Fix SMTP TLS CSV row serialization (include policy_domain/policy_type on failure-detail rows; prevent cross-policy field leakage) and ensure aggregate xml_schema never becomes empty/None.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/test_init.py	Adds regression tests for aggregate xml_schema fallback behavior and SMTP TLS CSV-row policy identity/leak fixes.
tests/test_gsecops.py	New unit tests for UDM mapping and Chronicle client batching/bisection behavior.
tests/test_cli.py	Adds config parsing tests for new [gsecops] section and env var resolution.
pyproject.toml	Adds google-auth dependency required for Chronicle API auth/session support.
parsedmarc/gsecops.py	Implements UDM event mapping + Chronicle events.import client with batching and 400-bisect retry logic.
parsedmarc/cli.py	Wires [gsecops] config parsing, path expansion, and output publishing into the CLI execution path.
parsedmarc/init.py	Fixes SMTP TLS CSV row policy identity + prevents per-policy field leakage; ensures aggregate xml_schema defaults to non-empty draft when <version> is empty/whitespace.
google_secops_parser/README.md	New documentation for the collector-based SecOps CBN parser path, installation, mappings, and validation samples.
google_secops_parser/parsedmarc.conf	New CBN parser that maps parsedmarc syslog JSON to UDM for aggregate/failure/SMTP TLS shapes.
docs/source/usage.md	Documents the new [gsecops] output and links to the SecOps parser alternative.
CHANGELOG.md	Records the new SecOps features and the paired SMTP TLS + aggregate schema bug fixes under “Unreleased”.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[seanthegeek]

Addressed in 579626d (pushed together with a merge of current master that resolves the branch conflicts): the 400 handler now logs only productEventType and productLogId from the rejected event's metadata, keeping response.text for debugging. The bisect regression test now also pins the log content — identifier present, event body absent.

This comment was written by Claude on behalf of the maintainer.