Anchor dedup window at first alert and make prune non-critical

The suppression window is now decided entirely from first_seen at read
time: no row or a fully elapsed row notifies and (re-)anchors the window
at now, while an alert inside the window is counted without touching the
anchor. A flood of alerts can no longer slide the window forward and
silence a package.

Because the read path already re-notifies past an elapsed window, prune
is pure housekeeping. It drops rows on the same window and needs no
particular schedule, so RETENTION_SECONDS goes away and
DEDUP_TTL_SECONDS is the only knob left.

Also drop the per-row CVE list. It cost a JSON parse and dump on the
suppression hot path with no consumer, and the CVEs are already in
alerts.json and the indexer.

Add test_dedup.py, covering one email per key under a 1000-alert burst,
no window slide while alerts keep arriving, a fresh email plus re-anchor
once the window elapses, and prune taking only elapsed rows.

Author: leonfullxr

integrations/vulnerability_email_dedup/README.md

@@ -29,13 +29,17 @@ becomes hundreds of near-identical messages.
 This integration is a drop-in replacement for a `custom-email` script that
 collapses the burst:
 
-- The first alert for a given agent, package and status sends one email.
-- Every subsequent CVE for that same agent and package inside the suppression
-  window is recorded and counted, but no email is sent.
+- The first alert for a given agent, package and status sends one email and
+  opens a 24 hour window.
+- Every subsequent CVE for that same agent and package inside the window is
+  counted, but no email is sent.
+- The window is measured from that first alert and never slides. More alerts
+  arriving inside it do not extend it and do not start a new one.
+- Once 24 hours have passed since the first alert, the next detection emails
+  again and opens a fresh window, so a package that stays vulnerable stays
+  visible without becoming noise.
 - Non-vulnerability alerts are passed through and emailed unchanged, so the
   script can also serve as the general email path.
-- The window expires (24 hours by default), so a package that is still
-  vulnerable the next day notifies again and recurring problems stay visible.
 
 How much noise this removes depends on how densely CVEs cluster per package. In
 a synthetic load test, 2,000 vulnerability alerts spread across 300 distinct
@@ -64,29 +68,41 @@ concurrent writers.
 
 ### How It Works
 
+Each `(agent_id, package_name, status)` gets one row, anchored at `first_seen`:
+the moment its first alert arrived. Every alert resolves to one of three cases,
+decided entirely at read time:
+
+| State of the row | Decision |
+| --- | --- |
+| No row exists | Send one email, create the row anchored at now. |
+| `now - first_seen >= 24h` | The window has elapsed. Send one email, re-anchor `first_seen` at now. |
+| Inside the window | Suppress. Increment the counter and **leave `first_seen` untouched**. |
+
+That last line is the whole de-duplication rule. Because a suppressed alert
+never moves the anchor, no volume of alerts can push the window forward, which
+is the failure mode of a last-seen window: a package detected every few hours
+would slide its own window indefinitely and never notify again.
+
+The package **version** is shown in the email body (read from the triggering
+alert, not persisted) but is deliberately kept out of the key, so a version bump
+that is still vulnerable does not re-notify. Move it into the key if you want
+per-version notifications.
+
 `wazuh-integratord` runs an integration script **once per alert, as a separate
 OS process**. A burst of 1,000 alerts is therefore 1,000 short-lived processes,
-not one process handling 1,000 items. Two properties make that safe and fast:
+not one process handling 1,000 items. Three properties make that safe and cheap:
 
-- **Cheap lookup.** Suppressed alerts hit one indexed SQLite lookup and exit in
-  under a millisecond, without ever opening an SMTP connection. Only the winning
-  alert pays the cost of sending mail.
+- **Cheap lookup.** Per alert the script does one indexed SQLite lookup and one
+  small write. Suppressed alerts exit in under a millisecond, without ever
+  opening an SMTP connection. Only the notifying alert pays for sending mail.
 - **Safe concurrency.** The store runs in WAL mode with a busy timeout, and the
   check-and-write happens inside a single `BEGIN IMMEDIATE` transaction.
   Concurrent processes for the same package serialize on the write lock and
-  exactly one wins, so a burst produces no `database is locked` errors, no lost
-  updates, and no duplicate emails.
-
-The de-duplication key is `(agent_id, package_name, status)`. The package
-**version** is shown in the email body (read from the triggering alert, not
-persisted) but is deliberately kept out of the key, so a version bump that is
-still vulnerable does not re-notify. Move it into the key if you want
-per-version notifications.
-
-The suppression window is anchored at the **first** notification rather than at
-the last alert seen. This matters: anchoring on the last alert would let a
-steady drip of CVEs slide the window forward indefinitely and silence a package
-forever.
+  exactly one wins at each window boundary, so a burst produces no
+  `database is locked` errors, no lost updates, and no duplicate emails.
+- **Nothing runs between alerts.** No daemon, no polling, no in-memory state.
+  The table holds one row per currently active package and agent, so it stays in
+  the kilobytes.
 
 If the database cannot be opened or the de-duplication check raises, the script
 **fails open** and sends the email. Losing a notification is worse than sending
@@ -97,10 +113,10 @@ graph TD
     A[Vulnerability alert] --> B{Has package name?}
     B -- No --> C[Send email, stock behaviour]
     B -- Yes --> D[BEGIN IMMEDIATE on SQLite]
-    D --> E{Key seen inside window?}
-    E -- No --> F[Insert or reset row, COMMIT]
+    D --> E{Row exists and<br/>within 24h of first_seen?}
+    E -- No --> F[Insert or re-anchor<br/>first_seen = now, COMMIT]
     F --> G[Send one email]
-    E -- Yes --> H[Increment cve_count, append CVE, COMMIT]
+    E -- Yes --> H[Increment cve_count,<br/>first_seen unchanged, COMMIT]
     H --> I[Exit, no email]
 ```
 
@@ -110,14 +126,15 @@ graph TD
 
 #### Using the Integration Files
 
-This integration ships two files:
+This integration ships three files:
 
 | File | Purpose |
 | --- | --- |
 | `custom-email.py` | The integration logic. |
 | `custom-email` | Standard Wazuh shell wrapper that invokes the script with the embedded Python interpreter. |
+| `test_dedup.py` | Self-check for the window logic. Not deployed to the manager. |
 
-Copy both to the manager:
+Copy the two runtime files to the manager:
 
 ```bash
 cp custom-email custom-email.py /var/ossec/integrations/
@@ -146,8 +163,11 @@ redirects mail and storage into a sandbox without editing the file.
 | `RECEIVER_EMAIL` | `VULN_RECEIVER` | `soc@example.com` | Destination address. |
 | `DB_PATH` | `VULN_DEDUP_DB` | `/var/ossec/logs/vuln_dedup.db` | De-duplication store. |
 | `LOG_PATH` | `VULN_DEDUP_LOG` | `/var/ossec/logs/custom-email_integration.log` | Script log file. |
-| `DEDUP_TTL_SECONDS` | `VULN_DEDUP_TTL` | `86400` (24h) | Suppression window per agent, package and status. |
-| `RETENTION_SECONDS` | `VULN_RETENTION` | `604800` (7d) | Age at which `prune` deletes expired rows. |
+| `DEDUP_TTL_SECONDS` | `VULN_DEDUP_TTL` | `86400` (24h) | Window per agent, package and status, measured from the first alert. |
+
+`DEDUP_TTL_SECONDS` is the only behavioural knob. Set it to `43200` for a 12
+hour window or `172800` for 48 hours; nothing else in the script needs changing,
+and the prune job picks up the new value automatically.
 
 The script does not perform SMTP authentication or STARTTLS. If your relay
 requires either, extend `send_email()` with `server.starttls()` and
@@ -186,13 +206,21 @@ Notes on this block:
 #### Scheduled Maintenance
 
 Cleanup is an explicit maintenance mode rather than opportunistic work on the
-alert hot path. `custom-email prune` deletes rows whose window expired more than
-`RETENTION_SECONDS` ago and runs a `VACUUM` to reclaim file space.
-
-Keep this in perspective: the table is bounded by the number of **distinct**
-`(agent, package, status)` combinations, not by alert volume. One thousand CVEs
-for a single package is still one row. This is housekeeping, not a growth
-problem, and once a day is ample.
+alert hot path. `custom-email prune` deletes rows whose window has fully elapsed
+and runs a `VACUUM` to reclaim file space.
+
+**Prune is not load-bearing.** Every de-duplication decision is made from
+`first_seen` when the alert arrives, and the read path already treats an elapsed
+row as "notify and re-anchor". So if prune runs late, or skips a day, or never
+runs at all, the only consequence is some rows for packages that went quiet
+sitting around until the next run. No missed emails, no duplicate emails. That
+also means the schedule needs no particular timing: no midnight pin, no
+alignment with scan windows.
+
+Keep the growth in perspective: the table is bounded by the number of
+**distinct** `(agent, package, status)` combinations seen in the last window, not
+by alert volume. One thousand CVEs for a single package is still one row. Prune
+is one `DELETE` on an indexed column, so once a day is ample.
 
 Since the script lives on the manager, schedule it with the manager-native
 `command` wodle rather than system cron. Add to `/var/ossec/etc/ossec.conf`:
@@ -263,7 +291,29 @@ seen package or records the CVE and exits silently.
 
 ### Integration Testing
 
-#### Test 1: First alert for a package sends an email
+#### Test 1: Window logic self-check
+
+Run the bundled self-check from the integration directory. It exercises the
+window rules against a temporary database, sends no mail, and needs nothing
+outside the standard library:
+
+```bash
+python3 test_dedup.py
+```
+
+Expected:
+
+```
+OK: all de-duplication window checks passed
+```
+
+It asserts the behaviours that would otherwise take a day of wall-clock time to
+observe: that a burst of 1,000 alerts for one package produces exactly one
+email with no lost counts, that a flood arriving inside the window neither
+notifies nor moves the anchor, that the next alert after the window elapses
+notifies and re-anchors, and that prune removes only fully elapsed rows.
+
+#### Test 2: First alert for a package sends an email
 
 Write a sample alert and invoke the script directly:
 
@@ -293,7 +343,7 @@ Expected:
 2026-07-18T10:00:01 INFO Sent vuln alert package=openssl agent=001 cve=CVE-2026-0001
 ```
 
-#### Test 2: A second CVE for the same package is suppressed
+#### Test 3: A second CVE for the same package is suppressed
 
 ```bash
 sed 's/CVE-2026-0001/CVE-2026-0002/g' /tmp/alert1.json > /tmp/alert2.json
@@ -305,16 +355,19 @@ default `INFO` level. The row instead shows the incremented count:
 
 ```bash
 sqlite3 /var/ossec/logs/vuln_dedup.db \
-  "SELECT agent_id, package, status, cve_count FROM dedup;"
+  "SELECT agent_id, package, status, cve_count,
+          CAST((strftime('%s','now') - first_seen) / 3600 AS INT) AS window_age_h
+   FROM dedup;"
 ```
 
-Expected:
+Expected, with the anchor still showing zero hours of age because the second
+alert did not move it:
 
 ```
-001|openssl|Active|2
+001|openssl|Active|2|0
 ```
 
-#### Test 3: Burst behaviour and concurrency
+#### Test 4: Burst behaviour and concurrency
 
 Generate a burst of alerts for a handful of packages and confirm that no update
 is lost. The invariant to check is that the sum of `cve_count` across all rows
@@ -330,17 +383,17 @@ sqlite3 /var/ossec/logs/vuln_dedup.db \
   "SELECT agent_id, package, status, cve_count FROM dedup ORDER BY cve_count DESC LIMIT 10;"
 ```
 
-#### Test 4: Maintenance mode
+#### Test 5: Maintenance mode
 
 ```bash
 sudo -u wazuh /var/ossec/integrations/custom-email prune
 tail -n 1 /var/ossec/logs/custom-email_integration.log
 ```
 
-Expected:
+Expected, since the rows just created are still inside their window:
 
 ```
-2026-07-18T10:05:00 INFO Prune complete: removed 0 expired row(s).
+2026-07-18T10:05:00 INFO Prune complete: removed 0 elapsed row(s).
 ```
 
 ---
@@ -353,7 +406,8 @@ Expected:
 | `has write permissions` in `ossec.log` | Permissions are too broad. Re-apply `chmod 750` and `chown root:wazuh`. |
 | One email per CVE, no suppression | Alerts are not matching the expected schema. Confirm `data.vulnerability.package.name` is present in the raw alert. |
 | `DB open failed ... sending without dedup` | The database path is not writable by the `wazuh` user. Check `DB_PATH` and its parent directory. |
-| Emails stop for a package that is still vulnerable | Expected inside the window. Lower `DEDUP_TTL_SECONDS` if you want more frequent reminders. |
+| Emails stop for a package that is still vulnerable | Expected inside the window. Check `first_seen` for that row: one fresh email is sent on the first detection after it plus `DEDUP_TTL_SECONDS`. Lower the TTL if you want more frequent reminders. |
+| The dedup table is larger than expected | Prune has not run recently. It is housekeeping only, so this affects nothing but disk. Run `custom-email prune` and confirm the wodle is enabled. |
 | SMTP errors in the script log | Verify relay reachability with `nc -zv <SMTP_HOST> 25`. |
 
 ---
@@ -366,10 +420,10 @@ Expected:
 - **Adapted by**: Leon Fuller.
 - **Tested versions**: Wazuh manager 4.x with Vulnerability Detection enabled
   and the 4.8+ vulnerability alert schema. Python 3 standard library only, no
-  third-party dependencies. Logic validated on Python 3.11 against a synthetic
-  2,000-alert burst at 64-way concurrency: 300 distinct keys produced 300
-  emails, `SUM(cve_count)` matched the 2,000 alerts fed in, and no lock errors
-  occurred.
+  third-party dependencies. Logic validated on Python 3.11 by the bundled
+  `test_dedup.py`, plus a synthetic 2,000-alert burst at 64-way concurrency: 300
+  distinct keys produced 300 emails, `SUM(cve_count)` matched the 2,000 alerts
+  fed in, and no lock errors occurred.
 - **Maintainer**: Leon Fuller.
 - **Support boundary**: Community-maintained and provided as is. Not covered by
   Wazuh commercial support.