Add vulnerability email de-duplication integration

[leonfullxr]

Summary

Adds integrations/vulnerability_email_dedup/, a drop-in replacement for a custom-email integration that suppresses duplicate vulnerability notifications.

The Vulnerability Detection module emits one alert per (package, CVE) pair. A single package often carries dozens of CVEs, so one scan produces a burst of near-identical alerts for the same package on the same agent. Routed to email through the stock integration, that burst becomes hundreds of near-identical messages.

This integration sends one email per affected package per agent instead of one per CVE. Every subsequent CVE for the same key inside the window is counted, but not mailed. Non-vulnerability alerts pass through unchanged.

De-duplication model

Each (agent_id, package_name, status) gets one row anchored at first_seen, the moment its first alert arrived. Every alert resolves to one of three cases, decided entirely at read time:

State of the row	Decision
No row exists	Send one email, create the row anchored at now.
now - first_seen >= 24h	Window elapsed. Send one email, re-anchor first_seen at now.
Inside the window	Suppress. Increment the counter and leave first_seen untouched.

That last line is the whole rule. A suppressed alert never moves the anchor, so no volume of alerts can push the window forward. This is the failure mode of a last-seen window, where a package detected every few hours slides its own window indefinitely and never notifies again.

DEDUP_TTL_SECONDS is the only behavioural knob, so a 12h or 48h window is a one-line change.

Design notes

• Execution model. wazuh-integratord runs the script once per alert as a separate process, so a 1000-alert burst is 1000 short-lived processes rather than one process handling 1000 items. Correctness under the burst comes from the storage layer, not from concurrency inside the script.
• Cost per alert. One indexed SQLite lookup plus one small write. Suppressed alerts exit in under a millisecond and never open an SMTP connection; smtplib is imported lazily so the suppression path does not pay the import cost. Nothing runs between alerts: no daemon, no polling, no in-memory state. The table holds one row per currently active package and agent, so it stays in the kilobytes.
• Concurrency. State lives in a SQLite store in WAL mode with a busy timeout. The check-and-write runs inside a single BEGIN IMMEDIATE transaction, so concurrent processes for the same key serialize on the write lock and exactly one wins at each window boundary.
• Prune is not load-bearing. Because the decision is made from first_seen at read time and an elapsed row is already treated as "notify and re-anchor", prune only reclaims space. If it runs late, skips a day, or never runs, the only consequence is stale rows sitting around until the next run: no missed emails, no duplicates. The wodle therefore needs no particular timing, so no midnight pin and no alignment with scan windows.
• Fail open. If the store cannot be opened or the de-duplication check raises, the alert is emailed. A duplicate is preferable to a dropped notification.

Files

• custom-email.py - integration logic, Python standard library only.
• custom-email - standard Wazuh shell wrapper for the embedded interpreter.
• README.md - installation, configuration table, ossec.conf blocks for both the integration and the prune wodle, testing steps, troubleshooting, provenance.

Testing

The window rules were verified by ageing a row's first_seen with sqlite3, which confirms in seconds what would otherwise take a day of wall-clock time. The README documents the procedure:

• With the anchor aged to 23 hours, replaying an alert sends no email, increments the counter, and leaves window_age_h at 23. The anchor does not move, so replaying alerts can never push the window forward.
• Ageing it two hours further and replaying sends one fresh email and re-anchors the row, so the counter returns to 1 and the age to 0.
• Prune leaves the re-anchored row alone and removes it once aged past the window.

Also verified end to end on Python 3.11 as real process invocations: the first vulnerability alert attempts one send, two further CVEs for the same key exit silently with the counter at 3, a non-vulnerability alert is emailed unchanged, invocation with no arguments exits 1 with a usage error, and prune reports cleanly.

Earlier load test of the storage layer, unchanged by this revision: 2,000 alerts across 300 distinct (agent, package, status) combinations at 64-way concurrency produced 300 emails, one per distinct key, an 85% reduction. SUM(cve_count) equalled 2,000 and no lock errors occurred, at roughly 665 alerts/sec.

Not yet exercised against a live vulnerability scan on a production manager; the ossec.conf wiring in the README is the documented path for that.

[Copilot]

Pull request overview

Adds a new Wazuh integration (integrations/vulnerability_email_dedup/) that acts as a drop-in custom-email replacement to suppress duplicate Vulnerability Detection emails by de-duplicating on (agent_id, package, status) within a fixed time window, while still passing non-vulnerability alerts through unchanged.

Changes:

• Introduces a SQLite/WAL-backed de-duplication store to suppress duplicate vulnerability emails during bursts.
• Adds a prune maintenance mode intended to be run via a scheduled Wazuh command wodle.
• Provides installation/configuration/testing documentation and a standard Wazuh shell wrapper.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.

File	Description
integrations/vulnerability_email_dedup/README.md	Documents the integration behavior, configuration, ossec.conf wiring, and testing/maintenance steps.
integrations/vulnerability_email_dedup/custom-email.py	Implements per-alert de-duplication logic with SQLite WAL and a prune maintenance entry point.
integrations/vulnerability_email_dedup/custom-email	Provides the standard Wazuh wrapper to invoke the Python script via the embedded interpreter.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

integrations/vulnerability_email_dedup/custom-email.py:159

• prune_old_rows() starts an IMMEDIATE transaction but doesn’t roll it back on exceptions. If the DELETE or COMMIT fails, the connection can keep a write lock until process exit, which can block alert-path writers unnecessarily.

    conn.execute("BEGIN IMMEDIATE")
    cur = conn.execute("DELETE FROM dedup WHERE first_seen < ?",
                       (time.time() - DEDUP_TTL_SECONDS,))
    deleted = cur.rowcount
    conn.execute("COMMIT")

integrations/vulnerability_email_dedup/custom-email.py:209

• In prune mode, if open_db() or prune_old_rows() raises, the SQLite connection isn’t closed (because conn.close() is inside the happy path). This can leave a write lock held longer than necessary and also leaks the file descriptor until process exit.

    if len(sys.argv) > 1 and sys.argv[1] == 'prune':
        try:
            conn = open_db()
            deleted = prune_old_rows(conn, vacuum=True)
            conn.close()
            logging.info("Prune complete: removed %d elapsed row(s).", deleted)

integrations/vulnerability_email_dedup/custom-email.py:183

• smtplib.SMTP() is created without a timeout, so a stalled TCP connect or hung SMTP session can block the per-alert process indefinitely. Since wazuh-integratord can spawn many processes during bursts, this can cause process pileups when the relay is down/unresponsive.

    with smtplib.SMTP(SMTP_SERVER, SMTP_PORT) as server:
        server.send_message(msg)