Add silent agent monitoring integration - #112
Open
leonfullxr wants to merge 8 commits into
Open
Conversation
Detects agents that stay registered, and often still active, while log ingestion from them has stopped. For every agent of a target group the check reads the newest indexed event time, writes one SILENT record when it is older than the threshold, and one RESTORED record when events arrive again. Local state keeps an unchanged condition from being reported on every run. Records are JSON lines ingested through a <localfile> block, so the built-in JSON decoder handles them and no custom decoder is needed. Rules 100121 and 100122 carry alert_by_email and are routed to email and to Telegram through the included integration script. Verified end to end on Wazuh 4.14.6: group lookup, silence detection, repeat suppression, recovery, ingestion, rules, real email delivered through Postfix and accepted upstream, Telegram delivery, the wrong-index-pattern safety stop, and the command wodle schedule. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds status_text to both records, so a dispatcher that maps dotted paths to labels can render the Status line without a literal of its own, and documents the two dispatch-table entries plus the per-entry header override that reproduce the requested layout on an existing custom-telegram script. The bundled script is now the option for environments with no Telegram integration yet. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Telegram script built a default SSL context, which trusts nothing on the Wazuh embedded interpreter because its OpenSSL looks for roots in a path that does not exist, so every delivery failed with CERTIFICATE_VERIFY_FAILED. It now prefers the certifi bundle that ships with that interpreter and falls back to the default store elsewhere, with verification still on. --selftest asserted against the shipped 24h threshold while reading the module globals, so an install with a shorter window failed the check on valid logic. The assertions now pin the threshold and lookback they were written for. A zero-length gap made silence_duration render as "unknown" through a truthiness test on a timedelta. The README is cut to the shape the other integrations use: install, configure, schedule, rules, routing, test, troubleshoot.
The header block on each file says what the script is and how it is invoked. Below that, only the non-obvious parts keep a comment: why the field is named event_status, why a failed query exits without touching the state, why every agent silent at once is a safety stop, why the selftest pins its thresholds, and why the Telegram context needs certifi. Everything else the code says by itself.
The group is the one setting that changes per deployment and per wodle, so it belongs in ossec.conf next to the schedule rather than in a file that a manager upgrade can replace. A --group argument now overrides SAM_GROUP, which in turn overrides the shipped default, and the README documents one wodle per group with its own state file and output log.
--group now takes a comma-separated list, so a single wodle covers every group that should be watched instead of one wodle, one state file and one output log per group. Agents are deduplicated across the listed groups, so an agent that belongs to two of them is checked once and its record names both. The indexer still sees one aggregation per run, whatever the number of groups.
A second block with sample values next to the placeholder one makes the shape of hook_url and api_key obvious. The multi-group example loses its space: a wodle command is split on whitespace, so 'Server, Windows' would pass one trailing comma group and a stray argument.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a new silent agent monitoring integration that detects when Wazuh agents remain registered/active but have stopped shipping events by querying the indexer for each agent’s most recent @timestamp, emitting JSONL SILENT / RESTORED records, and providing rules + optional Telegram notification formatting.
Changes:
- Adds a manager-side Python monitor (
silent_agent_monitor.py) that queries Wazuh API + indexer, suppresses repeat notifications via local state, and writes JSONL events for ingestion via<localfile>. - Adds custom Wazuh rules (
silent_agent_monitor-rules.xml) to classify and alert onSILENT/RESTORED. - Adds an optional Telegram integration wrapper + script (
custom-server-telegram*) and documentation (README.md) for deployment/configuration/testing.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| integrations/silent_agent_monitoring/silent_agent_monitor.py | Manager-side scheduled checker: agent discovery, indexer aggregation query, stateful SILENT/RESTORED emission |
| integrations/silent_agent_monitoring/silent_agent_monitor-rules.xml | Rules to decode JSON and alert on event_status transitions |
| integrations/silent_agent_monitoring/custom-server-telegram.py | Formats and delivers alerts to Telegram via Bot API |
| integrations/silent_agent_monitoring/custom-server-telegram | Shell wrapper to run the Python integration via embedded interpreter |
| integrations/silent_agent_monitoring/README.md | Installation/configuration instructions and troubleshooting |
Suppressed comments (1)
integrations/silent_agent_monitoring/silent_agent_monitor-rules.xml:32
- The rule description hard-codes "Server" even though the integration monitors agents in arbitrary groups. Using a neutral term like "Agent" avoids misleading recovery text.
<rule id="100122" level="5">
<if_sid>100120</if_sid>
<field name="event_status">^RESTORED$</field>
<description>Server $(agent_name) (ID $(agent_id)) resumed logging after $(silence_duration)</description>
<options>alert_by_email</options>
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+41
to
+65
| data = alert.get("data", {}) | ||
| rule = alert.get("rule", {}) | ||
| rule_id = str(rule.get("id", "")) | ||
|
|
||
| if rule_id == SILENT_RULE: | ||
| return (f"⚠ <b>Server Logging Alert</b>\n" | ||
| f"<b>Name:</b> {data.get('agent_name', 'unknown')}\n" | ||
| f"<b>Agent ID:</b> {data.get('agent_id', 'unknown')}\n" | ||
| f"<b>Status:</b> {data.get('status_text', 'No logs received')}\n" | ||
| f"<b>Last Log Received:</b> {data.get('last_log', 'unknown')}\n" | ||
| f"<b>No Logs For:</b> {data.get('no_logs_for', 'unknown')}") | ||
|
|
||
| if rule_id == RESTORED_RULE: | ||
| return (f"✅ <b>Server Logging Restored</b>\n" | ||
| f"<b>Name:</b> {data.get('agent_name', 'unknown')}\n" | ||
| f"<b>Agent ID:</b> {data.get('agent_id', 'unknown')}\n" | ||
| f"<b>Status:</b> {data.get('status_text', 'Logs received')}\n" | ||
| f"<b>Restored At:</b> {data.get('restored_at', 'unknown')}\n" | ||
| f"<b>No Logs Duration:</b> {data.get('silence_duration', 'unknown')}") | ||
|
|
||
| agent = alert.get("agent", {}) | ||
| return (f"<b>Wazuh alert</b>\n" | ||
| f"<b>Rule:</b> {rule_id} (level {rule.get('level', '')})\n" | ||
| f"<b>Description:</b> {rule.get('description', '')}\n" | ||
| f"<b>Agent:</b> {agent.get('name', 'manager')} ({agent.get('id', '000')})") |
The offline assertions and their fixture agent were development scaffolding. The script now has one run mode, so the header states a single usage line and argparse carries only --group. The README's testing section keeps the checks that matter to an operator: a manual run, a forced notification with a lowered threshold, and a wazuh-logtest line for the rules. The rules file loses its header comment as well; the per-rule comments and the README already say what each rule matches and that the IDs can be moved.
leonfullxr
marked this pull request as ready for review
September 4, 2026 17:08
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
integrations/silent_agent_monitoring/, which detects agents that stay registered, and often still show as active, while log ingestion from them has stopped. Agent connection status does not cover this: an agent can hold its connection to the manager and still have stopped shipping events.For every agent of the monitored groups, a scheduled script on the manager reads the newest indexed event timestamp. When it is older than the threshold the script writes one SILENT record, and when events arrive again it writes one RESTORED record. Local state keeps an unchanged condition from being reported on every run, and durations are measured against real log timestamps rather than the moment the script noticed.
Records are JSON lines ingested through a
<localfile>block, so the built-in JSON decoder handles them and no custom decoder is needed. Rules 100121 and 100122 carryalert_by_emailand are routed to email and to Telegram.Contents
silent_agent_monitor.py- the check, scheduled by a command wodle. Standard library only, so it runs on the embedded interpreter with nopip install.silent_agent_monitor-rules.xml- rules 100120 (level 0 parent), 100121 (SILENT, level 12) and 100122 (RESTORED, level 5).custom-server-telegram,custom-server-telegram.py- Telegram notification, for environments with no Telegram integration yet. The README documents the two dispatch-table entries that reproduce the same layout on an existing integration script.README.md- install, configure, schedule, rules, routing, test, troubleshoot.Design notes
termsaggregation onagent.idwith amaxon@timestampreturns every agent's last event time at once, so the cost does not grow with the fleet or with the number of groups.--grouptakes a comma-separated list, so the wodle inossec.confowns the group names and one schedule covers several groups. Agents are deduplicated across them, so an agent in two listed groups is checked once and its record names both.event_status, notstatus.statusis a static Wazuh field name and a rule matching it fails to load withField 'status' is static.Testing
Verified end to end on a Wazuh 4.14.6 all-in-one, against two agent groups with an overlapping member:
pip install.000andnever_connectedagents.--group rhelfound 2 agents,--group rhel,ops1found 3 rather than 4, and the shared agent's record readgroup=rhel,ops1.alerts.jsonas rule 100121 level 12 or 100122 level 5, with the description fully interpolated.wazuh-integratorddelivered both message layouts to a real bot, HTTP 200.No agents in group(s) '<name>'and exits cleanly.wazuh-modulesdran the command on its interval after a manager restart, and suppression held across the restart.Rule IDs 100120-100122 are in the custom range and can be moved if maintainers prefer a different allocation.