diff --git a/integrations/silent_agent_monitoring/README.md b/integrations/silent_agent_monitoring/README.md new file mode 100644 index 0000000..759cda0 --- /dev/null +++ b/integrations/silent_agent_monitoring/README.md @@ -0,0 +1,191 @@ +# Silent Agent Monitoring + +## Table of Contents +* [Introduction](#introduction) +* [Prerequisites](#prerequisites) +* [Integration Steps](#integration-steps) + * [Add the integration files](#add-the-integration-files) + * [Script configuration](#script-configuration) + * [Wazuh manager configuration](#wazuh-manager-configuration) + * [Add custom rules](#add-custom-rules) + * [Email notifications](#email-notifications) + * [Telegram notifications](#telegram-notifications) +* [Testing](#testing) +* [Troubleshooting](#troubleshooting) +* [Sources](#sources) + +## Introduction +This script runs on the Wazuh manager and detects agents that are still registered, and often still active, while log ingestion from them has stopped. + +For every agent of a target group it reads the timestamp of the most recent indexed event. When that timestamp is older than the threshold it appends a `SILENT` record to a local JSON log, and when events start arriving again it appends a `RESTORED` record. State is kept locally so an unchanged condition is reported once, not once per run. + +The records are plain JSON lines ingested through a `` block, so the built-in JSON decoder handles them and no custom decoder is needed. They trigger rules 100121 and 100122, which are routed to email and to Telegram. + +## Prerequisites +- Wazuh manager 4.4 or later, with an API user that can read `/agents`. +- Wazuh indexer reachable from the manager, with a user that can search the alerts (or archives) indices. +- An agent group to monitor. +- The scripts use only the standard library, so the Wazuh embedded interpreter at `/var/ossec/framework/python/bin/python3` is enough and there is no `pip install` step. + +In a cluster, install on the master node only: running the script on several nodes duplicates every notification and splits the state. + +## Integration Steps + +### Add the integration files +``` +cp silent_agent_monitor.py /var/ossec/wodles/ +cp custom-server-telegram custom-server-telegram.py /var/ossec/integrations/ + +chown root:wazuh /var/ossec/wodles/silent_agent_monitor.py /var/ossec/integrations/custom-server-telegram* +chmod 750 /var/ossec/wodles/silent_agent_monitor.py /var/ossec/integrations/custom-server-telegram* +``` +A manager upgrade can replace the contents of `/var/ossec/wodles`, so keep a copy of the configured script outside `/var/ossec`. + +### Script configuration +Edit the `CONFIGURATION` block at the top of `silent_agent_monitor.py`, or set the matching environment variables and leave the file untouched: + +| Setting | Variable | Default | +| --- | --- | --- | +| Wazuh API URL | `SAM_API_URL` | `https://127.0.0.1:55000` | +| Wazuh API user / password | `SAM_API_USER`, `SAM_API_PASSWORD` | `wazuh-wui` / `CHANGE_ME` | +| Indexer URL | `SAM_INDEXER_URL` | `https://127.0.0.1:9200` | +| Indexer user / password | `SAM_INDEXER_USER`, `SAM_INDEXER_PASSWORD` | `admin` / `CHANGE_ME` | +| Index pattern | `SAM_INDEX_PATTERN` | `wazuh-alerts-*` | +| Agent groups | `--group`, `SAM_GROUP` | `Server` | +| Silence threshold, hours | `SAM_THRESHOLD_HOURS` | `24` | +| Lookback window, days | `SAM_LOOKBACK_DAYS` | `7` | +| State file | `SAM_STATE_FILE` | `/var/ossec/var/silent_agents_state.json` | +| Output log | `SAM_OUTPUT_LOG` | `/var/ossec/logs/silent_agents.json` | +| Script log | `SAM_SCRIPT_LOG` | `/var/ossec/logs/silent_agent_monitor.log` | +| Verify TLS certificates | `SAM_VERIFY_SSL` | `no` | + +The groups are normally passed with `--group` from the wodle, so `ossec.conf` owns them and the script needs no edit to change them. The file holds credentials, so keep it root-owned and `chmod 750`. `SAM_LOOKBACK_DAYS` must stay larger than the threshold: it bounds the indexer query, and an agent with nothing inside it is reported as silent for "more than" that window. + +The index pattern decides what counts as a log. `wazuh-alerts-*` is available everywhere but only sees alerts, so an agent that ships logs normally while producing no alert for a full day is reported as silent. `wazuh-archives-*` is the exact answer to "no logs received" but needs `` enabled and the archives indexed. Use archives when they are available; otherwise confirm that every agent in the group normally produces alerts within the threshold. + +### Wazuh manager configuration +Add to `/var/ossec/etc/ossec.conf`: + +```xml + + no + silent-agent-monitor + /var/ossec/framework/python/bin/python3 /var/ossec/wodles/silent_agent_monitor.py --group Server,Windows + 1h + yes + 300 + yes + + + + json + /var/ossec/logs/silent_agents.json + +``` +`--group` takes a comma-separated list, so one wodle covers several groups: `--group Server,Windows,DMZ`. Agents are deduplicated across them, so an agent in two of the listed groups is checked once and its record names both. A separate wodle per group also works, but each one then needs its own `SAM_STATE_FILE` and `SAM_OUTPUT_LOG`, because a run rewrites the whole state file. + +One run per hour bounds detection lag and recovery lag to an hour each, at one indexer query per hour whatever the number of agents. With `run_on_start`, the first run after a restart can reach the API before it finishes starting and log `HTTP Error 500`; nothing is lost, because a failed run never writes state. + +### Add custom rules +In the Wazuh dashboard go to Server Management > Rules > Add new rules file, name it `silent_agent_monitor-rules.xml`, add the content of `silent_agent_monitor-rules.xml` and save. Then restart the manager. + +| Rule | Level | Fires when | +| --- | --- | --- | +| 100120 | 0 | Any record from this integration. Classification only. | +| 100121 | 12 | `event_status` is `SILENT`. | +| 100122 | 5 | `event_status` is `RESTORED`. | + +The matched field is `event_status`, not `status`: `status` is a static Wazuh field name and a rule matching it fails to load. Both children carry `alert_by_email`, which forces the email regardless of the global ``; without it the level 5 recovery alert is dropped by the default threshold of 12. Move the IDs into a free range if 100120-100122 are already used. + +### Email notifications +Global email must already be configured. Then route the two rules: + +```xml + + soc-team@example.com + 100121,100122 + + full + +``` +The `full` format prints the record's fields one per line, so the email already carries the agent name, the agent ID, the last log timestamp and the duration. + +### Telegram notifications +Use the bundled script only when there is no Telegram integration yet: + +```xml + + custom-server-telegram + 100121,100122 + https://api.telegram.org/bot<BOT_TOKEN>/sendMessage + <CHAT_ID> + json + + + + + custom-server-telegram + 100121,100122 + https://api.telegram.org/bot8454124324:niwefn76t5safuef8s76tg/sendMessage + 123123123123 + json + +``` +`` is the full `sendMessage` endpoint of the bot and `` is the numeric chat ID. Both are passed to the script as arguments and override its defaults. The messages it produces: + +``` +⚠️ Server Logging Alert ✅ Server Logging Restored +Name: File2 Name: File2 +Agent ID: 152 Agent ID: 152 +Status: No logs received Status: Logs received +Last Log Received: ... Restored At: ... +No Logs For: 25h 40m No Logs Duration: 25h 40m +``` + +When a Telegram integration already exists, keep it and add `100121,100122` to its `` (or `silent_agent_monitoring` to its ``). A dispatcher that maps dotted paths to labels needs one entry per event type, matching on the `server_silent` and `server_restored` rule groups, with the fields listed above; the `Status` line is `data.status_text`, so no literal is needed in the dispatcher. + +`wazuh-integratord` runs integration scripts as the `wazuh` user, so any path the script writes, including a custom `TELEGRAM_LOG`, must be writable by it. + +## Testing +Run the check by hand against the live environment: +```bash +/var/ossec/framework/python/bin/python3 /var/ossec/wodles/silent_agent_monitor.py --group Server +# Checked 12 agent(s) in 'Server': 0 silent, 0 event(s) written to /var/ossec/logs/silent_agents.json. +``` + +To force a notification without waiting, drop the threshold for one run: +```bash +SAM_THRESHOLD_HOURS=0.05 /var/ossec/framework/python/bin/python3 \ + /var/ossec/wodles/silent_agent_monitor.py --group Server +tail -1 /var/ossec/logs/silent_agents.json +tail -f /var/ossec/logs/alerts/alerts.log | grep -A5 100121 +``` +Running with the real threshold again produces the `RESTORED` notification, which confirms the recovery path and the message formatting in one go. Delete `/var/ossec/var/silent_agents_state.json` afterwards so the test does not leave agents marked silent. + +The rules can be checked without running the script: +```bash +echo '{"integration":"silent-agent-monitor","event_status":"SILENT","agent_id":"152","agent_name":"File2","last_log":"2026-08-18 08:35:12 CEST","no_logs_for":"25h 40m"}' \ + | /var/ossec/bin/wazuh-logtest +``` + +## Troubleshooting +| Symptom | Cause and fix | +| --- | --- | +| `No events found for any of the N agents` and no alerts | Deliberate safety stop: every agent silent at once is almost always a wrong index pattern or wrong indexer credentials. Check `SAM_INDEX_PATTERN` and the indexer user. | +| `Indexer query failed` or `Wazuh API query failed` | The run exits without touching the state, so nothing is reported as silent or recovered on a failed query. Check connectivity and credentials. | +| `No agents in group(s) 'X'` | The group does not exist or is empty. Check with `/var/ossec/bin/agent_groups -l`. | +| Records in `silent_agents.json` but no alerts | The `` block is missing, points elsewhere, or sits on a node that is not running the script. | +| Alerts fire but no email | Global email is not enabled, or the rules lost `alert_by_email`. Check `/var/ossec/logs/ossec.log` for `wazuh-maild`. | +| Alerts fire but no Telegram message | Check `/var/ossec/logs/integrations.log`. A missing chat ID or hook URL, or an HTTP error from the bot API, is logged with the rule ID. | +| `Field 'status' is static` | The rule was edited to match `status` instead of `event_status`. | +| A healthy agent is reported silent | It produced no alerts within the threshold. Point `SAM_INDEX_PATTERN` at `wazuh-archives-*`, or raise the threshold. | +| Every agent reported again after a manager rebuild | The state file was lost, so the first run re-reports the conditions that are still true. One repeat, then quiet again. | + +## Sources +- [Wazuh - command wodle](https://documentation.wazuh.com/current/user-manual/reference/ossec-conf/wodle-command.html) +- [Wazuh - localfile configuration](https://documentation.wazuh.com/current/user-manual/reference/ossec-conf/localfile.html) +- [Wazuh - integration configuration](https://documentation.wazuh.com/current/user-manual/reference/ossec-conf/integration.html) +- [Wazuh - granular email alerts](https://documentation.wazuh.com/current/user-manual/manager/manual-email-report/index.html) +- [Wazuh - rules syntax](https://documentation.wazuh.com/current/user-manual/ruleset/ruleset-xml-syntax/rules.html) +- [Wazuh API - agents](https://documentation.wazuh.com/current/user-manual/api/reference.html#tag/Agents) +- [Wazuh - archiving alerts and events](https://documentation.wazuh.com/current/user-manual/manager/event-logging.html) diff --git a/integrations/silent_agent_monitoring/custom-server-telegram b/integrations/silent_agent_monitoring/custom-server-telegram new file mode 100755 index 0000000..fc3023b --- /dev/null +++ b/integrations/silent_agent_monitoring/custom-server-telegram @@ -0,0 +1,38 @@ +#!/bin/sh +# Copyright (C) 2015, Wazuh Inc. +# Created by Wazuh, Inc. . +# This program is free software; you can redistribute it and/or modify it under the terms of GPLv2 + +WPYTHON_BIN="framework/python/bin/python3" + +SCRIPT_PATH_NAME="$0" + +DIR_NAME="$(cd $(dirname ${SCRIPT_PATH_NAME}); pwd -P)" +SCRIPT_NAME="$(basename ${SCRIPT_PATH_NAME})" + +case ${DIR_NAME} in + */active-response/bin | */wodles*) + if [ -z "${WAZUH_PATH}" ]; then + WAZUH_PATH="$(cd ${DIR_NAME}/../..; pwd)" + fi + + PYTHON_SCRIPT="${DIR_NAME}/${SCRIPT_NAME}.py" + ;; + */bin) + if [ -z "${WAZUH_PATH}" ]; then + WAZUH_PATH="$(cd ${DIR_NAME}/..; pwd)" + fi + + PYTHON_SCRIPT="${WAZUH_PATH}/framework/scripts/$(echo ${SCRIPT_NAME} | sed 's/\-/_/g').py" + ;; + */integrations) + if [ -z "${WAZUH_PATH}" ]; then + WAZUH_PATH="$(cd ${DIR_NAME}/..; pwd)" + fi + + PYTHON_SCRIPT="${DIR_NAME}/${SCRIPT_NAME}.py" + ;; +esac + + +${WAZUH_PATH}/${WPYTHON_BIN} ${PYTHON_SCRIPT} "$@" diff --git a/integrations/silent_agent_monitoring/custom-server-telegram.py b/integrations/silent_agent_monitoring/custom-server-telegram.py new file mode 100755 index 0000000..a05102f --- /dev/null +++ b/integrations/silent_agent_monitoring/custom-server-telegram.py @@ -0,0 +1,125 @@ +#!/var/ossec/framework/python/bin/python3 +# Copyright (C) 2015, Wazuh Inc. +# This program is free software; you can redistribute it and/or modify it +# under the terms of GPLv2. +# +# custom-server-telegram.py +# Wazuh Telegram integration for the silent agent monitoring rules. Formats +# rules 100121 (no logs received) and 100122 (logging restored) and posts them +# to a Telegram chat. Any other rule routed here falls back to a generic +# message. wazuh-integratord calls this as: +# custom-server-telegram +# so carries the chat ID and the bot sendMessage URL. + +import json +import logging +import os +import sys +import ssl +import urllib.request + +# === CONFIGURATION === +# Used only when ossec.conf passes nothing, or for a manual test run. +CHAT_ID = os.environ.get("TELEGRAM_CHAT_ID", "") +HOOK_URL = os.environ.get("TELEGRAM_HOOK_URL", "") +LOG_PATH = os.environ.get("TELEGRAM_LOG", "/var/ossec/logs/integrations.log") +VERIFY_SSL = os.environ.get("TELEGRAM_VERIFY_SSL", "yes").lower() in ("yes", "true", "1") +TIMEOUT = 15 + +SILENT_RULE = "100121" +RESTORED_RULE = "100122" + +_LOG_ARGS = {"format": "%(asctime)s custom-server-telegram %(levelname)s %(message)s", + "datefmt": "%Y-%m-%dT%H:%M:%S", "level": logging.INFO} +try: + logging.basicConfig(filename=LOG_PATH, filemode="a", **_LOG_ARGS) +except OSError: + logging.basicConfig(stream=sys.stderr, **_LOG_ARGS) + + +def build_message(alert): + data = alert.get("data", {}) + rule = alert.get("rule", {}) + rule_id = str(rule.get("id", "")) + + if rule_id == SILENT_RULE: + return (f"⚠ Server Logging Alert\n" + f"Name: {data.get('agent_name', 'unknown')}\n" + f"Agent ID: {data.get('agent_id', 'unknown')}\n" + f"Status: {data.get('status_text', 'No logs received')}\n" + f"Last Log Received: {data.get('last_log', 'unknown')}\n" + f"No Logs For: {data.get('no_logs_for', 'unknown')}") + + if rule_id == RESTORED_RULE: + return (f"✅ Server Logging Restored\n" + f"Name: {data.get('agent_name', 'unknown')}\n" + f"Agent ID: {data.get('agent_id', 'unknown')}\n" + f"Status: {data.get('status_text', 'Logs received')}\n" + f"Restored At: {data.get('restored_at', 'unknown')}\n" + f"No Logs Duration: {data.get('silence_duration', 'unknown')}") + + agent = alert.get("agent", {}) + return (f"Wazuh alert\n" + f"Rule: {rule_id} (level {rule.get('level', '')})\n" + f"Description: {rule.get('description', '')}\n" + f"Agent: {agent.get('name', 'manager')} ({agent.get('id', '000')})") + + +def ssl_context(): + if not VERIFY_SSL: + context = ssl.create_default_context() + context.check_hostname = False + context.verify_mode = ssl.CERT_NONE + return context + # The Wazuh embedded interpreter's OpenSSL looks for roots in a path that + # does not exist, so a default context trusts nothing. certifi ships with + # it; a system interpreter without certifi already has a working store. + try: + import certifi + return ssl.create_default_context(cafile=certifi.where()) + except ImportError: + return ssl.create_default_context() + + +def send(hook_url, chat_id, message): + payload = json.dumps({"chat_id": chat_id, "text": message, + "parse_mode": "HTML"}).encode() + req = urllib.request.Request(hook_url, data=payload, method="POST") + req.add_header("Content-Type", "application/json") + with urllib.request.urlopen(req, timeout=TIMEOUT, context=ssl_context()) as resp: + return resp.status + + +def main(): + if len(sys.argv) < 2: + logging.error("Usage: %s [chat_id] [hook_url]", sys.argv[0]) + sys.exit(1) + + try: + with open(sys.argv[1]) as f: + alert = json.load(f) + except (OSError, ValueError) as err: + logging.error("Failed to read alert file '%s': %s", sys.argv[1], err) + sys.exit(1) + + chat_id = sys.argv[2] if len(sys.argv) > 2 and sys.argv[2] else CHAT_ID + hook_url = sys.argv[3] if len(sys.argv) > 3 and sys.argv[3] else HOOK_URL + if not chat_id or not hook_url: + logging.error("Missing chat ID or hook URL. Set and " + "in the block.") + sys.exit(1) + + message = build_message(alert) + try: + status = send(hook_url, chat_id, message) + except Exception as err: + logging.error("Telegram delivery failed for rule %s: %s", + alert.get("rule", {}).get("id", ""), err) + sys.exit(1) + + logging.info("Sent rule %s to chat %s (HTTP %s).", + alert.get("rule", {}).get("id", ""), chat_id, status) + + +if __name__ == "__main__": + main() diff --git a/integrations/silent_agent_monitoring/silent_agent_monitor-rules.xml b/integrations/silent_agent_monitoring/silent_agent_monitor-rules.xml new file mode 100644 index 0000000..0f8cd08 --- /dev/null +++ b/integrations/silent_agent_monitoring/silent_agent_monitor-rules.xml @@ -0,0 +1,30 @@ + + + + + json + ^silent-agent-monitor$ + Silent agent monitor event + + + + + 100120 + ^SILENT$ + Server $(agent_name) (ID $(agent_id)) has sent no logs for $(no_logs_for) + alert_by_email + no_full_log + server_silent,service_availability, + + + + + 100120 + ^RESTORED$ + Server $(agent_name) (ID $(agent_id)) resumed logging after $(silence_duration) + alert_by_email + no_full_log + server_restored,service_availability, + + + diff --git a/integrations/silent_agent_monitoring/silent_agent_monitor.py b/integrations/silent_agent_monitoring/silent_agent_monitor.py new file mode 100755 index 0000000..66988c8 --- /dev/null +++ b/integrations/silent_agent_monitoring/silent_agent_monitor.py @@ -0,0 +1,260 @@ +#!/var/ossec/framework/python/bin/python3 +# +# silent_agent_monitor.py +# Detects Wazuh agents that are still registered but have stopped shipping +# logs. For every agent in a target group it reads the timestamp of the most +# recent indexed event and, when that timestamp is older than the threshold, +# appends a SILENT record to a local JSON log that Wazuh ingests through a +# block. When events start arriving again it appends a matching +# RESTORED record. State is kept locally so an unchanged condition is reported +# once, not once per run. Standard library only. +# +# Usage: silent_agent_monitor.py --group Server,Windows + +import argparse +import base64 +import json +import logging +import os +import ssl +import sys +import urllib.error +import urllib.request +from datetime import datetime, timedelta, timezone + +# === CONFIGURATION === +API_URL = os.environ.get("SAM_API_URL", "https://127.0.0.1:55000") +API_USER = os.environ.get("SAM_API_USER", "wazuh-wui") +API_PASSWORD = os.environ.get("SAM_API_PASSWORD", "CHANGE_ME") + +INDEXER_URL = os.environ.get("SAM_INDEXER_URL", "https://127.0.0.1:9200") +INDEXER_USER = os.environ.get("SAM_INDEXER_USER", "admin") +INDEXER_PASSWORD = os.environ.get("SAM_INDEXER_PASSWORD", "CHANGE_ME") + +INDEX_PATTERN = os.environ.get("SAM_INDEX_PATTERN", "wazuh-alerts-*") + +# Comma-separated. Overridden by --group, so the wodle owns the group names. +TARGET_GROUPS = [g.strip() for g in os.environ.get("SAM_GROUP", "Server").split(",") + if g.strip()] +SILENCE_THRESHOLD = timedelta(hours=float(os.environ.get("SAM_THRESHOLD_HOURS", "24"))) +LOOKBACK = timedelta(days=float(os.environ.get("SAM_LOOKBACK_DAYS", "7"))) + +STATE_FILE = os.environ.get("SAM_STATE_FILE", "/var/ossec/var/silent_agents_state.json") +OUTPUT_LOG = os.environ.get("SAM_OUTPUT_LOG", "/var/ossec/logs/silent_agents.json") +SCRIPT_LOG = os.environ.get("SAM_SCRIPT_LOG", "/var/ossec/logs/silent_agent_monitor.log") + +VERIFY_SSL = os.environ.get("SAM_VERIFY_SSL", "no").lower() in ("yes", "true", "1") +PAGE_SIZE = 500 +HTTP_TIMEOUT = 30 + +_LOG_ARGS = {"format": "%(asctime)s %(levelname)s %(message)s", + "datefmt": "%Y-%m-%dT%H:%M:%S", "level": logging.INFO} +try: + logging.basicConfig(filename=SCRIPT_LOG, filemode="a", **_LOG_ARGS) +except OSError: + logging.basicConfig(stream=sys.stderr, **_LOG_ARGS) + +SSL_CONTEXT = ssl.create_default_context() +if not VERIFY_SSL: + SSL_CONTEXT.check_hostname = False + SSL_CONTEXT.verify_mode = ssl.CERT_NONE + + +def http_json(url, method="GET", body=None, token=None, basic=None): + data = json.dumps(body).encode() if body is not None else None + req = urllib.request.Request(url, data=data, method=method) + req.add_header("Content-Type", "application/json") + if token: + req.add_header("Authorization", f"Bearer {token}") + if basic: + raw = base64.b64encode(f"{basic[0]}:{basic[1]}".encode()).decode() + req.add_header("Authorization", f"Basic {raw}") + with urllib.request.urlopen(req, timeout=HTTP_TIMEOUT, context=SSL_CONTEXT) as resp: + return json.loads(resp.read().decode()) + + +def get_token(): + url = f"{API_URL}/security/user/authenticate" + return http_json(url, method="POST", basic=(API_USER, API_PASSWORD))["data"]["token"] + + +def fetch_group_agents(token, group): + agents, offset = [], 0 + while True: + url = (f"{API_URL}/agents?group={group}&limit={PAGE_SIZE}&offset={offset}" + f"&sort=%2Bid&select=id,name,status,lastKeepAlive") + data = http_json(url, token=token).get("data", {}) + agents.extend(a for a in data.get("affected_items", []) + if a.get("id") != "000" and a.get("status") != "never_connected") + offset += PAGE_SIZE + if offset >= data.get("total_affected_items", 0): + break + return agents + + +def fetch_last_event_times(agent_ids): + """{agent_id: datetime}. Agents with no event inside LOOKBACK are absent.""" + query = { + "size": 0, + "query": {"bool": {"filter": [ + {"terms": {"agent.id": agent_ids}}, + {"range": {"@timestamp": {"gte": f"now-{int(LOOKBACK.total_seconds())}s"}}}, + ]}}, + "aggs": {"per_agent": { + "terms": {"field": "agent.id", "size": len(agent_ids)}, + "aggs": {"last_event": {"max": {"field": "@timestamp"}}}, + }}, + } + url = f"{INDEXER_URL}/{INDEX_PATTERN}/_search" + result = http_json(url, method="POST", body=query, + basic=(INDEXER_USER, INDEXER_PASSWORD)) + buckets = result.get("aggregations", {}).get("per_agent", {}).get("buckets", []) + return {b["key"]: datetime.fromtimestamp(b["last_event"]["value"] / 1000, timezone.utc) + for b in buckets if b["last_event"]["value"]} + + +def format_duration(delta): + minutes = int(delta.total_seconds() // 60) + hours, minutes = divmod(minutes, 60) + return f"{hours}h {minutes}m" if minutes else f"{hours}h" + + +def local_time(dt): + return dt.astimezone().strftime("%Y-%m-%d %H:%M:%S %Z") + + +def decide(agent, last_log, previous, now): + """Pure decision for one agent. Returns (event or None, new state entry).""" + agent_id, name = agent["id"], agent.get("name", "unknown") + silent = last_log is None or (now - last_log) >= SILENCE_THRESHOLD + was_silent = previous.get("status") == "SILENT" + + # event_status, not status: "status" is a static Wazuh field name and a + # rule cannot match it with . + common = { + "integration": "silent-agent-monitor", + "group": ",".join(agent.get("groups", [])), + "agent_id": agent_id, + "agent_name": name, + "agent_status": agent.get("status", "unknown"), + } + state = {"status": "SILENT" if silent else "OK", "name": name, + "last_log": last_log.isoformat() if last_log else previous.get("last_log")} + + if silent and not was_silent: + gap = (now - last_log) if last_log else LOOKBACK + event = dict(common, event_status="SILENT", + status_text="No logs received", + last_log=local_time(last_log) if last_log else "unknown", + no_logs_for=format_duration(gap) if last_log + else f"more than {format_duration(LOOKBACK)}", + no_logs_seconds=int(gap.total_seconds()), + message=f"Agent {name} (ID {agent_id}) has sent no logs " + f"for more than {format_duration(SILENCE_THRESHOLD)}.") + return event, state + + if not silent and was_silent: + # Measured from the last log before the gap to the first log after it, + # not from the moment this script noticed. + previous_log = previous.get("last_log") + gap = (last_log - datetime.fromisoformat(previous_log)) if previous_log else None + event = dict(common, event_status="RESTORED", + status_text="Logs received", + restored_at=local_time(last_log), + silence_duration=format_duration(gap) if gap is not None else "unknown", + silence_seconds=int(gap.total_seconds()) if gap is not None else 0, + message=f"Agent {name} (ID {agent_id}) has resumed sending logs.") + return event, state + + return None, state + + +def load_state(): + try: + with open(STATE_FILE) as f: + return json.load(f) + except FileNotFoundError: + return {} + except (OSError, ValueError) as err: + logging.error("Could not read state file '%s': %s. Starting empty.", STATE_FILE, err) + return {} + + +def save_state(state): + tmp = f"{STATE_FILE}.tmp" + with open(tmp, "w") as f: + json.dump(state, f, indent=2) + os.replace(tmp, STATE_FILE) + + +def append_events(events): + with open(OUTPUT_LOG, "a") as f: + for event in events: + f.write(json.dumps(event, ensure_ascii=False) + "\n") + + +def main(): + now = datetime.now(timezone.utc) + groups = ",".join(TARGET_GROUPS) + # An agent in two target groups is checked once and reports both names. + found = {} + try: + token = get_token() + for group in TARGET_GROUPS: + for agent in fetch_group_agents(token, group): + found.setdefault(agent["id"], dict(agent, groups=[]))["groups"].append(group) + except (urllib.error.URLError, OSError, KeyError, ValueError) as err: + logging.error("Wazuh API query failed: %s", err) + sys.exit(1) + + agents = list(found.values()) + if not agents: + logging.info("No agents in group(s) '%s'. Nothing to do.", groups) + return + + agent_ids = [a["id"] for a in agents] + try: + last_events = fetch_last_event_times(agent_ids) + except (urllib.error.URLError, OSError, KeyError, ValueError) as err: + # Exit without touching the state: a failed query must never be read as + # "every agent went silent", nor as "every agent recovered". + logging.error("Indexer query failed: %s", err) + sys.exit(1) + + if not last_events and len(agents) > 1: + # Every agent silent at once is far more likely a wrong index pattern + # or wrong credentials than a real outage. Refuse to send the storm. + logging.error("No events found for any of the %d agents in '%s' over the last %s. " + "Check SAM_INDEX_PATTERN and the indexer credentials. No alerts sent.", + len(agents), groups, format_duration(LOOKBACK)) + sys.exit(1) + + state = load_state() + events, new_state = [], {} + for agent in agents: + event, entry = decide(agent, last_events.get(agent["id"]), + state.get(agent["id"], {}), now) + new_state[agent["id"]] = entry + if event: + events.append(event) + + if events: + append_events(events) + save_state(new_state) + + silent = sum(1 for e in new_state.values() if e["status"] == "SILENT") + logging.info("Checked %d agent(s) in '%s': %d silent, %d new event(s) written.", + len(agents), groups, silent, len(events)) + print(f"Checked {len(agents)} agent(s) in '{groups}': " + f"{silent} silent, {len(events)} event(s) written to {OUTPUT_LOG}.") + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description="Detect Wazuh agents that stopped sending logs.") + parser.add_argument("--group", default=",".join(TARGET_GROUPS), + help="comma-separated agent groups to monitor " + f"(default: {','.join(TARGET_GROUPS)})") + args = parser.parse_args() + + TARGET_GROUPS = [g.strip() for g in args.group.split(",") if g.strip()] + main()