A NOC/SOC-style triage bot that takes raw infrastructure alerts, enriches them with threat intel, and produces a severity (P1–P4), a human-readable reason, and an escalation target — using a deterministic rule engine first, an LLM second. Every decision is logged and visualized in a Streamlit dashboard.
TL;DR: Rules decide. AI confirms or overrides with reasoning. Nothing ships without a paper trail.
Modern infra teams get buried in alerts. A noisy L1 tier over-escalates, under-escalates, and gets inconsistent across shifts. This bot is an experiment in layering rather than replacing human judgment:
- A rule engine (fast, deterministic, auditable) proposes a severity.
- An LLM (Groq, JSON-constrained) either confirms the call or overrides it with a written reason.
- If the LLM fails or returns bad JSON, the rules win. The pipeline never crashes.
The end result: a structured decision per alert, ready to be paged, filed, or reviewed.
Evaluated against 100 synthetic alerts with hand-labeled ground truth.
| Engine | Accuracy |
|---|---|
| Rule-Based | 96% |
| AI-Assisted (Groq LLM) | 76% |
The rules stayed near-perfect across every alert type. The LLM over-escalated — most sharply on beaconing (10/100) and host-down (6/100). That's the headline finding of this project: the cheap deterministic layer is the one to trust; the LLM is a reasoning assistant, not a primary decision-maker.
Rules decide first. AI confirms or overrides with reasoning. Every decision is logged.
- Raw Alert arrives (failed login, host down, disk space, beaconing).
- IP Reputation Enrichment via AbuseIPDB — assigns a 0–100 abuse confidence score.
- Rule Engine applies deterministic thresholds and returns
(severity, reason, escalate_to). - AI Layer (Groq LLM) receives the rule suggestion and either confirms or overrides with its own JSON-shaped reasoning.
- Decision Log + Streamlit Dashboard record the call, including whether the AI agreed with the rules.
| Alert | Rule logic |
|---|---|
failed_logins |
P1 if source IP reputation > 50; P2 if count > 30; P3 otherwise |
host_down |
P2, escalate to NOC Shift Lead |
disk_space |
P2 if > 90% used; P4 otherwise |
beaconing |
P1 — regular-interval connections, escalate to SOC L2 / Security on-call |
The rule thresholds live in triage.py and are intentionally simple so they can be tuned by a human in seconds.
streamlit run dashboard.py shows:
- Total alerts logged, P1 count, AI-assisted decisions
- Severity distribution
- The full decision log (sortable)
- A side-by-side accuracy view from
evaluation_results.json
alert-triage-bot/
├── triage.py # Rule engine + IP reputation + decision logging
├── ai_triage.py # LLM confirmation / override layer (Groq)
├── beacon_detect.py # Detect regular-interval connections in a connection log
├── generate_alerts.py # Generate synthetic alerts with hand-coded ground truth
├── evaluate.py # Run both engines against ground truth, write results JSON
├── dashboard.py # Streamlit dashboard over triage_log.csv
├── synthetic_alerts.json # 100-alert evaluation dataset
├── evaluation_results.json
├── triage_log.csv # Every triage decision ever made
├── .env.example # Copy to .env, fill in keys
└── assets/ # README images
# 1. Clone
git clone https://github.com/ryuk27/alert-triage-bot.git
cd alert-triage-bot
# 2. Create venv and install deps
python -m venv venv
source venv/bin/activate
pip install requests python-dotenv groq pandas streamlit
# 3. Add your API keys
cp .env.example .env
# ABUSEIPDB_KEY=... (https://www.abuseipdb.com/account/api)
# GROQ_API_KEY=... (https://console.groq.com/keys)# Generate the synthetic alert dataset
python generate_alerts.py
# Run the rule engine only (no API calls beyond AbuseIPDB)
python triage.py
# Run the full pipeline with the LLM layer
python ai_triage.py
# Detect beaconing in a connection log and triage the hits
python beacon_detect.py
# Evaluate both engines against the labeled ground truth
python evaluate.py
# View the dashboard
streamlit run dashboard.pyai_triage.py calls Groq's openai/gpt-oss-120b with:
- A system prompt that frames the LLM as a NOC/SOC L1 triage analyst.
- The alert + the rule-based suggestion.
response_format={"type": "json_object"}to force valid JSON output.temperature=0.2andreasoning_effort="medium"for stable, fast calls.
If the model returns empty, malformed, or anything other than valid JSON, the pipeline falls back to the rule result and logs the decision with source=rule_fallback. AI outages never break the bot.
The interesting result is the gap, not the absolute numbers. A 20-point accuracy drop from the AI layer is consistent with what you see in production: LLMs are good at producing plausible severity calls, but they over-escalate security-flavored language (any mention of "beaconing" or "brute force" tends to get pushed to P1). For a paging system, that bias is expensive — every false P1 burns an on-call rotation.
The architecture here is one answer: use the LLM to write the explanation, but let the deterministic layer write the verdict. This is also why the dashboard surfaces rule_suggested and ai_agreed_with_rules on every row.
- Wire a real alert source (Prometheus Alertmanager webhook, Datadog, etc.)
- Per-customer rule profiles (thresholds vary by environment)
- Feedback loop: when humans override a decision, retrain the rule thresholds
- Deeper beaconing detection (sliding-window entropy, jitter tolerance)
MIT.