Skip to content

README.md

Latest commit

 

History

History
209 lines (145 loc) · 5.15 KB

README.md

File metadata and controls

209 lines (145 loc) · 5.15 KB
Raw

Kernolog

**Real-time Linux log monitoring and semantic search **

Kernolog streams live system logs from journalctl, deduplicates repeated entries using the Drain3 log template miner, embeds them with a SentenceTransformer model, and stores them in a hybrid SQLite + binary vector store for fast semantic similarity search.

Features

  • Live log ingestion via journalctl in JSON mode, capturing message, priority, and unit
  • Log normalization & deduplication using Drain3 — repeated log lines are collapsed into templates (e.g. "User <*> logged in")
  • Parameter extraction — variables like usernames, device names, and IPs are stored separately and highlighted in results
  • AI embeddings using all-MiniLM-L6-v2 (via sentence-transformers)
  • Three-tier classification — logs are bucketed into error (priority ≤ 3), warning (priority 4), and debug (priority ≥ 5)
  • Two-phase semantic search — a fast broad pass over template vectors, followed by live re-ranking with hydrated (parameter-restored) sentences
  • Recency-biased search — use keywords like now, latest, or recent to surface the most recent relevant logs
  • Desktop alerts via notify-send for critical errors
  • Interactive shell for querying logs in real time

Architecture

journalctl (JSON)
      │
      ▼
 [Collector]       collector/core.py
      │  raw log dicts
      ▼
 [Normalizer]      normalizer/core.py
      │  template + params + priority
      ▼
 [Engine]          engine.py
      │  batched embedding + storage
      ▼
 [Storage]         storage.py
   ├── {category}.sqlite   (templates, occurrences, parameters)
   └── {category}.bin      (float32 embedding vectors)
      │
      ▼
 [Shell]           shell.py
   └── semantic search CLI

Requirements

  • Python 3.8+
  • Linux with systemd (for journalctl)
  • notify-send (optional, for desktop alerts on critical errors)

Installation & Setup

1. Clone the repository

git clone https://github.com/iririthik/Kernolog.git
cd Kernolog

2. (Recommended) Create a virtual environment

python3 -m venv venv
source venv/bin/activate

3. Install dependencies

pip install sentence-transformers drain3 colorama numpy

Note: The first run will download the all-MiniLM-L6-v2 model (~90 MB) from HuggingFace automatically.

4. (Optional) Install desktop notifications

# Debian/Ubuntu
sudo apt install libnotify-bin

# Arch
sudo apt install libnotify

5. Verify journalctl access

Kernolog reads from journalctl. If you get a permissions error, add your user to the systemd-journal group:

sudo usermod -aG systemd-journal $USER
# Log out and back in for this to take effect

Quick Start

Run everything (recommended)

python boot.py

This starts the engine in the background and opens the search shell in a new terminal window. Indexed data is written to ./gen_data/.

Or run components separately

Terminal 1 — start the engine:

python engine.py

Terminal 2 — open the search shell:

python shell.py

Search Shell Usage

Kernolog> search <category> <query>

Categories: error, warning, debug

Example command What it does
search error disk failure Semantic search for disk-related errors
search warning latest Most recent warnings
search error usb device now Recent USB errors, time-prioritized
search debug network Debug logs related to networking

Tips:

  • Use now, latest, recent, last, today, or current to sort results by time instead of relevance score
  • Type clear to clear the screen
  • Type exit or quit to close the shell

Example output:

--- ERROR Results ---
[Score:0.87] 14:23:01.042 | kernel: usb disconnect, device number 3
[Score:0.74] 14:19:55.118 | kernel: EXT4-fs error on sda1
--------------------------------------------------

Matched parameters are highlighted in yellow.

Data Storage

All indexed data lives in ./gen_data/:

File Contents
error.sqlite Templates, occurrences, and extracted parameters for errors
error.bin Raw float32 embedding vectors (384-dim)
warning.sqlite / warning.bin Same for warnings
debug.sqlite / debug.bin Same for debug/info logs

Using as a Library

Watch live logs with a custom callback:

from collector import watch

def my_handler(log):
    print(log['priority'], log['message'])

watch(my_handler)

Run the full normalized pipeline:

from normalizer import process_stream

def my_handler(log_line):
    print(log_line)

process_stream(my_handler)

License

AGPL-3.0 — if you run a modified version as a network service, you must make your source code available.

Contributing

Contributions are welcome. See the repository's contributing guidelines.

Acknowledgements

Built with Drain3, sentence-transformers, and NumPy.