Skip to content

Commit 8a075b1

Browse files
feat!: rewrite to local JSONL logging, replace Analysis Server architecture
Changes: - Replace AnalysisFilter/AnalysisService/AnalysisConfig with RequestLogFilter/RequestLogService/RequestLogConfig - Rename env vars from ANALYSIS_* to REQUEST_LOG_* - Remove Guzzle dependency, queue system, and AnalysisSend command - Write JSONL directly to writable/logs/analysis.log with daily rotation, gzip compression, and retention pruning - Add requestlog:rotate spark command for manual/cron rotation - Add docs/wazuh.md with agent config, JSON decoder, and detection rules (suspicious file upload, SQL injection, etc.) - Update README, CHANGELOG (v2.0.0), .gitignore, composer.json BREAKING CHANGE: Filter alias changes from 'analysis' to 'requestlog'. Config class Analysis -> RequestLog. Env vars ANALYSIS_* -> REQUEST_LOG_*. No Guzzle/server/queue required.
1 parent 2a73a8a commit 8a075b1

13 files changed

Lines changed: 666 additions & 946 deletions

.ai/INSTRUCTION.md

Lines changed: 0 additions & 481 deletions
This file was deleted.

.gitignore

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -31,5 +31,4 @@ Thumbs.db
3131
/build/
3232
/dist/
3333

34-
# Analysis queue (local spool)
35-
/analysis_queue/
34+
/ai/

CHANGELOG.md

Lines changed: 30 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -5,37 +5,39 @@ All notable changes to this project will be documented in this file.
55
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
66
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
77

8+
## [2.0.0] - 2026-08-31
9+
10+
### Changed
11+
12+
- **Architecture overhaul**: logs are no longer sent to an external Analysis
13+
Server. The filter now writes directly to the CodeIgniter `writable/logs`
14+
directory as JSON Lines.
15+
- Configuration renamed from `ANALYSIS_*` to `REQUEST_LOG_*`; config class
16+
`Analysis``RequestLog`.
17+
- Filter renamed `AnalysisFilter``RequestLogFilter` (alias `requestlog`).
18+
- Service renamed `AnalysisService``RequestLogService`.
19+
- Single log file `analysis.log` with automatic daily rotation + gzip
20+
compression + retention pruning.
21+
22+
### Removed
23+
24+
- `AnalysisSend` command, local queue, and `failed/` retry handling.
25+
- Guzzle dependency and `ANALYSIS_SERVER_URL` / `ANALYSIS_API_KEY` config.
26+
27+
### Added
28+
29+
- `requestlog:rotate` spark command — manual/cron fallback for rotation,
30+
gzip compression and pruning of old logs.
31+
- `REQUEST_LOG_RETENTION_DAYS` config (default 30) to control how many days of
32+
compressed logs are kept.
33+
834
## [1.0.0] - 2026-08-31
935

1036
### Added
1137

12-
- `AnalysisFilter` (CI4 filter) — manual per-route attachment, enabled check,
13-
IP whitelist skip, non-blocking queue + background send.
14-
- `AnalysisService` — request metadata collection (`timestamp`, `domain`,
15-
`path`, `method`, `srcip`, `user_agent`, `query_string`, `headers`,
16-
`raw_body`, `file_count`, `file_names`, `file_metadata`).
17-
- Sensitive-field redaction (recursive, case-insensitive; default:
18-
`password`, `nik`, `Api-Key`, `no_telp`).
19-
- Body truncation at `ANALYSIS_MAX_BODY_SIZE` with configurable suffix.
20-
- File upload metadata (original name, size, MIME, extension, SHA-256 hash,
21-
double-extension detection) — binary content is never sent.
22-
- Local file-based queue with `ANALYSIS_MAX_QUEUE` enforcement (oldest dropped).
23-
- `AnalysisSend` spark command — sends queued entries, retries on failure,
24-
moves items to `failed/` after `ANALYSIS_MAX_RETRIES`.
25-
- `Analysis` config class reading all `ANALYSIS_*` environment variables.
26-
- IP/CIDR whitelist (RFC1918 + localhost by default) via `AnalysisService`.
27-
- Multi-tenant identification from `HTTP_HOST` subdomain.
28-
- Guzzle 7 HTTP client with configurable timeout.
29-
- PHP 7.4 and PHP 8.4+ support.
30-
31-
## [Unreleased]
32-
33-
### Planned
34-
35-
- Unit tests (PHPUnit) for `AnalysisService` and `AnalysisSend`.
36-
- Analysis Server (Docker: PHP-FPM + Nginx on Alpine) with daily gzip log
37-
rotation and Wazuh integration.
38-
- k6 load test script.
39-
- Sample CI4 application showing integration.
38+
- Initial release: `AnalysisFilter`, `AnalysisService`, `Analysis` config and
39+
`AnalysisSend` command with queue-based asynchronous delivery to an Analysis
40+
Server. Superseded by 2.0.0.
4041

42+
[2.0.0]: https://github.com/mrnaeem4/ci4-request-analysis/releases/tag/v2.0.0
4143
[1.0.0]: https://github.com/mrnaeem4/ci4-request-analysis/releases/tag/v1.0.0

README.md

Lines changed: 100 additions & 93 deletions
Original file line numberDiff line numberDiff line change
@@ -1,36 +1,34 @@
11
# ci4-request-analysis
22

33
CodeIgniter 4 library that intercepts incoming HTTP requests, extracts structured
4-
metadata (headers, body, files, tenant, source IP, etc.), and asynchronously sends a
5-
JSON log to a dedicated Analysis Server — with local file-based queuing, retry,
6-
sensitive-field redaction, body truncation, and IP whitelisting.
4+
metadata (headers, body, files, tenant, source IP, etc.), and writes one JSON
5+
Line per request directly to the CI4 `writable/logs` directory — with
6+
sensitive-field redaction, body truncation, IP whitelisting, and daily log
7+
rotation with gzip compression.
78

8-
> Client side only. The Analysis Server (Docker: PHP-FPM + Nginx on Alpine, log
9-
> rotation, Wazuh consumption) is delivered in the `server/` directory of this
10-
> repository.
9+
> Local logging only. No external server, no queue, no Guzzle.
1110
1211
## Features
1312

1413
- Attachable per-route (manual), not global.
15-
- Non-blocking delivery: log is spooled locally, then sent by a background
16-
`analysis:send` command (triggered by the filter and/or cron).
17-
- Local file-based queue with max-size enforcement (oldest dropped) and
18-
exponential retry; items are moved to `failed/` after `max_retries`.
19-
- Multi-tenant identification via subdomain (`HTTP_HOST`).
14+
- Writes directly to `writable/logs/analysis.log` as JSON Lines (JSONL).
15+
- Daily rotation: a file from a previous day is renamed and gzip-compressed
16+
automatically on the next write.
17+
- Retention pruning: compressed logs older than `REQUEST_LOG_RETENTION_DAYS`
18+
(default 30) are deleted.
2019
- Configurable sensitive-field redaction (default: `password`, `nik`, `Api-Key`, `no_telp`).
2120
- Raw body truncation at 3 MB (configurable) with `... [truncated]` suffix.
2221
- File upload metadata captured without binary content (name, size, MIME,
2322
extension, SHA-256 hash, double-extension detection).
2423
- IP/CIDR whitelist to skip private/internal traffic.
25-
- API-key authentication on the Analysis Server.
24+
- Multi-tenant identification via subdomain (`HTTP_HOST`).
2625
- PHP 7.4 and PHP 8.4+.
2726

2827
## Requirements
2928

30-
- PHP 7.4+ / 8.4+ (with `exec` enabled)
29+
- PHP 7.4+ / 8.4+
3130
- CodeIgniter 4 (>= 4.3)
32-
- Guzzle 7
33-
- A writable local queue directory
31+
- A writable `writable/logs` directory
3432

3533
## Installation
3634

@@ -44,34 +42,28 @@ Set the following environment variables in your CI4 application `.env`
4442
(or system environment):
4543

4644
```ini
47-
ANALYSIS_ENABLED = true
48-
ANALYSIS_SERVER_URL = "http://host.docker.internal:8081/analyze"
49-
ANALYSIS_API_KEY = "your-shared-api-key"
50-
ANALYSIS_TIMEOUT = 2
51-
ANALYSIS_MAX_QUEUE = 10000
52-
ANALYSIS_QUEUE_DIR = "/tmp/analysis_queue"
53-
ANALYSIS_REDACT_FIELDS = "password,nik,Api-Key,no_telp"
54-
ANALYSIS_MAX_BODY_SIZE = 3145728
55-
ANALYSIS_WHITELIST_IPS = "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.1"
56-
ANALYSIS_TRUNCATE_SUFFIX = "... [truncated]"
57-
ANALYSIS_MAX_RETRIES = 5
45+
REQUEST_LOG_ENABLED = true
46+
REQUEST_LOG_DIR = "writable/logs"
47+
REQUEST_LOG_FILE = "analysis.log"
48+
REQUEST_LOG_REDACT_FIELDS = "password,nik,Api-Key,no_telp"
49+
REQUEST_LOG_MAX_BODY_SIZE = 3145728
50+
REQUEST_LOG_WHITELIST_IPS = "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.1"
51+
REQUEST_LOG_TRUNCATE_SUFFIX = "... [truncated]"
52+
REQUEST_LOG_RETENTION_DAYS = 30
5853
```
5954

6055
### Config reference
6156

6257
| Variable | Default | Description |
6358
|---|---|---|
64-
| `ANALYSIS_ENABLED` | `false` | Master switch for the filter. |
65-
| `ANALYSIS_SERVER_URL` | `''` | Analysis Server endpoint (POST `/analyze`). |
66-
| `ANALYSIS_API_KEY` | `''` | Shared static API key sent in the `X-API-Key` header. |
67-
| `ANALYSIS_TIMEOUT` | `2` | HTTP timeout (seconds) when sending a log entry. |
68-
| `ANALYSIS_MAX_QUEUE` | `10000` | Max queued items; oldest are dropped when exceeded. |
69-
| `ANALYSIS_QUEUE_DIR` | `sys_get_temp_dir()/analysis_queue` | Directory for the local spool. |
70-
| `ANALYSIS_REDACT_FIELDS` | `password,nik,Api-Key,no_telp` | Comma-separated sensitive fields (case-insensitive). |
71-
| `ANALYSIS_MAX_BODY_SIZE` | `3145728` (3 MB) | `raw_body` truncation length (bytes). |
72-
| `ANALYSIS_WHITELIST_IPS` | RFC1918 + localhost | CIDR ranges to skip. |
73-
| `ANALYSIS_TRUNCATE_SUFFIX` | `... [truncated]` | Appended when the body is truncated. |
74-
| `ANALYSIS_MAX_RETRIES` | `5` | Send attempts before moving to `failed/`. |
59+
| `REQUEST_LOG_ENABLED` | `false` | Master switch for the filter. |
60+
| `REQUEST_LOG_DIR` | `''` (→ `writable/logs`) | Directory for the log file. |
61+
| `REQUEST_LOG_FILE` | `analysis.log` | Log file name (single file, rotated daily). |
62+
| `REQUEST_LOG_REDACT_FIELDS` | `password,nik,Api-Key,no_telp` | Comma-separated sensitive fields (case-insensitive). |
63+
| `REQUEST_LOG_MAX_BODY_SIZE` | `3145728` (3 MB) | `raw_body` truncation length (bytes). |
64+
| `REQUEST_LOG_WHITELIST_IPS` | RFC1918 + localhost | CIDR ranges to skip. |
65+
| `REQUEST_LOG_TRUNCATE_SUFFIX` | `... [truncated]` | Appended when the body is truncated. |
66+
| `REQUEST_LOG_RETENTION_DAYS` | `30` | Days of compressed logs kept before pruning. |
7567

7668
## Usage
7769

@@ -82,112 +74,127 @@ In `app/Config/Filters.php`:
8274
```php
8375
public $aliases = [
8476
// ...
85-
'analysis' => \MrNaeem\Ci4RequestAnalysis\Filters\AnalysisFilter::class,
77+
'requestlog' => \MrNaeem\Ci4RequestAnalysis\Filters\RequestLogFilter::class,
8678
];
8779
```
8880

8981
### 2. Attach to specific routes
9082

9183
```php
92-
$routes->group('api', ['filter' => 'analysis'], function ($routes) {
84+
$routes->group('api', ['filter' => 'requestlog'], function ($routes) {
9385
$routes->post('profile/update', 'Profile::update');
9486
$routes->post('search', 'Search::index');
9587
$routes->post('upload/avatar', 'Upload::avatar');
9688
});
9789
```
9890

99-
Only these routes will be analyzed — nothing is logged globally.
91+
Only these routes are logged — nothing is logged globally.
10092

101-
### 3. Ensure the queue directory is writable
93+
### 3. Verify writable logs directory
10294

103-
The directory must be writable by the web server. It is created automatically,
104-
but on a persistent/containerized deployment mount it to a volume:
95+
The web server must be able to write to `writable/logs`. The directory is
96+
created automatically if missing:
10597

10698
```bash
107-
mkdir -p /tmp/analysis_queue && chown -R www-data:www-data /tmp/analysis_queue
99+
mkdir -p writable/logs && chown -R www-data:www-data writable/logs
108100
```
109101

110-
### 4. Run the sender
102+
### 4. (Optional) Run rotation via cron
111103

112-
The filter spawns `analysis:send` in the background on each analyzed request.
113-
For retries of failed items, schedule it via cron (every minute):
104+
Rotation happens automatically on write. For a guaranteed nightly pass and
105+
retention cleanup, schedule `requestlog:rotate` daily:
114106

115107
```cron
116-
* * * * * cd /path/to/app && php spark analysis:send >/dev/null 2>&1
108+
0 0 * * * cd /path/to/app && php spark requestlog:rotate >/dev/null 2>&1
117109
```
118110

119111
Or run it manually:
120112

121113
```bash
122-
php spark analysis:send
114+
php spark requestlog:rotate
123115
```
124116

125117
## Log payload
126118

127-
Each entry sent to the Analysis Server is a JSON object:
119+
Each line in `analysis.log` is a JSON object (envelope + `log_data`):
128120

129121
```json
130122
{
131-
"timestamp": "2026-08-31T02:15:04+00:00",
132-
"domain": "app.example.com",
133-
"path": "/api/profile/update",
134-
"method": "POST",
135-
"srcip": "203.0.113.10",
136-
"user_agent": "Mozilla/5.0 ...",
137-
"query_string": "page=1",
138-
"headers": { "Content-Type": "application/json", ... },
139-
"raw_body": "{\"name\":\"User\",\"email\":\"user@example.com\",\"password\":\"***REDACTED***\"}",
140-
"file_count": 1,
141-
"file_names": ["shell.php.jpg"],
142-
"file_metadata": [
143-
{
144-
"original_name": "shell.php.jpg",
145-
"size": 20480,
146-
"mime_type": "image/jpeg",
147-
"extension": "jpg",
148-
"hash": "3c98...",
149-
"has_double_extension": true
150-
}
151-
]
123+
"log_data": {
124+
"timestamp": "2026-08-31T02:15:04+00:00",
125+
"domain": "app.example.com",
126+
"path": "/api/profile/update",
127+
"method": "POST",
128+
"srcip": "203.0.113.10",
129+
"user_agent": "Mozilla/5.0 ...",
130+
"query_string": "page=1",
131+
"headers": { "Content-Type": "application/json", ... },
132+
"raw_body": "{\"name\":\"User\",\"email\":\"user@example.com\",\"password\":\"***REDACTED***\"}",
133+
"file_count": 1,
134+
"file_names": ["shell.php.jpg"],
135+
"file_metadata": [
136+
{
137+
"original_name": "shell.php.jpg",
138+
"size": 20480,
139+
"mime_type": "image/jpeg",
140+
"extension": "jpg",
141+
"hash": "3c98...",
142+
"has_double_extension": true
143+
}
144+
]
145+
},
146+
"retry_count": 0,
147+
"last_attempt": null,
148+
"created_at": "2026-08-31T02:15:04+00:00"
152149
}
153150
```
154151

152+
## Log rotation
153+
154+
- Active file: `writable/logs/analysis.log`
155+
- Rotated file: `writable/logs/analysis-YYYY-MM-DD.log.gz`
156+
157+
When a write detects the active file belongs to a previous day, it is renamed
158+
to `analysis-YYYY-MM-DD.log`, compressed with gzip, and the original removed.
159+
Compressed files older than `REQUEST_LOG_RETENTION_DAYS` are pruned.
160+
155161
## How it works
156162

157163
```
158-
Client (CI4)
159-
Filter (before hook)
160-
├─ enabled? ──no──► done
161-
├─ IP whitelisted? ──yes──► done
162-
├─ collect: headers, body, files, tenant, srcip...
163-
│ ├─ redact sensitive fields
164-
│ ├─ truncate body at max size
165-
│ └─ extract file metadata (no binary)
166-
├─ queue: write JSON to ANALYSIS_QUEUE_DIR
167-
└─ triggerSend: background `php spark analysis:send`
168-
169-
AnalysisSend (spark command)
170-
└─ for each queue file:
171-
├─ send POST to ANALYSIS_SERVER_URL (X-API-Key header)
172-
├─ success → delete file
173-
└─ failure → retry_count++, move to failed/ after max_retries
164+
Request → RequestLogFilter (before hook)
165+
├─ enabled? ──no──► done
166+
├─ IP whitelisted? ──yes──► done
167+
├─ collect: headers, body, files, tenant, srcip...
168+
│ ├─ redact sensitive fields
169+
│ ├─ truncate body at max size
170+
│ └─ extract file metadata (no binary)
171+
├─ rotate if the active log is from a previous day (rename + gzip + prune)
172+
└─ append one JSONL line to writable/logs/analysis.log
174173
```
175174

175+
The write is a single append with `LOCK_EX`, so it does not block the request
176+
and is safe for concurrent PHP-FPM workers.
177+
176178
## Security notes
177179

178-
- Redaction happens client-side before the body is sent; binary file content is
179-
never transmitted.
180+
- Redaction happens before the body is written; binary file content is never
181+
stored.
180182
- Do not log `Authorization`/`Cookie` values unless required — redact them by
181-
adding their field names to `ANALYSIS_REDACT_FIELDS`.
182-
- Keep `ANALYSIS_API_KEY` out of version control (use `.env`).
183-
- The `exec` PHP function must be enabled for background sending; otherwise use
184-
cron only.
183+
adding their field names to `REQUEST_LOG_REDACT_FIELDS`.
184+
- `writable/logs` should not be publicly accessible (CI4 already blocks it by
185+
default); logs are git-ignored via `/writable/`.
185186

186187
## Multi-tenant
187188

188189
The tenant is derived from the subdomain via `HTTP_HOST` and stored in the
189190
`domain` field of the payload. No extra configuration is required.
190191

192+
## Wazuh integration
193+
194+
The library ships with a ready-to-use Wazuh setup: a JSON decoder for the
195+
`analysis.log` envelope, detection rules, and agent config that points to
196+
`writable/logs/analysis.log`. See [docs/wazuh.md](docs/wazuh.md).
197+
191198
## Testing
192199

193200
```bash

composer.json

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,18 @@
11
{
22
"name": "mrnaeem4/ci4-request-analysis",
3-
"description": "CodeIgniter 4 library that intercepts incoming HTTP requests and asynchronously sends structured JSON request logs to an Analysis Server.",
3+
"description": "CodeIgniter 4 library that intercepts incoming HTTP requests and writes structured JSON request logs (JSONL) to the CI4 logs directory with daily rotation and gzip compression.",
44
"type": "library",
55
"license": "MIT",
66
"keywords": [
77
"codeigniter4",
88
"request-log",
9-
"analysis",
9+
"jsonl",
1010
"filter",
11-
"async",
12-
"guzzle"
11+
"log-rotation"
1312
],
1413
"require": {
1514
"php": "^7.4 || ^8.0",
16-
"codeigniter4/framework": "^4.3",
17-
"guzzlehttp/guzzle": "^7.0"
15+
"codeigniter4/framework": "^4.3"
1816
},
1917
"require-dev": {
2018
"phpunit/phpunit": "^9.5"

0 commit comments

Comments
 (0)