docs: report CRS conformance, and warn before blocking mode

.github/workflows/ci.yml

@@ -80,4 +80,11 @@ jobs:
           restore-keys: ${{ runner.os }}-cargo-test-
       - name: Install protoc
         run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
+      # The CRS integration tests fail loudly without this checkout rather than
+      # skipping, so that they cannot silently stop running again.
+      - name: Fetch OWASP CRS
+        run: |
+          mkdir -p test-rules
+          git clone --depth 1 https://github.com/coreruleset/coreruleset.git test-rules/crs
+          cp test-rules/crs/crs-setup.conf.example test-rules/crs/crs-setup.conf
       - run: cargo test --workspace

README.md

@@ -1,12 +1,17 @@
 # zentinel-agent-zentinelsec
 
-A pure Rust ModSecurity-compatible WAF agent for [Zentinel](https://github.com/zentinelproxy/zentinel) reverse proxy. Provides full OWASP Core Rule Set (CRS) support with **zero C dependencies** - no libmodsecurity required.
+A pure Rust ModSecurity-compatible WAF agent for [Zentinel](https://github.com/zentinelproxy/zentinel) reverse proxy. Loads the OWASP Core Rule Set with **zero C dependencies** - no libmodsecurity required.
 
-> **Note:** CRS compatibility depends on the [zentinel-modsec](https://github.com/zentinelproxy/zentinel-modsec) engine, a pure Rust reimplementation of libmodsecurity. If you encounter unsupported SecLang features, please [file an issue](https://github.com/zentinelproxy/zentinel-agent-zentinelsec/issues).
+> **CRS compatibility is measured, not assumed.** Detection comes from the
+> [zentinel-modsec](https://github.com/zentinelproxy/zentinel-modsec) engine, a
+> pure Rust reimplementation of libmodsecurity, which is run against the OWASP
+> CRS regression suite on every push. See
+> [CRS conformance](#crs-conformance) for the current number and the known gaps
+> before deploying this in blocking mode.
 
 ## Features
 
-- **Full OWASP CRS Compatibility**: Parse and execute 800+ CRS rules
+- **Loads the stock OWASP CRS**: all 703 rules of CRS 4.30 parse and execute; see [CRS conformance](#crs-conformance)
 - **Pure Rust Implementation**: No libmodsecurity or C dependencies
 - **Built-in SQLi/XSS Detection**: Native `@detectSQLi` and `@detectXSS` operators
 - **SecLang Support**: Load standard ModSecurity rule files
@@ -20,15 +25,50 @@ A pure Rust ModSecurity-compatible WAF agent for [Zentinel](https://github.com/z
 
 | Feature | ZentinelSec | ModSec | WAF |
 |---------|-------------|--------|-----|
-| Detection Rules | 800+ CRS rules | 800+ CRS rules | 285 rules |
+| Detection Rules | stock CRS, [conformance measured](#crs-conformance) | stock CRS (reference implementation) | 285 rules |
 | SecLang Support | Yes | Yes | No |
 | Custom Rules | Yes | Yes | No |
 | @detectSQLi/@detectXSS | Yes (pure Rust) | Yes (C lib) | No |
 | Dependencies | **Pure Rust** | libmodsecurity (C) | Pure Rust |
 | Binary Size | ~10MB | ~50MB | ~5MB |
 | Installation | `cargo install` | Requires libmodsecurity | `cargo install` |
 
-**ZentinelSec combines the best of both worlds**: Full CRS compatibility like ModSec, with zero-dependency installation like WAF.
+**ZentinelSec aims to combine the two**: the stock CRS rule set like ModSec, with
+zero-dependency installation like WAF. Unlike ModSec it is not the reference
+implementation of SecLang, so its CRS behaviour is
+[measured against the upstream regression suite](#crs-conformance) rather than
+taken as given.
+
+## CRS conformance
+
+Detection is provided by [zentinel-modsec](https://github.com/zentinelproxy/zentinel-modsec),
+which runs the OWASP CRS regression suite — roughly 5,000 request/expectation
+pairs naming the rule IDs that must or must not fire — on every push.
+
+| | |
+|---|---|
+| Corpus | CRS `main` (4.30.0-dev), 5,033 runnable cases |
+| Passing | **4,080 (81.1%)** |
+
+That corpus runs in `DetectionOnly` at paranoia level 4, which is how CRS
+documents it. It measures whether individual rules **match**; it does not
+measure whether the WAF **decides** correctly, because in that mode nothing
+blocks and the anomaly score never reaches rule 949110. Both properties are
+tested, separately, in that repository.
+
+### Before deploying in blocking mode
+
+`--block-mode` defaults to `true`. Check the open items below first — they
+affect what a stock CRS deployment does to real traffic:
+
+| Issue | Effect |
+|---|---|
+| [zentinel-modsec#29](https://github.com/zentinelproxy/zentinel-modsec/issues/29) | Stock CRS denies **every** request, `GET /` included, and anomaly scoring never accumulates |
+| [zentinel-modsec#34](https://github.com/zentinelproxy/zentinel-modsec/issues/34) | `+` is not decoded as a space in form-encoded arguments, so rules matching payloads containing whitespace can be evaded |
+| [zentinel-modsec#31](https://github.com/zentinelproxy/zentinel-modsec/issues/31) | `t:cmdLine` is incomplete, so Windows command-line rules (932xxx) do not match |
+
+Until #29 is fixed and released, run with `--block-mode false` and treat the
+output as detection only.
 
 ## Installation
 

src/lib.rs

@@ -142,13 +142,25 @@ pub struct ZentinelSecEngine {
 impl ZentinelSecEngine {
     /// Create a new ZentinelSec engine with the given configuration
     pub fn new(config: ZentinelSecConfig) -> Result<Self> {
-        // Build rules string from all rule files
+        // Hand the engine `Include` directives rather than the text of each
+        // rule file.
+        //
+        // Splicing file contents together loses the one thing the engine needs
+        // to resolve `@pmFromFile`/`@ipMatchFromFile` data paths and nested
+        // `Include`s: which file a directive came from. Those paths are
+        // relative to their own rule file, so a concatenated string sent them
+        // to the process's working directory instead. Stock CRS references
+        // `scanners-user-agents.data` and many others, which meant the
+        // documented deployment only started if it happened to be run from
+        // /etc/modsecurity/crs/rules.
         let mut rules_content = String::new();
 
         // Always enable the rule engine
         rules_content.push_str("SecRuleEngine On\n");
 
-        // Load rules from configured paths
+        // Resolve the configured patterns ourselves so a path that cannot be
+        // read is reported against the pattern that produced it, and so the
+        // file count is available for logging.
         let mut loaded_count = 0;
         for path_pattern in &config.rules_paths {
             // Handle glob patterns
@@ -159,13 +171,16 @@ impl ZentinelSecEngine {
                 match entry {
                     Ok(path) => {
                         if path.is_file() {
-                            let content = fs::read_to_string(&path).map_err(|e| {
-                                anyhow::anyhow!("Failed to read rule file {:?}: {}", path, e)
+                            // An absolute path keeps the include independent of
+                            // the working directory; quoting keeps a path with
+                            // spaces in one argument.
+                            let absolute = fs::canonicalize(&path).map_err(|e| {
+                                anyhow::anyhow!("Failed to resolve rule file {:?}: {}", path, e)
                             })?;
-                            rules_content.push_str(&content);
-                            rules_content.push('\n');
+                            rules_content
+                                .push_str(&format!("Include \"{}\"\n", absolute.display()));
                             loaded_count += 1;
-                            debug!(path = ?path, "Loaded rule file");
+                            debug!(path = ?absolute, "Including rule file");
                         }
                     }
                     Err(e) => {
@@ -176,7 +191,7 @@ impl ZentinelSecEngine {
         }
 
         // Create the ModSecurity engine
-        let modsec = if rules_content.trim().is_empty() || loaded_count == 0 {
+        let modsec = if loaded_count == 0 {
             // No rules loaded, create with just SecRuleEngine On
             ModSecurity::from_string("SecRuleEngine On")
                 .map_err(|e| anyhow::anyhow!("Failed to initialize ZentinelSec engine: {}", e))?
@@ -194,6 +209,20 @@ impl ZentinelSecEngine {
         Ok(Self { modsec, config })
     }
 
+    /// Number of rules the engine loaded.
+    pub fn rule_count(&self) -> usize {
+        self.modsec.rule_count()
+    }
+
+    /// The underlying ModSecurity engine.
+    ///
+    /// Exposed so tests can drive transactions through the rules this agent
+    /// actually loaded, rather than through a separately built engine that
+    /// might load them differently.
+    pub fn modsec(&self) -> &ModSecurity {
+        &self.modsec
+    }
+
     /// Check if path should be excluded
     pub fn is_excluded(&self, path: &str) -> bool {
         self.config
@@ -313,12 +342,17 @@ impl ZentinelSecAgent {
                     status = status,
                     "ZentinelSec intervention (headers)"
                 );
-                let rule_ids = tx.matched_rules().iter().map(|s| s.to_string()).collect();
-                return Ok(Some((
-                    status,
-                    "Blocked by ZentinelSec".to_string(),
-                    rule_ids,
-                )));
+                // The rules that caused the intervention, not every rule that
+                // matched: `matched_rules()` also lists CRS setup actions and
+                // chain starters whose chain never completed, so its first
+                // entry under stock CRS is a setup rule such as 900990 rather
+                // than the rule that blocked.
+                let rule_ids = intervention.rule_ids.clone();
+                let message = intervention
+                    .log
+                    .clone()
+                    .unwrap_or_else(|| "Blocked by ZentinelSec".to_string());
+                return Ok(Some((status, message, rule_ids)));
             }
         }
 
@@ -339,12 +373,12 @@ impl ZentinelSecAgent {
                             status = status,
                             "ZentinelSec intervention (body)"
                         );
-                        let rule_ids = tx.matched_rules().iter().map(|s| s.to_string()).collect();
-                        return Ok(Some((
-                            status,
-                            "Blocked by ZentinelSec".to_string(),
-                            rule_ids,
-                        )));
+                        let rule_ids = intervention.rule_ids.clone();
+                        let message = intervention
+                            .log
+                            .clone()
+                            .unwrap_or_else(|| "Blocked by ZentinelSec".to_string());
+                        return Ok(Some((status, message, rule_ids)));
                     }
                 }
             }