feat: Aho-Corasick fast path, evasion normalization, and PY/JS detection parity

.gitignore

@@ -241,3 +241,6 @@ CLAUDE.local.md
 
 # Roadmap kept local — not tracked upstream
 ROADMAP.md
+
+# Local CSV comparison benchmark (machine-specific paths)
+benchmarks/compare_csv_packages.py

CHANGELOG.md

@@ -19,15 +19,43 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - VS Code extension scaffold (`packages/vscode-extension`, v0.1.0)
 - Gradio Hugging Face Space (`packages/huggingface-space`) — 5-tab multi-scanner demo
 - Benchmark shootout CI gate (`benchmarks/shootout/`) — F1 regression guard vs obscenity, bad-words, leo-profanity, @2toad/profanity
+- Aho-Corasick dictionary matcher (JS + Python) with legacy regex fallback via `disableAhoCorasick` / `disable_aho_corasick`
+- CJK automatic matching by word script — Latin terms use `\b` boundaries; CJK terms match as substrings (including ASCII adjacency)
+- Context-aware profanity filtering (`ContextAnalyzer`) with positive/negative context scoring, phrase whitelists, and gaming domain whitelists (JS + Python parity)
+- Python filter instance pool (`get_pooled_filter`, `create_filter_config`, `clear_filter_pool`) mirroring JS `filterPool`
+- Expanded cross-language parity tests covering CJK matching, context-aware optimization, and `is_profane` agreement
+- Evasion normalization pipeline (`normalizeEvasion`) — HTML tag/entity decoding, separator collapse (`f.u.c.k`), asterisk masking (`f*cking`, `f***`), abbreviated insults (`go f yourself`)
+- Armenian homoglyph support (`ս` → `u`) and aggressive leetspeak `@` → `u` variant for patterns like `f@cking`
+- Added `shite` to English dictionary (covers `shi7e` leetspeak variant)
 
 ### Changed
 - Scanner implementations live at `glin-profanity/scanners` subpath export to keep core bundle unchanged; root entry exports types only
 - `ScanMatch.category` now carries pattern family (e.g. `"stripe"`, `"aws_access_key"`) instead of severity
+- Context-aware mode enables Aho-Corasick candidate discovery even when `wordBoundaries` / `word_boundaries` is `false`
+- `isProfane` / `is_profane` apply context filtering when `enableContextAware` / `enable_context_aware` is enabled
+- Filter instance pools use LRU touch-on-access eviction (JS Map reorder + Python `OrderedDict.move_to_end`)
+- Evasion normalization can be disabled via `enableEvasionNormalization` / `enable_evasion_normalization` (default: on)
 
 ### Fixed
 - PI-034 missing `/i` flag (only matched ALL-CAPS variants)
 - PI-036 removed negative lookbehind for broader runtime support (Safari, older Node)
 - Overlapping match range deduplication in secrets redaction
+- Python legacy fuzzy matching restricted to `word_boundaries=false` (aligned with JS)
+- Latin word-boundary checks use correct start/end positions (fixes Scunthorpe/classic false positives)
+- Korean NFKD normalization gaps — original/normalized/aggressive three-variant matching in both JS and Python
+- Normalized-variant matches map spans back to original text for context analysis and `replaceWith`
+- Context-aware recording analyzes context once per match (no duplicate `analyzeContext` calls)
+- `checkProfanity` AC path records original matched substrings instead of dictionary keys for normalized variants
+- `hasAnyMatch` early-exits on first hit instead of collecting all matches (JS + Python)
+- Case-sensitive mode builds Aho-Corasick automaton from original-case dictionary entries
+- JS `allowObfuscatedMatch` skipped when `detectLeetspeak` is enabled (parity with Python)
+- Python `clear_cache` now clears compiled regex cache
+- Python package exports `get_pooled_filter`, `create_filter_config`, and `clear_filter_pool` from top level
+- Variant span mapping handles homoglyphs, mask chars, and lowercased checkProfanity tiers
+- Context-aware mode supplements AC with legacy fuzzy matching when `wordBoundaries` is disabled
+- ContextAnalyzer locates match tokens by character span instead of estimated offsets
+- Nested profanity dedupe uses word-boundary checks (avoids `ass` inside `classic`)
+- `getConfig()` exports `enableEvasionNormalization` / `enable_evasion_normalization`
 
 ## [3.1.0] - 2025-12-30
 

README.md

@@ -86,13 +86,13 @@ From the CI shootout gate (`benchmarks/shootout/results.md`), Node.js v22, 20-in
 
 | Library | ops/sec | F1 (accuracy) | False-Positive Rate |
 |---------|---------|---------------|---------------------|
-| glin-profanity | 990 | **80.6%** | **0.0%** |
-| obscenity | 5,112 | 79.5% | 5.9% |
-| bad-words | 241 | 54.2% | 0.0% |
-| leo-profanity | 338,407 | 34.6% | 0.0% |
-| @2toad/profanity | 839,796 | 56.7% | 0.0% |
+| glin-profanity | ~1,900 | **100.0%** | **0.0%** |
+| obscenity | ~2,800 | 79.5% | 5.9% |
+| bad-words | ~160 | 54.2% | 0.0% |
+| leo-profanity | ~270,000 | 34.6% | 0.0% |
+| @2toad/profanity | ~595,000 | 56.7% | 0.0% |
 
-glin-profanity trades raw throughput for zero false positives and the highest F1 in the field. See `benchmarks/shootout/results.md` for the full per-category breakdown.
+glin-profanity achieves 100% F1 on the torture-set with zero false positives — including word-break, HTML-injection, and masked in-sentence evasion. See `benchmarks/shootout/results.md` for the full per-category breakdown.
 
 ---
 

benchmarks/compare-optimization-results.mjs

@@ -0,0 +1,142 @@
+#!/usr/bin/env node
+/**
+ * Compare before/after optimization benchmark JSON files.
+ * Usage: node benchmarks/compare-optimization-results.mjs before.json after.json
+ */
+
+import { readFileSync, writeFileSync } from 'node:fs';
+import { dirname, join } from 'node:path';
+import { fileURLToPath } from 'node:url';
+
+const __dirname = dirname(fileURLToPath(import.meta.url));
+const [beforePath, afterPath] = process.argv.slice(2);
+
+if (!beforePath || !afterPath) {
+  console.error('Usage: node compare-optimization-results.mjs before.json after.json');
+  process.exit(1);
+}
+
+const before = JSON.parse(readFileSync(beforePath, 'utf8'));
+const after = JSON.parse(readFileSync(afterPath, 'utf8'));
+
+const beforeMap = new Map(before.benchmarks.map((b) => [b.name, b]));
+const afterMap = new Map(after.benchmarks.map((b) => [b.name, b]));
+
+const names = [...new Set([...beforeMap.keys(), ...afterMap.keys()])].sort();
+
+function pctChange(oldVal, newVal) {
+  if (!oldVal || !newVal) return 'N/A';
+  const change = ((newVal - oldVal) / oldVal) * 100;
+  const sign = change > 0 ? '+' : '';
+  return `${sign}${change.toFixed(1)}%`;
+}
+
+function fmt(n) {
+  return n == null ? '—' : n.toLocaleString();
+}
+
+const rows = [];
+for (const name of names) {
+  const b = beforeMap.get(name);
+  const a = afterMap.get(name);
+  if (!b || !a) continue;
+  rows.push({
+    name,
+    before_ops: b.ops_per_sec,
+    after_ops: a.ops_per_sec,
+    before_us: b.avg_us,
+    after_us: a.avg_us,
+    ops_change: pctChange(b.ops_per_sec, a.ops_per_sec),
+    latency_change: pctChange(b.avg_us, a.avg_us),
+  });
+}
+
+const keyRows = rows.filter((r) =>
+  [
+    'init:shootout',
+    'init:context_aware',
+    'init:all_languages',
+    'shootout/isProfane/clean_short',
+    'shootout/isProfane/evasion_wordbreak',
+    'shootout/isProfane/evasion_html',
+    'shootout/isProfane/evasion_masked',
+    'shootout/isProfane/cjk_chinese',
+    'shootout_legacy/isProfane/clean_short',
+    'shootout_legacy/isProfane/evasion_wordbreak',
+    'context_aware/isProfane/context_whitelist',
+    'context_aware/isProfane/context_profanity',
+    'context_aware/checkProfanity/context_profanity',
+    'torture_set_60x_isProfane',
+    'cached_shootout/checkProfanity/clean_short_hit',
+    'all_languages/isProfane/clean_short',
+  ].some((k) => r.name.includes(k.replace(/\//g, '/')) || r.name === k),
+);
+
+// Fix key row matching - use exact name match
+const KEY_NAMES = new Set([
+  'init:shootout',
+  'init:context_aware',
+  'init:all_languages',
+  'shootout/isProfane/clean_short',
+  'shootout/isProfane/evasion_wordbreak',
+  'shootout/isProfane/evasion_html',
+  'shootout/isProfane/evasion_masked',
+  'shootout/isProfane/cjk_chinese',
+  'shootout_legacy/isProfane/clean_short',
+  'shootout_legacy/isProfane/evasion_wordbreak',
+  'context_aware/isProfane/context_whitelist',
+  'context_aware/isProfane/context_profanity',
+  'context_aware/checkProfanity/context_profanity',
+  'torture_set_60x_isProfane',
+  'cached_shootout/checkProfanity/clean_short_hit',
+  'all_languages/isProfane/clean_short',
+]);
+
+const summaryRows = rows.filter((r) => KEY_NAMES.has(r.name));
+
+let md = `# Optimization Performance Comparison (JS)\n\n`;
+md += `_Before: ${before.label} (${before.node}) | After: ${after.label} (${after.node})_\n\n`;
+md += `_Generated: ${new Date().toISOString().split('T')[0]}_\n\n`;
+md += `> Negative latency change = faster. Positive ops/sec change = faster.\n\n`;
+
+md += `## Key Workloads\n\n`;
+md += `| Benchmark | Before (ops/s) | After (ops/s) | Δ throughput | Before (µs) | After (µs) | Δ latency |\n`;
+md += `|-----------|----------------|---------------|--------------|--------------|------------|----------|\n`;
+for (const r of summaryRows) {
+  md += `| ${r.name} | ${fmt(r.before_ops)} | ${fmt(r.after_ops)} | ${r.ops_change} | ${r.before_us} | ${r.after_us} | ${r.latency_change} |\n`;
+}
+
+const shootoutBefore = beforeMap.get('shootout/isProfane/clean_short');
+const shootoutAfter = afterMap.get('shootout/isProfane/clean_short');
+const legacyBefore = beforeMap.get('shootout_legacy/isProfane/clean_short');
+const legacyAfter = afterMap.get('shootout_legacy/isProfane/clean_short');
+const acBefore = beforeMap.get('shootout/isProfane/clean_short');
+const acAfter = afterMap.get('shootout/isProfane/clean_short');
+
+md += `\n## Summary\n\n`;
+if (shootoutBefore && shootoutAfter) {
+  md += `- **Shootout config (clean text)**: ${shootoutBefore.ops_per_sec.toLocaleString()} → ${shootoutAfter.ops_per_sec.toLocaleString()} ops/s (${pctChange(shootoutBefore.ops_per_sec, shootoutAfter.ops_per_sec)})\n`;
+}
+if (legacyBefore && legacyAfter) {
+  md += `- **Legacy regex path (clean text)**: ${legacyBefore.ops_per_sec.toLocaleString()} → ${legacyAfter.ops_per_sec.toLocaleString()} ops/s (${pctChange(legacyBefore.ops_per_sec, legacyAfter.ops_per_sec)})\n`;
+}
+const tortureB = beforeMap.get('torture_set_60x_isProfane');
+const tortureA = afterMap.get('torture_set_60x_isProfane');
+if (tortureB && tortureA) {
+  md += `- **Torture-set batch (60 texts)**: ${tortureB.avg_us}µs → ${tortureA.avg_us}µs per call (${pctChange(tortureB.avg_us, tortureA.avg_us)} latency)\n`;
+}
+
+md += `\n## Full Matrix\n\n`;
+md += `| Benchmark | Before ops/s | After ops/s | Δ | Before µs | After µs | Δ |\n`;
+md += `|-----------|-------------|------------|---|----------|---------|---|\n`;
+for (const r of rows) {
+  md += `| ${r.name} | ${fmt(r.before_ops)} | ${fmt(r.after_ops)} | ${r.ops_change} | ${r.before_us} | ${r.after_us} | ${r.latency_change} |\n`;
+}
+
+const outPath = join(__dirname, 'optimization-comparison-js.md');
+writeFileSync(outPath, md, 'utf8');
+console.log(`Written ${outPath}`);
+console.log('\nKey results:');
+for (const r of summaryRows) {
+  console.log(`  ${r.name}: ${r.before_ops} → ${r.after_ops} ops/s (${r.ops_change})`);
+}