Merge pull request #23 from thinkgrid-labs/feat/reportable-prefilter

Feat/reportable prefilter

Author: akosidencio

.gitignore

@@ -63,3 +63,5 @@ npm-debug.log*
 yarn-debug.log*
 yarn-error.log*
 lerna-debug.log*
+
+PLAN.md

.pre-commit-hooks.yaml

@@ -3,7 +3,7 @@
 #
 #   repos:
 #     - repo: https://github.com/thinkgrid-labs/diffmind
-#       rev: v0.8.0
+#       rev: v0.9.0
 #       hooks:
 #         - id: diffmind
 #

CHANGELOG.md

@@ -0,0 +1,69 @@
+# Changelog
+
+## 0.9.0 — the reviewer's cockpit
+
+diffmind was built as a gate: run it, get a verdict, pass or fail. This release
+adds the other half — a place to *sit* while deciding what to say about someone
+else's branch. Both surfaces share one engine; neither replaces the other.
+
+### Breaking
+
+- **`--format json`: `stats.chunks` → `stats.units`**, plus `units_cached` and
+  `units_unparseable`. The unit of review is now a region of a file rather than
+  an arbitrary slice of lines, and the field names say so. Exit codes, SARIF and
+  Markdown output are unchanged, so CI gates are unaffected.
+- **Daemon protocol changed.** A running 0.8 daemon is ignored rather than
+  misused — the version gate already handled this — but you will want
+  `diffmind serve --stop` after upgrading.
+- **`chunk_diff` removed** from the `core-engine` public API, superseded by
+  `build_units`.
+
+### Added
+
+- **Reviewer's cockpit** (`diffmind --tui`). Analyses on launch. Each finding
+  shows the actual hunk the model reviewed and the context it was given.
+  `a` accepts (and copies a review comment via OSC 52, which works over SSH),
+  `d` dismisses, `w` marks wrong. Verdicts are written through immediately.
+- **Review standards as markdown** — `.diffmind/rules/*.md`, scoped by path
+  glob, committed to the repo. `diffmind rules init` / `rules list`. A finding
+  the model attributes to a rule set gets the ID `rulebook.<id>` and suppresses
+  like any other; an attribution naming a rule set that does not govern that
+  file is discarded.
+- **Reportable pre-filter.** Lockfiles, `linguist-generated` paths, `@generated`
+  banners, minified bundles, assets, snapshots and formatting-only hunks are
+  dropped before the model sees them — and the run says what it skipped:
+  `312 hunks → 74 reviewable (238 filtered: lockfiles, generated, formatting)`.
+  Whitespace inside a string literal counts as content; indentation is never
+  dismissed in Python or YAML.
+- **`diffmind stats`** — findings, cost and the accept-to-wrong ratio over
+  recorded runs. Every review is filed to `.diffmind/runs/<sha>/`.
+- **Cost reporting** — wall-clock and token counts in the footer, in JSON, and
+  in the run record. Estimated counts are marked `~` rather than passed off as
+  exact.
+- **Revision ranges** — `diffmind v1.2.0..HEAD`, or `--range`. Paths after a
+  range narrow it.
+- **`review.ignore`** globs in `.diffmind/config.toml`.
+- diffmind writes its own `.diffmind/.gitignore`, keeping generated state out of
+  git while leaving `rules/`, `rules.toml`, `config.toml` and `baseline.json`
+  committable. Your repository's `.gitignore` is not touched.
+
+### Fixed
+
+- **The result cache never worked across files.** Context was assembled once
+  from the whole diff and folded into every chunk's cache key, so editing one
+  file invalidated every other file's cached result — a re-review after a
+  force-push re-inferred the entire diff. Context is now assembled per unit.
+- **Each chunk was given the wrong context.** The six enclosing functions were
+  taken from whichever files sorted first, so most chunks paid for context about
+  files they did not contain.
+- **Editing one region of a file re-reviewed the whole file.** Units are grouped
+  by region, so a change in one function only invalidates that function.
+- **Glob matching could not express a directory prefix.** `src/api/**` matched
+  nothing, and `**/gen/**` was a substring test that also matched
+  `src/gen-legacy/`. Affects `rules.toml` `files` patterns as well.
+- **Conflicting input selectors were silently resolved by order.** Passing both
+  `--staged` and `--last` reviewed whichever the code checked first; it now
+  errors.
+- A diff dominated by a lockfile is no longer refused for size before the
+  lockfile is filtered out.
+- The TUI now honours the pre-filter and files a run record, which it did not.

README.md

@@ -27,16 +27,19 @@ Your source code never leaves your environment. Works offline. Ships as a **sing
 
 ## Features
 
+- **Deterministic pre-filter** — lockfiles, generated files, minified bundles, assets and formatting-only hunks are dropped before the model sees them, and the run tells you exactly what it skipped: `312 hunks → 74 reviewable (238 filtered: lockfiles, generated, formatting)`.
 - **Deterministic detectors** — commented-out code, removed-but-still-used declarations, and your own regex rules. These run before the model, cost nothing, and are the findings you can trust unconditionally.
 - **Security, bug, performance and maintainability review** by a local model
+- **Review standards as markdown** — commit your team's own rules to `.diffmind/rules/`, scoped by path. Prose the model reads, versioned next to the code it governs.
 - **Ticket-aware review** — check the diff actually implements the acceptance criteria (`--ticket`)
 - **Suppressions** — inline `// diffmind-ignore` comments and a project baseline, so one false positive doesn't get the whole gate deleted
 - **SARIF output** — inline PR annotations via GitHub Code Scanning, no bot account or token
 - **Pluggable backends** — the bundled GGUF, or your own Ollama / vLLM / LM Studio endpoint
 - **Daemon mode** — keep the model resident so reviews are near-instant
 - **Local RAG** — feeds the model the *enclosing function* of each hunk, not just the diff
 - **Reproducible** — greedy decoding with a fixed seed: the same diff always reviews the same way
-- **Interactive TUI** (`--tui`), JSON / Markdown output, and a proper CI gate
+- **Reviewer's cockpit** (`--tui`) — analyses on launch, shows the hunk and context behind each finding, and records accept / dismiss / wrong so the signal-to-noise ratio is measured rather than guessed
+- JSON / Markdown output, and a proper CI gate
 
 ---
 
@@ -53,7 +56,7 @@ Auto-detects your OS and CPU architecture, verifies the SHA-256 checksum, and in
 Pin a specific version — note that the variable goes on the **`bash`** side of the pipe, not the `curl` side:
 
 ```bash
-curl -fsSL https://github.com/thinkgrid-labs/diffmind/releases/latest/download/install.sh | VERSION=v0.8.0 bash
+curl -fsSL https://github.com/thinkgrid-labs/diffmind/releases/latest/download/install.sh | VERSION=v0.9.0 bash
 ```
 
 ### Windows
@@ -89,11 +92,14 @@ diffmind                   # review this branch against the repo's default branc
 
 `--branch` is no longer assumed to be `main`: diffmind reads the repository's default branch from `origin/HEAD` and falls back to whichever of `main`/`master`/`develop`/`trunk` actually exists.
 
+A bare `a..b` argument is recognised as a revision range without needing `--range`. The detection is strict — both endpoints must resolve as revisions and the string must not name an existing path — so `diffmind ../lib` and a file genuinely called `a..b` are still treated as paths. Paths after a range narrow it: `diffmind v1.2.0..HEAD src/api/`.
+
 ```bash
 diffmind --last            # just the last commit
 diffmind --staged          # just what's staged
+diffmind v1.2.0..HEAD      # an explicit revision range
 diffmind src/auth/         # just these paths
-diffmind --tui             # interactive browser
+diffmind --tui             # the reviewer's cockpit
 ```
 
 ---
@@ -150,6 +156,7 @@ Findings carry a stable rule ID, shown in the output, so you can silence exactly
 | `DM002`          | A declaration was removed but is still referenced   |
 | `DM900.<category>` | A model-authored finding of that category        |
 | `custom.<slug>`  | One of your `.diffmind/rules.toml` rules           |
+| `rulebook.<id>`  | A violation of one of your `.diffmind/rules/*.md`  |
 
 ### Inline
 
@@ -183,7 +190,7 @@ Commit `.diffmind/baseline.json`. Future runs report only new issues. The baseli
 ### GitHub Action
 
 ```yaml
-- uses: thinkgrid-labs/diffmind@v0.8.0
+- uses: thinkgrid-labs/diffmind@v0.9.0
   with:
     model: 1.5b
     fail-on: high
@@ -192,7 +199,7 @@ Commit `.diffmind/baseline.json`. Future runs report only new issues. The baseli
 That caches the model, installs the binary, reviews the PR diff, and uploads SARIF so findings appear inline on the diff. For a PR comment instead:
 
 ```yaml
-- uses: thinkgrid-labs/diffmind@v0.8.0
+- uses: thinkgrid-labs/diffmind@v0.9.0
   with:
     format: markdown
     comment: true
@@ -231,13 +238,83 @@ Or via [pre-commit](https://pre-commit.com):
 ```yaml
 repos:
   - repo: https://github.com/thinkgrid-labs/diffmind
-    rev: v0.8.0
+    rev: v0.9.0
     hooks:
       - id: diffmind
 ```
 
 ---
 
+## The cockpit — `diffmind --tui`
+
+```bash
+diffmind --tui
+```
+
+Analysis starts on launch. Each finding shows the **actual hunk the model
+reviewed** and the **context it was given** — a finding you cannot check is one
+you will eventually stop reading.
+
+| Key | Action |
+| --------- | ------------------------------------------------------------ |
+| `j` / `k` | Move through findings |
+| `a`       | Accept — records the verdict and copies a review comment |
+| `d`       | Dismiss — read, not worth raising |
+| `w`       | Wrong — the finding was incorrect |
+| `PgUp` / `PgDn` | Scroll the detail pane |
+| `r`       | Re-run |
+| `q`       | Quit |
+
+Verdicts are written through immediately, so closing the terminal mid-triage
+loses nothing. They feed `diffmind stats`.
+
+Accept copies via **OSC 52**, the terminal's own clipboard escape — no
+dependency, and it works over SSH. tmux and screen need clipboard passthrough
+enabled; if the copy fails you are told, so you never believe you have copied
+something you have not.
+
+Because the output is private to you, the tool can afford to be wrong
+occasionally: a bad finding costs one keystroke, not an author's afternoon.
+
+---
+
+## Run history
+
+Every review is filed to `.diffmind/runs/<sha>/` — the findings as `run.json`
+and `review.md`, plus what the run cost. `diffmind stats` reads them back:
+
+```
+  Runs             34
+  Median findings  3
+  Median time      6.2s
+  Median tokens    18420
+  Cache hits       61%
+
+  Verdicts         71 accepted · 44 dismissed · 12 wrong
+  Accept : wrong   5.9:1  (on target)
+
+  Most often wrong
+      7  DM900.maintainability
+      3  rulebook.house-style
+```
+
+The **accept-to-wrong ratio** is the number that decides whether the tool is
+earning its keep — noise is this category's known failure mode, and a reviewer
+who cannot measure it will just quietly stop running the reviewer. Verdicts come
+from the TUI. Dismissals are excluded from the ratio: choosing not to raise a
+correct observation is not the tool being wrong.
+
+Run snapshots are overwritten when a sha is reviewed again; verdicts are
+append-only and survive `diffmind stats --clear`, because the ratio is only
+meaningful over months.
+
+diffmind writes `.diffmind/.gitignore` covering `runs/`, `cache/`, `models/`,
+`symbols.json` and `daemon.json` — your review notes stay private, while
+`rules/`, `rules.toml`, `config.toml` and `baseline.json` remain committable.
+Your repository's own `.gitignore` is never touched.
+
+---
+
 ## Daemon mode
 
 Every invocation otherwise pays the model-load cost — seconds, every time.
@@ -267,6 +344,8 @@ triage        = "auto"     # two-pass triage on large diffs
 cache         = true
 temperature   = 0.0        # 0 = greedy and reproducible
 max_tokens    = 1024
+# Extra paths to drop, on top of the built-in noise rules.
+ignore        = ["**/legacy/**", "*.generated.ts"]
 
 [backend]
 kind        = "local"      # or "ollama" / "openai-compatible"
@@ -277,7 +356,50 @@ api_key_env = "DIFFMIND_API_KEY"
 
 Unknown keys are reported rather than silently ignored.
 
-### Team rules — `.diffmind/rules.toml`
+### Review standards — `.diffmind/rules/*.md`
+
+Rules that need judgement rather than a pattern. Written as prose, committed to
+the repo, and read by the model on every review — so a team's review culture
+becomes a reviewed artifact instead of tacit knowledge.
+
+```bash
+diffmind rules init     # scaffold .diffmind/rules/default.md
+diffmind rules list     # what would load, and what each governs
+```
+
+```markdown
+---
+scope: ["src/api/**/*.ts"]
+severity: high
+---
+
+# API conventions
+
+- Public handlers must return `ApiError`, never a bare string.
+- Any new endpoint needs a corresponding entry in `openapi.yaml`.
+- Reject changes that widen a response struct without a version bump.
+```
+
+| Key | Description |
+| ---------- | ----------------------------------------------------------------- |
+| `scope`    | Globs this rule set governs. Omit to cover the whole repository.  |
+| `severity` | **Ceiling** for findings attributed to it — never a promotion.     |
+| `id`       | Name used to attribute and suppress. Defaults to the file stem.    |
+
+A finding the model attributes to a rule set gets the rule ID
+`rulebook.<id>`, so it suppresses like any other:
+`// diffmind-ignore-next-line rulebook.api-conventions`. An attribution naming a
+rule set that does not govern that file is discarded — a 1.5B will invent a
+plausible name, and an invented one could never be suppressed.
+
+Rule bodies go in the *stable* half of the prompt and units are grouped by which
+rule sets govern them, so every unit in a group sends a byte-identical prefix.
+That is what keeps prompt-prefix caching possible; scoping rules per file would
+otherwise make every prompt unique.
+
+A rule set that fails to parse is reported and skipped, never silently ignored.
+
+### Pattern rules — `.diffmind/rules.toml`
 
 Regex rules run before the model: instant, deterministic, zero inference cost.
 
@@ -317,7 +439,11 @@ category = "security"
 diffmind describe                 # PR title, summary, and test plan
 diffmind commit                   # conventional commit message for staged changes
 diffmind commit --apply
+diffmind rules init               # scaffold .diffmind/rules/default.md
+diffmind rules list               # show which rule sets load
 diffmind index                    # build the symbol index
+diffmind stats                    # cost and signal over recorded runs
+diffmind stats --clear            # drop run snapshots (verdicts are kept)
 diffmind cache show               # cache location and size
 diffmind cache clear
 ```
@@ -335,6 +461,8 @@ Commands:
   describe       Generate a PR title and description
   commit         Suggest a conventional commit message
   baseline       Record current findings as accepted
+  rules          Manage the prose rule sets in .diffmind/rules/
+  stats          Findings, cost and accept/wrong ratio over recorded runs
   install-hooks  Install git hooks
   serve          Keep the model resident between runs
   cache          Inspect or clear the review cache
@@ -345,6 +473,7 @@ Options:
   -l, --last                     Review the last commit only
       --staged                   Review staged changes only
       --stdin                    Read the diff from stdin
+      --range <RANGE>            Review an explicit revision range, e.g. v1.2.0..HEAD
   -t, --tui                      Launch the interactive TUI
       --ticket <FILE_OR_TEXT>    Acceptance criteria to check against
       --min-severity <LEVEL>     Minimum severity to report [default: low]
@@ -371,13 +500,14 @@ Options:
 ## How It Works
 
 1. **Parse** — the diff is parsed once into typed per-file hunks with real pre/post-image line numbers.
-2. **Deterministic detectors** — commented-out code, removed-but-used declarations, and your regex rules. No model involved.
-3. **Context** — the enclosing function of each hunk (plus definitions of referenced symbols) is pulled from `.diffmind/symbols.json`.
-4. **Triage** — on large diffs, a cheap first pass decides which files carry real risk.
-5. **Chunked inference** — chunks are sized to the backend's *actual* context window, read from the GGUF metadata.
-6. **Constrained decoding** — the sampler consults a JSON state machine before committing each token, so the model cannot emit a preamble, an unbalanced brace, or a truncated string. Output that hits the token cap is repaired rather than discarded.
-7. **Anchoring** — findings pointing at a file not in the diff are dropped; off-by-N line numbers snap to the nearest changed line.
-8. **Suppression** — inline directives, the baseline, and `--min-confidence` are applied, then results are deduplicated and sorted.
+2. **Pre-filter** — lockfiles, `linguist-generated` paths, files carrying a `@generated` banner, minified bundles, assets, snapshots, your `ignore` globs, and hunks that only change whitespace are dropped. Costs nothing, typically removes most of a real branch, and the counts are reported rather than silently applied. Whitespace inside a string literal counts as content, and indentation is never dismissed in Python or YAML.
+3. **Deterministic detectors** — commented-out code, removed-but-used declarations, and your regex rules. No model involved.
+4. **Context** — the enclosing function of each hunk (plus definitions of referenced symbols) is pulled from `.diffmind/symbols.json`, assembled per chunk so one file's edit does not invalidate another's cached result.
+5. **Triage** — on large diffs, a cheap first pass decides which files carry real risk.
+6. **Review units** — hunks are grouped into regions of a file rather than cut wherever a line budget ran out, so related hunks are read together and an edit in one function only re-reviews that function. Units are sized to the backend's *actual* context window, read from the GGUF metadata.
+7. **Constrained decoding** — the sampler consults a JSON state machine before committing each token, so the model cannot emit a preamble, an unbalanced brace, or a truncated string. Output that hits the token cap is repaired rather than discarded.
+8. **Anchoring** — findings pointing at a file not in the diff are dropped; off-by-N line numbers snap to the nearest changed line.
+9. **Suppression** — inline directives, the baseline, and `--min-confidence` are applied, then results are deduplicated and sorted.
 
 ---
 
@@ -394,7 +524,10 @@ diffmind/
 │   ├── detectors.rs            # deterministic rules
 │   ├── diff.rs                 # unified-diff parser, chunking, finding anchoring
 │   ├── json_guard.rs           # constrained-decoding state machine
+│   ├── prefilter.rs            # deterministic noise removal, with counts
 │   ├── prompt.rs               # prompt construction
+│   ├── rulebook.rs             # prose rule sets (.diffmind/rules/*.md)
+│   ├── unit.rs                 # hunks → review units (the cached, reviewed thing)
 │   ├── sarif.rs                # SARIF 2.1.0 output
 │   └── suppression.rs          # inline directives and baselines
 └── apps/tui-cli/src/
@@ -412,7 +545,6 @@ diffmind/
 
 - [ ] Auto-fix patches (`diffmind fix`) — gated on remote backends landing, since a 1.5B's patches are not trustworthy enough to apply
 - [ ] Cross-file impact analysis — find callers of deleted or renamed functions
-- [ ] `diffmind stats` — findings over time
 - [ ] VS Code / JetBrains extensions, talking to the daemon
 - [ ] Homebrew tap and scoop manifest
 - [ ] Fine-tuned review model

apps/tui-cli/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "diffmind"
-version = "0.8.0"
+version = "0.9.0"
 edition = "2024"
 description = "Local-first AI code review agent — powered by on-device inference"