Skip to content

Commit 3792e47

Browse files
authored
feat(core): parallel scan with negation-aware directory pruning (#169)
Makes a scan of a large source (600k+ files, gitignored `node_modules`, an `include_patterns` entry) both responsive from t=0 and actually parallel. ## What was wrong - **~4 full-tree BFS passes before the walk even started.** `build_source_matcher` called `collect_ignore_files` once for `.gitignore` and again for `.ignore`, each an unpruned `std::fs::read_dir` sweep of the whole tree - and `build_walker` called `build_source_matcher` AGAIN for its prune closure. All of it ran synchronously on the tokio runtime thread inside `scan_with_progress`, before the walk's first yield point, which is why the app went dead ("Loading...") at scan start. - **Any single `!` rule disabled pruning for the entire source.** The prune gate was `SourceMatcher::has_negations()`, so one include pattern (or one `!` line in any gitignore tier) meant the walk descended every `node_modules` in the tree. There was an existing `TODO(perf): prune excluded dirs even with negations`. - **The walk was serial**, with inline stat/hash on one task: ~40% disk utilisation and no core above 85%. ## What changed - **One pruned pass collects both cascades.** `collect_ignore_scopes` does a single BFS that finds `.gitignore` and `.ignore` together and does not descend a directory that is excluded by the cascade built so far AND that no whitelist rule can reach under. Because BFS has already built every scope that can apply to a candidate directory (its ancestors), the prune decision there is exactly the decision the finished matcher would make. - **`SourceMatcher::negations_could_match_under(rel_dir)`** - true iff some `!` rule in some scope could match at or under that directory. Anchored patterns are run as a tiny segment NFA (`**` consumes zero or more segments, `*`/`?`/`[...]` match within one segment); unanchored patterns, unreadable ignore files, and any parse that disagrees with `Gitignore::num_whitelists` all answer `true`. Conservative by construction: a wrong `true` only costs a prune. - **The prune rule is now per directory** - `!is_included(d, true) && !negations_could_match_under(d)` - in `build_walker`, in the streaming exclusion preview (`stream_classify_tree`), and in the one-shot preview (`classify_tree`). - **The matcher is built once.** `build_walker_with_matcher(source, Arc<SourceMatcher>)` shares the scanner's matcher with the walker's `'static + Send + Sync` filter closure; `build_walker(source)` still works for other callers. - **The walk is parallel and off the runtime.** One `spawn_blocking` builds the matcher and drives `ignore::WalkParallel` (`available_parallelism()` clamped to 2..=8 threads). Workers do the per-entry work - include check, NFC `RelativePath`, stat, Windows placeholder + ADS probes, the BLAKE3 re-hash - and stream batched, typed results over a bounded channel. A single async consumer keeps everything order-sensitive: the `seen` set, NFC collision detection, progress ticks, errored-prefix bookkeeping, the latency reservoir. The orphan split also moved to `spawn_blocking` (600k stored rows means 600k matcher queries). - **Scope lookup is indexed by directory.** `is_included` used to scan every scope per file; it now consults only the scopes on the path's own ancestor chain, which matters once a source contains hundreds of nested ignore files. - Progress additionally ticks on the FIRST batch that carries anything, so the readout starts moving immediately instead of after a full 512-file stride. ## Behaviour change worth reviewing Pruned collection means a `!keep.txt` in a `.gitignore` **inside** an excluded, unreachable directory is no longer read - matching git, which never reads a `.gitignore` inside an ignored directory. `vendor/keep.txt` under a gitignored `vendor/` with no negation at or above it is now classified EXCLUDED rather than re-included. This is safe because the matcher and the walk derive from the same collected set, so they stay in lockstep: the scanner's orphan split reads a stored `file_state` row under a pruned directory as an `excluded_orphan` (no trash op), never as a deletion. Both halves are pinned by tests (`nested_negation_under_pruned_dir_is_orphan_not_deleted` and `negation_above_an_excluded_dir_keeps_the_file_out_of_deleted`). A negation at or above the excluded directory still disables the prune and is honoured exactly as before. ## Tests New: a truth table for `negations_could_match_under` (`.env`, `*/.env`, `/x/.env`, `/*/.env`, `/a/**/.env`, `/a/b/`, metacharacters, nested scopes, the no-negation fast path), `parse_negation_line` vectors, the single-segment glob matcher including its conservative malformed-class fallback, the end-to-end `/myrepo/.env` + gitignored `node_modules` case (target file backed up, the whole `node_modules` subtree never visited), the `/*/.env` depth-1 case (`node_modules/.env` backed up, `node_modules/a/.env` not, `node_modules/a/` pruned), plus the two P1-1 lockstep tests above and a third case in the exclusion preview's prune test. `cargo test -p driven-core` 373 + 24 + 11 pass; `cargo test -p driven-app --lib` 264 pass; `cargo clippy -p driven-core --all-targets -- -D warnings` and the same for `driven-app` clean; `cargo fmt --all` applied.
1 parent 51bb1f3 commit 3792e47

4 files changed

Lines changed: 1990 additions & 441 deletions

File tree

0 commit comments

Comments
 (0)