Commit 3792e47
authored
feat(core): parallel scan with negation-aware directory pruning (#169)
Makes a scan of a large source (600k+ files, gitignored `node_modules`,
an
`include_patterns` entry) both responsive from t=0 and actually
parallel.
## What was wrong
- **~4 full-tree BFS passes before the walk even started.**
`build_source_matcher`
called `collect_ignore_files` once for `.gitignore` and again for
`.ignore`,
each an unpruned `std::fs::read_dir` sweep of the whole tree - and
`build_walker`
called `build_source_matcher` AGAIN for its prune closure. All of it ran
synchronously on the tokio runtime thread inside `scan_with_progress`,
before the
walk's first yield point, which is why the app went dead ("Loading...")
at scan start.
- **Any single `!` rule disabled pruning for the entire source.** The
prune gate was
`SourceMatcher::has_negations()`, so one include pattern (or one `!`
line in any
gitignore tier) meant the walk descended every `node_modules` in the
tree. There
was an existing `TODO(perf): prune excluded dirs even with negations`.
- **The walk was serial**, with inline stat/hash on one task: ~40% disk
utilisation
and no core above 85%.
## What changed
- **One pruned pass collects both cascades.** `collect_ignore_scopes`
does a single
BFS that finds `.gitignore` and `.ignore` together and does not descend
a
directory that is excluded by the cascade built so far AND that no
whitelist rule
can reach under. Because BFS has already built every scope that can
apply to a
candidate directory (its ancestors), the prune decision there is exactly
the
decision the finished matcher would make.
- **`SourceMatcher::negations_could_match_under(rel_dir)`** - true iff
some `!` rule
in some scope could match at or under that directory. Anchored patterns
are run as
a tiny segment NFA (`**` consumes zero or more segments, `*`/`?`/`[...]`
match
within one segment); unanchored patterns, unreadable ignore files, and
any parse
that disagrees with `Gitignore::num_whitelists` all answer `true`.
Conservative by
construction: a wrong `true` only costs a prune.
- **The prune rule is now per directory** - `!is_included(d, true) &&
!negations_could_match_under(d)`
- in `build_walker`, in the streaming exclusion preview
(`stream_classify_tree`),
and in the one-shot preview (`classify_tree`).
- **The matcher is built once.** `build_walker_with_matcher(source,
Arc<SourceMatcher>)`
shares the scanner's matcher with the walker's `'static + Send + Sync`
filter
closure; `build_walker(source)` still works for other callers.
- **The walk is parallel and off the runtime.** One `spawn_blocking`
builds the
matcher and drives `ignore::WalkParallel` (`available_parallelism()`
clamped to
2..=8 threads). Workers do the per-entry work - include check, NFC
`RelativePath`,
stat, Windows placeholder + ADS probes, the BLAKE3 re-hash - and stream
batched,
typed results over a bounded channel. A single async consumer keeps
everything
order-sensitive: the `seen` set, NFC collision detection, progress
ticks,
errored-prefix bookkeeping, the latency reservoir. The orphan split also
moved to
`spawn_blocking` (600k stored rows means 600k matcher queries).
- **Scope lookup is indexed by directory.** `is_included` used to scan
every scope
per file; it now consults only the scopes on the path's own ancestor
chain, which
matters once a source contains hundreds of nested ignore files.
- Progress additionally ticks on the FIRST batch that carries anything,
so the
readout starts moving immediately instead of after a full 512-file
stride.
## Behaviour change worth reviewing
Pruned collection means a `!keep.txt` in a `.gitignore` **inside** an
excluded,
unreachable directory is no longer read - matching git, which never
reads a
`.gitignore` inside an ignored directory. `vendor/keep.txt` under a
gitignored
`vendor/` with no negation at or above it is now classified EXCLUDED
rather than
re-included.
This is safe because the matcher and the walk derive from the same
collected set,
so they stay in lockstep: the scanner's orphan split reads a stored
`file_state`
row under a pruned directory as an `excluded_orphan` (no trash op),
never as a
deletion. Both halves are pinned by tests
(`nested_negation_under_pruned_dir_is_orphan_not_deleted` and
`negation_above_an_excluded_dir_keeps_the_file_out_of_deleted`). A
negation at or
above the excluded directory still disables the prune and is honoured
exactly as
before.
## Tests
New: a truth table for `negations_could_match_under` (`.env`, `*/.env`,
`/x/.env`,
`/*/.env`, `/a/**/.env`, `/a/b/`, metacharacters, nested scopes, the
no-negation
fast path), `parse_negation_line` vectors, the single-segment glob
matcher
including its conservative malformed-class fallback, the end-to-end
`/myrepo/.env` + gitignored `node_modules` case (target file backed up,
the whole
`node_modules` subtree never visited), the `/*/.env` depth-1 case
(`node_modules/.env` backed up, `node_modules/a/.env` not,
`node_modules/a/`
pruned), plus the two P1-1 lockstep tests above and a third case in the
exclusion
preview's prune test.
`cargo test -p driven-core` 373 + 24 + 11 pass; `cargo test -p
driven-app --lib`
264 pass; `cargo clippy -p driven-core --all-targets -- -D warnings` and
the same
for `driven-app` clean; `cargo fmt --all` applied.1 parent 51bb1f3 commit 3792e47
4 files changed
Lines changed: 1990 additions & 441 deletions
File tree
- crates/driven-core/src
- src-tauri/src/commands
0 commit comments