feat(ui): instant exclusion-preview re-evaluation from an in-memory tree - #177
Merged
Merged
Conversation
Decouple WALKING the source folder from CLASSIFYING it under the candidate rules. A per-root, editor-scoped tree cache records each directory the first pass reads; every later pass replays it from memory and touches the disk only for subtrees earlier passes pruned and the new rules now reach into. - preview_exclusions_start returns its generation id immediately; the matcher build (which reads the ignore-file cascade off disk) moved into the spawned task and reports failures on a new exclusion_preview:error event. - A superseded pass is demoted to a silent cache builder rather than cancelled, bounded at one builder so fast edits cannot stack walks. - Cancelling the preview frees the cache; a root change drops it. - The store no longer blanks the tree or zeroes the totals on a rule edit: the incoming generation is built aside and swapped in whole on its first batch, with a dimmed "updating" affordance covering the gap. - Expansion and paging state now survive a rule edit.
Contributor
Coverage
Gate: passed - no coverage regression (epsilon 0.1 pp). |
This was referenced Jul 25, 2026
pmaxhogan
added a commit
that referenced
this pull request
Jul 26, 2026
Six new rows in "How Driven compares" for what landed since v2.3.0, plus a re-verification of every competitor cell I touched against current upstream docs. Docs-only; no code or `site-landing/` changes (the landing page has no mirror of this table). ## Rows added | Row | Why | | --- | --- | | Re-uploads backup copies deleted at the destination | #171 (audit) + #168 (live self-heal) | | Parallel, multi-threaded local scan | #169 | | OS-level CPU / disk I/O priority for backup work | #170, #173, #176, #179 | | Live preview of which files a rule keeps or drops | #172, #177 | | Rolling local logs plus a one-click diagnostics bundle | #167 | | Reproducible end-to-end benchmark suite in the repo | #178 | Also: a `:grey_question:` legend entry ("not documented"), so a closed-source client whose behaviour Google or Backblaze simply does not publish is marked honestly instead of being guessed at; the intro paragraph now mentions the priority work; the Features list gains the parallel scanner, the preview, the priority setting, the remote audit, and rolling logs; and `just bench` is in the recipe list with a pointer to `bench/README.md`. ## On the benchmark numbers I deliberately quoted **no** upload throughput from #178. That was a 16 MiB / 300-file smoke run - a pipeline proof, not a verdict - and a headline MiB/s from it would not survive scrutiny. The two numbers that did make it in are locally measured and honest at their scale: the exclusion-preview re-classification (536 ms cached vs 868 ms fresh walk on a 63k-entry tree, from #177) and the scan thread clamp. The bench suite appears as a row on its own merits, with the caveats left in `bench/README.md`. ## Competitor claims and where each was verified Versions checked: rclone 1.74.4, restic 0.19.1, Duplicati 2.3.0.4, Backblaze Personal Backup 10.0.2, Drive for desktop 128.0. **CPU / I/O priority - nobody else has it.** rclone has no priority code and closed both requests pointing at `ionice` (rclone/rclone#864). restic's FAQ answers "How to prioritize restic's IO and CPU time" entirely with `ionice`/`nice` recipes (https://restic.readthedocs.io/en/stable/faq.html). Duplicati still accepts `--thread-priority` but the shipping string is "has no effect, use the operating system controls to set the process priority" (`Duplicati/Library/Main/Options.cs` at the `v2.3.0.4_stable_2026-07-09` tag) - note its published docs page still lists the old text with no deprecation notice, so I cited the source, not the doc. Drive for desktop's entire preference surface is bandwidth rate limits plus pause (https://support.google.com/drive/answer/13470231), with no priority key in the admin policy list. Backblaze offers an automatic/manual bandwidth throttle and an upload-thread count (https://www.backblaze.com/computer-backup/docs/configure-performance-settings-windows). Bandwidth limiting is not I/O priority and the note says so. **Parallel scan.** rclone walks at `--checkers` (8) and transfers at `--transfers` (4) (https://rclone.org/docs/). Duplicati's `FileEnumerationProcess` is a single serial task with the concurrency downstream of it, so its walk is a genuine `:x:` (source at the stable tag). restic reads at `--read-concurrency`, default 2 (https://restic.readthedocs.io/en/stable/manual_rest.html). Google and Backblaze document nothing about scan concurrency - hence the new "not documented" marker rather than a guessed `:x:`. **Destination-side deletion.** rclone gets a `:white_check_mark:` here, not an `:x:`: it keeps no state, so `copy`/`sync` re-list the destination every run and re-transfer anything missing (https://rclone.org/commands/rclone_sync/). The note says so plainly, including that this is why it is slower on the incremental case. Drive for desktop is the one that is worse than absent - "any files you put in the trash are put in the trash everywhere" (https://support.google.com/drive/answer/2375102). Duplicati and restic detect damage but recovery is operator-driven (Duplicati's `RepairHandler.cs` refuses missing dblock files without `--rebuild-missing-dblock-files`; restic's troubleshooting doc says re-run `backup` to heal, https://restic.readthedocs.io/en/stable/077_troubleshooting.html). Backblaze documents nothing either way, so it is marked not-documented rather than `:x:`. **Live rule preview - and one claim I had to walk back.** Duplicati shipped server-evaluated inclusion state in its tree UI in 2.3.0.4 on 2026-07-09 (duplicati/duplicati#6955, closing a six-year-old request, duplicati/duplicati#4194). Six weeks ago "nobody else has this" would have been true; it is not any more, so Duplicati gets a `:white_check_mark:` and the note records the remaining difference (it marks the nodes you expand, with no whole-source counts). rclone and restic are `--dry-run` only. **Logs and diagnostics.** Driven is not unique here and the row shows that: Drive for desktop and Backblaze both get `:white_check_mark:` (https://knowledge.workspace.google.com/admin/drive/capture-google-drive-for-desktop-logs-for-support, https://www.backblaze.com/computer-backup/docs/send-logs-to-backblaze-windows). Duplicati is partial - its "Create bug report" bundle is real, but file logging is opt-in, warnings-only by default, and unrotated. restic has no log-file option at all. **Benchmark suite.** Negative evidence for all five (repo tree listings plus code search); the note names Duplicati's unreleased AutoTune harness and Backblaze's B2-not-client benchmark rather than pretending there is nothing adjacent. No existing rows were removed and none had become false. The trailing line now names the exact versions checked instead of just the month. ## Checks - 40 notes, sequential, every table marker resolves to one and every note is cited (validated by script). - All 27 table rows have 7 cells. - Zero em/en dashes or other dash-like non-ASCII; only the superscript digits the file already used. `git ls-files --eol` reports `w/lf`. - README is not covered by prettier or any markdown linter in CI (prettier runs against `ui/src` only), so there is no formatting gate to run. - `deploy-landing.yml`'s `TAGLINE_MARKER` is the README's first line, which is untouched. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01JLB3E2Jm7knNJd37fVpH8X Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
pmaxhogan
added a commit
that referenced
this pull request
Jul 26, 2026
🤖 I have created a release *beep* *boop* --- ## [2.4.0](v2.3.0...v2.4.0) (2026-07-26) ### Features * **cli:** add dump-client-creds to print an account's stored BYO OAuth client ([#166](#166)) ([4b6070b](4b6070b)) * **core:** parallel scan with negation-aware directory pruning ([#169](#169)) ([3792e47](3792e47)) * **core:** remote-existence audit heals files whose Drive objects vanished ([#171](#171)) ([5cb8e3a](5cb8e3a)) * **core:** run the scan walk at the configured io_priority ([#173](#173)) ([8f39961](8f39961)) * **core:** shape bundle-build file reads with the io_priority setting ([#179](#179)) ([af8f048](af8f048)) * **core:** shape upload I/O with the io_priority setting ([#176](#176)) ([badd9c9](badd9c9)) * **core:** wire the ioPriority setting to real OS thread priorities ([#170](#170)) ([51bb1f3](51bb1f3)) * persist rolling backend logs and capture frontend console into diagnostics ([#167](#167)) ([292e221](292e221)) * real-world benchmark suite comparing driven with rclone ([#178](#178)) ([85f6d6a](85f6d6a)) * **ui:** instant exclusion-preview re-evaluation from an in-memory tree ([#177](#177)) ([8f49570](8f49570)) * **ui:** sticky shell chrome, indeterminate scan progress and navigation cleanup ([#163](#163)) ([4896336](4896336)) * **ui:** transient in-app toast notifications ([#164](#164)) ([0104a8e](0104a8e)) * **ui:** warn when include patterns defeat directory pruning ([#162](#162)) ([6d3b4b2](6d3b4b2)) ### Bug Fixes * **core:** self-heal a stale drive_file_id when an update hits a definitive 404 ([#168](#168)) ([8b983b3](8b983b3)) ### Performance Improvements * **core:** per-directory decision cursor for the exclusion preview + NFC fast path ([#172](#172)) ([a587fed](a587fed)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
pmaxhogan
added a commit
that referenced
this pull request
Jul 29, 2026
#206) ## The 60GB was not Driven Investigated as a P0 memory blowup in the app. It was not the app. Primary evidence, from an artifact the OS wrote during the incident itself: `/Library/Logs/DiagnosticReports/JetsamEvent-2026-07-29-114254.ips`, a kernel memory-pressure snapshot that records every process's footprint. At that moment (73.2 GB of total system footprint): | process | pid | footprint | | --- | --- | --- | | `node` x11 | 77048-77118 | **4460-4532 MB each, 49.7 GB total** | | `node` (all 39, incl. the above) | - | 51.3 GB | | WindowServer | 422 | 1013 MB | | `driven-app` | 56224 | **45.9 MB** | | `com.apple.WebKit.WebContent` (the app's webview) | 63448 | **42.8 MB** | | `cargo-tauri` | 54805 | 32.1 MB | | `cargo` x2 | 80745, 81134 | 132.5 + 123.1 MB | The report's own `largestProcess` field is `node`. The whole `cargo tauri dev` tree is identifiable and contiguous in that snapshot - `zsh` 54803 -> `cargo-tauri` 54805 (32.1 MB) -> the vite chain `node` 54953/54959/55017 (62.4 + 55.8 + 167.6 MB) -> `driven-app` 56224 (45.9 MB) -> WebKit WebContent/GPU/Networking 63446-63448 (42.8 + 16.8 + 7.2 MB). **~431 MB for everything Driven owned, after an hour of running.** The eleven giants are pids 77048-77118, a separate burst ~53 minutes later. The eleven big `node` processes were spawned in a single burst at 11:39:59 and all died the same way: six crash reports in `~/Library/Logs/DiagnosticReports/` show `SIGABRT` through `node::OOMErrorHandler` -> `v8::internal::Heap::FatalProcessOutOfMemory`, i.e. each one hit V8's ~4.5 GB old-space ceiling. Their parent had already exited (all six report `parentProc: launchd`), so they were orphaned workers of a pool whose supervisor was gone. Crash reports do not record argv, and the burst started almost an hour after the app did, so they are not the app's vite dev server - that was a separate ~170 MB `node` in the same snapshot. ### What the app actually did `~/Library/Application Support/app.driven/logs/driven.2026-07-29.log` covers the incident run exactly: ``` 15:46:39.081Z rolling file logs active 15:46:39.289Z assembling per-account orchestrators accounts=0 sources=0 15:46:39.298Z updater periodic check started interval_secs=21600 15:46:39.298Z telemetry ping task started interval_secs=86400 15:46:39.881Z add-account wizard session opened ... one hour of complete silence ... 16:47:34Z (a different build's first line) ``` So the app booted with **zero accounts and zero sources**, parked on the add-account wizard, and logged nothing for the next hour. ### Reproduction Ran `cargo tauri dev` from this worktree and reached the identical state (`accounts=0 sources=0`, wizard session opened), then sampled the whole process tree every 10s. Over ~20 minutes idle on that screen: - `driven-app`: 141 MB -> 144 MB - its `WebContent`: 76 MB -> 76 MB Flat. No growth path exists in that state to begin with: with no accounts and no sources there is no scanner, no FSEvents watcher, and no tray sync animation, and the two periodic tasks that do start fire at 6h and 24h. ### Suspects ruled out - **#177 (exclusion-preview in-memory tree)** - needs a configured source; the incident had none. The cache is hard-capped at 4M entries and frees everything on overflow (`preview_cache.rs:178-203`). Real worst case is a few hundred MB, and only while the editor is open. (It did contain a separate, real leak - see below.) - **#167 (rolling logs + console capture)** - frontend ring is 500 entries x 2000 chars, ~1 MB ceiling; the backend appender is lossy-bounded at 128k buffered lines. On-disk log for the whole incident run was 1.1 KB. - **Scanner / watcher** - never ran (`sources=0`). - **Dev-build overhead** - the debug build measured 46 MB in the field and 141 MB under my own dev run. Also ran this repo's UI test suite (43 files, 530 tests) directly: 4.3s, no worker anywhere near a GB. It is not the source of the eleven OOMing workers. ## What this PR fixes A real, unbounded leak found while ruling out suspect #177. **It is not the cause of the 60 GB event** - it is bounded per open/close by `NODE_STREAM_CAP` and needs a lost race to trigger - but it is genuinely unbounded over a session and it lives in exactly the code that was suspected, so it should not be left in. `ExclusionPreviewTree` subscribes in `onMounted` via an awaited `preview.subscribe()` (three `listen()` round-trips) and stores the teardown handle afterwards. `onUnmounted` only calls the handle if it is already set. A component unmounted inside that window - and the editor mounts under `v-if` in both `SourceTable` and `AddSourceWizard`, so open-then-close is ordinary use - therefore tore down nothing, and the three listeners resolved into a permanently unreachable closure. That would be a bounded one-time cost if the listeners were scoped, but `onExclusionPreviewBatch` and friends use a plain `listen(name, cb)` (`ipc/events.ts:153-174`), which registers **globally by event name**. So the orphan keeps receiving every later preview's `exclusion_preview:batch`. Its `currentId` never resolves, so `ingestBatch` takes the pre-id park branch - an array only its own `start()` can drain. Every batch of every future preview accumulated there for the life of the process. The park's doc comment claimed it "cannot grow without limit"; that was true only for a controller that goes on to resolve an id. Two changes: 1. `ExclusionPreviewTree.vue` - guard the race with the same shape `activity.ts:640-668` already uses: flip a `subscribeWanted` intent flag, re-check it after the await, and invoke the resolved unlisteners inline if it flipped. Also suppresses the `restart()` that would otherwise start a full walk for a tree nobody is rendering. 2. `exclusionPreview.ts` - cap the pre-id park at `PRE_ID_PARK_CAP` (256, vs the ~125 batches one generation can legitimately produce), dropping the newest over the cap so the breadth-first ancestors are preserved and the overflow degrades to the already-handled `truncated` case. Defence in depth: it also bounds the other way to reach this state, a rejected `previewExclusionsStart`. ### Regression tests Three, all verified failing before the change (`git stash` of the two source files, tests kept): - `tears down every listener when unmounted while subscribe is still in flight` - gates `listen()` on a promise, unmounts inside the window, asserts all three unlisten spies fire and that no walk is started. Before: `expected "spy" to be called 1 times, but got 0 times`. - `caps the park so a controller that never resolves an id cannot grow without bound` - drives a controller whose `start` rejects, fires 4x the cap in batches, pins the retained count at `PRE_ID_PARK_CAP`. - `still parks and replays everything that arrives before a real id lands` - the legitimate park path still drains and folds into the tree. Plus `tears down every listener on an ordinary unmount`, which passes both ways and pins the non-racing path. ### Gates `vitest` 530 passed, `prettier --check`, `eslint`, `vue-tsc --noEmit`, `cargo fmt --all --check`, `cargo clippy --workspace --all-targets -D warnings`, `cargo test --workspace` - all clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Editing an exclusion rule used to re-walk the entire source folder from disk. On a large source that is minutes of I/O per keystroke, to answer a question that cannot possibly depend on what is on disk - and while it ran, the tree blanked and every count dropped to zero before repopulating.
This separates the two jobs the preview had conflated: WALKING the folder, and CLASSIFYING it under the candidate rules.
Backend
A per-root, editor-scoped folder-tree cache (
src-tauri/src/commands/preview_cache.rs) records each directory the first pass reads -rel_dir -> [{name, is_dir, size}], one entry per directory inread_dirorder. Every later pass replays it from memory and touches the disk only for directories the cache does not have.That single rule covers both cases the design called for. A directory an earlier pass PRUNED (excluded, no
!-rule reaching in) is simply absent from the map, so a rule that now reaches into it is a cache miss, gets walked lazily, and is appended to both the stream and the cache. Presence per directory is the completeness signal, so no separate frontier set or global complete flag is needed - noted below as a deviation from the brief's shape.The cache never stores a verdict. Classification runs fresh on every pass, threaded through the
DirDecisioncursor from #172 (root_decision()/descend()/is_included_at()), which now rides on the BFS queue entry.Also in this pass:
preview_exclusions_startreturns its generation id IMMEDIATELY. Building the matcher reads the source's ignore-file cascade off disk - seconds on a repo-of-repos - and that moved into the spawned task.sources.rssplitsresolve_preview_root_and_matcherinto the fast, fallibleresolve_preview_root(unchanged validation: globs, exactly-one-selector, dialog-token peek, readable-dir) and the slowbuild_preview_matcher. A matcher that fails to build now reports on a newexclusion_preview:errorevent carrying the same stable SPEC s24 code the rejection would have used.Frontend
start()no longer clears the tree or zeroes the totals. The incoming generation is built off to the side and swapped in whole on its FIRST batch, which with the cache lands within a frame or two. Arecomputingref dims the summary and tree behind an "Updating..." status in the meantime. Adonealso swaps (it is the generation's final word, even if it found nothing), but a CANCELLED one does not - that generation was abandoned and the tree on screen is still the best answer available.Expansion and paging state now survive a rule edit too. Re-classifying the same folder used to collapse the whole tree on every keystroke, throwing away the user's place in it.
Measured
On a real 63k-entry tree: first pass 2.79s, then a rule edit reclassifies in 536ms against 868ms for a fresh re-walk of the now-OS-cached tree - and the cached path issues zero
read_dircalls, so unlike the walk its cost does not scale with disk speed. On a 3.45M-entryDocumentsfolder the budget overflowed as designed and the classification came out byte-identical to the full walk (23,264 excluded either way); that measurement is what set the 4M ceiling.Deviations from the brief
Gates
cargo fmt --check,cargo clippy -p driven-app -p driven-core --all-targets -D warnings,cargo test -p driven-app --lib(303 passed),cargo test -p driven-core --lib exclude(50 passed),npx vitest run(526 passed),npx prettier --check src,npx eslint .(0 errors) all pass. Mergedorigin/main(including #172) before opening.Rust coverage added: cache-vs-fresh-walk equivalence over a matrix of 10 successive rule edits on one shared cache; a second pass proved to be pure-memory by DELETING the fixture between passes; lazy frontier walk matching a fresh walk exactly; overflow fallback staying correct; cancel freeing the cache; registry demotion, the two-walk bound, and both slots cancelling. #172's cursor-equivalence test now runs over the cached replay as well as the disk walk.
Vitest coverage added: the old tree staying visible until the new generation's first batch, totals never dipping through zero, done-swaps-empty vs cancelled-does-not, first-open publishing on batch 1, the error event (live, superseded, pre-id replay, cleared on restart), and component tests for the dimmed recomputing state and preserved expansion.