Skip to content

feat(ui): instant exclusion-preview re-evaluation from an in-memory tree - #177

Merged
pmaxhogan merged 2 commits into
mainfrom
feat/preview-cache
Jul 25, 2026
Merged

pmaxhogan merged 2 commits into
mainfrom
feat/preview-cache

Conversation

@pmaxhogan

Copy link
Copy Markdown
Owner

Editing an exclusion rule used to re-walk the entire source folder from disk. On a large source that is minutes of I/O per keystroke, to answer a question that cannot possibly depend on what is on disk - and while it ran, the tree blanked and every count dropped to zero before repopulating.

This separates the two jobs the preview had conflated: WALKING the folder, and CLASSIFYING it under the candidate rules.

Backend

A per-root, editor-scoped folder-tree cache (src-tauri/src/commands/preview_cache.rs) records each directory the first pass reads - rel_dir -> [{name, is_dir, size}], one entry per directory in read_dir order. Every later pass replays it from memory and touches the disk only for directories the cache does not have.

That single rule covers both cases the design called for. A directory an earlier pass PRUNED (excluded, no !-rule reaching in) is simply absent from the map, so a rule that now reaches into it is a cache miss, gets walked lazily, and is appended to both the stream and the cache. Presence per directory is the completeness signal, so no separate frontier set or global complete flag is needed - noted below as a deviation from the brief's shape.

The cache never stores a verdict. Classification runs fresh on every pass, threaded through the DirDecision cursor from #172 (root_decision() / descend() / is_included_at()), which now rides on the BFS queue entry.

Also in this pass:

  • preview_exclusions_start returns its generation id IMMEDIATELY. Building the matcher reads the source's ignore-file cascade off disk - seconds on a repo-of-repos - and that moved into the spawned task. sources.rs splits resolve_preview_root_and_matcher into the fast, fallible resolve_preview_root (unchanged validation: globs, exactly-one-selector, dialog-token peek, readable-dir) and the slow build_preview_matcher. A matcher that fails to build now reports on a new exclusion_preview:error event carrying the same stable SPEC s24 code the rejection would have used.
  • A superseded pass is DEMOTED to a silent cache builder rather than cancelled - its remaining walk is exactly the cache the new pass wants. At most one builder exists (the oldest survivor, furthest into the tree), so fast typing cannot stack more than two walks. Cancelling the preview stops both and frees the cache; a root change drops it.
  • Memory is bounded at 4M entries. Over the ceiling the cache marks itself overflowed, frees everything, and stops caching - every preview then walks from disk exactly as before.

Frontend

start() no longer clears the tree or zeroes the totals. The incoming generation is built off to the side and swapped in whole on its FIRST batch, which with the cache lands within a frame or two. A recomputing ref dims the summary and tree behind an "Updating..." status in the meantime. A done also swaps (it is the generation's final word, even if it found nothing), but a CANCELLED one does not - that generation was abandoned and the tree on screen is still the best answer available.

Expansion and paging state now survive a rule edit too. Re-classifying the same folder used to collapse the whole tree on every keystroke, throwing away the user's place in it.

Measured

On a real 63k-entry tree: first pass 2.79s, then a rule edit reclassifies in 536ms against 868ms for a fresh re-walk of the now-OS-cached tree - and the cached path issues zero read_dir calls, so unlike the walk its cost does not scale with disk speed. On a 3.45M-entry Documents folder the budget overflowed as designed and the classification came out byte-identical to the full walk (23,264 excluded either way); that measurement is what set the 4M ceiling.

Deviations from the brief

  1. The cache is a per-directory map rather than a BFS node list plus an explicit pruned-frontier set and completeness flag. Isomorphic, but it makes "walk this directory if we do not have it" the only rule, which is what unifies the cache-hit and lazy-frontier paths into one BFS.
  2. The brief's superseded-walk design has the new generation "fed as the walk advances". Here both passes share the cache and each walks its own misses, so the new generation is never blocked on the builder and is fed by whatever the builder has finished. The cost is that they can duplicate one directory when they race on it; the alternative (following the builder, waiting on directories it has claimed) blocks the very path this PR exists to make instant.
  3. Preserving expansion/paging state across a restart was not in the brief. It follows from "keep showing the previous result" - preserving the tree but collapsing it is still losing the user's place.

Gates

cargo fmt --check, cargo clippy -p driven-app -p driven-core --all-targets -D warnings, cargo test -p driven-app --lib (303 passed), cargo test -p driven-core --lib exclude (50 passed), npx vitest run (526 passed), npx prettier --check src, npx eslint . (0 errors) all pass. Merged origin/main (including #172) before opening.

Rust coverage added: cache-vs-fresh-walk equivalence over a matrix of 10 successive rule edits on one shared cache; a second pass proved to be pure-memory by DELETING the fixture between passes; lazy frontier walk matching a fresh walk exactly; overflow fallback staying correct; cancel freeing the cache; registry demotion, the two-walk bound, and both slots cancelling. #172's cursor-equivalence test now runs over the cached replay as well as the disk walk.

Vitest coverage added: the old tree staying visible until the new generation's first batch, totals never dipping through zero, done-swaps-empty vs cancelled-does-not, first-open publishing on batch 1, the error event (live, superseded, pre-id replay, cleared on restart), and component tests for the dimmed recomputing state and preserved expansion.

Decouple WALKING the source folder from CLASSIFYING it under the candidate
rules. A per-root, editor-scoped tree cache records each directory the first
pass reads; every later pass replays it from memory and touches the disk only
for subtrees earlier passes pruned and the new rules now reach into.

- preview_exclusions_start returns its generation id immediately; the matcher
  build (which reads the ignore-file cascade off disk) moved into the spawned
  task and reports failures on a new exclusion_preview:error event.
- A superseded pass is demoted to a silent cache builder rather than cancelled,
  bounded at one builder so fast edits cannot stack walks.
- Cancelling the preview frees the cache; a root change drops it.
- The store no longer blanks the tree or zeroes the totals on a rule edit: the
  incoming generation is built aside and swapped in whole on its first batch,
  with a dimmed "updating" affordance covering the gap.
- Expansion and paging state now survive a rule edit.
@github-actions

Copy link
Copy Markdown
Contributor

Coverage

Area main this PR delta
Rust (lib crates) 80.81% 80.92% +0.11 (OK)
UI (vue/ts) 91.33% 91.40% +0.07 (OK)

Gate: passed - no coverage regression (epsilon 0.1 pp).

@pmaxhogan
pmaxhogan merged commit 8f49570 into main Jul 25, 2026
18 checks passed
@pmaxhogan
pmaxhogan deleted the feat/preview-cache branch July 25, 2026 22:49
@github-project-automation github-project-automation Bot moved this from Todo to Done in Driven Jul 25, 2026
pmaxhogan added a commit that referenced this pull request Jul 26, 2026
Six new rows in "How Driven compares" for what landed since v2.3.0, plus
a
re-verification of every competitor cell I touched against current
upstream
docs. Docs-only; no code or `site-landing/` changes (the landing page
has no
mirror of this table).

## Rows added

| Row | Why |
| --- | --- |
| Re-uploads backup copies deleted at the destination | #171 (audit) +
#168 (live self-heal) |
| Parallel, multi-threaded local scan | #169 |
| OS-level CPU / disk I/O priority for backup work | #170, #173, #176,
#179 |
| Live preview of which files a rule keeps or drops | #172, #177 |
| Rolling local logs plus a one-click diagnostics bundle | #167 |
| Reproducible end-to-end benchmark suite in the repo | #178 |

Also: a `:grey_question:` legend entry ("not documented"), so a
closed-source
client whose behaviour Google or Backblaze simply does not publish is
marked
honestly instead of being guessed at; the intro paragraph now mentions
the
priority work; the Features list gains the parallel scanner, the
preview, the
priority setting, the remote audit, and rolling logs; and `just bench`
is in
the recipe list with a pointer to `bench/README.md`.

## On the benchmark numbers

I deliberately quoted **no** upload throughput from #178. That was a 16
MiB /
300-file smoke run - a pipeline proof, not a verdict - and a headline
MiB/s
from it would not survive scrutiny. The two numbers that did make it in
are
locally measured and honest at their scale: the exclusion-preview
re-classification (536 ms cached vs 868 ms fresh walk on a 63k-entry
tree,
from #177) and the scan thread clamp. The bench suite appears as a row
on its
own merits, with the caveats left in `bench/README.md`.

## Competitor claims and where each was verified

Versions checked: rclone 1.74.4, restic 0.19.1, Duplicati 2.3.0.4,
Backblaze Personal Backup 10.0.2, Drive for desktop 128.0.

**CPU / I/O priority - nobody else has it.** rclone has no priority code
and
closed both requests pointing at `ionice`
(rclone/rclone#864). restic's FAQ answers "How
to
prioritize restic's IO and CPU time" entirely with `ionice`/`nice`
recipes
(https://restic.readthedocs.io/en/stable/faq.html). Duplicati still
accepts
`--thread-priority` but the shipping string is "has no effect, use the
operating system controls to set the process priority"
(`Duplicati/Library/Main/Options.cs` at the `v2.3.0.4_stable_2026-07-09`
tag)
- note its published docs page still lists the old text with no
deprecation
notice, so I cited the source, not the doc. Drive for desktop's entire
preference surface is bandwidth rate limits plus pause
(https://support.google.com/drive/answer/13470231), with no priority key
in
the admin policy list. Backblaze offers an automatic/manual bandwidth
throttle
and an upload-thread count

(https://www.backblaze.com/computer-backup/docs/configure-performance-settings-windows).
Bandwidth limiting is not I/O priority and the note says so.

**Parallel scan.** rclone walks at `--checkers` (8) and transfers at
`--transfers` (4) (https://rclone.org/docs/). Duplicati's
`FileEnumerationProcess` is a single serial task with the concurrency
downstream of it, so its walk is a genuine `:x:` (source at the stable
tag).
restic reads at `--read-concurrency`, default 2
(https://restic.readthedocs.io/en/stable/manual_rest.html). Google and
Backblaze document nothing about scan concurrency - hence the new
"not documented" marker rather than a guessed `:x:`.

**Destination-side deletion.** rclone gets a `:white_check_mark:` here,
not an
`:x:`: it keeps no state, so `copy`/`sync` re-list the destination every
run
and re-transfer anything missing
(https://rclone.org/commands/rclone_sync/).
The note says so plainly, including that this is why it is slower on the
incremental case. Drive for desktop is the one that is worse than absent
-
"any files you put in the trash are put in the trash everywhere"
(https://support.google.com/drive/answer/2375102). Duplicati and restic
detect
damage but recovery is operator-driven (Duplicati's `RepairHandler.cs`
refuses
missing dblock files without `--rebuild-missing-dblock-files`; restic's
troubleshooting doc says re-run `backup` to heal,
https://restic.readthedocs.io/en/stable/077_troubleshooting.html).
Backblaze
documents nothing either way, so it is marked not-documented rather than
`:x:`.

**Live rule preview - and one claim I had to walk back.** Duplicati
shipped
server-evaluated inclusion state in its tree UI in 2.3.0.4 on 2026-07-09
(duplicati/duplicati#6955, closing a
six-year-old
request, duplicati/duplicati#4194). Six weeks
ago
"nobody else has this" would have been true; it is not any more, so
Duplicati
gets a `:white_check_mark:` and the note records the remaining
difference
(it marks the nodes you expand, with no whole-source counts). rclone and
restic are `--dry-run` only.

**Logs and diagnostics.** Driven is not unique here and the row shows
that:
Drive for desktop and Backblaze both get `:white_check_mark:`

(https://knowledge.workspace.google.com/admin/drive/capture-google-drive-for-desktop-logs-for-support,

https://www.backblaze.com/computer-backup/docs/send-logs-to-backblaze-windows).
Duplicati is partial - its "Create bug report" bundle is real, but file
logging is opt-in, warnings-only by default, and unrotated. restic has
no
log-file option at all.

**Benchmark suite.** Negative evidence for all five (repo tree listings
plus
code search); the note names Duplicati's unreleased AutoTune harness and
Backblaze's B2-not-client benchmark rather than pretending there is
nothing
adjacent.

No existing rows were removed and none had become false. The trailing
line
now names the exact versions checked instead of just the month.

## Checks

- 40 notes, sequential, every table marker resolves to one and every
note is
  cited (validated by script).
- All 27 table rows have 7 cells.
- Zero em/en dashes or other dash-like non-ASCII; only the superscript
digits
  the file already used. `git ls-files --eol` reports `w/lf`.
- README is not covered by prettier or any markdown linter in CI
(prettier
  runs against `ui/src` only), so there is no formatting gate to run.
- `deploy-landing.yml`'s `TAGLINE_MARKER` is the README's first line,
which is
  untouched.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01JLB3E2Jm7knNJd37fVpH8X

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
pmaxhogan added a commit that referenced this pull request Jul 26, 2026
🤖 I have created a release *beep* *boop*
---


## [2.4.0](v2.3.0...v2.4.0)
(2026-07-26)


### Features

* **cli:** add dump-client-creds to print an account's stored BYO OAuth
client ([#166](#166))
([4b6070b](4b6070b))
* **core:** parallel scan with negation-aware directory pruning
([#169](#169))
([3792e47](3792e47))
* **core:** remote-existence audit heals files whose Drive objects
vanished ([#171](#171))
([5cb8e3a](5cb8e3a))
* **core:** run the scan walk at the configured io_priority
([#173](#173))
([8f39961](8f39961))
* **core:** shape bundle-build file reads with the io_priority setting
([#179](#179))
([af8f048](af8f048))
* **core:** shape upload I/O with the io_priority setting
([#176](#176))
([badd9c9](badd9c9))
* **core:** wire the ioPriority setting to real OS thread priorities
([#170](#170))
([51bb1f3](51bb1f3))
* persist rolling backend logs and capture frontend console into
diagnostics ([#167](#167))
([292e221](292e221))
* real-world benchmark suite comparing driven with rclone
([#178](#178))
([85f6d6a](85f6d6a))
* **ui:** instant exclusion-preview re-evaluation from an in-memory tree
([#177](#177))
([8f49570](8f49570))
* **ui:** sticky shell chrome, indeterminate scan progress and
navigation cleanup
([#163](#163))
([4896336](4896336))
* **ui:** transient in-app toast notifications
([#164](#164))
([0104a8e](0104a8e))
* **ui:** warn when include patterns defeat directory pruning
([#162](#162))
([6d3b4b2](6d3b4b2))


### Bug Fixes

* **core:** self-heal a stale drive_file_id when an update hits a
definitive 404 ([#168](#168))
([8b983b3](8b983b3))


### Performance Improvements

* **core:** per-directory decision cursor for the exclusion preview +
NFC fast path ([#172](#172))
([a587fed](a587fed))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
pmaxhogan added a commit that referenced this pull request Jul 29, 2026
#206)

## The 60GB was not Driven

Investigated as a P0 memory blowup in the app. It was not the app.
Primary
evidence, from an artifact the OS wrote during the incident itself:
`/Library/Logs/DiagnosticReports/JetsamEvent-2026-07-29-114254.ips`, a
kernel
memory-pressure snapshot that records every process's footprint.

At that moment (73.2 GB of total system footprint):

| process | pid | footprint |
| --- | --- | --- |
| `node` x11 | 77048-77118 | **4460-4532 MB each, 49.7 GB total** |
| `node` (all 39, incl. the above) | - | 51.3 GB |
| WindowServer | 422 | 1013 MB |
| `driven-app` | 56224 | **45.9 MB** |
| `com.apple.WebKit.WebContent` (the app's webview) | 63448 | **42.8
MB** |
| `cargo-tauri` | 54805 | 32.1 MB |
| `cargo` x2 | 80745, 81134 | 132.5 + 123.1 MB |

The report's own `largestProcess` field is `node`.

The whole `cargo tauri dev` tree is identifiable and contiguous in that
snapshot - `zsh` 54803 -> `cargo-tauri` 54805 (32.1 MB) -> the vite
chain
`node` 54953/54959/55017 (62.4 + 55.8 + 167.6 MB) -> `driven-app` 56224
(45.9 MB) -> WebKit WebContent/GPU/Networking 63446-63448 (42.8 + 16.8 +
7.2 MB). **~431 MB for everything Driven owned, after an hour of
running.**

The eleven giants are pids 77048-77118, a separate burst ~53 minutes
later.

The eleven big `node` processes were spawned in a single burst at
11:39:59 and
all died the same way: six crash reports in
`~/Library/Logs/DiagnosticReports/`
show `SIGABRT` through `node::OOMErrorHandler` ->
`v8::internal::Heap::FatalProcessOutOfMemory`, i.e. each one hit V8's
~4.5 GB
old-space ceiling. Their parent had already exited (all six report
`parentProc: launchd`), so they were orphaned workers of a pool whose
supervisor was gone. Crash reports do not record argv, and the burst
started
almost an hour after the app did, so they are not the app's vite dev
server -
that was a separate ~170 MB `node` in the same snapshot.

### What the app actually did

`~/Library/Application Support/app.driven/logs/driven.2026-07-29.log`
covers
the incident run exactly:

```
15:46:39.081Z  rolling file logs active
15:46:39.289Z  assembling per-account orchestrators accounts=0 sources=0
15:46:39.298Z  updater periodic check started interval_secs=21600
15:46:39.298Z  telemetry ping task started interval_secs=86400
15:46:39.881Z  add-account wizard session opened
   ... one hour of complete silence ...
16:47:34Z     (a different build's first line)
```

So the app booted with **zero accounts and zero sources**, parked on the
add-account wizard, and logged nothing for the next hour.

### Reproduction

Ran `cargo tauri dev` from this worktree and reached the identical state
(`accounts=0 sources=0`, wizard session opened), then sampled the whole
process tree every 10s. Over ~20 minutes idle on that screen:

- `driven-app`: 141 MB -> 144 MB
- its `WebContent`: 76 MB -> 76 MB

Flat. No growth path exists in that state to begin with: with no
accounts and
no sources there is no scanner, no FSEvents watcher, and no tray sync
animation, and the two periodic tasks that do start fire at 6h and 24h.

### Suspects ruled out

- **#177 (exclusion-preview in-memory tree)** - needs a configured
source; the
  incident had none. The cache is hard-capped at 4M entries and frees
everything on overflow (`preview_cache.rs:178-203`). Real worst case is
a
  few hundred MB, and only while the editor is open. (It did contain a
  separate, real leak - see below.)
- **#167 (rolling logs + console capture)** - frontend ring is 500
entries x
2000 chars, ~1 MB ceiling; the backend appender is lossy-bounded at 128k
  buffered lines. On-disk log for the whole incident run was 1.1 KB.
- **Scanner / watcher** - never ran (`sources=0`).
- **Dev-build overhead** - the debug build measured 46 MB in the field
and
  141 MB under my own dev run.

Also ran this repo's UI test suite (43 files, 530 tests) directly: 4.3s,
no
worker anywhere near a GB. It is not the source of the eleven OOMing
workers.

## What this PR fixes

A real, unbounded leak found while ruling out suspect #177. **It is not
the
cause of the 60 GB event** - it is bounded per open/close by
`NODE_STREAM_CAP` and needs a lost race to trigger - but it is genuinely
unbounded over a session and it lives in exactly the code that was
suspected,
so it should not be left in.

`ExclusionPreviewTree` subscribes in `onMounted` via an awaited
`preview.subscribe()` (three `listen()` round-trips) and stores the
teardown
handle afterwards. `onUnmounted` only calls the handle if it is already
set.
A component unmounted inside that window - and the editor mounts under
`v-if`
in both `SourceTable` and `AddSourceWizard`, so open-then-close is
ordinary
use - therefore tore down nothing, and the three listeners resolved into
a
permanently unreachable closure.

That would be a bounded one-time cost if the listeners were scoped, but
`onExclusionPreviewBatch` and friends use a plain `listen(name, cb)`
(`ipc/events.ts:153-174`), which registers **globally by event name**.
So the
orphan keeps receiving every later preview's `exclusion_preview:batch`.
Its
`currentId` never resolves, so `ingestBatch` takes the pre-id park
branch - an
array only its own `start()` can drain. Every batch of every future
preview
accumulated there for the life of the process. The park's doc comment
claimed
it "cannot grow without limit"; that was true only for a controller that
goes
on to resolve an id.

Two changes:

1. `ExclusionPreviewTree.vue` - guard the race with the same shape
`activity.ts:640-668` already uses: flip a `subscribeWanted` intent
flag,
re-check it after the await, and invoke the resolved unlisteners inline
if
it flipped. Also suppresses the `restart()` that would otherwise start a
   full walk for a tree nobody is rendering.
2. `exclusionPreview.ts` - cap the pre-id park at `PRE_ID_PARK_CAP`
(256, vs
the ~125 batches one generation can legitimately produce), dropping the
newest over the cap so the breadth-first ancestors are preserved and the
   overflow degrades to the already-handled `truncated` case. Defence in
   depth: it also bounds the other way to reach this state, a rejected
   `previewExclusionsStart`.

### Regression tests

Three, all verified failing before the change (`git stash` of the two
source
files, tests kept):

- `tears down every listener when unmounted while subscribe is still in
flight` - gates `listen()` on a promise, unmounts inside the window,
asserts
  all three unlisten spies fire and that no walk is started.
  Before: `expected "spy" to be called 1 times, but got 0 times`.
- `caps the park so a controller that never resolves an id cannot grow
without
bound` - drives a controller whose `start` rejects, fires 4x the cap in
  batches, pins the retained count at `PRE_ID_PARK_CAP`.
- `still parks and replays everything that arrives before a real id
lands` -
  the legitimate park path still drains and folds into the tree.

Plus `tears down every listener on an ordinary unmount`, which passes
both
ways and pins the non-racing path.

### Gates

`vitest` 530 passed, `prettier --check`, `eslint`, `vue-tsc --noEmit`,
`cargo fmt --all --check`, `cargo clippy --workspace --all-targets -D
warnings`, `cargo test --workspace` - all clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant