feat(core): run the scan walk at the configured io_priority - #173
Merged
Merged
Conversation
Adopts the SPEC s22 io_priority setting (PR #170's mechanism) in the parallel scanner from #169: the walk's pooled spawn_blocking coordinator takes an RAII begin_background_work guard (covering the matcher build's ignore-file collection I/O), and each transient walk worker thread applies the priority once on its first visit - workers die with the walk, so no restore is needed there. The orchestrator passes its live PriorityCell value, so a settings change applies from the next scan. This extends I/O+CPU shaping from bundled uploads to the scan + deep-verify hashing path.
Contributor
Coverage
Gate: passed - no coverage regression (epsilon 0.1 pp). |
This was referenced Jul 25, 2026
pmaxhogan
added a commit
that referenced
this pull request
Jul 26, 2026
Six new rows in "How Driven compares" for what landed since v2.3.0, plus a re-verification of every competitor cell I touched against current upstream docs. Docs-only; no code or `site-landing/` changes (the landing page has no mirror of this table). ## Rows added | Row | Why | | --- | --- | | Re-uploads backup copies deleted at the destination | #171 (audit) + #168 (live self-heal) | | Parallel, multi-threaded local scan | #169 | | OS-level CPU / disk I/O priority for backup work | #170, #173, #176, #179 | | Live preview of which files a rule keeps or drops | #172, #177 | | Rolling local logs plus a one-click diagnostics bundle | #167 | | Reproducible end-to-end benchmark suite in the repo | #178 | Also: a `:grey_question:` legend entry ("not documented"), so a closed-source client whose behaviour Google or Backblaze simply does not publish is marked honestly instead of being guessed at; the intro paragraph now mentions the priority work; the Features list gains the parallel scanner, the preview, the priority setting, the remote audit, and rolling logs; and `just bench` is in the recipe list with a pointer to `bench/README.md`. ## On the benchmark numbers I deliberately quoted **no** upload throughput from #178. That was a 16 MiB / 300-file smoke run - a pipeline proof, not a verdict - and a headline MiB/s from it would not survive scrutiny. The two numbers that did make it in are locally measured and honest at their scale: the exclusion-preview re-classification (536 ms cached vs 868 ms fresh walk on a 63k-entry tree, from #177) and the scan thread clamp. The bench suite appears as a row on its own merits, with the caveats left in `bench/README.md`. ## Competitor claims and where each was verified Versions checked: rclone 1.74.4, restic 0.19.1, Duplicati 2.3.0.4, Backblaze Personal Backup 10.0.2, Drive for desktop 128.0. **CPU / I/O priority - nobody else has it.** rclone has no priority code and closed both requests pointing at `ionice` (rclone/rclone#864). restic's FAQ answers "How to prioritize restic's IO and CPU time" entirely with `ionice`/`nice` recipes (https://restic.readthedocs.io/en/stable/faq.html). Duplicati still accepts `--thread-priority` but the shipping string is "has no effect, use the operating system controls to set the process priority" (`Duplicati/Library/Main/Options.cs` at the `v2.3.0.4_stable_2026-07-09` tag) - note its published docs page still lists the old text with no deprecation notice, so I cited the source, not the doc. Drive for desktop's entire preference surface is bandwidth rate limits plus pause (https://support.google.com/drive/answer/13470231), with no priority key in the admin policy list. Backblaze offers an automatic/manual bandwidth throttle and an upload-thread count (https://www.backblaze.com/computer-backup/docs/configure-performance-settings-windows). Bandwidth limiting is not I/O priority and the note says so. **Parallel scan.** rclone walks at `--checkers` (8) and transfers at `--transfers` (4) (https://rclone.org/docs/). Duplicati's `FileEnumerationProcess` is a single serial task with the concurrency downstream of it, so its walk is a genuine `:x:` (source at the stable tag). restic reads at `--read-concurrency`, default 2 (https://restic.readthedocs.io/en/stable/manual_rest.html). Google and Backblaze document nothing about scan concurrency - hence the new "not documented" marker rather than a guessed `:x:`. **Destination-side deletion.** rclone gets a `:white_check_mark:` here, not an `:x:`: it keeps no state, so `copy`/`sync` re-list the destination every run and re-transfer anything missing (https://rclone.org/commands/rclone_sync/). The note says so plainly, including that this is why it is slower on the incremental case. Drive for desktop is the one that is worse than absent - "any files you put in the trash are put in the trash everywhere" (https://support.google.com/drive/answer/2375102). Duplicati and restic detect damage but recovery is operator-driven (Duplicati's `RepairHandler.cs` refuses missing dblock files without `--rebuild-missing-dblock-files`; restic's troubleshooting doc says re-run `backup` to heal, https://restic.readthedocs.io/en/stable/077_troubleshooting.html). Backblaze documents nothing either way, so it is marked not-documented rather than `:x:`. **Live rule preview - and one claim I had to walk back.** Duplicati shipped server-evaluated inclusion state in its tree UI in 2.3.0.4 on 2026-07-09 (duplicati/duplicati#6955, closing a six-year-old request, duplicati/duplicati#4194). Six weeks ago "nobody else has this" would have been true; it is not any more, so Duplicati gets a `:white_check_mark:` and the note records the remaining difference (it marks the nodes you expand, with no whole-source counts). rclone and restic are `--dry-run` only. **Logs and diagnostics.** Driven is not unique here and the row shows that: Drive for desktop and Backblaze both get `:white_check_mark:` (https://knowledge.workspace.google.com/admin/drive/capture-google-drive-for-desktop-logs-for-support, https://www.backblaze.com/computer-backup/docs/send-logs-to-backblaze-windows). Duplicati is partial - its "Create bug report" bundle is real, but file logging is opt-in, warnings-only by default, and unrotated. restic has no log-file option at all. **Benchmark suite.** Negative evidence for all five (repo tree listings plus code search); the note names Duplicati's unreleased AutoTune harness and Backblaze's B2-not-client benchmark rather than pretending there is nothing adjacent. No existing rows were removed and none had become false. The trailing line now names the exact versions checked instead of just the month. ## Checks - 40 notes, sequential, every table marker resolves to one and every note is cited (validated by script). - All 27 table rows have 7 cells. - Zero em/en dashes or other dash-like non-ASCII; only the superscript digits the file already used. `git ls-files --eol` reports `w/lf`. - README is not covered by prettier or any markdown linter in CI (prettier runs against `ui/src` only), so there is no formatting gate to run. - `deploy-landing.yml`'s `TAGLINE_MARKER` is the README's first line, which is untouched. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01JLB3E2Jm7knNJd37fVpH8X Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
pmaxhogan
added a commit
that referenced
this pull request
Jul 26, 2026
🤖 I have created a release *beep* *boop* --- ## [2.4.0](v2.3.0...v2.4.0) (2026-07-26) ### Features * **cli:** add dump-client-creds to print an account's stored BYO OAuth client ([#166](#166)) ([4b6070b](4b6070b)) * **core:** parallel scan with negation-aware directory pruning ([#169](#169)) ([3792e47](3792e47)) * **core:** remote-existence audit heals files whose Drive objects vanished ([#171](#171)) ([5cb8e3a](5cb8e3a)) * **core:** run the scan walk at the configured io_priority ([#173](#173)) ([8f39961](8f39961)) * **core:** shape bundle-build file reads with the io_priority setting ([#179](#179)) ([af8f048](af8f048)) * **core:** shape upload I/O with the io_priority setting ([#176](#176)) ([badd9c9](badd9c9)) * **core:** wire the ioPriority setting to real OS thread priorities ([#170](#170)) ([51bb1f3](51bb1f3)) * persist rolling backend logs and capture frontend console into diagnostics ([#167](#167)) ([292e221](292e221)) * real-world benchmark suite comparing driven with rclone ([#178](#178)) ([85f6d6a](85f6d6a)) * **ui:** instant exclusion-preview re-evaluation from an in-memory tree ([#177](#177)) ([8f49570](8f49570)) * **ui:** sticky shell chrome, indeterminate scan progress and navigation cleanup ([#163](#163)) ([4896336](4896336)) * **ui:** transient in-app toast notifications ([#164](#164)) ([0104a8e](0104a8e)) * **ui:** warn when include patterns defeat directory pruning ([#162](#162)) ([6d3b4b2](6d3b4b2)) ### Bug Fixes * **core:** self-heal a stale drive_file_id when an update hits a definitive 404 ([#168](#168)) ([8b983b3](8b983b3)) ### Performance Improvements * **core:** per-directory decision cursor for the exclusion preview + NFC fast path ([#172](#172)) ([a587fed](a587fed)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adopts #170's priority mechanism in #169's parallel scanner - the follow-up called out in #170's PR body ('the broad user-visible win lands when the scanner's dedicated worker threads call apply_to_current_thread').
A sibling PR (in flight) extends shaping to the large-file upload pipeline.
Gates: cargo fmt, clippy -p driven-core -D warnings clean, cargo test -p driven-core 393+ green.
🤖 Generated with Claude Code
https://claude.ai/code/session_01JLB3E2Jm7knNJd37fVpH8X