feat(core): shape bundle-build file reads with the io_priority setting - #179
Merged
Merged
Conversation
Closes the asymmetry left by #176. Large-file upload reads are shaped by the per-handle I/O priority hint, but bundled small-file reads were not: they go through build_bundle's own opens, covered only by the thread guard from #170 - and on Windows "low" maps to THREAD_PRIORITY_BELOW_NORMAL, which is CPU-only. So a "low" backup of many small files still read at normal I/O priority. - build_bundle takes a WorkPriority and hints each member's handle as it opens. - The executor passes the same value it already reads for the thread guard, so both levers use one level and a settings change still applies per bundle. - std::fs::read is split into a read_member helper purely so there IS a handle to hint before the reads it shapes; same default share mode (File::open is what std::fs::read uses internally) and the same stat-sized allocation, so the open's sharing/locking semantics are unchanged. The size argument is a capacity hint only, never a read bound - the caller's post-read coherency stat stays the sole judge of whether the bytes are a usable snapshot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JLB3E2Jm7knNJd37fVpH8X
Contributor
Coverage
Gate: passed - no coverage regression (epsilon 0.1 pp). |
This was referenced Jul 25, 2026
pmaxhogan
added a commit
that referenced
this pull request
Jul 26, 2026
Six new rows in "How Driven compares" for what landed since v2.3.0, plus a re-verification of every competitor cell I touched against current upstream docs. Docs-only; no code or `site-landing/` changes (the landing page has no mirror of this table). ## Rows added | Row | Why | | --- | --- | | Re-uploads backup copies deleted at the destination | #171 (audit) + #168 (live self-heal) | | Parallel, multi-threaded local scan | #169 | | OS-level CPU / disk I/O priority for backup work | #170, #173, #176, #179 | | Live preview of which files a rule keeps or drops | #172, #177 | | Rolling local logs plus a one-click diagnostics bundle | #167 | | Reproducible end-to-end benchmark suite in the repo | #178 | Also: a `:grey_question:` legend entry ("not documented"), so a closed-source client whose behaviour Google or Backblaze simply does not publish is marked honestly instead of being guessed at; the intro paragraph now mentions the priority work; the Features list gains the parallel scanner, the preview, the priority setting, the remote audit, and rolling logs; and `just bench` is in the recipe list with a pointer to `bench/README.md`. ## On the benchmark numbers I deliberately quoted **no** upload throughput from #178. That was a 16 MiB / 300-file smoke run - a pipeline proof, not a verdict - and a headline MiB/s from it would not survive scrutiny. The two numbers that did make it in are locally measured and honest at their scale: the exclusion-preview re-classification (536 ms cached vs 868 ms fresh walk on a 63k-entry tree, from #177) and the scan thread clamp. The bench suite appears as a row on its own merits, with the caveats left in `bench/README.md`. ## Competitor claims and where each was verified Versions checked: rclone 1.74.4, restic 0.19.1, Duplicati 2.3.0.4, Backblaze Personal Backup 10.0.2, Drive for desktop 128.0. **CPU / I/O priority - nobody else has it.** rclone has no priority code and closed both requests pointing at `ionice` (rclone/rclone#864). restic's FAQ answers "How to prioritize restic's IO and CPU time" entirely with `ionice`/`nice` recipes (https://restic.readthedocs.io/en/stable/faq.html). Duplicati still accepts `--thread-priority` but the shipping string is "has no effect, use the operating system controls to set the process priority" (`Duplicati/Library/Main/Options.cs` at the `v2.3.0.4_stable_2026-07-09` tag) - note its published docs page still lists the old text with no deprecation notice, so I cited the source, not the doc. Drive for desktop's entire preference surface is bandwidth rate limits plus pause (https://support.google.com/drive/answer/13470231), with no priority key in the admin policy list. Backblaze offers an automatic/manual bandwidth throttle and an upload-thread count (https://www.backblaze.com/computer-backup/docs/configure-performance-settings-windows). Bandwidth limiting is not I/O priority and the note says so. **Parallel scan.** rclone walks at `--checkers` (8) and transfers at `--transfers` (4) (https://rclone.org/docs/). Duplicati's `FileEnumerationProcess` is a single serial task with the concurrency downstream of it, so its walk is a genuine `:x:` (source at the stable tag). restic reads at `--read-concurrency`, default 2 (https://restic.readthedocs.io/en/stable/manual_rest.html). Google and Backblaze document nothing about scan concurrency - hence the new "not documented" marker rather than a guessed `:x:`. **Destination-side deletion.** rclone gets a `:white_check_mark:` here, not an `:x:`: it keeps no state, so `copy`/`sync` re-list the destination every run and re-transfer anything missing (https://rclone.org/commands/rclone_sync/). The note says so plainly, including that this is why it is slower on the incremental case. Drive for desktop is the one that is worse than absent - "any files you put in the trash are put in the trash everywhere" (https://support.google.com/drive/answer/2375102). Duplicati and restic detect damage but recovery is operator-driven (Duplicati's `RepairHandler.cs` refuses missing dblock files without `--rebuild-missing-dblock-files`; restic's troubleshooting doc says re-run `backup` to heal, https://restic.readthedocs.io/en/stable/077_troubleshooting.html). Backblaze documents nothing either way, so it is marked not-documented rather than `:x:`. **Live rule preview - and one claim I had to walk back.** Duplicati shipped server-evaluated inclusion state in its tree UI in 2.3.0.4 on 2026-07-09 (duplicati/duplicati#6955, closing a six-year-old request, duplicati/duplicati#4194). Six weeks ago "nobody else has this" would have been true; it is not any more, so Duplicati gets a `:white_check_mark:` and the note records the remaining difference (it marks the nodes you expand, with no whole-source counts). rclone and restic are `--dry-run` only. **Logs and diagnostics.** Driven is not unique here and the row shows that: Drive for desktop and Backblaze both get `:white_check_mark:` (https://knowledge.workspace.google.com/admin/drive/capture-google-drive-for-desktop-logs-for-support, https://www.backblaze.com/computer-backup/docs/send-logs-to-backblaze-windows). Duplicati is partial - its "Create bug report" bundle is real, but file logging is opt-in, warnings-only by default, and unrotated. restic has no log-file option at all. **Benchmark suite.** Negative evidence for all five (repo tree listings plus code search); the note names Duplicati's unreleased AutoTune harness and Backblaze's B2-not-client benchmark rather than pretending there is nothing adjacent. No existing rows were removed and none had become false. The trailing line now names the exact versions checked instead of just the month. ## Checks - 40 notes, sequential, every table marker resolves to one and every note is cited (validated by script). - All 27 table rows have 7 cells. - Zero em/en dashes or other dash-like non-ASCII; only the superscript digits the file already used. `git ls-files --eol` reports `w/lf`. - README is not covered by prettier or any markdown linter in CI (prettier runs against `ui/src` only), so there is no formatting gate to run. - `deploy-landing.yml`'s `TAGLINE_MARKER` is the README's first line, which is untouched. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01JLB3E2Jm7knNJd37fVpH8X Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
pmaxhogan
added a commit
that referenced
this pull request
Jul 26, 2026
🤖 I have created a release *beep* *boop* --- ## [2.4.0](v2.3.0...v2.4.0) (2026-07-26) ### Features * **cli:** add dump-client-creds to print an account's stored BYO OAuth client ([#166](#166)) ([4b6070b](4b6070b)) * **core:** parallel scan with negation-aware directory pruning ([#169](#169)) ([3792e47](3792e47)) * **core:** remote-existence audit heals files whose Drive objects vanished ([#171](#171)) ([5cb8e3a](5cb8e3a)) * **core:** run the scan walk at the configured io_priority ([#173](#173)) ([8f39961](8f39961)) * **core:** shape bundle-build file reads with the io_priority setting ([#179](#179)) ([af8f048](af8f048)) * **core:** shape upload I/O with the io_priority setting ([#176](#176)) ([badd9c9](badd9c9)) * **core:** wire the ioPriority setting to real OS thread priorities ([#170](#170)) ([51bb1f3](51bb1f3)) * persist rolling backend logs and capture frontend console into diagnostics ([#167](#167)) ([292e221](292e221)) * real-world benchmark suite comparing driven with rclone ([#178](#178)) ([85f6d6a](85f6d6a)) * **ui:** instant exclusion-preview re-evaluation from an in-memory tree ([#177](#177)) ([8f49570](8f49570)) * **ui:** sticky shell chrome, indeterminate scan progress and navigation cleanup ([#163](#163)) ([4896336](4896336)) * **ui:** transient in-app toast notifications ([#164](#164)) ([0104a8e](0104a8e)) * **ui:** warn when include patterns defeat directory pruning ([#162](#162)) ([6d3b4b2](6d3b4b2)) ### Bug Fixes * **core:** self-heal a stale drive_file_id when an update hits a definitive 404 ([#168](#168)) ([8b983b3](8b983b3)) ### Performance Improvements * **core:** per-directory decision cursor for the exclusion preview + NFC fast path ([#172](#172)) ([a587fed](a587fed)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the asymmetry I flagged in #176.
The gap
After #176,
io_priorityshaped large-file upload reads through the per-handlehint. Bundled small-file reads were still not shaped: they go through
build_bundle's own opens, covered only by the thread guard from #170 - andon Windows
lowmaps toTHREAD_PRIORITY_BELOW_NORMAL, which is CPU-only. So alowbackup of a folder full of small files still read at normal I/O priority,which is exactly the workload someone is most likely to reach for when testing
the setting.
(At
idlethis was already covered:THREAD_MODE_BACKGROUND_BEGINlowers I/Oand memory priority alongside CPU. The gap was
low-only.)The change
build_bundletakes aWorkPriorityand hints each member's file handle as itopens it. The executor passes the same value it already reads for the thread
guard, so both levers run at one level and a settings change still takes effect
per bundle with no restart.
The bundle path is the one place that visibly needs both levers, which is
what makes the two-lever split concrete: the thread guard covers the gzip CPU,
the handle hint covers the reads.
Why
std::fs::readhad to gostd::fs::readopens and reads in one call and never exposes theFile, sothere is no handle to hint - and the hint has to land on the handle before the
reads it is meant to shape. It is now split into a small
read_memberhelperthat opens, hints, then reads. Behaviour is deliberately identical:
std::fs::readusesFile::openinternally, so theopen's sharing/locking semantics are unchanged - which matters because the
locked-file/VSS logic elsewhere depends on those semantics.
prestat the loop already took instead of re-statting.
does not short-circuit the read. Treating it as a bound would hide a
grew-mid-read member from the caller's post-read coherency stat, which remains
the sole judge of whether the bytes are a usable snapshot.
Nothing else in the build loop moved: the pre-stat size re-validation, the
accumulated-bytes ceiling, the post-read coherency check, and the skip
bookkeeping are untouched.
Tests
priority_does_not_change_the_archive_or_the_members- builds the same inputs(including one member that must be skipped) at all three levels and asserts
the
.tar.gzbytes, the packed members, and the skip list are identical.Byte-equality is a meaningful assertion here because the gzip layer is written
with a zeroed mtime for reproducibility, so two builds over the same inputs
are bit-identical. This is the guard that
io_prioritystays a purescheduling hint.
read_member_matches_fs_read_and_ignores_a_wrong_size_hint- all bytes comeback at size hints of 0, exact, and oversized, and a missing file errors
rather than returning short, matching
std::fs::readexactly.Gates
cargo fmt --all -- --checkcleancargo clippy --workspace --all-targets -- -D warningscleancargo test -p driven-core: 461 passed, 0 failedcargo test -p driven-app: 296 passed;driven-chaos: 44 passedui/changes.Unlike #176, this diff contains no
cfg-gated code -read_memberis plainstd, and the per-OS branching all lives inside the already-cross-checkedapply_to_file_handle. So there is no platform-divergent surface for theWindows-only local run to have missed; CI's ubuntu and macos legs are the
confirmation.
Platform reality, unchanged from #176
This is a Windows-only win. Linux and macOS have no per-descriptor I/O priority
apply_to_file_handleis a no-op there andbundle reads keep whatever the thread guard gives them.
Docs
design/DESIGN.mds11.2 previously namedopen_sharedas the single hint site.Updated: both
open_sharedandbuild_bundlehint handles now, with a note onwhy the bundle path needs both levers and why the hint is redundant-but-harmless
at
idle.Generated with Claude Code
https://claude.ai/code/session_01JLB3E2Jm7knNJd37fVpH8X