Skip to content

feat(core): shape bundle-build file reads with the io_priority setting - #179

Merged
pmaxhogan merged 1 commit into
mainfrom
feat/bundle-handle-priority
Jul 25, 2026
Merged

pmaxhogan merged 1 commit into
mainfrom
feat/bundle-handle-priority

Conversation

@pmaxhogan

Copy link
Copy Markdown
Owner

Closes the asymmetry I flagged in #176.

The gap

After #176, io_priority shaped large-file upload reads through the per-handle
hint. Bundled small-file reads were still not shaped: they go through
build_bundle's own opens, covered only by the thread guard from #170 - and
on Windows low maps to THREAD_PRIORITY_BELOW_NORMAL, which is CPU-only. So a
low backup of a folder full of small files still read at normal I/O priority,
which is exactly the workload someone is most likely to reach for when testing
the setting.

(At idle this was already covered: THREAD_MODE_BACKGROUND_BEGIN lowers I/O
and memory priority alongside CPU. The gap was low-only.)

The change

build_bundle takes a WorkPriority and hints each member's file handle as it
opens it. The executor passes the same value it already reads for the thread
guard, so both levers run at one level and a settings change still takes effect
per bundle with no restart.

The bundle path is the one place that visibly needs both levers, which is
what makes the two-lever split concrete: the thread guard covers the gzip CPU,
the handle hint covers the reads.

Why std::fs::read had to go

std::fs::read opens and reads in one call and never exposes the File, so
there is no handle to hint - and the hint has to land on the handle before the
reads it is meant to shape. It is now split into a small read_member helper
that opens, hints, then reads. Behaviour is deliberately identical:

  • Same share mode. std::fs::read uses File::open internally, so the
    open's sharing/locking semantics are unchanged - which matters because the
    locked-file/VSS logic elsewhere depends on those semantics.
  • Same allocation. The buffer is still sized from a stat, reusing the pre
    stat the loop already took instead of re-statting.
  • The size argument is a capacity hint, never a read bound. It deliberately
    does not short-circuit the read. Treating it as a bound would hide a
    grew-mid-read member from the caller's post-read coherency stat, which remains
    the sole judge of whether the bytes are a usable snapshot.

Nothing else in the build loop moved: the pre-stat size re-validation, the
accumulated-bytes ceiling, the post-read coherency check, and the skip
bookkeeping are untouched.

Tests

  • priority_does_not_change_the_archive_or_the_members - builds the same inputs
    (including one member that must be skipped) at all three levels and asserts
    the .tar.gz bytes, the packed members, and the skip list are identical.
    Byte-equality is a meaningful assertion here because the gzip layer is written
    with a zeroed mtime for reproducibility, so two builds over the same inputs
    are bit-identical. This is the guard that io_priority stays a pure
    scheduling hint.
  • read_member_matches_fs_read_and_ignores_a_wrong_size_hint - all bytes come
    back at size hints of 0, exact, and oversized, and a missing file errors
    rather than returning short, matching std::fs::read exactly.

Gates

  • cargo fmt --all -- --check clean
  • cargo clippy --workspace --all-targets -- -D warnings clean
  • cargo test -p driven-core: 461 passed, 0 failed
  • cargo test -p driven-app: 296 passed; driven-chaos: 44 passed
  • LF endings, ASCII dashes only. No ui/ changes.

Unlike #176, this diff contains no cfg-gated code - read_member is plain
std, and the per-OS branching all lives inside the already-cross-checked
apply_to_file_handle. So there is no platform-divergent surface for the
Windows-only local run to have missed; CI's ubuntu and macos legs are the
confirmation.

Platform reality, unchanged from #176

This is a Windows-only win. Linux and macOS have no per-descriptor I/O priority

  • both scope it to the thread - so apply_to_file_handle is a no-op there and
    bundle reads keep whatever the thread guard gives them.

Docs

design/DESIGN.md s11.2 previously named open_shared as the single hint site.
Updated: both open_shared and build_bundle hint handles now, with a note on
why the bundle path needs both levers and why the hint is redundant-but-harmless
at idle.

Generated with Claude Code

https://claude.ai/code/session_01JLB3E2Jm7knNJd37fVpH8X

Closes the asymmetry left by #176. Large-file upload reads are shaped by the
per-handle I/O priority hint, but bundled small-file reads were not: they go
through build_bundle's own opens, covered only by the thread guard from #170 -
and on Windows "low" maps to THREAD_PRIORITY_BELOW_NORMAL, which is CPU-only.
So a "low" backup of many small files still read at normal I/O priority.

- build_bundle takes a WorkPriority and hints each member's handle as it opens.
- The executor passes the same value it already reads for the thread guard, so
  both levers use one level and a settings change still applies per bundle.
- std::fs::read is split into a read_member helper purely so there IS a handle
  to hint before the reads it shapes; same default share mode (File::open is
  what std::fs::read uses internally) and the same stat-sized allocation, so
  the open's sharing/locking semantics are unchanged.

The size argument is a capacity hint only, never a read bound - the caller's
post-read coherency stat stays the sole judge of whether the bytes are a usable
snapshot.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JLB3E2Jm7knNJd37fVpH8X
@github-actions

Copy link
Copy Markdown
Contributor

Coverage

Area main this PR delta
Rust (lib crates) 80.81% 80.83% +0.02 (OK)
UI (vue/ts) 91.33% 91.33% +0.00 (OK)

Gate: passed - no coverage regression (epsilon 0.1 pp).

@pmaxhogan
pmaxhogan merged commit af8f048 into main Jul 25, 2026
18 checks passed
@pmaxhogan
pmaxhogan deleted the feat/bundle-handle-priority branch July 25, 2026 22:57
@github-project-automation github-project-automation Bot moved this from Todo to Done in Driven Jul 25, 2026
pmaxhogan added a commit that referenced this pull request Jul 26, 2026
Six new rows in "How Driven compares" for what landed since v2.3.0, plus
a
re-verification of every competitor cell I touched against current
upstream
docs. Docs-only; no code or `site-landing/` changes (the landing page
has no
mirror of this table).

## Rows added

| Row | Why |
| --- | --- |
| Re-uploads backup copies deleted at the destination | #171 (audit) +
#168 (live self-heal) |
| Parallel, multi-threaded local scan | #169 |
| OS-level CPU / disk I/O priority for backup work | #170, #173, #176,
#179 |
| Live preview of which files a rule keeps or drops | #172, #177 |
| Rolling local logs plus a one-click diagnostics bundle | #167 |
| Reproducible end-to-end benchmark suite in the repo | #178 |

Also: a `:grey_question:` legend entry ("not documented"), so a
closed-source
client whose behaviour Google or Backblaze simply does not publish is
marked
honestly instead of being guessed at; the intro paragraph now mentions
the
priority work; the Features list gains the parallel scanner, the
preview, the
priority setting, the remote audit, and rolling logs; and `just bench`
is in
the recipe list with a pointer to `bench/README.md`.

## On the benchmark numbers

I deliberately quoted **no** upload throughput from #178. That was a 16
MiB /
300-file smoke run - a pipeline proof, not a verdict - and a headline
MiB/s
from it would not survive scrutiny. The two numbers that did make it in
are
locally measured and honest at their scale: the exclusion-preview
re-classification (536 ms cached vs 868 ms fresh walk on a 63k-entry
tree,
from #177) and the scan thread clamp. The bench suite appears as a row
on its
own merits, with the caveats left in `bench/README.md`.

## Competitor claims and where each was verified

Versions checked: rclone 1.74.4, restic 0.19.1, Duplicati 2.3.0.4,
Backblaze Personal Backup 10.0.2, Drive for desktop 128.0.

**CPU / I/O priority - nobody else has it.** rclone has no priority code
and
closed both requests pointing at `ionice`
(rclone/rclone#864). restic's FAQ answers "How
to
prioritize restic's IO and CPU time" entirely with `ionice`/`nice`
recipes
(https://restic.readthedocs.io/en/stable/faq.html). Duplicati still
accepts
`--thread-priority` but the shipping string is "has no effect, use the
operating system controls to set the process priority"
(`Duplicati/Library/Main/Options.cs` at the `v2.3.0.4_stable_2026-07-09`
tag)
- note its published docs page still lists the old text with no
deprecation
notice, so I cited the source, not the doc. Drive for desktop's entire
preference surface is bandwidth rate limits plus pause
(https://support.google.com/drive/answer/13470231), with no priority key
in
the admin policy list. Backblaze offers an automatic/manual bandwidth
throttle
and an upload-thread count

(https://www.backblaze.com/computer-backup/docs/configure-performance-settings-windows).
Bandwidth limiting is not I/O priority and the note says so.

**Parallel scan.** rclone walks at `--checkers` (8) and transfers at
`--transfers` (4) (https://rclone.org/docs/). Duplicati's
`FileEnumerationProcess` is a single serial task with the concurrency
downstream of it, so its walk is a genuine `:x:` (source at the stable
tag).
restic reads at `--read-concurrency`, default 2
(https://restic.readthedocs.io/en/stable/manual_rest.html). Google and
Backblaze document nothing about scan concurrency - hence the new
"not documented" marker rather than a guessed `:x:`.

**Destination-side deletion.** rclone gets a `:white_check_mark:` here,
not an
`:x:`: it keeps no state, so `copy`/`sync` re-list the destination every
run
and re-transfer anything missing
(https://rclone.org/commands/rclone_sync/).
The note says so plainly, including that this is why it is slower on the
incremental case. Drive for desktop is the one that is worse than absent
-
"any files you put in the trash are put in the trash everywhere"
(https://support.google.com/drive/answer/2375102). Duplicati and restic
detect
damage but recovery is operator-driven (Duplicati's `RepairHandler.cs`
refuses
missing dblock files without `--rebuild-missing-dblock-files`; restic's
troubleshooting doc says re-run `backup` to heal,
https://restic.readthedocs.io/en/stable/077_troubleshooting.html).
Backblaze
documents nothing either way, so it is marked not-documented rather than
`:x:`.

**Live rule preview - and one claim I had to walk back.** Duplicati
shipped
server-evaluated inclusion state in its tree UI in 2.3.0.4 on 2026-07-09
(duplicati/duplicati#6955, closing a
six-year-old
request, duplicati/duplicati#4194). Six weeks
ago
"nobody else has this" would have been true; it is not any more, so
Duplicati
gets a `:white_check_mark:` and the note records the remaining
difference
(it marks the nodes you expand, with no whole-source counts). rclone and
restic are `--dry-run` only.

**Logs and diagnostics.** Driven is not unique here and the row shows
that:
Drive for desktop and Backblaze both get `:white_check_mark:`

(https://knowledge.workspace.google.com/admin/drive/capture-google-drive-for-desktop-logs-for-support,

https://www.backblaze.com/computer-backup/docs/send-logs-to-backblaze-windows).
Duplicati is partial - its "Create bug report" bundle is real, but file
logging is opt-in, warnings-only by default, and unrotated. restic has
no
log-file option at all.

**Benchmark suite.** Negative evidence for all five (repo tree listings
plus
code search); the note names Duplicati's unreleased AutoTune harness and
Backblaze's B2-not-client benchmark rather than pretending there is
nothing
adjacent.

No existing rows were removed and none had become false. The trailing
line
now names the exact versions checked instead of just the month.

## Checks

- 40 notes, sequential, every table marker resolves to one and every
note is
  cited (validated by script).
- All 27 table rows have 7 cells.
- Zero em/en dashes or other dash-like non-ASCII; only the superscript
digits
  the file already used. `git ls-files --eol` reports `w/lf`.
- README is not covered by prettier or any markdown linter in CI
(prettier
  runs against `ui/src` only), so there is no formatting gate to run.
- `deploy-landing.yml`'s `TAGLINE_MARKER` is the README's first line,
which is
  untouched.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01JLB3E2Jm7knNJd37fVpH8X

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
pmaxhogan added a commit that referenced this pull request Jul 26, 2026
🤖 I have created a release *beep* *boop*
---


## [2.4.0](v2.3.0...v2.4.0)
(2026-07-26)


### Features

* **cli:** add dump-client-creds to print an account's stored BYO OAuth
client ([#166](#166))
([4b6070b](4b6070b))
* **core:** parallel scan with negation-aware directory pruning
([#169](#169))
([3792e47](3792e47))
* **core:** remote-existence audit heals files whose Drive objects
vanished ([#171](#171))
([5cb8e3a](5cb8e3a))
* **core:** run the scan walk at the configured io_priority
([#173](#173))
([8f39961](8f39961))
* **core:** shape bundle-build file reads with the io_priority setting
([#179](#179))
([af8f048](af8f048))
* **core:** shape upload I/O with the io_priority setting
([#176](#176))
([badd9c9](badd9c9))
* **core:** wire the ioPriority setting to real OS thread priorities
([#170](#170))
([51bb1f3](51bb1f3))
* persist rolling backend logs and capture frontend console into
diagnostics ([#167](#167))
([292e221](292e221))
* real-world benchmark suite comparing driven with rclone
([#178](#178))
([85f6d6a](85f6d6a))
* **ui:** instant exclusion-preview re-evaluation from an in-memory tree
([#177](#177))
([8f49570](8f49570))
* **ui:** sticky shell chrome, indeterminate scan progress and
navigation cleanup
([#163](#163))
([4896336](4896336))
* **ui:** transient in-app toast notifications
([#164](#164))
([0104a8e](0104a8e))
* **ui:** warn when include patterns defeat directory pruning
([#162](#162))
([6d3b4b2](6d3b4b2))


### Bug Fixes

* **core:** self-heal a stale drive_file_id when an update hits a
definitive 404 ([#168](#168))
([8b983b3](8b983b3))


### Performance Improvements

* **core:** per-directory decision cursor for the exclusion preview +
NFC fast path ([#172](#172))
([a587fed](a587fed))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant