Skip to content

feat(core): adaptive upload parallelism with throughput probe and disk-saturation gate - #143

Merged
pmaxhogan merged 1 commit into
mainfrom
feat/adaptive-parallelism
Jul 20, 2026
Merged

pmaxhogan merged 1 commit into
mainfrom
feat/adaptive-parallelism

Conversation

@pmaxhogan

Copy link
Copy Markdown
Owner

Adaptive upload parallelism (DESIGN §11.4.7 / §18.2), default-on with a kill-switch.

Closes the control loop around the one Drive concurrency knob that matters - how many files are in flight at once. A fixed pool is a guess: too few wastes the link, too many overloads Drive's edge so each upload takes longer and net throughput falls (§11.4.7's pathological case). The pool now floats within [1, 32], driven by measured throughput and a per-OS disk-busy gate.

DESIGN conformance

  • §11.4.7 ThroughputProbe - a lock-free byte accumulator the executor feeds at every completed upload; the controller drains it once per 30 s window and divides by the injected-clock interval for aggregate bytes/sec. Shrinks when a window's throughput collapses below 50% of the previous one; grows while lifting the pool is still paying off.
  • §18.2 disk-saturation signal - sampled every 5 s, per-OS, in-process; > 80 % busy = saturated. Gates growth (and blocks a shrink from firing on a disk-bound drop).
  • §11.4.2 bounds - hard cap 32, floor 1. Start size = the user's default_concurrent_uploads (finally wired), else min(available_parallelism*2, 16).
  • Kill-switch - adaptive_parallelism_enabled, default true. Off = today's exact fixed-pool behavior (no controller is built; the pool stays pinned at the start size).

Controller rules (pure decide, exhaustively unit-tested)

Window condition Decision
Pool was not the bottleneck, or zero throughput (non-representative window) Hold
Throughput < 50% of previous, pacer not throttling, disk not saturated, above floor Shrink
Throughput still improving (> 1.05× previous, or the first bootstrap window), disk has headroom, not throttling, below cap Grow
Plateau, or any gate blocks Hold

Additive-increase / multiplicative-decrease: growth continues only while it pays off and settles at a plateau. A pacer throttle or disk saturation explains a throughput drop, so neither triggers a shrink.

Resolved ambiguities

  • "At the pool's ceiling" (§11.4.7): operationalized as pool pinned AND throughput still improving window-over-window. A literal "pinned" reading would grow every steady window straight to the cap; requiring improvement gives a stable AIMD loop.
  • "disk + CPU headroom": §18.2 defines a concrete signal only for the disk. Uploads are network-bound and their CPU work (hash/encrypt) runs on a separate rayon pool sized to leave a reactor core (§11.4.5), so the disk gate is the operative hardware-headroom signal; no separate dynamic CPU probe is introduced.

Per-OS disk-busy backends (driven-diskstat, mirrors driven-power's cfg-gated shape)

  • Windows - PDH \PhysicalDisk(_Total)\% Disk Time, added via PdhAddEnglishCounterW (locale-independent), PDH_FMT_NOCAP100 so a genuinely-over-100% _Total reads honestly, /100 → fraction. Compiles and passes clippy -D warnings on a real Windows host; the CI tauri-compile Windows matrix compiles it as well.
  • Linux - /proc/diskstats field 10 ("time spent doing I/Os", ms) delta for the device backing the source root (resolved via stat(2) st_dev → major:minor), over the wall-clock interval.
  • macOS - IOKit IOBlockStorageDriver Statistics Total Time (Read/Write) ns deltas summed across drivers (best-effort; can over-report, which is safe - it only holds the pool at its start size, never strangles it).
  • Fail-open (load-bearing) - any unreadable reader (no baseline, parse/FFI error, unsupported target) returns Unknown, which maps to not saturated. A broken disk reader must never pin the pool small.

Known limitation: the disk-busy reader binds to the first source's root, so a multi-source account spanning different physical disks monitors only the first. Fail-open keeps this harmless (the worst case degrades to the fixed-pool behavior); a per-source disk gate is a future refinement.

Resize mechanism

The pool is a resizable tokio::sync::Semaphore. Grow = add_permits(1). Shrink = acquire one permit and forget() it, done inline under a 100 ms timeout (never a detached task - the repo forbids orphanable spawns); on timeout the size is left honest and the next window retries. The executor and controller share the same Arc<UploadPool>, so a resize is immediately seen by the executor's per-file acquire path.

Tests

Full workspace suite green on a Windows host: clippy --workspace --all-targets -D warnings clean, cargo test --workspace all-pass (both adaptive e2e tests included), cargo fmt --check clean; UI lint / vue-tsc / 266 vitest specs green. The Linux /proc/diskstats parser and the macOS IOKit backend compile per-target under CI's Linux test job + the 3-OS tauri-compile matrix.

  • Pure decide: every branch (hold/shrink/grow, each gate, floor/cap, bootstrap) covered.
  • UploadPool grow-to-cap / shrink-to-floor / contention counting.
  • AdaptiveController end-to-end on a FakeClock + FakeDiskBusyProbe: shrinks under induced latency then recovers (the §11.4.7 acceptance behavior, deterministically, no real time); holds below a full window; does not shrink on a throttle-explained drop.
  • e2e_fake wiring seam: a real executor running a real multi-file plan feeds the injected ThroughputProbe the full uploaded byte total and passes uploads through the injected pool (the ROADMAP M3 adaptive-parallelism acceptance row).
  • UI: a Settings mount test toggles the kill-switch and asserts the emitted patch.

Refs #34

🤖 Generated with Claude Code

@github-project-automation github-project-automation Bot moved this to Todo in Driven Jul 20, 2026
@pmaxhogan
pmaxhogan force-pushed the feat/adaptive-parallelism branch from 12517ae to d0df17f Compare July 20, 2026 20:20
@github-actions

github-actions Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Coverage

Area main this PR delta
Rust (lib crates) 78.81% 79.01% +0.20 (OK)
UI (vue/ts) 89.49% 89.52% +0.03 (OK)

Gate: passed - no coverage regression (epsilon 0.1 pp).

@pmaxhogan
pmaxhogan force-pushed the feat/adaptive-parallelism branch from d0df17f to 198ba1d Compare July 20, 2026 21:08
@pmaxhogan
pmaxhogan enabled auto-merge (squash) July 20, 2026 21:10
…k-saturation gate

Close the control loop around the in-flight-file count (DESIGN 11.4.7 /
18.2). A 30s-window ThroughputProbe fed by the executor drives a resizable
UploadPool within [1, 32]: shrink when a window's throughput collapses
below 50% of the prior one (and neither the pacer nor the disk explains the
drop), grow while lifting the pool still improves throughput. A per-OS
disk-busy gate (new driven-diskstat crate: PDH on Windows, /proc/diskstats
on Linux, IOKit on macOS; fail-open to not-saturated so a broken reader
never strangles uploads) blocks growth when the disk is the bound.

Default-on with an adaptive_parallelism_enabled kill-switch; off = today's
fixed pool at default_concurrent_uploads (now wired as the start size). The
control law is a pure, exhaustively unit-tested decide(); an e2e_fake test
asserts a real shrink-then-regrow transition through the shared
executor/probe/pool wiring.

Refs #34

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QZQVP2tUuTLh8oL31D8heC
@pmaxhogan
pmaxhogan force-pushed the feat/adaptive-parallelism branch from 198ba1d to 9971dc3 Compare July 20, 2026 21:15
@pmaxhogan
pmaxhogan merged commit 8ecced6 into main Jul 20, 2026
18 checks passed
@pmaxhogan
pmaxhogan deleted the feat/adaptive-parallelism branch July 20, 2026 21:36
@github-project-automation github-project-automation Bot moved this from Todo to Done in Driven Jul 20, 2026
pmaxhogan added a commit that referenced this pull request Jul 20, 2026
The FS mtime-granularity write-stat probe ran inline on a source's FIRST
scan. That first scan is the source's initial backup cycle, where every file
is new and each path's first upload must commit its file_state row cleanly.
The probe's stat/sleep I/O (up to ~2.2s on a coarse FS) plus temp-file churn
in the source root added latency/contention to that hot path; on the
windows-latest hermetic chaos runner it tipped a pre-existing
create-during-upload race - the executor's SkipPostUpload on a CREATE leaves
a live orphan object the once-per-boot reconcile never adopts mid-run - into
a duplicate upload, failing the `frequent-edits` scenario ("expected exactly
1 object after edit soak, found 2"). The underlying orphan race is pre-existing
and tracked separately (#144).

Defer the probe: the first scan (no completed scan yet -> last_full_scan_at
is None) trusts mtime with no probe and no coarse fallback, persisting
nothing; the probe runs on the next scan, once a completed scan exists. This
mirrors the existing probe-failure degradation ("trust mtime this cycle,
re-probe next") and costs at most a one-cycle delay before the coarse
fallback first engages on a coarse FS (DESIGN s5.2). Adds a scanner-level
regression test that scan #1 performs no probe and no coarse-fallback re-hash
(and that the probe runs from scan #2).

Also rebases onto origin/main (#142 shared-drive, #143 adaptive parallelism)
and renumbers the migration 0009_mtime_granularity.sql ->
0012_mtime_granularity.sql (contiguous above the applied 0011; sqlx
auto-discovers by filename). Regenerates the .sqlx query cache for the merged
backup_sources INSERTs that now carry both drive_id and mtime_granularity_ns.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QZQVP2tUuTLh8oL31D8heC
pmaxhogan added a commit that referenced this pull request Jul 24, 2026
🤖 I have created a release *beep* *boop*
---


## [2.1.0](v2.0.1...v2.1.0)
(2026-07-24)


### Features

* **core:** adaptive upload parallelism with throughput probe and
disk-saturation gate
([#143](#143))
([8ecced6](8ecced6))
* **core:** filesystem timestamp-granularity probe with ctime fallback
and per-directory gitignore cascade
([#141](#141))
([344262c](344262c))
* **drive:** support Google Shared Drive destinations end-to-end
([#142](#142))
([d9c3161](d9c3161))
* **net:** native OS reachability backends with automatic fallback
([#138](#138))
([319e85f](319e85f))
* **net:** SOCKS5 and PAC proxy support for all outbound connections
([#145](#145))
([2f0b7d1](2f0b7d1))
* **net:** support a custom corporate root CA for all outbound
connections ([#134](#134))
([929e93d](929e93d))
* per-source toggle to back up OneDrive cloud-only placeholder files
([#133](#133))
([6863ea3](6863ea3))
* **telemetry:** capture latency percentiles and add rollup query
endpoint ([#132](#132))
([4e9fde6](4e9fde6))
* **telemetry:** preview exactly what a telemetry ping sends
([#139](#139))
([95fbd9a](95fbd9a))


### Bug Fixes

* **core:** commit file_state for a create that skipped post-upload so
the next scan updates instead of re-creating
([#146](#146))
([f5230d1](f5230d1))
* **deps:** bump tauri-winrt-notification to drop vulnerable quick-xml
(closes [#89](#89))
([#129](#129))
([232fd8f](232fd8f))
* **telemetry:** exclude pre-schema rows from latency rollup
([#137](#137))
([1ae6220](1ae6220))
* **ui:** add cursor pointer to buttons and link-buttons
([#136](#136))
([dbd4809](dbd4809))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant