Skip to content

Commit 8ecced6

Browse files
pmaxhoganclaude
andauthored
feat(core): adaptive upload parallelism with throughput probe and disk-saturation gate (#143)
Adaptive upload parallelism (DESIGN §11.4.7 / §18.2), default-on with a kill-switch. Closes the control loop around the one Drive concurrency knob that matters - how many files are in flight at once. A fixed pool is a guess: too few wastes the link, too many overloads Drive's edge so each upload takes longer and *net* throughput falls (§11.4.7's pathological case). The pool now floats within `[1, 32]`, driven by measured throughput and a per-OS disk-busy gate. ## DESIGN conformance - **§11.4.7 ThroughputProbe** - a lock-free byte accumulator the executor feeds at every completed upload; the controller drains it once per 30 s window and divides by the injected-clock interval for aggregate bytes/sec. Shrinks when a window's throughput collapses below 50% of the previous one; grows while lifting the pool is still paying off. - **§18.2 disk-saturation signal** - sampled every 5 s, per-OS, in-process; `> 80 %` busy = saturated. Gates growth (and blocks a shrink from firing on a disk-bound drop). - **§11.4.2 bounds** - hard cap 32, floor 1. Start size = the user's `default_concurrent_uploads` (finally wired), else `min(available_parallelism*2, 16)`. - **Kill-switch** - `adaptive_parallelism_enabled`, default **true**. Off = today's exact fixed-pool behavior (no controller is built; the pool stays pinned at the start size). ## Controller rules (pure `decide`, exhaustively unit-tested) | Window condition | Decision | |---|---| | Pool was not the bottleneck, or zero throughput (non-representative window) | **Hold** | | Throughput < 50% of previous, pacer not throttling, disk not saturated, above floor | **Shrink** | | Throughput still improving (> 1.05× previous, or the first bootstrap window), disk has headroom, not throttling, below cap | **Grow** | | Plateau, or any gate blocks | **Hold** | Additive-increase / multiplicative-decrease: growth continues only while it pays off and settles at a plateau. A pacer throttle or disk saturation *explains* a throughput drop, so neither triggers a shrink. ### Resolved ambiguities - **"At the pool's ceiling"** (§11.4.7): operationalized as *pool pinned AND throughput still improving window-over-window*. A literal "pinned" reading would grow every steady window straight to the cap; requiring improvement gives a stable AIMD loop. - **"disk + CPU headroom"**: §18.2 defines a concrete signal only for the disk. Uploads are network-bound and their CPU work (hash/encrypt) runs on a separate rayon pool sized to leave a reactor core (§11.4.5), so the disk gate is the operative hardware-headroom signal; no separate dynamic CPU probe is introduced. ## Per-OS disk-busy backends (`driven-diskstat`, mirrors `driven-power`'s cfg-gated shape) - **Windows** - PDH `\PhysicalDisk(_Total)\% Disk Time`, added via `PdhAddEnglishCounterW` (locale-independent), `PDH_FMT_NOCAP100` so a genuinely-over-100% `_Total` reads honestly, `/100` → fraction. Compiles and passes `clippy -D warnings` on a real Windows host; the CI `tauri-compile` Windows matrix compiles it as well. - **Linux** - `/proc/diskstats` field 10 ("time spent doing I/Os", ms) delta for the device backing the source root (resolved via `stat(2)` `st_dev` → major:minor), over the wall-clock interval. - **macOS** - IOKit `IOBlockStorageDriver` `Statistics` `Total Time (Read/Write)` ns deltas summed across drivers (best-effort; can over-report, which is safe - it only holds the pool at its start size, never strangles it). - **Fail-open (load-bearing)** - any unreadable reader (no baseline, parse/FFI error, unsupported target) returns `Unknown`, which maps to *not saturated*. A broken disk reader must never pin the pool small. **Known limitation:** the disk-busy reader binds to the first source's root, so a multi-source account spanning different physical disks monitors only the first. Fail-open keeps this harmless (the worst case degrades to the fixed-pool behavior); a per-source disk gate is a future refinement. ## Resize mechanism The pool is a resizable `tokio::sync::Semaphore`. **Grow** = `add_permits(1)`. **Shrink** = acquire one permit and `forget()` it, done inline under a 100 ms timeout (never a detached task - the repo forbids orphanable spawns); on timeout the size is left honest and the next window retries. The executor and controller share the *same* `Arc<UploadPool>`, so a resize is immediately seen by the executor's per-file acquire path. ## Tests Full workspace suite green on a Windows host: `clippy --workspace --all-targets -D warnings` clean, `cargo test --workspace` all-pass (both adaptive e2e tests included), `cargo fmt --check` clean; UI `lint` / `vue-tsc` / 266 vitest specs green. The Linux `/proc/diskstats` parser and the macOS IOKit backend compile per-target under CI's Linux test job + the 3-OS `tauri-compile` matrix. - Pure `decide`: every branch (hold/shrink/grow, each gate, floor/cap, bootstrap) covered. - `UploadPool` grow-to-cap / shrink-to-floor / contention counting. - `AdaptiveController` end-to-end on a `FakeClock` + `FakeDiskBusyProbe`: shrinks under induced latency then recovers (the §11.4.7 acceptance behavior, deterministically, no real time); holds below a full window; does not shrink on a throttle-explained drop. - `e2e_fake` wiring seam: a real executor running a real multi-file plan feeds the injected `ThroughputProbe` the full uploaded byte total and passes uploads through the injected pool (the ROADMAP M3 adaptive-parallelism acceptance row). - UI: a Settings mount test toggles the kill-switch and asserts the emitted patch. Refs #34 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent d9c3161 commit 8ecced6

27 files changed

Lines changed: 2562 additions & 81 deletions

File tree

Cargo.lock

Lines changed: 12 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Cargo.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ members = [
55
"crates/driven-drive",
66
"crates/driven-crypto",
77
"crates/driven-power",
8+
"crates/driven-diskstat",
89
"crates/driven-vss",
910
"crates/driven-vss-helper",
1011
"crates/driven-net",

crates/driven-core/Cargo.toml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,10 @@ sqlx = { version = "0.9", default-features = false, features = [
4747
driven-drive = { path = "../driven-drive" }
4848
driven-crypto = { path = "../driven-crypto" }
4949
driven-power = { path = "../driven-power" }
50+
# Per-OS disk-busy reader for the adaptive upload-parallelism controller
51+
# (DESIGN s11.4.7 / s18.2). Same acyclic shape as driven-power: it carries no
52+
# driven-core dep.
53+
driven-diskstat = { path = "../driven-diskstat" }
5054
# M3.5 Windows VSS reads for exclusively-locked files (ROADMAP M3.5, DESIGN
5155
# s5.3). The executor consults the `VssProvider` seam on its open path; the
5256
# orchestrator owns the per-cycle snapshot lifecycle + orphan cleanup. The

0 commit comments

Comments
 (0)