Commit 8ecced6
feat(core): adaptive upload parallelism with throughput probe and disk-saturation gate (#143)
Adaptive upload parallelism (DESIGN §11.4.7 / §18.2), default-on with a
kill-switch.
Closes the control loop around the one Drive concurrency knob that
matters - how many files are in flight at once. A fixed pool is a guess:
too few wastes the link, too many overloads Drive's edge so each upload
takes longer and *net* throughput falls (§11.4.7's pathological case).
The pool now floats within `[1, 32]`, driven by measured throughput and
a per-OS disk-busy gate.
## DESIGN conformance
- **§11.4.7 ThroughputProbe** - a lock-free byte accumulator the
executor feeds at every completed upload; the controller drains it once
per 30 s window and divides by the injected-clock interval for aggregate
bytes/sec. Shrinks when a window's throughput collapses below 50% of the
previous one; grows while lifting the pool is still paying off.
- **§18.2 disk-saturation signal** - sampled every 5 s, per-OS,
in-process; `> 80 %` busy = saturated. Gates growth (and blocks a shrink
from firing on a disk-bound drop).
- **§11.4.2 bounds** - hard cap 32, floor 1. Start size = the user's
`default_concurrent_uploads` (finally wired), else
`min(available_parallelism*2, 16)`.
- **Kill-switch** - `adaptive_parallelism_enabled`, default **true**.
Off = today's exact fixed-pool behavior (no controller is built; the
pool stays pinned at the start size).
## Controller rules (pure `decide`, exhaustively unit-tested)
| Window condition | Decision |
|---|---|
| Pool was not the bottleneck, or zero throughput (non-representative
window) | **Hold** |
| Throughput < 50% of previous, pacer not throttling, disk not
saturated, above floor | **Shrink** |
| Throughput still improving (> 1.05× previous, or the first bootstrap
window), disk has headroom, not throttling, below cap | **Grow** |
| Plateau, or any gate blocks | **Hold** |
Additive-increase / multiplicative-decrease: growth continues only while
it pays off and settles at a plateau. A pacer throttle or disk
saturation *explains* a throughput drop, so neither triggers a shrink.
### Resolved ambiguities
- **"At the pool's ceiling"** (§11.4.7): operationalized as *pool pinned
AND throughput still improving window-over-window*. A literal "pinned"
reading would grow every steady window straight to the cap; requiring
improvement gives a stable AIMD loop.
- **"disk + CPU headroom"**: §18.2 defines a concrete signal only for
the disk. Uploads are network-bound and their CPU work (hash/encrypt)
runs on a separate rayon pool sized to leave a reactor core (§11.4.5),
so the disk gate is the operative hardware-headroom signal; no separate
dynamic CPU probe is introduced.
## Per-OS disk-busy backends (`driven-diskstat`, mirrors
`driven-power`'s cfg-gated shape)
- **Windows** - PDH `\PhysicalDisk(_Total)\% Disk Time`, added via
`PdhAddEnglishCounterW` (locale-independent), `PDH_FMT_NOCAP100` so a
genuinely-over-100% `_Total` reads honestly, `/100` → fraction. Compiles
and passes `clippy -D warnings` on a real Windows host; the CI
`tauri-compile` Windows matrix compiles it as well.
- **Linux** - `/proc/diskstats` field 10 ("time spent doing I/Os", ms)
delta for the device backing the source root (resolved via `stat(2)`
`st_dev` → major:minor), over the wall-clock interval.
- **macOS** - IOKit `IOBlockStorageDriver` `Statistics` `Total Time
(Read/Write)` ns deltas summed across drivers (best-effort; can
over-report, which is safe - it only holds the pool at its start size,
never strangles it).
- **Fail-open (load-bearing)** - any unreadable reader (no baseline,
parse/FFI error, unsupported target) returns `Unknown`, which maps to
*not saturated*. A broken disk reader must never pin the pool small.
**Known limitation:** the disk-busy reader binds to the first source's
root, so a multi-source account spanning different physical disks
monitors only the first. Fail-open keeps this harmless (the worst case
degrades to the fixed-pool behavior); a per-source disk gate is a future
refinement.
## Resize mechanism
The pool is a resizable `tokio::sync::Semaphore`. **Grow** =
`add_permits(1)`. **Shrink** = acquire one permit and `forget()` it,
done inline under a 100 ms timeout (never a detached task - the repo
forbids orphanable spawns); on timeout the size is left honest and the
next window retries. The executor and controller share the *same*
`Arc<UploadPool>`, so a resize is immediately seen by the executor's
per-file acquire path.
## Tests
Full workspace suite green on a Windows host: `clippy --workspace
--all-targets -D warnings` clean, `cargo test --workspace` all-pass
(both adaptive e2e tests included), `cargo fmt --check` clean; UI `lint`
/ `vue-tsc` / 266 vitest specs green. The Linux `/proc/diskstats` parser
and the macOS IOKit backend compile per-target under CI's Linux test job
+ the 3-OS `tauri-compile` matrix.
- Pure `decide`: every branch (hold/shrink/grow, each gate, floor/cap,
bootstrap) covered.
- `UploadPool` grow-to-cap / shrink-to-floor / contention counting.
- `AdaptiveController` end-to-end on a `FakeClock` +
`FakeDiskBusyProbe`: shrinks under induced latency then recovers (the
§11.4.7 acceptance behavior, deterministically, no real time); holds
below a full window; does not shrink on a throttle-explained drop.
- `e2e_fake` wiring seam: a real executor running a real multi-file plan
feeds the injected `ThroughputProbe` the full uploaded byte total and
passes uploads through the injected pool (the ROADMAP M3
adaptive-parallelism acceptance row).
- UI: a Settings mount test toggles the kill-switch and asserts the
emitted patch.
Refs #34
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>1 parent d9c3161 commit 8ecced6
27 files changed
Lines changed: 2562 additions & 81 deletions
File tree
- crates
- driven-core
- src
- migrations
- tests
- driven-diskstat
- src
- driven-test-fixtures
- src
- src-tauri
- src
- commands
- ui/src
- __tests__
- ipc
- locales
- views
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
| 8 | + | |
8 | 9 | | |
9 | 10 | | |
10 | 11 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
47 | 47 | | |
48 | 48 | | |
49 | 49 | | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
50 | 54 | | |
51 | 55 | | |
52 | 56 | | |
| |||
0 commit comments