Skip to content

Commit 2d9d763

Browse files
pmaxhoganclaude
andauthored
feat: live bottleneck indicator on the Activity dashboard (#311)
## Summary Closes #308. Backend classifier (new `src-tauri/src/bottleneck_hub.rs`, 1s cadence, idle suppression) that names the current limiting pipeline stage: `not_backing_up` / `disk` / `network` / `api` / `cpu` / `mixed`. Streams as a new `sync:bottleneck` event with a `bottleneck_status` hydration command. **Heuristic** (documented in `classify()`'s doc comment, unit-tested over all six states + boundary cases): 1. No account mid-cycle => `not_backing_up`. 2. A mid-cycle account's rate pacer is backing off (or the orchestrator itself is in the `Backoff` circuit-breaker state) => `api`, naming the destination + remaining ms. Checked before rates because a paced account can otherwise show misleadingly healthy numbers between gated requests. 3. Otherwise compare disk-read / net-wire-accepted / blake3-hashed bytes/sec. Stages at or below a 32 KB/s idle floor are dropped from consideration. Among the remaining active stages, the slowest is the bottleneck, provided the fastest active stage clears it by >= 1.5x; short of that there's no clear winner => `mixed`. **Signals wired in:** - Disk/net rates: existing `IoCounters` (`crates/driven-core/src/iostat.rs`), diffed by the sampler - untouched by the AdaptiveController's separate drain-based `ThroughputProbe`. - `api`: new public `AimdPacer::backoff_remaining_ms()` getter (non-blocking read of the private `backoff_until_ms` deadline against the injected clock), exposed through the `Pacer` trait (default `None`) and two new default `Orchestrator` trait methods (`pacer_backoff_remaining_ms`, `backend_label`) so the sampler can poll every account's orchestrator through its existing `Arc<dyn Orchestrator>` handle with no structural changes to orchestrator.rs/executor.rs. - `cpu`: net-new cumulative hash-byte counter added alongside `IoCounters`' existing disk/net fields (`IoCounters::add_hashed`), credited from the upload pipeline's `cpu_stage` (streamed path) and `inline_upload` (buffered small-file path), plus the scanner's deep-verify re-hash (threaded through a new optional `io_counters` param on `scan_with_priority`). A relaxed atomic add on the existing counter - no new hot-path allocation. ## Frontend - `BottleneckStatTile.vue`: a plain-value sibling of `ThroughputStatTile` (same `STAT_TILE` chrome, `dt`/`dd` typography); no sparkline since the backend classifies one current state, not a series. - `stores/bottleneck.ts`: subscribe()/hydrate() pattern like `stores/iostat.ts`, plus a debounce/hysteresis gate - a new state must hold for 5s (measured off the snapshots' own `tsMs`, not `Date.now()`, so it's deterministic under test) before the tile adopts it, so brief flapping never reaches the UI. - Activity stat grid: 7 tiles now (Bottleneck joined), so both the real grid and its loading skeleton moved to `lg:grid-cols-6` (previously mismatched at 5 vs 6). - README: mentions the new tile where the Activity dashboard's features are listed. ## Test plan - [x] `cargo test -p driven-core -p driven-app --lib` - 557 + 455 passed - [x] `cargo clippy -p driven-core -p driven-app --all-targets -- -D warnings` - clean - [x] `cargo fmt --check` - clean - [x] `pnpm -C ui run test:unit` - 805 passed (61 files), including new `bottleneck-store.test.ts` (7 tests) and `bottleneck-stat-tile.test.ts` (9 tests) - [x] `npx vue-tsc --noEmit` / `eslint` / `prettier --check` - clean - [x] Visual baselines regenerated via the `just visual-update` Docker image, scoped to `activity.spec.ts` (light/dark x populated/empty/loading; `error.png` unchanged) - screenshots reviewed, tile renders correctly in both themes at the new 6-column grid. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_019xKUm9vH4ifb5LHR5szy1v --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent 7e87341 commit 2d9d763

29 files changed

Lines changed: 1888 additions & 16 deletions

File tree

README.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -180,10 +180,12 @@ These move: check each project's current docs before relying on a cell.
180180
snapshot (Settings > macOS), which does not help with a Full Disk Access
181181
denial; there is no Linux equivalent.
182182
- In-app restore browser with full-text file-name search and streaming decrypt.
183-
- Activity dashboard with a live tail, filterable history, and real-time
183+
- Activity dashboard with a live tail, filterable history, real-time
184184
disk-read and network-upload throughput graphs (probe-fed, one-second
185185
resolution - they move during every phase of a backup, including crash
186-
recovery).
186+
recovery), and a live Bottleneck tile naming which stage - disk, network,
187+
a rate-limited destination, or CPU hashing - is presently the limiting
188+
factor (debounced a few seconds so it does not flicker between readings).
187189
- Rolling local log files covering both the backend and the webview console,
188190
collected into a one-click diagnostics bundle alongside a redacted summary of
189191
in-flight upload recovery state and a trailing window of process-memory

crates/driven-core/src/executor.rs

Lines changed: 74 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2746,6 +2746,10 @@ impl DefaultExecutor {
27462746
.map_err(UploadError::from_read)?;
27472747
if let Some(io) = self.io_counters.as_ref() {
27482748
io.add_disk_read(plaintext_len);
2749+
// issue #308 bottleneck classifier: this buffered path both reads
2750+
// and blake3-hashes the whole file in one pass, so both counters
2751+
// move together here.
2752+
io.add_hashed(plaintext_len);
27492753
}
27502754

27512755
// --- post-read fstat identity check (SPEC s8 defence #3) -----------
@@ -2862,7 +2866,7 @@ impl DefaultExecutor {
28622866
self.mem_gauge.clone(),
28632867
self.io_counters.clone(),
28642868
);
2865-
let cpu = cpu_stage(raw_rx, out_tx, crypto, size);
2869+
let cpu = cpu_stage(raw_rx, out_tx, crypto, size, self.io_counters.clone());
28662870
let uploader = self.upload_stage(
28672871
target,
28682872
existing_file_id,
@@ -7292,20 +7296,29 @@ async fn cpu_stage(
72927296
out_tx: tokio::sync::mpsc::Sender<Bytes>,
72937297
crypto: Option<Arc<dyn SourceCryptoSuite>>,
72947298
size: u64,
7299+
// issue #308 bottleneck classifier: credited alongside the disk/net
7300+
// counters the reader/uploader stages already feed, so the "cpu" state
7301+
// has a real hash-bytes/sec rate to compare against them.
7302+
io_counters: Option<Arc<crate::iostat::IoCounters>>,
72957303
) -> Result<CpuOutput, StageError> {
72967304
use md5::{Digest, Md5};
72977305

72987306
let mut hasher = blake3::Hasher::new();
72997307
let use_rayon = size >= RAYON_HASH_THRESHOLD;
73007308
let mut md5 = Md5::new();
73017309

7302-
// Hash a plaintext chunk into blake3, multi-core for big files.
7310+
// Hash a plaintext chunk into blake3, multi-core for big files. Credits
7311+
// the hash counter (issue #308) on every chunk regardless of path
7312+
// (encrypted or not) since both arms below call this closure.
73037313
let hash_chunk = |h: &mut blake3::Hasher, chunk: &[u8]| {
73047314
if use_rayon {
73057315
h.update_rayon(chunk);
73067316
} else {
73077317
h.update(chunk);
73087318
}
7319+
if let Some(io) = io_counters.as_ref() {
7320+
io.add_hashed(chunk.len() as u64);
7321+
}
73097322
};
73107323

73117324
if let Some(suite) = crypto {
@@ -13265,6 +13278,65 @@ mod tests {
1326513278
);
1326613279
}
1326713280

13281+
/// issue #308 bottleneck classifier: a fresh small-file upload (the
13282+
/// BUFFERED `inline_upload` path, below [`PIPELINE_THRESHOLD`]) credits
13283+
/// the hash counter with the whole plaintext, alongside the existing
13284+
/// disk-read credit - both happen in the same `read_hash_encrypt` pass.
13285+
#[tokio::test]
13286+
async fn inline_upload_credits_hashed_bytes() {
13287+
let h = harness().await;
13288+
let body = vec![7u8; 4096];
13289+
let (rel, size) = h.write_file("small.bin", &body);
13290+
13291+
let io = Arc::new(crate::iostat::IoCounters::default());
13292+
let exec = h.executor().with_io_counters(io.clone());
13293+
let out = exec
13294+
.execute(
13295+
&h.source,
13296+
&h.upload_plan(&rel, size),
13297+
&noop_progress,
13298+
&noop_outcome,
13299+
)
13300+
.await
13301+
.unwrap();
13302+
assert!(matches!(out[0], OpOutcome::Done { .. }), "got {:?}", out[0]);
13303+
13304+
let snap = io.snapshot();
13305+
assert_eq!(snap.hashed_bytes, size, "the whole plaintext was hashed");
13306+
assert_eq!(snap.disk_read_bytes, size, "and read from disk");
13307+
}
13308+
13309+
/// issue #308 bottleneck classifier: a fresh large-file upload (the
13310+
/// STREAMING `cpu_stage` path, at/above [`PIPELINE_THRESHOLD`]) credits
13311+
/// the hash counter chunk-by-chunk as it streams, summing to the whole
13312+
/// plaintext by the time the upload completes.
13313+
#[tokio::test]
13314+
async fn stream_upload_credits_hashed_bytes() {
13315+
let h = harness().await;
13316+
let size_bytes = (PIPELINE_THRESHOLD + 64 * 1024) as usize;
13317+
let body: Vec<u8> = (0..size_bytes).map(|i| (i % 251) as u8).collect();
13318+
let (rel, size) = h.write_file("streamed.bin", &body);
13319+
13320+
let io = Arc::new(crate::iostat::IoCounters::default());
13321+
let exec = h.executor().with_io_counters(io.clone());
13322+
let out = exec
13323+
.execute(
13324+
&h.source,
13325+
&h.upload_plan(&rel, size),
13326+
&noop_progress,
13327+
&noop_outcome,
13328+
)
13329+
.await
13330+
.unwrap();
13331+
assert!(matches!(out[0], OpOutcome::Done { .. }), "got {:?}", out[0]);
13332+
13333+
let snap = io.snapshot();
13334+
assert_eq!(
13335+
snap.hashed_bytes, size,
13336+
"every streamed chunk's bytes were credited to the hash counter"
13337+
);
13338+
}
13339+
1326813340
/// [`ResumeAcc`]'s gauge accounting is symmetric across push / partial
1326913341
/// drain / clear, the drain clamps at the buffered length, and DROP
1327013342
/// refunds whatever is left - the guarantee the error-unwind test relies

crates/driven-core/src/iostat.rs

Lines changed: 27 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -21,11 +21,17 @@
2121
//! completion for single-request uploads). Each byte is credited exactly
2222
//! once; bundle members are covered by their bundle's wire push, never
2323
//! double-counted at completion.
24+
//! - `hashed`: plaintext bytes blake3-hashed (issue #308 bottleneck
25+
//! classifier, 2026-08-17 follow-up). Credited from the two hot hashing
26+
//! paths - the upload pipeline's cpu stage (streamed and buffered) and the
27+
//! scanner's deep-verify re-hash - so the "cpu" bottleneck state has a real
28+
//! rate to compare against `disk_read` and `net_wire`. Deliberately its own
29+
//! counter rather than folded into `disk_read`: a deep-verify re-hash of an
30+
//! already-synced file hashes bytes without any corresponding upload, so
31+
//! conflating the two would make a hash-only scan look like disk activity.
2432
//!
25-
//! v1 scope notes: the scanner's deep-verify hashing and the restore path do
26-
//! not credit `disk_read` yet, and bundle ASSEMBLY reads (tar-ing members)
27-
//! are approximated by the bundle's wire push rather than counted at read
28-
//! time.
33+
//! v1 scope notes: bundle ASSEMBLY reads (tar-ing members) are approximated
34+
//! by the bundle's wire push rather than counted at read time.
2935
3036
use std::sync::atomic::{AtomicU64, Ordering};
3137

@@ -36,6 +42,7 @@ use std::sync::atomic::{AtomicU64, Ordering};
3642
pub struct IoCounters {
3743
disk_read: AtomicU64,
3844
net_wire: AtomicU64,
45+
hashed: AtomicU64,
3946
}
4047

4148
/// One peek of the cumulative totals.
@@ -45,6 +52,8 @@ pub struct IoSnapshot {
4552
pub disk_read_bytes: u64,
4653
/// Total wire bytes accepted by the destination.
4754
pub net_wire_bytes: u64,
55+
/// Total plaintext bytes blake3-hashed (issue #308).
56+
pub hashed_bytes: u64,
4857
}
4958

5059
impl IoCounters {
@@ -58,11 +67,20 @@ impl IoCounters {
5867
self.net_wire.fetch_add(n, Ordering::Relaxed);
5968
}
6069

61-
/// Peek both totals. Never resets - samplers diff consecutive snapshots.
70+
/// Credit `n` plaintext bytes blake3-hashed (issue #308 bottleneck
71+
/// classifier's cpu signal). A single relaxed atomic add on the same
72+
/// buffer the hashing path already owns - zero measurable overhead in
73+
/// the hot loop.
74+
pub fn add_hashed(&self, n: u64) {
75+
self.hashed.fetch_add(n, Ordering::Relaxed);
76+
}
77+
78+
/// Peek all totals. Never resets - samplers diff consecutive snapshots.
6279
pub fn snapshot(&self) -> IoSnapshot {
6380
IoSnapshot {
6481
disk_read_bytes: self.disk_read.load(Ordering::Relaxed),
6582
net_wire_bytes: self.net_wire.load(Ordering::Relaxed),
83+
hashed_bytes: self.hashed.load(Ordering::Relaxed),
6684
}
6785
}
6886
}
@@ -78,15 +96,18 @@ mod tests {
7896
c.snapshot(),
7997
IoSnapshot {
8098
disk_read_bytes: 0,
81-
net_wire_bytes: 0
99+
net_wire_bytes: 0,
100+
hashed_bytes: 0,
82101
}
83102
);
84103
c.add_disk_read(100);
85104
c.add_net_wire(40);
86105
c.add_disk_read(1);
106+
c.add_hashed(7);
87107
let s1 = c.snapshot();
88108
assert_eq!(s1.disk_read_bytes, 101);
89109
assert_eq!(s1.net_wire_bytes, 40);
110+
assert_eq!(s1.hashed_bytes, 7);
90111
// Peek-only: a second reader sees the same cumulative totals.
91112
assert_eq!(c.snapshot(), s1);
92113
}

0 commit comments

Comments
 (0)