fix(core): stream the resumable-upload resume instead of buffering the whole file (#279)

## Incident (2026-08-14)

Quitting Driven mid-upload of an 88.6 GB file (`dev-drives/dev.vhdx`)
and relaunching made the app consume >10 GB of RAM in ~30 s (at disk
read speed) until Windows killed it - and it repeated on every relaunch.
Diagnostics zip + the laptop's `pending_ops` row confirmed the
mechanism:

- The startup **reconcile** (runs before any scan, so the UI still shows
the indeterminate state) found the persisted resumable session
(`size=88655003648`, `acked_offset=7436500992`), passed the
resume-identity gate, and called `read_hash_encrypt` - which **buffers
the entire upload body into one `Vec`**. The live upload path has
streamed with bounded channels since DESIGN s11.4.3; only the
restart-resume path (and the adopt re-hash) still used the legacy
buffered read.
- The OOM kill happens before `delete_pending_op`, so the op survives
and every launch retries the same 88 GB read.
- The whole fatal phase logged **nothing** at INFO - the diagnostics
bundle was blind to it.

## Fix

- **`resume_persisted` now streams** via a pass-based design: one
sequential read that hashes every byte, discards the already-acked
prefix, and pushes the tail in bounded wire chunks (~2 in flight,
`ResumeAcc` RAII-guarded MemGauge accounting that stays balanced even on
`?` error unwinds). The pacer is charged only for pushed wire bytes.
- **Backend offset semantics** are interpreted against the streaming
window (`offset` always equals the file position of the accumulator's
first byte): driven-s3's one-shot post-hydration rewind-to-0 and
sftp/localfs resyncs RESTART the pass from the store's stated offset
(one-restart budget); in-window forward offsets drain exactly the newly
covered bytes; an exact stall abandons. Without this, every S3 resume
with prior progress degraded to a full re-upload.
- **Legacy rows** (recorded hash, no identity) are verified by a
hash-only prepass before any byte is pushed; identity-carrying rows get
a cheap EOF re-fstat that aborts a mid-stream-modified file before the
finalizing chunk (md5-vs-store remains the correctness backstop).
- **`rehash_local_plaintext` streams** (hash-only): its old callee's
plaintext arm buffered the whole body despite the comment claiming
otherwise.
- **Excluded paths**: reconcile skips only the *resume* for a path the
user has since excluded and lets adopt-or-requeue run - a finalized
object is adopted into a `file_state` row (never left as an invisible
untracked orphan on the remote), an unfinalized one requeues into
nothing (the next scan excludes the path; the abandoned session is
GC'd).
- **`push_chunks`** (fresh-session uploads) gets the same no-progress
stall guard the resume path has.

## Instrumentation (make the next diagnostics zip self-diagnosing)

- INFO breadcrumbs: cycle start with tick source, orchestrator state
transitions (variant *name* only - never the Error variant's free-text
details, which could leak partially-redacted paths), reconcile entry
with pending-op count, resume start/abandon/restart with sizes + offsets
+ reasons.
- RSS watchdog task (`driven::app::memlog`): samples every 15 s on the
blocking pool, logs on >=128 MiB movement, keeps a trailing ~15 min
window.
- Diagnostic bundle: new `pending_ops.txt` (op shapes, sizes, offsets,
ages, recovery-flag presence; session URLs are capability secrets and
are never included) and `memory.txt` (current + peak RSS, a
`sample=fresh|stale-or-unread` honesty flag, and the trailing sample
window). Redaction policy text updated in the same change.

## Review

An ultra multi-agent review ran against the first cut; its confirmed
findings (S3 rewind misread, MemGauge leak on error unwind, excluded-op
orphan gap, push-before-validate on legacy rows, state-transition Debug
leak, plus several consistency nits) are fixed in the follow-up commit.
Refuted findings (e.g. mid-stream prefix modification "undetected" - the
full-stream md5 vs the store's stored md5 catches exactly that) are
documented in code comments.

## Tests

- `crash_mid_upload_resumes_persisted_session_byte_for_byte` asserts
MemGauge `peak > 0` (proves the resume routes through the instrumented
streaming loop) and `peak <= 3 * WIRE_CHUNK` (boundedness).
- New: `resume_honours_a_one_shot_rewind_to_zero` (S3 contract; red on
the stall-guard misread),
`resume_error_midstream_keeps_the_mem_gauge_balanced` (red without the
RAII guard), `reconcile_skips_resuming_a_now_excluded_paths_session` (+
control proving the gate),
`reconcile_adopts_finalized_orphan_even_when_path_now_excluded` (red on
a drop-the-op design),
`pending_ops_summary_names_resumable_sessions_without_urls`, memlog
sampler tests.
- Full workspace suite green locally (macOS): driven-core 547 lib + 23
e2e + integration suites, driven-app 437.

README checked - no changes needed (it does not enumerate bundle
contents, and no feature claims changed).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01A7q3CvJzL4zZmDA9CbXyQQ

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Author: pmaxhogan

Cargo.lock

@@ -99,6 +99,24 @@ const VSS_ORPHAN_SETTING_KEY: &str = "vss.orphans";
 /// `tokio::sync::Mutex` (held across the `.await`s of the DB read + write) makes
 /// each account's whole RMW atomic with respect to the others. `OnceLock` so
 /// every orchestrator in the process shares the same instance.
+/// The bare variant name of an [`OrchestratorState`], for the transition log
+/// (2026-08-14 incident). Deliberately NOT the Debug form: `Error` carries
+/// free-text details that can embed local paths, which must never enter a log
+/// line un-redacted.
+fn state_name(state: &OrchestratorState) -> &'static str {
+    match state {
+        OrchestratorState::Idle { .. } => "idle",
+        OrchestratorState::PowerCheck => "power_check",
+        OrchestratorState::Scanning { .. } => "scanning",
+        OrchestratorState::Planning { .. } => "planning",
+        OrchestratorState::Executing { .. } => "executing",
+        OrchestratorState::Verifying { .. } => "verifying",
+        OrchestratorState::Backoff { .. } => "backoff",
+        OrchestratorState::Paused { .. } => "paused",
+        OrchestratorState::Error { .. } => "error",
+    }
+}
+
 fn orphan_registry_lock() -> &'static tokio::sync::Mutex<()> {
     static LOCK: std::sync::OnceLock<tokio::sync::Mutex<()>> = std::sync::OnceLock::new();
     LOCK.get_or_init(|| tokio::sync::Mutex::new(()))
@@ -932,7 +950,28 @@ impl SyncOrchestrator {
     /// a tray-facing event - the next [`Orchestrator::state`] read still
     /// reflects the stored state.
     async fn transition(&self, next: OrchestratorState) {
-        *self.state_machine.write().await = next.clone();
+        let changed = {
+            let mut guard = self.state_machine.write().await;
+            let changed = *guard != next;
+            *guard = next.clone();
+            changed
+        };
+        // 2026-08-14 incident: log every REAL state change at INFO so a
+        // diagnostics bundle shows which phase (reconcile / scan / plan /
+        // execute) the app was in when something went sideways - the
+        // incident's bundle had no phase breadcrumbs at all. A handful of
+        // lines per cycle; same-state re-writes (e.g. Idle timestamp bumps on
+        // consecutive quiet cycles carry differing `last_run_at`, so those
+        // still log once per cycle) are the volume ceiling.
+        //
+        // Log the variant NAME only, never the Debug dump: the `Error`
+        // variant carries free-text error details that routinely embed local
+        // paths, and the bundle redactor's unquoted-run scanner truncates a
+        // path at its first space - so a raw `?next` here would leak
+        // partially-redacted paths into exported bundles.
+        if changed {
+            tracing::info!(target: TARGET, account_id = %self.account_id, state = state_name(&next), "state transition");
+        }
         let _ = self
             .events
             .send(OrchestratorEvent::StateChanged { state: next });
@@ -2340,6 +2379,12 @@ impl SyncOrchestrator {
             return Ok(());
         }
 
+        // 2026-08-14 incident: one INFO breadcrumb per cycle naming what
+        // triggered it. The incident's diagnostics bundle could not even show
+        // whether the user's "Sync now" click had reached the orchestrator -
+        // the whole fatal phase (reconcile, pre-scan) logged nothing at INFO.
+        tracing::info!(target: TARGET, account_id = %self.account_id, ?tick, "cycle start");
+
         // P1-6 (DESIGN s5.6, s5.7): the startup reconcile (DESIGN s5.6) is a
         // REMOTE pass - it issues Drive find/metadata calls to adopt orphaned
         // objects - so it MUST come AFTER the power / network / manual gates,

crates/driven-core/src/executor.rs

@@ -947,6 +947,17 @@ async fn crash_mid_upload_resumes_persisted_session_byte_for_byte() {
 
     // --- phase 2: a fresh executor reconciles -> resumes the session --------
     // The network drop was single-shot, so phase 2's requests all succeed.
+    //
+    // 2026-08-14 OOM regression guard: the resume re-read must STREAM, never
+    // buffer the body (the original implementation read the ENTIRE file into
+    // one `Vec` during reconcile - an interrupted 88 GB upload then OOM-killed
+    // the app on every launch). The MemGauge is bumped by the resume loop as
+    // bytes are buffered and released as Drive acks them; `peak > 0` proves
+    // the resume actually routed through the instrumented streaming path (a
+    // non-instrumented buffered path would read 0 and pass a bare `<=` bound
+    // vacuously), and the ceiling proves boundedness: ~2 wire chunks in
+    // flight, far below the ~16 MiB tail this file resumes.
+    let resume_gauge = Arc::new(MemGauge::default());
     let exec2 = DefaultExecutor::with_clock(
         ExecutorDeps {
             remote: remote.clone(),
@@ -957,9 +968,21 @@ async fn crash_mid_upload_resumes_persisted_session_byte_for_byte() {
             network: None,
         },
         clock.clone(),
-    );
+    )
+    .with_mem_gauge(resume_gauge.clone());
     exec2.reconcile(&src).await.unwrap();
 
+    let peak = resume_gauge.peak();
+    assert!(
+        peak > 0,
+        "the resume must route through the gauge-instrumented streaming loop"
+    );
+    assert!(
+        peak <= 3 * WIRE_CHUNK as u64,
+        "resume memory must stay bounded at a few wire chunks regardless of \
+         file size; peak was {peak} bytes for a {total_len}-byte file"
+    );
+
     // The upload completed via byte-level resume: exactly one object, and it
     // carries the full byte count (proving the resumed tail bytes landed, NOT a
     // truncated or from-zero re-do).

crates/driven-core/src/orchestrator.rs

@@ -193,6 +193,11 @@ windows-sys = { version = "0.61", features = [
     # DwmSetWindowAttribute: force a dark titlebar/border so the native window
     # chrome matches Driven's dark theme instead of the user's Windows accent.
     "Win32_Graphics_Dwm",
+    # memlog.rs (2026-08-14 incident): K32GetProcessMemoryInfo +
+    # GetCurrentProcess for the RSS watchdog behind the diagnostic bundle's
+    # memory samples.
+    "Win32_System_ProcessStatus",
+    "Win32_System_Threading",
 ] }
 
 [dev-dependencies]