ci(chaos): windows-only PR gate, 3-OS on tags, fuzz/soak local + recheck-2 fixes

Maintainer budget decision: trim chaos Actions spend.
- chaos-hermetic + chaos-fake-drive run windows-only on PR/main pushes (Windows
  is the primary platform); the full 3-OS matrix runs ONLY on `v*` tag pushes
  via a startsWith(github.ref,'refs/tags/') matrix switch.
- Remove the weekly 6h soak cron + in-CI fuzz. The long fuzz + soak-gated
  massive-input rows run locally via `just chaos-soak` / `just chaos-fuzz`.
- justfile: add chaos-fake-drive + chaos-soak recipes.

Folds in the two fixable M3.7 recheck round-2 (final) codex findings:
- fuzz --duration was silently capped at the 60s SCENARIO_WALL_CAP. run_fuzz now
  takes an explicit wall_cap so `fuzz --duration D` soaks by wall-clock for D
  (a local --duration 6h actually soaks 6h); fuzz-smoke stays bounded by a small
  step budget + 60s. Verified: --duration 8s ran 2038 steps over 8.6s.
- central duplicate-client_op_uuid check counts over list_folder_with_trashed
  (including trashed), so "create two for one op then trash one" is caught;
  legitimate trash-then-recreate uses distinct uuids so never false-positives.

The remaining recheck-2 findings (deferred-create pending-op exemption,
daily-quota midnight-resume = M4, uncapped setup, huge-file cacheable) are
documented as accepted residuals in design/CODEX_NOTES.md (recheck cap reached;
none are regressions or affect the green gates).

Gates: fmt; clippy -D warnings; test --workspace; hermetic 56 PASS/29 SKIP/0
FAIL; fault-injection 27 PASS/0 FAIL.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8meqeTo8bcZ3zjgKjBnJ4

Author: pmaxhogan

.github/workflows/chaos.yml

@@ -2,35 +2,39 @@ name: Chaos
 
 # Driven stress / chaos harness (design/STRESS_HARNESS.md s7).
 #
+# COST POLICY (maintainer budget): the chaos jobs run WINDOWS-ONLY on every PR
+# and push to main (Windows is the primary target platform). The full 3-OS
+# matrix [ubuntu, macos, windows] runs ONLY on `v*` tag pushes (release gates),
+# where the extra macOS/Windows runner cost is worth the release-time coverage.
+# The weekly 6-hour fuzz soak is NOT run in CI - it is a local/on-demand task
+# (`just chaos-fuzz` / `just chaos-soak`); the bounded `fuzz-smoke` row still
+# runs in the per-PR sweep. Unix-shaped rows (POSIX perms, case-sensitive FS)
+# are exercised on the ubuntu leg of the tag-push 3-OS run and locally on Linux.
+#
 # Jobs:
-#   chaos-hermetic   - every PR, 3-OS matrix [ubuntu, macos, windows]. Runs
-#                      every scenario whose requires() is satisfiable on a stock
-#                      non-elevated runner of that OS; the rest SKIP cleanly.
-#                      Linux + macOS exercise the Unix-shaped rows; Windows adds
-#                      the NTFS / Win32-shaped rows. None need real Drive or
-#                      elevation (the admin / VSS / real-Drive rows SKIP).
-#                      ROADMAP M3.7 acceptance: Linux + macOS + Windows.
-#   chaos-fake-drive - every PR, 3-OS matrix. The dedicated fault-injection gate
-#                      (STRESS_HARNESS s7 / ROADMAP M3.7): runs ONLY the s3.7
-#                      Drive-side hazards + s4.2/s5 drive-side mutator faults
-#                      against InMemoryRemoteStore (`run-all --fault-injection`).
-#                      These rows are also in the hermetic sweep; this focused
-#                      gate gives the separately-named, faster fault-injection
-#                      check the acceptance requires.
+#   chaos-hermetic   - every PR/push: windows-only; on a `v*` tag: 3-OS matrix.
+#                      Runs every scenario whose requires() is satisfiable on a
+#                      stock non-elevated runner; the rest SKIP cleanly.
+#   chaos-fake-drive - every PR/push: windows-only; on a `v*` tag: 3-OS matrix.
+#                      The dedicated fault-injection gate (STRESS_HARNESS s7):
+#                      runs ONLY the s3.7 Drive-side hazards + s4.2/s5 drive-side
+#                      mutator faults against InMemoryRemoteStore
+#                      (`run-all --fault-injection`).
 #   chaos-real-drive - WIRED but SKIPPED until M4: gated on the
 #                      DRIVEN_E2E_REFRESH_TOKEN secret, which is absent until
 #                      the GoogleDriveStore (M4) lands. It skips cleanly, it
 #                      does NOT fail (STRESS_HARNESS s7 "Gating on M4").
-#   chaos-soak       - weekly cron: a long seeded fuzz run. Informational.
+#
+# The long fuzz soak + soak-gated massive-input rows (million-files-nested,
+# tiny-files-100k) run LOCALLY via the justfile (`just chaos-soak`,
+# `just chaos-fuzz`), not in CI.
 
 on:
   push:
     branches: [main]
+    tags: ["v*"]
   pull_request:
     branches: [main]
-  schedule:
-    # Weekly soak, Monday 06:00 UTC (STRESS_HARNESS s7 chaos-soak).
-    - cron: "0 6 * * 1"
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}
@@ -47,17 +51,14 @@ env:
 jobs:
   chaos-hermetic:
     name: chaos hermetic (${{ matrix.os }})
-    # Skip the PR run on the scheduled trigger; the soak job covers cron.
-    if: github.event_name != 'schedule'
     strategy:
       fail-fast: false
       matrix:
-        # ROADMAP M3.7 acceptance: the hermetic harness must pass on all three
-        # OS families. Each runner capability-gates its own scenario set - Linux
-        # + macOS run the Unix-shaped rows, Windows runs the NTFS / Win32 rows;
-        # the admin / VSS / real-Drive rows SKIP cleanly on every unelevated
-        # runner (recorded, never red).
-        os: [ubuntu-latest, macos-latest, windows-latest]
+        # WINDOWS-ONLY on a normal PR/push (the primary platform, cheapest
+        # coverage); the FULL 3-OS matrix only on a `v*` tag push, where Linux +
+        # macOS add the Unix-shaped rows for the release gate. The admin / VSS /
+        # real-Drive rows SKIP cleanly on every unelevated runner (never red).
+        os: ${{ startsWith(github.ref, 'refs/tags/') && fromJSON('["ubuntu-latest","macos-latest","windows-latest"]') || fromJSON('["windows-latest"]') }}
     runs-on: ${{ matrix.os }}
     steps:
       - uses: actions/checkout@v4
@@ -70,23 +71,22 @@ jobs:
             libayatana-appindicator3-dev librsvg2-dev libsoup-3.0-dev javascriptcoregtk-4.1
       - uses: Swatinem/rust-cache@v2
       # run-all capability-gates every scenario: on a stock non-elevated runner
-      # the admin / NTFS / real-Drive rows SKIP (recorded, never red); the rest
-      # run against the headless core + InMemoryRemoteStore.
+      # the admin / real-Drive rows SKIP (recorded, never red); the rest run
+      # against the headless core + InMemoryRemoteStore.
       # Exit code: 0 = all pass/skip, 1 = any fail (STRESS_HARNESS s9).
       - name: chaos run-all (hermetic)
         run: cargo run -p driven-chaos -- run-all --hermetic
 
   chaos-fake-drive:
     name: chaos fake-drive (${{ matrix.os }})
-    # The dedicated fault-injection gate (ROADMAP M3.7 acceptance: a distinct
-    # chaos-fake-drive job on Linux + macOS + Windows). Runs ONLY the s3.7 /
-    # s4.2 / s5 fault-injection subset against InMemoryRemoteStore; capability
-    # gated rows (the daily-quota real-Drive path) SKIP cleanly, never red.
-    if: github.event_name != 'schedule'
+    # The dedicated fault-injection gate (STRESS_HARNESS s7). Runs ONLY the
+    # s3.7 / s4.2 / s5 fault-injection subset against InMemoryRemoteStore;
+    # capability-gated rows SKIP cleanly, never red. Same cost policy as
+    # hermetic: windows-only normally, 3-OS on a `v*` tag.
     strategy:
       fail-fast: false
       matrix:
-        os: [ubuntu-latest, macos-latest, windows-latest]
+        os: ${{ startsWith(github.ref, 'refs/tags/') && fromJSON('["ubuntu-latest","macos-latest","windows-latest"]') || fromJSON('["windows-latest"]') }}
     runs-on: ${{ matrix.os }}
     steps:
       - uses: actions/checkout@v4
@@ -129,37 +129,3 @@ jobs:
       # throwaway Drive folder; everything else runs as in the hermetic job.
       - name: chaos run-all (real Drive)
         run: cargo run -p driven-chaos -- run-all --hermetic
-
-  chaos-soak:
-    name: chaos soak (weekly fuzz)
-    # Weekly cron only; informational, does not gate merges (STRESS_HARNESS s7).
-    if: github.event_name == 'schedule'
-    runs-on: ubuntu-latest
-    # Opt into the soak-grade massive-input rows (million-files-nested,
-    # tiny-files-100k); they SKIP in the per-PR matrix and run here.
-    env:
-      DRIVEN_CHAOS_SOAK: "1"
-    steps:
-      - uses: actions/checkout@v4
-      - uses: dtolnay/rust-toolchain@stable
-      - name: Install Linux Tauri deps
-        run: |
-          sudo apt-get update
-          sudo apt-get install -y libwebkit2gtk-4.1-dev libxdo-dev libssl-dev \
-            libayatana-appindicator3-dev librsvg2-dev libsoup-3.0-dev javascriptcoregtk-4.1
-      - uses: Swatinem/rust-cache@v2
-      # First the soak-grade massive-input scenarios (skipped in PR), then the
-      # long seeded fuzz run. Any invariant violation writes
-      # target/chaos-fuzz-failures/<seed>.json for bit-reproducible replay
-      # (STRESS_HARNESS s4.3).
-      - name: chaos run-all (soak-gated rows)
-        run: cargo run -p driven-chaos -- run-all --hermetic
-      - name: chaos fuzz (seeded soak)
-        run: cargo run -p driven-chaos -- fuzz --seed ${{ github.run_id }} --duration 30m
-      - name: upload fuzz failure replays
-        if: failure()
-        uses: actions/upload-artifact@v4
-        with:
-          name: chaos-fuzz-failures
-          path: target/chaos-fuzz-failures/
-          if-no-files-found: ignore

crates/driven-chaos/src/dispatch.rs

@@ -257,18 +257,12 @@ pub mod exit_code {
 /// subcommand can re-print it (STRESS_HARNESS s2.2 / s6.2).
 const LAST_RUN_JSON: &str = "target/chaos-runs/last-run.json";
 
-/// A default fuzz step budget for a CLI `fuzz` invocation with no
-/// `--duration`. The weekly soak (s7) drives a much larger budget via the
-/// `--duration` -> step-budget mapping below.
+/// A default fuzz step budget for a CLI `fuzz` invocation with no `--duration`
+/// (a bare `fuzz` stays quick: this many steps under the 60s safety cap). A
+/// `fuzz --duration D` instead soaks by wall-clock for D with a very large step
+/// budget (see the `Fuzz` arm in [`run`]).
 const DEFAULT_FUZZ_STEPS: u64 = 200;
 
-/// Map a `--duration` in seconds onto a fuzz step budget. Each second of
-/// requested soak buys a fixed number of mutation steps; the run's wall-clock
-/// cap inside `run_fuzz` still bounds an over-long request.
-fn steps_for_duration(secs: u64) -> u64 {
-    secs.saturating_mul(50).max(DEFAULT_FUZZ_STEPS)
-}
-
 /// Persist a finished run so `report` can re-print it. Best-effort: a write
 /// failure must not flip an otherwise-green run red, so it only warns.
 fn persist_last_run(report: &RunReport) {
@@ -372,11 +366,20 @@ pub async fn run(command: Command, caps: &CapabilitySet) -> i32 {
                     .map(|d| d.as_secs())
                     .unwrap_or(0)
             });
-            let steps = duration_secs
-                .map(steps_for_duration)
-                .unwrap_or(DEFAULT_FUZZ_STEPS);
-            println!("driven-chaos fuzz: seed={seed} steps={steps}");
-            match mutator_scenarios::run_fuzz(seed, steps).await {
+            // `--duration D` governs the run by WALL-CLOCK: soak for D with a
+            // very large step budget so the duration is the binding bound (a
+            // local `fuzz --duration 6h` actually soaks 6h). Without a duration,
+            // the run is bounded by a small default step budget + the 60s safety
+            // cap so a bare `fuzz` invocation stays quick.
+            let (steps, wall_cap) = match duration_secs {
+                Some(secs) => (u64::MAX, std::time::Duration::from_secs(secs)),
+                None => (DEFAULT_FUZZ_STEPS, mutator_scenarios::SCENARIO_WALL_CAP),
+            };
+            println!(
+                "driven-chaos fuzz: seed={seed} step_budget={steps} wall_cap={}s",
+                wall_cap.as_secs()
+            );
+            match mutator_scenarios::run_fuzz(seed, steps, wall_cap).await {
                 Ok(report) => {
                     if let Some(violation) = &report.violation {
                         match mutator_scenarios::write_fuzz_failure(&report) {

crates/driven-chaos/src/scenarios/mutator.rs

@@ -108,7 +108,7 @@ pub fn scenarios() -> Vec<Box<dyn Scenario>> {
 /// bounded ITERATION count + tiny intervals rather than the 5-minute soak
 /// default (which is reserved for the `fuzz --duration` CLI path). This cap is
 /// a backstop the driver can enforce; exceeding it is `harness.timeout`.
-const SCENARIO_WALL_CAP: Duration = Duration::from_secs(60);
+pub const SCENARIO_WALL_CAP: Duration = Duration::from_secs(60);
 
 /// Cycles a registered soak scenario interleaves with mutation (small, fast,
 /// deterministic - not the 5-min soak).
@@ -1260,12 +1260,19 @@ pub struct FuzzReport {
 /// sequence of filesystem mutations interleaved with sync cycles against a
 /// synthesised source tree, then assert the s6.3 post-condition invariants.
 ///
-/// `step_budget` bounds the run (the CLI maps `--duration` onto a step budget
-/// so a registered run stays fast and a soak run goes long). On any invariant
-/// violation the returned report carries `violation = Some(..)` and the full
-/// mutation log, which the caller writes to
+/// `step_budget` and `wall_cap` BOTH bound the run, whichever is hit first: the
+/// registered `fuzz-smoke` row passes a small step budget + the 60s
+/// [`SCENARIO_WALL_CAP`] so it stays fast, while the `fuzz --duration D` CLI
+/// path passes a very large step budget + `wall_cap = D` so a local soak
+/// actually runs for the requested wall-clock (`--duration 6h` soaks 6h, not
+/// 60s). On any invariant violation the returned report carries
+/// `violation = Some(..)` and the full mutation log, which the caller writes to
 /// `target/chaos-fuzz-failures/<seed>.json` for bit-reproducible replay.
-pub async fn run_fuzz(seed: u64, step_budget: u64) -> anyhow::Result<FuzzReport> {
+pub async fn run_fuzz(
+    seed: u64,
+    step_budget: u64,
+    wall_cap: Duration,
+) -> anyhow::Result<FuzzReport> {
     let mut rng = Rng::new(seed);
     let state_dir = tempfile::tempdir()?;
     let root_dir = tempfile::tempdir()?;
@@ -1283,7 +1290,7 @@ pub async fn run_fuzz(seed: u64, step_budget: u64) -> anyhow::Result<FuzzReport>
 
     let started = std::time::Instant::now();
     for step_index in 0..step_budget {
-        if started.elapsed() > SCENARIO_WALL_CAP {
+        if started.elapsed() > wall_cap {
             break;
         }
         let step = pick_step(&mut rng);
@@ -1478,7 +1485,9 @@ impl Scenario for FuzzSmokeScenario {
     }
 
     async fn run_assertions(&self, _handle: &DrivenHandle) -> anyhow::Result<Outcome> {
-        let report = run_fuzz(self.seed, self.steps).await?;
+        // The registered smoke row stays bounded by BOTH a small step budget
+        // and the 60s wall cap, so it can never balloon into a long CI step.
+        let report = run_fuzz(self.seed, self.steps, SCENARIO_WALL_CAP).await?;
         if let Some(reason) = &report.violation {
             // Persist the failure for bit-reproducible replay, then fail.
             let path = write_fuzz_failure(&report)?;
@@ -1584,7 +1593,11 @@ mod tests {
     /// real headless core).
     #[tokio::test]
     async fn fuzz_smoke_holds_invariants() {
-        let report = run_fuzz(0xABCD_1234, 80).await.expect("fuzz run");
+        // 80 steps complete in well under the 60s wall cap, so the step budget
+        // is the binding bound and the full budget runs.
+        let report = run_fuzz(0xABCD_1234, 80, SCENARIO_WALL_CAP)
+            .await
+            .expect("fuzz run");
         assert_eq!(report.steps, 80, "ran the full budget");
         assert!(
             report.violation.is_none(),

crates/driven-chaos/src/scenarios/reporting.rs

@@ -200,15 +200,19 @@ pub async fn assert_invariants(
     // leaves the remote in a LATCHED fault state (e.g. auth.invalid_grant, which
     // the fake applies to read calls too) can still have its terminal-state
     // invariants verified instead of the sweep itself erroring on the fault.
-    let live: Vec<_> = remote
-        .list_folder_with_trashed(folder_id)
-        .into_iter()
-        .filter(|e| !e.trashed)
-        .collect();
+    let all: Vec<_> = remote.list_folder_with_trashed(folder_id);
+    let live: Vec<_> = all.iter().filter(|e| !e.trashed).cloned().collect();
     let live_object_count = live.len() as u64;
 
+    // No duplicate Drive objects per client_op_uuid (s6.3). Count over ALL
+    // objects INCLUDING trashed ones: "two objects created for one op, then one
+    // trashed" is still evidence of a duplicate-create bug, so filtering trashed
+    // out before counting would hide it. Each upload op stamps a FRESH
+    // client_op_uuid, so a legitimate trash-then-recreate carries two distinct
+    // uuids and never collides here; only a genuine duplicate create does.
+    // (Mirrors the mutator checker, which already counts with-trashed.)
     let mut uuid_counts: HashMap<String, u64> = HashMap::new();
-    for entry in &live {
+    for entry in &all {
         if let Some(uuid) = entry.app_properties.get(CLIENT_OP_UUID_KEY) {
             *uuid_counts.entry(uuid.clone()).or_insert(0) += 1;
         }