test(chaos): extend the fault-injection harness to the S3 and local-folder backends

[pmaxhogan]

TL;DR

The chaos harness only ever drove one destination: Google Drive, through
InMemoryRemoteStore. This adds nine rows covering the S3 and local-folder
backends, injects the S3 faults on the wire against the real
driven_s3::S3Store, constructs the local-folder faults byte-for-byte on
disk, and - the part that actually matters - makes the SPEC s6.3 invariant
sweep backend-independent instead of structurally welded to the fake.

No production behaviour changes. One additive test-double method
(InMemoryRemoteStore::latch_dest_folder_missing) and nothing else outside
driven-chaos - hence test(chaos):. Rebased onto current main (post-#212
and #203).

The load-bearing change: assert_invariants is no longer Drive-only

assert_invariants took a &InMemoryRemoteStore and called two INHERENT
methods on it. So the claim "the s6.3 invariants are backend-independent" was
untestable by construction: a second backend could only be covered by a second,
forked copy of the checker - and two copies of "no duplicate client_op_uuid"
is exactly the drift s6.3 exists to prevent.

It is now generic over one narrow trait, with impls for all three backends:

#[async_trait]
pub trait InvariantSurface: Send + Sync {
    async fn invariant_listing(&self, folder_id: &str) -> anyhow::Result<Vec<RemoteEntry>>;
    async fn retained_content(&self, id: &str) -> Option<ObjectContent>;
}

All 34 existing call sites are unchanged (a delegating impl for Arc<T>
covers the ones holding an Arc). There is still exactly one implementation of
each invariant, now shared by every backend.

Two properties of that seam are deliberate:

• It is fault-free. A row that ends with a latched fault
  (auth.invalid_grant, NoSuchBucket, an unplugged drive) would make the
  sweep's own read fail, and the harness would report "could not verify" as
  though it were "verified nothing wrong".
• It is not RemoteStore. On S3, list_folder returns keys, sizes and
  ETags but no user metadata, so a duplicate-client_op_uuid check written
  against it would be silently vacuous on that backend - always green, never
  looking. It is also non-recursive on every backend.

S3: why the faults are injected on the wire

The S3 failures that lose data rather than merely erroring live below the
trait, in the HTTP layer. Injected at the trait seam they arrive already
classified, which proves nothing about the code that does the classifying:

• 503 SlowDown is throttling wearing a 5xx costume. Read by status alone it
  becomes a bounded-retry transient fault, and a throttled bucket strands files.
• RequestTimeout arrives as a 400 and must still be retried.
• CompleteMultipartUpload can answer HTTP 200 with an error document,
  because S3 holds the connection open while assembling parts.
• A multipart upload cut between two UploadPart calls, or between the last
  UploadPart and CompleteMultipartUpload, has no trait-level representation
  at all.

So crates/driven-chaos/src/s3_server.rs runs an in-process, loopback-bound
HTTP/1.1 FaultyS3Server speaking the S3 subset driven-s3 uses, and the rows
drive the real S3Store at it: real SigV4 presigning, real reqwest round
trips, real XML parsing, real classify_s3_response.

Why not MinIO behind a proxy. driven-s3's own integration suite already
covers real MinIO and real R2. The chaos jobs are a different question: per
chaos.yml they run Windows-only on every PR, and minio is installed only
on the Linux legs of ci.yml / coverage.yml. A MinIO-gated chaos row would
SKIP in the one job that blocks a merge. In-process needs no external binary, no
port coordination beyond :0, and behaves the same on all three platforms.

No SigV4 validation, deliberately - S3Store signs as presigned URLs, so
the server never has to verify anything. Validating signatures would test
rusty-s3, not Driven.

The server keeps real S3 semantics where a fault row's meaning depends on them:
Content-MD5 is verified and a mismatch answered 400 BadDigest; a single-PUT
ETag is the content md5 while a multipart ETag is md5(concat(part md5s))-N;
CompleteMultipartUpload assembles only the parts the body names and
discards the rest (the property the crash-resume path relies on);
ListObjectsV2 honours max-keys and emits NextContinuationToken.

A contract smoke test (a_real_s3_store_round_trips_against_the_in_process_server)
pins the whole non-faulted surface, so a broken server can never be mistaken for
a broken backend.

Local folder: no protocol server and no fault seam needed

Its destination IS a directory, so crates/driven-chaos/src/localfs_fixture.rs
constructs each post-crash state byte-for-byte as the failure produces it and
reads the ground truth back with std::fs - no production code has to know it is
under test:

• swap_marker_identity() - the marker now names a different destination id,
  i.e. a different stick at the same mount point.
• block_target_path() - an object-shaped DIRECTORY at the target path, so the
  store really writes and F_FULLFSYNCes its temp file and the real
  fsx::commit_rename really fails. That is how the row reaches the instant
  between the sync and the rename.
• plant_orphan_temp_file(bytes, age) - the residue a killed process leaves,
  with a controllable mtime so both sides of the sweep window are testable.

LocalFsOracle is the fault-free listing. It must reproduce the backend's own
notion of what is NOT an object - .driven-meta/, .driven-destination.json,
.driven-tmp-*, and the macOS AppleDouble ._* shadows the OS writes on
exFAT/FAT32 - so it delegates to the backend's own layout::is_control_entry
rather than re-deriving the list. ._* files never appear on a tempdir, which is
exactly why an oracle that forgot them would look correct in CI and lie on real
hardware.

Faults now injectable, per backend

S3 (wire-level, against the real store): 503 SlowDown single-shot and as
a burst; 400 RequestTimeout; 500 InternalError; 404 NoSuchBucket
(latching); 403 AllAccessDisabled (latching); QuotaExceeded past a byte
budget, charged per part so it can run out mid-upload; connection dropped
mid-UploadPart; connection dropped at CompleteMultipartUpload; HTTP 200
carrying an <Error> document; 404 NoSuchUpload at completion; per-response
latency for race widening; clear_faults() to model recovery.

Local folder (on-disk states): a marker naming a different destination;
a commit whose rename cannot land; an abandoned temp file at any age.

Google Drive (unchanged, plus one): the existing s5 with_* builders, plus a
runtime latch_dest_folder_missing() in the same shape as the already-present
arm_session_invalidated_after. "The destination vanished" is a mid-run event:
asserting nothing already backed up is lost needs a healthy first cycle to
establish a synced baseline, which a construction-time-only surface cannot
express without rebuilding the store and discarding the very objects the
assertion is about.

The rows

Row	Fault	Asserts
s3-multipart-interrupted-between-parts	connection dies mid-UploadPart, one part already committed	recovers to exactly one byte-exact object; no gap, no duplicate
s3-multipart-interrupted-before-complete	connection dies at CompleteMultipartUpload	nothing published and nothing synced until a completion really succeeds
s3-complete-200-with-error-document	200 + <Error>, nothing published	the status is not trusted; no false synced row; recovery afterwards
s3-slow-down-throttling	10 consecutive 503 SlowDown	the upload still completes (see below - a single one proves nothing)
s3-request-timeout-retryable-400	400 RequestTimeout mid-transfer	retried despite the 4xx; the file is not abandoned
s3-kill-mid-upload-then-reboot	upload cut, orchestrator dropped with no graceful shutdown, fresh handle over the same DB + bucket	no orphan with no file_state row, no duplicate, nothing falsely synced
s3-destination-full-mid-upload	byte budget exhausted with a part already uploaded	drive.quota_exhausted; nothing partial published; freeing space finishes the backup
localfs-crash-between-temp-write-and-rename	a commit whose rename cannot land, then the temp file a killed process leaves holding the FULL bytes	nothing readable at the target, no sidecar, nothing synced, no temp residue from the failed commit; the orphan is invisible to list_folder AND to the audit; stale swept, fresh preserved; a further cycle changes nothing
destination-vanished-across-backends	destination vanishes mid-cycle on all three backends, one body	drive.dest_folder_missing; nothing written after the destination went away; the baseline survives

Every row ends in the shared assert_invariants sweep, plus one assertion the
sweep cannot make: for every synced row, the destination's bytes equal the
local file's bytes. A half-assembled multipart object, a dropped part, or a
completion that published a truncated object is caught even when every recorded
digest is self-consistent.

Registered in both registry() and fault_injection_registry(), so the
chaos-fake-drive job - the one named after fault injection - actually runs
them.

Two details worth calling out

The fresh-temp-file assertion in the localfs crash row is a real
requirement, not symmetry for its own sake: the sweep uses the trait's 6-day
session window as its cutoff, so a sweep that reaped a fresh temp file would
destroy an upload the executor could still resume. Both sides are asserted.

That row also asserts on the data file's absence, never on a sidecar's
presence, because the backend commits data first and sidecar second on
purpose. A crash between them is benign by design, and a row that asserted the
opposite ordering would encode a bug as the expected behaviour.

Making the rows discriminate, not just pass

Two rows initially passed for the wrong reason. Both are fixed, and the fixes
are the most interesting part of this PR.

s3-slow-down-throttling did not discriminate. With ONE injected
SlowDown, the row passes whether or not classify_s3_response special-cases
the S3 code, because a transient 5xx is also retried. Only a burst longer than
the executor's finite 5xx budget (MAX_TRANSIENT_RETRIES = 6) separates them:
rate limiting retries indefinitely under the pacer and completes; a transient
5xx gives up and strands the file. The row now injects 10, on a small file so
the buffered create path answers with the retry POLICY rather than the next
scan cycle happening to succeed.

The localfs crash row's audit assertion was vacuous. "The audit returned
nothing" is trivially true on an empty destination. The row now recovers to a
real object FIRST, then plants the orphans, and asserts the audit returns
exactly one result - which is the difference between "the temp files were
filtered" and "the query found nothing".

Every row now proves its fault fired. The S3 server keeps per-fault firing
counters (slow_down_fired, complete_error_documents_sent, quota_refusals,
bucket_missing_refusals, ...) and each row asserts on the one it armed; the
localfs rows assert an error was surfaced. This is the append-only-log lesson
applied to fault injection: a green run that never reached its fault is not a
passing test, it is a test that did nothing.

Race windows are forced open, not hoped for (the #192 technique): the S3
rows whose fault lands mid-transfer arm a 3 ms per-response delay so the
intended path is taken on every run.

Negative controls

Each high-value row was re-run with the production defence it guards
deliberately disabled, and had to go RED. All controls were reverted;
crates/driven-s3 and crates/driven-localfs are untouched on this branch.

Row	Defence disabled	Result
localfs-crash-between-temp-write-and-rename	layout::is_control_entry stops filtering .driven-tmp-*	RED: an abandoned temp file must not appear as an object: expected the 1 real object, got ["backups/report.bin", "backups/.driven-tmp-..."]
destination-vanished-across-backends (localfs arm)	LocalFsStore::guard_root stops comparing the destination id	RED: a marker belonging to a different volume must raise drive.dest_folder_missing, got []
s3-complete-200-with-error-document	all three of driven-s3's defences, one at a time	GREEN until all three were gone, then RED with big.bin recorded as synced with no object at the destination

That last one is a finding in itself: a lying 200 is stopped three independent
ways, any one of which suffices - (1) complete_multipart scans the body for
<Error even on a 2xx; (2) failing that, the composed-ETag check finds no
<ETag> in an error document and reports ChecksumMismatch; (3) failing both,
drain_and_maybe_complete HEADs the key afterwards and an object that was never
published answers 404. The row therefore asserts the OUTCOME rather than pinning
one of the three, so a future refactor that consolidated the defences while
keeping the guarantee does not produce a false failure.

Finding: an abandoned S3 multipart upload leaks its parts

The very first run of s3-multipart-interrupted-between-parts failed on
"multipart upload left in flight", which turned out to be real.

S3Store aborts a multipart upload on exactly two paths: the non-resumable
multipart_stream failure path, and a completion failure is_session_fatal
calls terminal. A transport failure mid-UploadPart on the resumable
path is neither - resume_chunk propagates the error and leaves the upload id
alive. That part is correct on its own: the startup
reconcile -> resume_persisted path may still resume that exact session after
a crash, and aborting eagerly would destroy a recoverable upload.

The leak is on the other side. executor.rs::upload_stage_resumable - the
streaming path every file >= 4 MiB takes - calls open_resumable_session
unconditionally at the top of each attempt, so the next cycle's retry mints
a fresh CreateMultipartUpload and never touches the previous upload id again.
Its parts stay on the bucket, billed, one abandoned upload per failed
attempt, and S3Store::new performs no sweep - unlike driven-localfs, which
explicitly sweeps abandoned temp files at construction using the trait's own
6-day session window (a mechanism this PR's localfs row now covers).

Not data loss, so no s6.3 invariant is violated. It is real money on a real
bucket, and it is unbounded across repeated failures.

How the rows treat it: they do NOT fail on it (that would make a
merge-blocking gate red over a known, unfixed cost issue), and they do NOT
ignore it. settle_and_assert takes a per-row cap, so the leak is BOUNDED - a
row fails if more uploads strand than its faults account for, which is what
would catch it growing. Every occurrence is reported as an explicit
FINDING (cost, not data loss) note carrying the cause. The fix belongs in
driven-s3 (a ListMultipartUploads sweep at construction, mirroring the
localfs one), not in the harness, so it is left for a follow-up fix(s3):
rather than smuggled into a test(chaos): PR.

What is NOT covered

The localfs ENOSPC errno path, end to end. The out-of-space behaviour IS
covered: the local-folder errno table maps ENOSPC/EDQUOT onto exactly the
DriveError::StorageQuota that s3-destination-full-mid-upload's
QuotaExceeded produces, and that row drives the whole chain mid-multipart with
bytes already at the destination (account pauses, nothing partial published,
recovery once space is freed).

What is missing is the errno itself, because a real ENOSPC needs a real
constrained VOLUME: unprivileged on macOS (hdiutil), root-only on Linux
(losetup), admin-only on Windows (New-VHD). The existing disk-full-target
row sets the precedent - capability-gated behind
DRIVEN_CHAOS_ALLOW_DISK_MOUNT - and a row gated the same way would SKIP in the
Windows-only PR gate, the one job that blocks a merge. The errno table is already
unit-tested in driven-localfs; what a chaos row adds beyond that is the
orchestrator behaviour, which the S3 row proves. Recorded as a deliberate gap
rather than shipped as a scenario that is always SKIPPED.

Non-flakiness

Sequential runs against the prebuilt debug binary (M5 MacBook Pro, macOS 26).
Sequential on purpose: these rows bind loopback ports and share a debug-build
CPU budget, so parallel runs would measure contention rather than flakiness.

Row	Runs	Pass	Fail
s3-multipart-interrupted-between-parts	100	100	0
s3-multipart-interrupted-before-complete	100	100	0
s3-complete-200-with-error-document	100	100	0
s3-slow-down-throttling	100	100	0
s3-request-timeout-retryable-400	100	100	0
s3-kill-mid-upload-then-reboot	100	100	0
s3-destination-full-mid-upload	100	100	0
localfs-crash-between-temp-write-and-rename	100	100	0
destination-vanished-across-backends	100 (+100 pre-localfs-arm)	100	0
total	900	900	0

Plus 22 full run-all --fault-injection sweeps (12 before the local-folder
rows at 33 PASS / 2 SKIP, 10 after at 34 PASS / 2 SKIP), 0 FAIL / 0 FLAKY
every time - so the new rows do not interfere with the existing ones (shared
ports, shared target/chaos-fixtures/).

Worth stating plainly rather than implying the counts alone earned the
confidence: every one of these rows is deterministic by construction. The
fault is an armed flag or a constructed on-disk state, not a timing race, and the
fault-fired counters prove the intended path was taken on each run. That is what
makes 100 runs meaningful instead of 100 coin flips that happened to land the
same way - and it is the same technique #192 used to convert
append-only-log from a 5-10% flake into a deterministic row.

Per-row wall clock is 65-610 ms locally and 236-779 ms on the Windows runner
(see the CI section), so the nine rows add ~5 s across both Windows chaos jobs.

CI on windows-latest, the platform the gate actually runs

All nine new rows pass on windows-latest, which matters because the S3 server
hand-rolls HTTP/1.1 framing and the local-folder rows depend on rename
semantics and directory handling:

s3-multipart-interrupted-between-parts        pass  721 ms
s3-multipart-interrupted-before-complete      pass  726 ms
s3-complete-200-with-error-document           pass  662 ms
s3-slow-down-throttling                       pass  241 ms
s3-request-timeout-retryable-400              pass  629 ms
s3-kill-mid-upload-then-reboot                pass  779 ms
s3-destination-full-mid-upload                pass  664 ms
localfs-crash-between-temp-write-and-rename   pass  236 ms
destination-vanished-across-backends          pass  455 ms

• chaos fake-drive (windows-latest): 36 PASS / 0 SKIP / 0 FAIL / 0 FLAKY
• chaos hermetic (windows-latest): 84 PASS / 10 SKIP / 0 FAIL / 0 FLAKY

Total added cost: ~5.1 s across both Windows jobs.

Pre-existing failure, macOS-only, unrelated to this branch

run-all --hermetic on the final rebased tree reports one FAIL on this
macOS host: noaccess-file ("exactly one local.io_error (the unreadable file);
got 0").
Verified pre-existing by stashing every change and running it against unmodified
main, where it fails identically - and it PASSES on windows-latest in this
PR's own run, which confirms it is a macOS permissions-model gap rather than
anything this branch touched. The chaos gate is Windows-only on a PR and macOS
only joins on a v* tag, so it does not block here, but it is a real gap worth
its own issue: on a v* tag the macOS leg would go red.

Gates

Local (M5 MacBook Pro, on the rebased tree):

• SQLX_OFFLINE=true cargo test --workspace - green (53 test binaries, 0
  failures)
• cargo clippy --workspace --all-targets -- -D warnings - clean
• cargo fmt --all -- --check - clean
• cargo run -p driven-chaos -- run-all --fault-injection - 34 PASS / 2 SKIP /
  0 FAIL (the 2 SKIPs are the Windows-only mutator rows)
• ASCII-only, LF endings verified via git ls-files --eol

CI (all against the pushed commit, which matches local HEAD): rustfmt,
cargo deny, coverage, ui build + lint + unit, all three tauri compile
legs, CodeQL / Analyze (rust, actions, javascript-typescript),
chaos hermetic (windows-latest), chaos fake-drive (windows-latest), and all
three cargo test + clippy legs - every check green. The only non-passing
row is real-drive e2e (tag-only), which skips by design outside a v* tag.

pnpm --dir ui test:unit was not run locally: ui/node_modules is absent
in this worktree and the diff touches no UI file, so CI's ui build + lint + unit job (green) is the honest check rather than a local install. Stated
rather than claimed.

[github-actions]

Coverage

Area	main	this PR	delta
Rust (lib crates)	82.84%	82.83%	-0.01 (OK)
UI (vue/ts)	92.29%	92.29%	+0.00 (OK)

Gate: passed - no coverage regression (epsilon 0.1 pp).