test(chaos): deterministically gate the #144 create-orphan race in the append-only-log soak - #192
Merged
Merged
Conversation
Widen the append-only-log soak's first-CREATE upload window with a 3ms per-request remote delay so the changed-after-upload path is exercised on every run instead of ~3% of runs, and refresh the stale pre-#146 comments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qu8GxMwkuxF7JBzwRjtcw7
Contributor
Coverage
Gate: passed - no coverage regression (epsilon 0.1 pp). |
This was referenced Jul 29, 2026
Closed
pmaxhogan
added a commit
that referenced
this pull request
Jul 29, 2026
…older backends (#221) ## TL;DR The chaos harness only ever drove one destination: Google Drive, through `InMemoryRemoteStore`. This adds nine rows covering the S3 and local-folder backends, injects the S3 faults **on the wire against the real `driven_s3::S3Store`**, constructs the local-folder faults **byte-for-byte on disk**, and - the part that actually matters - makes the SPEC s6.3 invariant sweep backend-independent instead of structurally welded to the fake. No production behaviour changes. One additive test-double method (`InMemoryRemoteStore::latch_dest_folder_missing`) and nothing else outside `driven-chaos` - hence `test(chaos):`. Rebased onto current `main` (post-#212 and #203). ## The load-bearing change: `assert_invariants` is no longer Drive-only `assert_invariants` took a `&InMemoryRemoteStore` and called two INHERENT methods on it. So the claim "the s6.3 invariants are backend-independent" was untestable by construction: a second backend could only be covered by a second, forked copy of the checker - and two copies of "no duplicate `client_op_uuid`" is exactly the drift s6.3 exists to prevent. It is now generic over one narrow trait, with impls for all three backends: ```rust #[async_trait] pub trait InvariantSurface: Send + Sync { async fn invariant_listing(&self, folder_id: &str) -> anyhow::Result<Vec<RemoteEntry>>; async fn retained_content(&self, id: &str) -> Option<ObjectContent>; } ``` **All 34 existing call sites are unchanged** (a delegating impl for `Arc<T>` covers the ones holding an `Arc`). There is still exactly one implementation of each invariant, now shared by every backend. Two properties of that seam are deliberate: - **It is fault-free.** A row that ends with a latched fault (`auth.invalid_grant`, `NoSuchBucket`, an unplugged drive) would make the sweep's own read fail, and the harness would report "could not verify" as though it were "verified nothing wrong". - **It is not `RemoteStore`.** On S3, `list_folder` returns keys, sizes and ETags but **no user metadata**, so a duplicate-`client_op_uuid` check written against it would be silently vacuous on that backend - always green, never looking. It is also non-recursive on every backend. ## S3: why the faults are injected on the wire The S3 failures that lose data rather than merely erroring live **below** the trait, in the HTTP layer. Injected at the trait seam they arrive already classified, which proves nothing about the code that does the classifying: - `503 SlowDown` is throttling wearing a 5xx costume. Read by status alone it becomes a bounded-retry transient fault, and a throttled bucket strands files. - `RequestTimeout` arrives as a **400** and must still be retried. - `CompleteMultipartUpload` can answer **HTTP 200 with an error document**, because S3 holds the connection open while assembling parts. - A multipart upload cut between two `UploadPart` calls, or between the last `UploadPart` and `CompleteMultipartUpload`, has no trait-level representation at all. So `crates/driven-chaos/src/s3_server.rs` runs an in-process, loopback-bound HTTP/1.1 `FaultyS3Server` speaking the S3 subset `driven-s3` uses, and the rows drive the **real** `S3Store` at it: real SigV4 presigning, real `reqwest` round trips, real XML parsing, real `classify_s3_response`. **Why not MinIO behind a proxy.** `driven-s3`'s own integration suite already covers real MinIO and real R2. The chaos jobs are a different question: per `chaos.yml` they run **Windows-only** on every PR, and `minio` is installed only on the Linux legs of `ci.yml` / `coverage.yml`. A MinIO-gated chaos row would SKIP in the one job that blocks a merge. In-process needs no external binary, no port coordination beyond `:0`, and behaves the same on all three platforms. **No SigV4 validation, deliberately** - `S3Store` signs as presigned URLs, so the server never has to verify anything. Validating signatures would test `rusty-s3`, not Driven. The server keeps real S3 semantics where a fault row's meaning depends on them: `Content-MD5` is verified and a mismatch answered `400 BadDigest`; a single-`PUT` ETag is the content md5 while a multipart ETag is `md5(concat(part md5s))-N`; `CompleteMultipartUpload` assembles **only** the parts the body names and discards the rest (the property the crash-resume path relies on); `ListObjectsV2` honours `max-keys` and emits `NextContinuationToken`. A contract smoke test (`a_real_s3_store_round_trips_against_the_in_process_server`) pins the whole non-faulted surface, so a broken server can never be mistaken for a broken backend. ## Local folder: no protocol server and no fault seam needed Its destination IS a directory, so `crates/driven-chaos/src/localfs_fixture.rs` constructs each post-crash state byte-for-byte as the failure produces it and reads the ground truth back with `std::fs` - no production code has to know it is under test: - `swap_marker_identity()` - the marker now names a **different** destination id, i.e. a different stick at the same mount point. - `block_target_path()` - an object-shaped DIRECTORY at the target path, so the store really writes and `F_FULLFSYNC`es its temp file and the real `fsx::commit_rename` really fails. That is how the row reaches the instant between the sync and the rename. - `plant_orphan_temp_file(bytes, age)` - the residue a killed process leaves, with a controllable mtime so both sides of the sweep window are testable. `LocalFsOracle` is the fault-free listing. It must reproduce the backend's own notion of what is NOT an object - `.driven-meta/`, `.driven-destination.json`, `.driven-tmp-*`, and the macOS AppleDouble `._*` shadows the OS writes on exFAT/FAT32 - so it delegates to the backend's own `layout::is_control_entry` rather than re-deriving the list. `._*` files never appear on a tempdir, which is exactly why an oracle that forgot them would look correct in CI and lie on real hardware. ## Faults now injectable, per backend **S3 (wire-level, against the real store):** `503 SlowDown` single-shot **and as a burst**; `400 RequestTimeout`; `500 InternalError`; `404 NoSuchBucket` (latching); `403 AllAccessDisabled` (latching); `QuotaExceeded` past a byte budget, charged per part so it can run out mid-upload; connection dropped mid-`UploadPart`; connection dropped at `CompleteMultipartUpload`; **HTTP 200 carrying an `<Error>` document**; `404 NoSuchUpload` at completion; per-response latency for race widening; `clear_faults()` to model recovery. **Local folder (on-disk states):** a marker naming a different destination; a commit whose `rename` cannot land; an abandoned temp file at any age. **Google Drive (unchanged, plus one):** the existing s5 `with_*` builders, plus a runtime `latch_dest_folder_missing()` in the same shape as the already-present `arm_session_invalidated_after`. "The destination vanished" is a mid-run event: asserting nothing already backed up is lost needs a healthy first cycle to establish a synced baseline, which a construction-time-only surface cannot express without rebuilding the store and discarding the very objects the assertion is about. ## The rows | Row | Fault | Asserts | |---|---|---| | `s3-multipart-interrupted-between-parts` | connection dies mid-`UploadPart`, one part already committed | recovers to exactly one byte-exact object; no gap, no duplicate | | `s3-multipart-interrupted-before-complete` | connection dies at `CompleteMultipartUpload` | nothing published and nothing `synced` until a completion really succeeds | | `s3-complete-200-with-error-document` | 200 + `<Error>`, nothing published | the status is not trusted; no false `synced` row; recovery afterwards | | `s3-slow-down-throttling` | **10** consecutive `503 SlowDown` | the upload still completes (see below - a single one proves nothing) | | `s3-request-timeout-retryable-400` | `400 RequestTimeout` mid-transfer | retried despite the 4xx; the file is not abandoned | | `s3-kill-mid-upload-then-reboot` | upload cut, orchestrator dropped with no graceful shutdown, fresh handle over the same DB + bucket | no orphan with no `file_state` row, no duplicate, nothing falsely `synced` | | `s3-destination-full-mid-upload` | byte budget exhausted with a part already uploaded | `drive.quota_exhausted`; nothing partial published; freeing space finishes the backup | | `localfs-crash-between-temp-write-and-rename` | a commit whose `rename` cannot land, then the temp file a killed process leaves holding the FULL bytes | nothing readable at the target, no sidecar, nothing `synced`, no temp residue from the failed commit; the orphan is invisible to `list_folder` AND to the audit; stale swept, **fresh preserved**; a further cycle changes nothing | | `destination-vanished-across-backends` | destination vanishes mid-cycle on **all three** backends, one body | `drive.dest_folder_missing`; nothing written after the destination went away; the baseline survives | Every row ends in the shared `assert_invariants` sweep, plus one assertion the sweep cannot make: **for every `synced` row, the destination's bytes equal the local file's bytes.** A half-assembled multipart object, a dropped part, or a completion that published a truncated object is caught even when every recorded digest is self-consistent. Registered in **both** `registry()` and `fault_injection_registry()`, so the `chaos-fake-drive` job - the one named after fault injection - actually runs them. ### Two details worth calling out The **fresh-temp-file** assertion in the localfs crash row is a real requirement, not symmetry for its own sake: the sweep uses the trait's 6-day session window as its cutoff, so a sweep that reaped a fresh temp file would destroy an upload the executor could still resume. Both sides are asserted. That row also asserts on the **data file's absence, never on a sidecar's presence**, because the backend commits data first and sidecar second on purpose. A crash between them is benign by design, and a row that asserted the opposite ordering would encode a bug as the expected behaviour. ## Making the rows discriminate, not just pass Two rows initially passed for the wrong reason. Both are fixed, and the fixes are the most interesting part of this PR. **`s3-slow-down-throttling` did not discriminate.** With ONE injected `SlowDown`, the row passes whether or not `classify_s3_response` special-cases the S3 code, because a transient 5xx is also retried. Only a burst **longer than the executor's finite 5xx budget** (`MAX_TRANSIENT_RETRIES = 6`) separates them: rate limiting retries indefinitely under the pacer and completes; a transient 5xx gives up and strands the file. The row now injects 10, on a small file so the buffered `create` path answers with the retry POLICY rather than the next scan cycle happening to succeed. **The localfs crash row's audit assertion was vacuous.** "The audit returned nothing" is trivially true on an empty destination. The row now recovers to a real object FIRST, then plants the orphans, and asserts the audit returns **exactly one** result - which is the difference between "the temp files were filtered" and "the query found nothing". **Every row now proves its fault fired.** The S3 server keeps per-fault firing counters (`slow_down_fired`, `complete_error_documents_sent`, `quota_refusals`, `bucket_missing_refusals`, ...) and each row asserts on the one it armed; the localfs rows assert an error was surfaced. This is the `append-only-log` lesson applied to fault injection: a green run that never reached its fault is not a passing test, it is a test that did nothing. **Race windows are forced open, not hoped for** (the #192 technique): the S3 rows whose fault lands mid-transfer arm a 3 ms per-response delay so the intended path is taken on every run. ## Negative controls Each high-value row was re-run with the production defence it guards deliberately disabled, and had to go RED. All controls were reverted; `crates/driven-s3` and `crates/driven-localfs` are untouched on this branch. | Row | Defence disabled | Result | |---|---|---| | `localfs-crash-between-temp-write-and-rename` | `layout::is_control_entry` stops filtering `.driven-tmp-*` | **RED**: `an abandoned temp file must not appear as an object: expected the 1 real object, got ["backups/report.bin", "backups/.driven-tmp-..."]` | | `destination-vanished-across-backends` (localfs arm) | `LocalFsStore::guard_root` stops comparing the destination id | **RED**: `a marker belonging to a different volume must raise drive.dest_folder_missing, got []` | | `s3-complete-200-with-error-document` | all three of `driven-s3`'s defences, one at a time | **GREEN** until all three were gone, then **RED** with `big.bin recorded as synced with no object at the destination` | That last one is a finding in itself: a lying 200 is stopped **three independent ways**, any one of which suffices - (1) `complete_multipart` scans the body for `<Error` even on a 2xx; (2) failing that, the composed-ETag check finds no `<ETag>` in an error document and reports `ChecksumMismatch`; (3) failing both, `drain_and_maybe_complete` HEADs the key afterwards and an object that was never published answers 404. The row therefore asserts the OUTCOME rather than pinning one of the three, so a future refactor that consolidated the defences while keeping the guarantee does not produce a false failure. ## Finding: an abandoned S3 multipart upload leaks its parts The very first run of `s3-multipart-interrupted-between-parts` failed on "multipart upload left in flight", which turned out to be real. `S3Store` aborts a multipart upload on exactly two paths: the non-resumable `multipart_stream` failure path, and a completion failure `is_session_fatal` calls terminal. A **transport** failure mid-`UploadPart` on the **resumable** path is neither - `resume_chunk` propagates the error and leaves the upload id alive. That part is correct on its own: the startup `reconcile` -> `resume_persisted` path may still resume that exact session after a crash, and aborting eagerly would destroy a recoverable upload. The leak is on the other side. `executor.rs::upload_stage_resumable` - the streaming path every file >= 4 MiB takes - calls `open_resumable_session` **unconditionally** at the top of each attempt, so the next cycle's retry mints a fresh `CreateMultipartUpload` and never touches the previous upload id again. Its parts stay on the bucket, **billed**, one abandoned upload per failed attempt, and `S3Store::new` performs no sweep - unlike `driven-localfs`, which explicitly sweeps abandoned temp files at construction using the trait's own 6-day session window (a mechanism this PR's localfs row now covers). Not data loss, so no s6.3 invariant is violated. It is real money on a real bucket, and it is unbounded across repeated failures. **How the rows treat it:** they do NOT fail on it (that would make a merge-blocking gate red over a known, unfixed cost issue), and they do NOT ignore it. `settle_and_assert` takes a per-row cap, so the leak is BOUNDED - a row fails if more uploads strand than its faults account for, which is what would catch it growing. Every occurrence is reported as an explicit `FINDING (cost, not data loss)` note carrying the cause. The fix belongs in `driven-s3` (a `ListMultipartUploads` sweep at construction, mirroring the localfs one), not in the harness, so it is left for a follow-up `fix(s3):` rather than smuggled into a `test(chaos):` PR. ## What is NOT covered **The localfs `ENOSPC` errno path, end to end.** The out-of-space *behaviour* IS covered: the local-folder errno table maps `ENOSPC`/`EDQUOT` onto exactly the `DriveError::StorageQuota` that `s3-destination-full-mid-upload`'s `QuotaExceeded` produces, and that row drives the whole chain mid-multipart with bytes already at the destination (account pauses, nothing partial published, recovery once space is freed). What is missing is the errno itself, because a real `ENOSPC` needs a real constrained VOLUME: unprivileged on macOS (`hdiutil`), root-only on Linux (`losetup`), admin-only on Windows (`New-VHD`). The existing `disk-full-target` row sets the precedent - capability-gated behind `DRIVEN_CHAOS_ALLOW_DISK_MOUNT` - and a row gated the same way would SKIP in the Windows-only PR gate, the one job that blocks a merge. The errno table is already unit-tested in `driven-localfs`; what a chaos row adds beyond that is the orchestrator behaviour, which the S3 row proves. Recorded as a deliberate gap rather than shipped as a scenario that is always SKIPPED. ## Non-flakiness Sequential runs against the prebuilt debug binary (M5 MacBook Pro, macOS 26). Sequential on purpose: these rows bind loopback ports and share a debug-build CPU budget, so parallel runs would measure contention rather than flakiness. | Row | Runs | Pass | Fail | |---|---|---|---| | `s3-multipart-interrupted-between-parts` | 100 | 100 | 0 | | `s3-multipart-interrupted-before-complete` | 100 | 100 | 0 | | `s3-complete-200-with-error-document` | 100 | 100 | 0 | | `s3-slow-down-throttling` | 100 | 100 | 0 | | `s3-request-timeout-retryable-400` | 100 | 100 | 0 | | `s3-kill-mid-upload-then-reboot` | 100 | 100 | 0 | | `s3-destination-full-mid-upload` | 100 | 100 | 0 | | `localfs-crash-between-temp-write-and-rename` | 100 | 100 | 0 | | `destination-vanished-across-backends` | 100 (+100 pre-localfs-arm) | 100 | 0 | | **total** | **900** | **900** | **0** | Plus **22** full `run-all --fault-injection` sweeps (12 before the local-folder rows at `33 PASS / 2 SKIP`, 10 after at `34 PASS / 2 SKIP`), `0 FAIL / 0 FLAKY` every time - so the new rows do not interfere with the existing ones (shared ports, shared `target/chaos-fixtures/`). Worth stating plainly rather than implying the counts alone earned the confidence: **every one of these rows is deterministic by construction.** The fault is an armed flag or a constructed on-disk state, not a timing race, and the fault-fired counters prove the intended path was taken on each run. That is what makes 100 runs meaningful instead of 100 coin flips that happened to land the same way - and it is the same technique #192 used to convert `append-only-log` from a 5-10% flake into a deterministic row. Per-row wall clock is 65-610 ms locally and 236-779 ms on the Windows runner (see the CI section), so the nine rows add ~5 s across both Windows chaos jobs. ## CI on windows-latest, the platform the gate actually runs All nine new rows pass on `windows-latest`, which matters because the S3 server hand-rolls HTTP/1.1 framing and the local-folder rows depend on `rename` semantics and directory handling: ``` s3-multipart-interrupted-between-parts pass 721 ms s3-multipart-interrupted-before-complete pass 726 ms s3-complete-200-with-error-document pass 662 ms s3-slow-down-throttling pass 241 ms s3-request-timeout-retryable-400 pass 629 ms s3-kill-mid-upload-then-reboot pass 779 ms s3-destination-full-mid-upload pass 664 ms localfs-crash-between-temp-write-and-rename pass 236 ms destination-vanished-across-backends pass 455 ms ``` - `chaos fake-drive (windows-latest)`: **36 PASS / 0 SKIP / 0 FAIL / 0 FLAKY** - `chaos hermetic (windows-latest)`: **84 PASS / 10 SKIP / 0 FAIL / 0 FLAKY** Total added cost: ~5.1 s across both Windows jobs. ## Pre-existing failure, macOS-only, unrelated to this branch `run-all --hermetic` on the final rebased tree reports **one** FAIL on this macOS host: `noaccess-file` ("exactly one local.io_error (the unreadable file); got 0"). Verified pre-existing by stashing every change and running it against unmodified `main`, where it fails identically - and it PASSES on `windows-latest` in this PR's own run, which confirms it is a macOS permissions-model gap rather than anything this branch touched. The chaos gate is Windows-only on a PR and macOS only joins on a `v*` tag, so it does not block here, but it is a real gap worth its own issue: on a `v*` tag the macOS leg would go red. ## Gates Local (M5 MacBook Pro, on the rebased tree): - `SQLX_OFFLINE=true cargo test --workspace` - green (53 test binaries, 0 failures) - `cargo clippy --workspace --all-targets -- -D warnings` - clean - `cargo fmt --all -- --check` - clean - `cargo run -p driven-chaos -- run-all --fault-injection` - 34 PASS / 2 SKIP / 0 FAIL (the 2 SKIPs are the Windows-only mutator rows) - ASCII-only, LF endings verified via `git ls-files --eol` CI (all against the pushed commit, which matches local `HEAD`): `rustfmt`, `cargo deny`, `coverage`, `ui build + lint + unit`, all three `tauri compile` legs, `CodeQL` / `Analyze (rust, actions, javascript-typescript)`, `chaos hermetic (windows-latest)`, `chaos fake-drive (windows-latest)`, and all three `cargo test + clippy` legs - **every check green**. The only non-passing row is `real-drive e2e (tag-only)`, which skips by design outside a `v*` tag. `pnpm --dir ui test:unit` was **not** run locally: `ui/node_modules` is absent in this worktree and the diff touches no UI file, so CI's `ui build + lint + unit` job (green) is the honest check rather than a local install. Stated rather than claimed.
pmaxhogan
added a commit
that referenced
this pull request
Jul 30, 2026
…229) Closes #218. The v2.5.0 tag's Chaos run failed on `noaccess-file` on BOTH ubuntu and macOS with: ``` scenario errored: exactly one local.io_error (the unreadable file); got 0 ``` ## Why #195 deliberately reclassified unix permission-denied opens: EACCES/EPERM now produce a graceful SKIP carrying `local.permission_denied` (a WARN, not counted in the cycle's error total) instead of failing as `local.io_error`. That was the point - on macOS a TCC denial is the common case, and "Driven hit a disk error" sent users to check a healthy disk. Windows keeps ERROR_ACCESS_DENIED as `local.io_error`, because elevation can genuinely read around some ACL denials. `posix-mode-000` was updated in #195. This sibling row was missed. ## Why it went unnoticed CI runs the chaos jobs on Windows, where the row still passes. So the suite was green while the row was broken for every unix contributor - and it only surfaced when the `v*` tag fired the full matrix. That is exactly the failure mode #192 fixed for the append-only-log flake: a row nobody can get green locally is a row everyone learns to ignore. ## The fix Both the run-time assertion and the declared `ExpectedOutcome` are now per-platform. They had to move together: the harness compares the declared outcome against the codes the run actually observed, so fixing only the assertion left it failing with `expected graceful failure with local.io_error but observed codes: [LocalPermissionDenied]`. ## Evidence - The row passes 5/5 consecutive runs on macOS. - Full hermetic suite locally: **63 PASS / 31 SKIP / 0 FAIL** - the first clean local chaos run on this machine. - **Negative control:** restoring the old `io_errors == 1` expectation makes it fail again, so the new assertion is load-bearing rather than self-confirming. Note this does not change any product behaviour - only what the harness expects. The skip-and-report path it now pins is the one #195 shipped and that the GUI release gate confirmed end to end (a chmod 000 file produced `reason=Denied code=local.permission_denied` and raised the Full Disk Access banner). --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR - the orphan race is already fixed; what was missing is a gate that proves it
The brief asked for a
fix(core)closing the #144 create-orphan race behind theappend-only-logchaos flake. I reproduced the race, then established thatPR #146 (merged 2026-07-24) already closed it. The stale piece was the
scenario itself: it only hit the racy code path by luck, so it neither caught
the bug reliably before the fix nor proves the fix now. This PR makes it
deterministic. No core behaviour changes - hence
test(chaos):, notfix(core):.What the race is
The first upload of
app.logis a plain CREATE (nodrive_file_idyet). Anappend that lands between the hash and the SPEC s8 post-upload
fstatmakesthe executor settle it as changed-after-upload. Pre-#146 that left a LIVE
object with no
file_staterow, adoptable only by the startup-gated reconcilepass, so the next mid-session scan planned a SECOND create. The scenario's
"after reconcile, expected exactly 1 log object, found N" assertion is exactly
that duplicate showing up.
Since #146 the executor instead commits a force-rescan
file_staterowpointing at the just-created object and drops the op in one transaction
(
settle_post_upload_change->commit_create_result), so the next scanUPDATEs that object. One object, no restart needed.
Evidence (M4 Mac, debug build, 4-6 way parallel,
driven-chaos scenario run append-only-log)I gated the #146 settle behind a temporary env switch to get a true negative
control, then removed it.
Two things fall out of that table:
reverted, and is gone with the fix in - so the flake really was Executor create-SkipPostUpload orphan can duplicate objects until next restart (frequent-edits race) #144, and
fix(core): commit file_state for a create that skipped post-upload so the next scan updates instead of re-creating #146 really closed it.
occasionally. A green run therefore did not mean the create-orphan path
was tested; it usually meant the window never opened. That is the actual
remaining defect.
The change
AppendOnlyLognow boots overInMemoryRemoteStore::with_slow_responses(3 ms)(
CREATE_RACE_DELAY), deliberately under the 4 msMUTATE_EVERYmutationcadence, so at least one append lands inside the create's upload window on
every run. This is the same widening technique the
mid-upload-*rowsalready use with
SLOW_REMOTE.world ("leaves an orphan ... which is accepted behaviour"). The reconcile +
drain step stays - it now stands in for the app restart that backstops the
arms fix(core): commit file_state for a create that skipped post-upload so the next scan updates instead of re-creating #146 deliberately left to reconcile (crash between upload and settle,
versioned create, ambiguous
DeferToReconcile) rather than papering over aroutine duplicate.
Cost: +246 ms of wall clock per run of one scenario on the Windows CI runner
(measured below).
Verification on windows-latest (the platform the flake was reported on)
The measurements above are from macOS, so here is the CI run of this branch's
chaos hermetic (windows-latest)job:append-only-log: pass,duration_ms: 523. Baseline for the samescenario on the same runner image, from an unmodified branch (fix(scanner): route the deep-verify hash through the platform-open helper #193's run):
duration_ms: 277. So the real cost of forcing the window open is +246 mson the 2-core Windows runner, not the ~150 ms I estimated from the Mac.
window does not destabilise the drain cap or any sibling row.
chaos fake-drive (windows-latest)also green (this row is hermetic-only, butthe sibling
mutator-fs-append-only-logruns there: pass, unchanged).Verification on macOS
driven-chaos scenario run append-only-logx 300 consecutive runs: 300 pass, 0 failwith this change applied.
driven-chaos run-all --hermetic: 54 PASS / 31 SKIP / 0 FAIL / 0 FLAKY.driven-chaos run-all --fault-injection: 25 PASS / 2 SKIP / 0 FAIL / 0 FLAKY.SQLX_OFFLINE=true cargo test --workspace: green.cargo clippy --workspace --all-targets -- -D warnings: clean.cargo fmt --all -- --check: clean.Follow-up worth knowing
The
driven-append-only-log-chaos-flaketriage note ("re-run the Chaos job, itclears ~85-90% of the time") predates #146 and is now wrong: a duplicate-object
failure on this row should be treated as a real regression, not a re-run.