Skip to content

fix(core): commit file_state for a create that skipped post-upload so the next scan updates instead of re-creating - #146

Merged
pmaxhogan merged 2 commits into
mainfrom
fix/create-skippostupload-orphan
Jul 24, 2026
Merged

pmaxhogan merged 2 commits into
mainfrom
fix/create-skippostupload-orphan

Conversation

@pmaxhogan

Copy link
Copy Markdown
Owner

Mechanism (#144)

When a plain first-CREATE upload's local file changes during the hash->upload->post-check window, the executor returned SkipPostUpload and kept the pending op with no file_state row, relying on the startup-gated reconcile pass to adopt the just-created object by op-uuid. Mid-run nothing adopts it, so the next scan saw a row-less path and planned a second create - duplicating live objects until the next app restart. The chaos frequent-edits gate saw ~20 duplicate hot.txt objects under a 3ms injected store delay (found during #141's CI run; pre-existing in v2.0.1, not introduced by #141).

Fix - symmetry with reconcile's adopt-requeue arm

upload_and_commit's post-upload identity recheck now routes through settle_post_upload_change. For a plain create (no pre-existing row, not a versioned change) it durably commits the same force-rescan file_state row that reconcile's adopt_reconciled requeue arm would eventually write:

  • drive_file_id = the just-created object,
  • mtime_ns = REQUEUE_FORCE_RESCAN_MTIME_NS (i64::MIN) so the next FastPath scan always re-emits the path,
  • the uploaded (now-stale) blake3, the on-Drive md5/size, status Pending,

via commit_create_result, which upserts the row and deletes the op in one transaction. The next scan re-uploads as an UPDATE against the same object: exactly one live object per path, no restart required.

Design note

The issue sketched "thread the created id through the error path". The id is already in scope at the detection point, so the helper commits in place and returns Ok(Skipped) - a smaller seam into the hardened UploadError enum with an identical durable outcome.

Invariants preserved

  • Atomic durable commit: row upsert + op delete are one transaction (commit_create_result) - no split write.
  • Crash-safety fallback: a crash before the commit leaves the op (with its op-uuid) and no row, so the pre-Executor create-SkipPostUpload orphan can duplicate objects until next restart (frequent-edits race) #144 orphan+reconcile adoption still runs on the next boot (covered by crash_mid_upload_adopts_orphan_without_duplicate).
  • Reconcile idempotency: the commit deletes the op atomically with the row, so reconcile never finds this op again - no double-adopt (verified vacuous + covered by a restart test).
  • UPDATE and versioned-create keep the pre-Executor create-SkipPostUpload orphan can duplicate objects until next restart (frequent-edits race) #144 SkipPostUpload path (their existing row already points at an object; the next scan updates/versions against it, never re-creating the path).
  • Encryption: the created id + ciphertext encrypted_remote_path flow into the row identically to a plaintext source.
  • DeferToReconcile (ambiguous create) is untouched - it has no known id to point at, so it stays reconcile-driven (comment updated to explain the distinction).

Tests

  • create_changed_after_upload_commits_force_rescan_row_no_duplicate - deterministic race, then the NEXT scan in the same process UPDATEs the same id (exactly one live object).
  • create_skip_post_upload_row_survives_reconcile_no_double_adopt - restart mid-window: committed row + reconcile pass stays one object.
  • update_changed_after_upload_keeps_existing_id_no_duplicate - update carve-out unchanged.
  • create_changed_after_upload_encrypted_commits_force_rescan_row - encrypted path.
  • frequent_edits_slow_store_holds_one_object_144 - chaos frequent-edits with a 3ms injected store delay holds exactly one object.

Full driven-core lib suite (336) + e2e_fake acceptance suite (24) + clippy green.

Closes #144

🤖 Generated with Claude Code

https://claude.ai/code/session_01QZQVP2tUuTLh8oL31D8heC

@github-actions

github-actions Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Coverage

Area main this PR delta
Rust (lib crates) 79.16% 79.26% +0.10 (OK)
UI (vue/ts) 89.71% 89.71% +0.00 (OK)

Gate: passed - no coverage regression (epsilon 0.1 pp).

pmaxhogan and others added 2 commits July 24, 2026 16:34
… the next scan updates instead of re-creating

When a plain first-CREATE upload's local file changes during the
hash->upload->post-check window, the executor returned SkipPostUpload and
kept the pending op with NO file_state row, relying on the startup-gated
reconcile pass to adopt the just-created object by op-uuid. Mid-run nothing
adopts it, so the next scan saw a row-less path and planned a SECOND create -
duplicating live objects until the next app restart (issue #144; the chaos
frequent-edits gate saw ~20 duplicate hot.txt objects under a 3ms store
delay).

Make create-SkipPostUpload symmetric with reconcile's adopt-requeue arm: for
a plain create (no pre-existing row, not a versioned change),
settle_post_upload_change durably commits the SAME force-rescan file_state
row that arm would eventually write - drive_file_id = the just-created
object, mtime_ns = REQUEUE_FORCE_RESCAN_MTIME_NS (i64::MIN) so the next
FastPath scan always re-emits the path, the uploaded (now-stale) blake3, the
on-Drive md5/size, status Pending - via commit_create_result, which upserts
the row and deletes the op in ONE transaction. The next scan then re-uploads
as an UPDATE against the same object: exactly one live object per path, no
restart required.

Design deviates slightly from the issue's "thread the id through the error"
sketch: the object id is available at the detection point, so the helper
commits in place and returns Ok(Skipped) - a smaller seam into the hardened
UploadError enum with an identical durable outcome.

Invariants preserved:
- Durable-commit ordering / atomicity: the row upsert + op delete are one
  transaction (commit_create_result), so no split write.
- Crash-safety fallback: a crash before the commit leaves the op (still
  carrying its op-uuid) with no row, so the pre-#144 orphan+reconcile
  adoption runs on the next boot exactly as before.
- Reconcile idempotency: the commit deletes the op atomically with the row,
  so reconcile never finds this op again - no double-adopt.
- In-place UPDATE and versioned-create keep the pre-#144 SkipPostUpload path
  (their existing row already points at an object; the next scan updates or
  versions against it, never re-creating the path).
- Encryption: the created id + ciphertext encrypted_remote_path flow into the
  row identically to a plaintext source.
- No-delete-on-metadata-error and the DeferToReconcile ambiguous-create path
  are untouched (that path has no known id to point at, so it stays
  reconcile-driven).

Tests:
- create_changed_after_upload_commits_force_rescan_row_no_duplicate: the
  deterministic race (post-upload hook) then the NEXT scan in the same process
  UPDATEs the same object id - exactly one live object.
- create_skip_post_upload_row_survives_reconcile_no_double_adopt: restart
  mid-window - the committed row + a reconcile pass stays one object.
- update_changed_after_upload_keeps_existing_id_no_duplicate: the update
  carve-out is unchanged.
- create_changed_after_upload_encrypted_commits_force_rescan_row: the
  encrypted path.
- frequent_edits_slow_store_holds_one_object_144: the chaos frequent-edits
  scenario with a 3ms injected store delay holds exactly one object.

Closes #144

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QZQVP2tUuTLh8oL31D8heC
@pmaxhogan
pmaxhogan force-pushed the fix/create-skippostupload-orphan branch from 139efe3 to f58fcf4 Compare July 24, 2026 21:34
@pmaxhogan
pmaxhogan enabled auto-merge (squash) July 24, 2026 21:57
@pmaxhogan
pmaxhogan merged commit f5230d1 into main Jul 24, 2026
18 checks passed
@pmaxhogan
pmaxhogan deleted the fix/create-skippostupload-orphan branch July 24, 2026 21:57
@github-project-automation github-project-automation Bot moved this from Todo to Done in Driven Jul 24, 2026
pmaxhogan added a commit that referenced this pull request Jul 24, 2026
🤖 I have created a release *beep* *boop*
---


## [2.1.0](v2.0.1...v2.1.0)
(2026-07-24)


### Features

* **core:** adaptive upload parallelism with throughput probe and
disk-saturation gate
([#143](#143))
([8ecced6](8ecced6))
* **core:** filesystem timestamp-granularity probe with ctime fallback
and per-directory gitignore cascade
([#141](#141))
([344262c](344262c))
* **drive:** support Google Shared Drive destinations end-to-end
([#142](#142))
([d9c3161](d9c3161))
* **net:** native OS reachability backends with automatic fallback
([#138](#138))
([319e85f](319e85f))
* **net:** SOCKS5 and PAC proxy support for all outbound connections
([#145](#145))
([2f0b7d1](2f0b7d1))
* **net:** support a custom corporate root CA for all outbound
connections ([#134](#134))
([929e93d](929e93d))
* per-source toggle to back up OneDrive cloud-only placeholder files
([#133](#133))
([6863ea3](6863ea3))
* **telemetry:** capture latency percentiles and add rollup query
endpoint ([#132](#132))
([4e9fde6](4e9fde6))
* **telemetry:** preview exactly what a telemetry ping sends
([#139](#139))
([95fbd9a](95fbd9a))


### Bug Fixes

* **core:** commit file_state for a create that skipped post-upload so
the next scan updates instead of re-creating
([#146](#146))
([f5230d1](f5230d1))
* **deps:** bump tauri-winrt-notification to drop vulnerable quick-xml
(closes [#89](#89))
([#129](#129))
([232fd8f](232fd8f))
* **telemetry:** exclude pre-schema rows from latency rollup
([#137](#137))
([1ae6220](1ae6220))
* **ui:** add cursor pointer to buttons and link-buttons
([#136](#136))
([dbd4809](dbd4809))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
pmaxhogan added a commit that referenced this pull request Jul 24, 2026
#147)

## Why

Main CI run 30129432441 (post-#146 merge) flaked on windows-latest:
`scan_reports_probe_only_when_unpersisted` asserted the probe returns
exactly `Some(0)` but got `Some(2511100)` (~2.5ms).

NTFS stamps file times from the interrupt-time clock, which advances in
~0.5-16ms ticks. When the probe's 8-rewrite burst lands inside a single
tick, phase 2's first 2ms sleep sees the mtime advance and correctly
reports a millisecond-scale window - a coin flip on fast runners. The
probe is right; the assertion was too strict.
`probe_classifies_local_fs_fine` had the same latent `== 0` assertion.

## Fix

Both tests now assert the probed granularity is present and
**non-coarse** (`!granularity_is_coarse(g)`, i.e. <= 1s) - the boundary
that actually drives behavior (the re-hash fallback) - instead of
exactly zero. Test-only change; no production code touched.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_016YnYLjHgYhhnJjUxojsRhn

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
pmaxhogan added a commit that referenced this pull request Jul 29, 2026
…e append-only-log soak (#192)

## TL;DR - the orphan race is already fixed; what was missing is a gate
that proves it

The brief asked for a `fix(core)` closing the #144 create-orphan race
behind the
`append-only-log` chaos flake. I reproduced the race, then established
that
**PR #146 (merged 2026-07-24) already closed it**. The stale piece was
the
scenario itself: it only hit the racy code path by luck, so it neither
caught
the bug reliably before the fix nor proves the fix now. This PR makes it
deterministic. **No core behaviour changes** - hence `test(chaos):`, not
`fix(core):`.

## What the race is

The first upload of `app.log` is a plain CREATE (no `drive_file_id`
yet). An
append that lands between the hash and the SPEC s8 post-upload `fstat`
makes
the executor settle it as changed-after-upload. Pre-#146 that left a
LIVE
object with no `file_state` row, adoptable only by the startup-gated
reconcile
pass, so the next mid-session scan planned a SECOND create. The
scenario's
"after reconcile, expected exactly 1 log object, found N" assertion is
exactly
that duplicate showing up.

Since #146 the executor instead commits a force-rescan `file_state` row
pointing at the just-created object and drops the op in one transaction
(`settle_post_upload_change` -> `commit_create_result`), so the next
scan
UPDATEs that object. One object, no restart needed.

## Evidence (M4 Mac, debug build, 4-6 way parallel, `driven-chaos
scenario run append-only-log`)

I gated the #146 settle behind a temporary env switch to get a true
negative
control, then removed it.

| build | injected remote delay | runs | pass | duplicate-object
failures |
|---|---|---|---|---|
| main (with #146) | none | 690 | 690 | **0** |
| main (with #146) | 3 ms | 240 | 240 | **0** |
| main (with #146) | 10 ms | 20 | 20 | **0** |
| #146 settle reverted | none | 200 | 194 | **6** (3.0%) |
| #146 settle reverted | 3 ms | 20 | 0 | **20** (100%, 2-14 objects
each) |

Two things fall out of that table:

1. The historical ~5-10% flake is reproduced at 3% on this host with the
fix
reverted, and is gone with the fix in - so the flake really was #144,
and
   #146 really closed it.
2. Without an injected delay the scenario reaches the racy code path
only
occasionally. A green run therefore did **not** mean the create-orphan
path
was tested; it usually meant the window never opened. That is the actual
   remaining defect.

## The change

- `AppendOnlyLog` now boots over
`InMemoryRemoteStore::with_slow_responses(3 ms)`
(`CREATE_RACE_DELAY`), deliberately under the 4 ms `MUTATE_EVERY`
mutation
cadence, so at least one append lands inside the create's upload window
on
  every run. This is the same widening technique the `mid-upload-*` rows
  already use with `SLOW_REMOTE`.
- Refreshed the two stale comment blocks that still described the
pre-#146
world ("leaves an orphan ... which is accepted behaviour"). The
reconcile +
drain step stays - it now stands in for the app restart that backstops
the
arms #146 deliberately left to reconcile (crash between upload and
settle,
versioned create, ambiguous `DeferToReconcile`) rather than papering
over a
  routine duplicate.

Cost: +246 ms of wall clock per run of one scenario on the Windows CI
runner
(measured below).

## Verification on windows-latest (the platform the flake was reported
on)

The measurements above are from macOS, so here is the CI run of this
branch's
`chaos hermetic (windows-latest)` job:

- `append-only-log`: **pass**, `duration_ms: 523`. Baseline for the same
scenario on the same runner image, from an unmodified branch (#193's
run):
`duration_ms: 277`. So the real cost of forcing the window open is
**+246 ms**
on the 2-core Windows runner, not the ~150 ms I estimated from the Mac.
- Whole suite: **75 PASS / 10 SKIP / 0 FAIL / 0 FLAKY**, so widening
this row's
  window does not destabilise the drain cap or any sibling row.
- `chaos fake-drive (windows-latest)` also green (this row is
hermetic-only, but
  the sibling `mutator-fs-append-only-log` runs there: pass, unchanged).

## Verification on macOS

- `driven-chaos scenario run append-only-log` x **300 consecutive runs:
300 pass, 0 fail**
  with this change applied.
- `driven-chaos run-all --hermetic`: 54 PASS / 31 SKIP / 0 FAIL / 0
FLAKY.
- `driven-chaos run-all --fault-injection`: 25 PASS / 2 SKIP / 0 FAIL /
0 FLAKY.
- `SQLX_OFFLINE=true cargo test --workspace`: green.
- `cargo clippy --workspace --all-targets -- -D warnings`: clean.
- `cargo fmt --all -- --check`: clean.

## Follow-up worth knowing

The `driven-append-only-log-chaos-flake` triage note ("re-run the Chaos
job, it
clears ~85-90% of the time") predates #146 and is now wrong: a
duplicate-object
failure on this row should be treated as a **real regression**, not a
re-run.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Executor create-SkipPostUpload orphan can duplicate objects until next restart (frequent-edits race)

1 participant