Skip to content

fix(rocksdb): make snapshot restore retryable - #3293

Merged
imbajin merged 11 commits into
apache:masterfrom
hugegraph:task/rocksdb-recovery-20261005
Oct 9, 2026
Merged

imbajin merged 11 commits into
apache:masterfrom
hugegraph:task/rocksdb-recovery-20261005

Conversation

@imbajin

@imbajin imbajin commented Oct 8, 2026 •

Copy link
Copy Markdown
Member

A failed RocksDB snapshot restore can leave live sessions referring to a closed database or lose the checkpoint needed for retry. This change keeps a shared database owner and preserves the checkpoint until restore and reopening both succeed.

  • Hold the recovery lock across close, restore and reopen; synchronize marker-directory changes on Linux.
  • Keep pending recovery state after failures so the next attempt can retry.
  • Clean the checkpoint only after successful reopening.
image

Based on current Apache master, preserving the merged ToplingDB provider checks and native resource management. This improves the shared RocksDB-compatible recovery path; it does not add a ToplingDB feature or promise recovery from arbitrary data corruption.

Validation: Java 17 compilation and focused RocksDB session, snapshot restore and directory-sync failure regressions pass. Optional kernel double-bind fixture cases were not run in this environment. The documented recovery scope covers I/O failures and process interruption; complete power-loss durability of copied data/WAL files is outside this change. See docs/rocksdb-recovery.md for recovery and retry behavior.

Doc - Done.

imbajin added 11 commits October 5, 2026 02:40
- Keep the newest run for each workflow and PR
- Verify source freshness before automatic retries
- Test retry decisions and document the CI policy
- hold physical database leases through close and reopen
- bind pending metadata to its checkpoint and WAL
- validate complete records before repairing aliases
- reclaim only unused operation-owned checkpoint copies
- cover interrupted retry and physical ownership boundaries
- check pending state before creating another checkpoint
- hold recovery ownership through copy creation and cleanup
- preserve existing pending sources without directory growth
- verify repeated failures and multi-database cleanup
- follow the requested Markdown paragraph style
- retain the validated recovery and compatibility guidance
- keep code snippets and path references intact
- incorporate the current Java 17 baseline
- retain the scoped PR changes
- preserve branch history without rewriting commits
- preserve WAL tails across physical directory aliases
- reject unsafe links before destructive recovery
- fail closed when marker existence is unknown
- release failed stores for retry and strengthen tests
- write and force owned metadata staging before publication
- publish pending markers without replacing existing records
- preserve live data and retry after partial write failures
- document same-directory hard-link requirements
- compare aliased ancestors and relative suffixes
- recheck identity before install and owned cleanup
- preserve live data on unsafe checkpoint layouts
- gate permission faults by observed access checks
- preserve merged Topling provider and resource handling
- retain retryable snapshot recovery and shared ownership
- adapt regression tests to current exception packages
- sync Linux marker entries before replacing live data
- retain checkpoints when directory sync fails
- verify ordering and describe durability limits
@codecov

codecov Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 18.48617% with 560 lines in your changes missing coverage. Please review.
✅ Project coverage is 41.40%. Comparing base (5039e5b) to head (2d335c9).

Files with missing lines Patch % Lines
.../backend/store/rocksdb/RocksDBSnapshotRestore.java 5.86% 377 Missing and 8 partials ⚠️
...raph/backend/store/rocksdb/RocksDBStdSessions.java 38.41% 91 Missing and 10 partials ⚠️
.../hugegraph/backend/store/rocksdb/RocksDBStore.java 14.70% 26 Missing and 3 partials ⚠️
.../hugegraph/backend/store/rocksdb/RecoveryLock.java 50.98% 17 Missing and 8 partials ⚠️
...hugegraph/backend/store/rocksdb/OpenedRocksDB.java 32.14% 15 Missing and 4 partials ⚠️
...gegraph/backend/store/rocksdb/RocksDBSessions.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3293      +/-   ##
============================================
- Coverage     41.87%   41.40%   -0.48%     
+ Complexity     6969     6926      -43     
============================================
  Files           768      762       -6     
  Lines         66436    66773     +337     
  Branches       8831     8939     +108     
============================================
- Hits          27823    27646     -177     
- Misses        35468    35968     +500     
- Partials       3145     3159      +14     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@imbajin
imbajin merged commit d00efce into apache:master Oct 9, 2026
30 of 32 checks passed
@imbajin
imbajin deleted the task/rocksdb-recovery-20261005 branch October 10, 2026 10:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants