Skip to content

perf(db): let SnapshotRoot cache refills share a read lock - #7104

Open
halibobo1205 wants to merge 1 commit into
tronprotocol:release_v4.8.3from
halibobo1205:feature/snapshot-root-refill-rwlock
Open

halibobo1205 wants to merge 1 commit into
tronprotocol:release_v4.8.3from
halibobo1205:feature/snapshot-root-refill-rwlock

Conversation

@halibobo1205

Copy link
Copy Markdown
Collaborator

What does this PR do?

Follow-up to #7078. Replaces the SnapshotRoot cache monitor with a ReentrantReadWriteLock.

  • A refill rechecks the generation and puts into the cache under the read lock, so concurrent refills on the same store are no longer serialized by the SnapshotRoot cache lock.
  • The writer advances the generation under the write lock, then publishes the cache entries outside it, as before.
  • Cache hits do not acquire the SnapshotRoot cache lock. The refill protocol and the write ordering from feat(db): improve SnapshotRoot cache consistency #7078 are unchanged.

Why are these changes required?

#7078 puts every cache-miss refill of a store behind one monitor. Before it, Guava's segments let those refills run in parallel. A refill only has to exclude the generation advance, not other refills.

Local microbenchmark, not node-level: JDK 17 on a 10-core Apple Silicon machine, Guava 33.6.0 with the account / storage-row cache spec (maximumSize=20000, expireAfterAccess=30s, concurrencyLevel=CPUS), a cold-key workload with approximately 99% cache misses, the DB read is a busy-wait, and the writer advances the generation every 50 ms. In production, caught-up block processing normally refreshes each store once per block, roughly every 3 seconds; catch-up and startup have different write patterns.

Reader threads DB read Reads/s before #7078 #7078 This PR Writer wait p99, #7078 This PR
10 2 µs 1,901k 1,045k 1,818k 238 µs 72 µs
10 10 µs 774k 702k 759k 237 µs 37 µs
32 2 µs 2,077k 1,003k 1,944k 843 µs 108 µs
32 10 µs 791k 680k 769k 1.5 ms 71 ms
  • Read throughput is within about 6% of the pre-feat(db): improve SnapshotRoot cache consistency #7078 numbers. feat(db): improve SnapshotRoot cache consistency #7078 cuts it by 45–52% with a 2 µs read and by 9–14% with a 10 µs read.
  • Trade-off: the writer now waits for every in-flight refill to finish. With 32 busy reader threads on 10 cores, its p99 was higher than with feat(db): improve SnapshotRoot cache consistency #7078 in 3 of 4 runs (19–71 ms). The table shows the cold-key runs; the other two used a 50% hot-key mix. This regression was observed under oversubscribed benchmark load. A refill descheduled while holding the read lock can delay the writer; the impact under production workloads has not been measured.
  • Each cell is a single run: 1.5 s warm-up, then 6 s measured. Keys are 21 random bytes drawn uniformly from 2M keys (the hot-key runs draw half the reads from 5,000 keys). Each run has about 120 writer samples, so the writer p99 is close to the second-largest sample.

This PR has been tested by:

  • Unit Tests (JDK 17, arm64): SnapshotRootCacheRaceTest (22 cases, rerun 5 times), the org.tron.core.db2 package, TronCacheTest, AccountAssetStoreTest, BandwidthProcessorTest.
  • Manual Testing: mutation checks. Each of these fails the race tests: dropping the recheck under the lock; refilling under the write lock or without a lock; advancing the generation without the lock, under the read lock, or after publishing; releasing the read lock only when the recheck passes.

Follow up

None.

Extra details

Test changes:

  • staleReadWaitingOnLockIsSkipped holds the write lock and waits until the reader is queued on it (hasQueuedThread). A thread waiting on a java.util.concurrent lock is WAITING, not BLOCKED.
  • putWaitsForInFlightRefill / mergeListWaitsForInFlightRefill detect the queued writer the same way.
  • New concurrentRefillsDoNotBlockEachOther: a refill completes and is cached while another refill holds the read lock.
  • assertUnlocked checks that no read or write hold is left after a refill.
  • park() now fails if it is never resumed, so a missed interleaving cannot pass silently.

Alternatives considered

  • Exclusive ReentrantLock with tryLock, skipping a refill that finds the lock taken. Refills do not wait for the SnapshotRoot cache lock, but colliding refills are dropped (51–99% in the same benchmark). Dropped refills can cause additional DB reads if those keys are accessed again before being cached. Its higher raw throughput comes from skipping the cache put, which matters less as DB latency grows. Its writer p99 at 10 threads was also higher (0.7–4.2 ms).
  • readLock().tryLock() without a timeout. It bypasses the reader admission policy and can barge past a queued writer, so a steady stream of refills could keep the flush thread waiting.
  • readLock().tryLock(0, TimeUnit.NANOSECONDS). With the default nonfair lock it follows the same reader admission policy as lock(), which is not strictly fair. It needs InterruptedException handling, and its writer wait was the longest of all variants (p50 1–39 ms). A refill that captured the generation before the writer's advance fails the recheck after waiting anyway, so skipping it would not change what that refill caches.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

topic:DB Database

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

1 participant