Skip to content

Fix concurrent JVM thread state metric refresh - #18950

Merged
jt2594838 merged 1 commit into
masterfrom
fix/jvm-thread-metrics-npe
Oct 21, 2026
Merged

jt2594838 merged 1 commit into
masterfrom
fix/jvm-thread-metrics-npe

Conversation

@jt2594838

Copy link
Copy Markdown
Contributor

Description

Concurrent JVM thread-state gauge reads can overlap with clearing and rebuilding the shared EnumMap. Its non-atomic getOrDefault checks can return null during repopulation, causing an NPE when the count is unboxed.

Synchronize the complete refresh-and-read operation with a class-level lock, matching the static cache's scope. Missing states continue to return zero, and the sampling interval remains unchanged.

Add regression coverage for missing states, null thread information, cached sampling, and a concurrent read while a refresh is paused after clearing the cache.

Validation

  • mvn spotless:apply -pl iotdb-core/metrics/interface -am -Drat.skip=true
  • mvn test -pl iotdb-core/metrics/interface -am -Dtest=JvmThreadMetricsTest -Dsurefire.failIfNoSpecifiedTests=false -Drat.skip=true: 2 tests passed; reactor build succeeded, including Checkstyle and Spotless checks.
  • Running the regression tests against the original implementation fails the concurrent-refresh case with an expected count of 1 and an actual count of 0. The test deterministically covers intermediate cache visibility rather than directly reproducing the narrower NPE interleaving.

This PR has:

  • been self-reviewed for concurrent reads and writes.
  • added comments explaining the synchronization requirement and test scenarios.
  • added unit tests covering the changed behavior.
Key changed/added classes
  • JvmThreadMetrics
  • JvmThreadMetricsTest


// VisibleForTesting
static long getThreadStateCount(ThreadMXBean threadBean, Thread.State state) {
static synchronized long getThreadStateCount(ThreadMXBean threadBean, Thread.State state) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Synchronize the entire cache refresh and lookup because the shared EnumMap is mutated in place. This class-level lock makes the read atomic with respect to clear/compute, preventing a concurrent gauge collection from observing a transient null value and unboxing it as a long. The 10-second sampling behavior is unchanged; the targeted JUnit regression passes in the reactor test run.

// Pause a refresh after clearing the cache, then read the same state concurrently. Both calls
// must return the complete sample rather than a temporarily absent entry.
@Test
public void testConcurrentReadDuringRefresh() throws Exception {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This regression pauses the first refresh after the cache is cleared and starts a concurrent gauge read. The expected count verifies that the reader waits for the refresh and never observes the intermediate empty EnumMap; the missing-state case above separately verifies the documented zero default. The original implementation fails this test with expected 1 but actual 0, while the synchronized implementation passes.

@Caideyipi Caideyipi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the implementation and regression tests. No blocking issues found; formatting check is clean.

@codecov

codecov Bot commented Oct 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 47.06%. Comparing base (f8135bb) to head (0522ab5).
⚠️ Report is 53 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master   #18950      +/-   ##
============================================
+ Coverage     46.03%   47.06%   +1.02%     
- Complexity      712      737      +25     
============================================
  Files          5509     5539      +30     
  Lines        400878   405207    +4329     
  Branches      52375    53028     +653     
============================================
+ Hits         184536   190701    +6165     
+ Misses       216342   214506    -1836     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jt2594838
jt2594838 merged commit 5a437f5 into master Oct 21, 2026
39 of 40 checks passed
@jt2594838
jt2594838 deleted the fix/jvm-thread-metrics-npe branch October 21, 2026 04:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants