Skip to content

[PIPE][Subscription] Fix pipe races and subscription WAL replay wakeup - #18904

Open
Caideyipi wants to merge 5 commits into
apache:masterfrom
Caideyipi:codex/fix-pipe-auto-drop-start-race
Open

Caideyipi wants to merge 5 commits into
apache:masterfrom
Caideyipi:codex/fix-pipe-auto-drop-start-race

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Description

This PR contains three reliability fixes on the current branch:

  • Preserve historical TsFile mods after compaction has removed the original resource, and release exactly the reference acquired during historical selection.
  • Remove redundant START calls from the finite pipe auto-drop integration test, because CREATE starts those pipes automatically and the extra calls can race with auto-drop.
  • Wake subscription WAL replay after a realtime consensus request is rejected by the shared request-memory limit.

Subscription replay wakeup

When a realtime consensus request hit CONSENSUS_REQUEST_MEMORY_LIMIT, the request remained available in the WAL but WakeableIndexedConsensusQueue left the prefetch worker dormant. The arthas dump showed both prefetch workers waiting in LinkedBlockingQueue.take() while WAL replay was yielded and no realtime request was accepted, matching this lost wakeup. Retryable rejection paths now wake the prefetch worker; inactive or closed queues retain their existing behavior.

The consensus queue regression test now asserts that both a successful enqueue and a request-memory rejection invoke the wakeup hook.

Validation

  • mvn spotless:apply -pl iotdb-core/datanode passed.
  • mvn -nsu -o surefire:test -pl iotdb-core/datanode -Dtest=ConsensusPrefetchingQueueTest#testPendingQueueUsesSharedConsensusMemoryBudgetAndClearReleasesIt -DforkCount=0 ... passed: 1 test, 0 failures, 0 errors.
  • git diff --check passed.

@Caideyipi Caideyipi changed the title [Test] Fix empty finite pipe auto-drop startup race [PIPE] Preserve historical TsFile mods and fix finite pipe auto-drop test race Oct 19, 2026
@Caideyipi Caideyipi changed the title [PIPE] Preserve historical TsFile mods and fix finite pipe auto-drop test race [PIPE][Subscription] Fix pipe races and subscription WAL replay wakeup Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants