Skip to content

[Query] Share fragment exchange budgets and consume ready Collect inputs - #18868

Closed
ColinLeeo wants to merge 3 commits into
apache:masterfrom
ColinLeeo:codex/exchange-fanin-budget
Closed

ColinLeeo wants to merge 3 commits into
apache:masterfrom
ColinLeeo:codex/exchange-fanin-budget

Conversation

@ColinLeeo

Copy link
Copy Markdown
Collaborator

Description

Exchange memory is divided among individual handles, and Collect consumes children in sequence. With many inputs, this can strand memory in per-handle quotas and leave ready inputs waiting behind a blocked child. This change shares each fragment instance's exchange budget by direction and lets Collect consume a ready input while preserving query correctness after unordered fan-in.

Shared exchange budgets

  • Split the fragment exchange budget F into fixed INPUT = F / 2 and OUTPUT = F - INPUT budgets shared by all pipelines in that fragment.
  • Charge remote sources to the receiving fragment's INPUT budget and sinks/local shared queues to the producing fragment's OUTPUT budget. Local queues are charged once, independent of creation order.
  • Maintain global, fragment-side, and plan-node accounting across reservation, rollback, cancellation, and release. Retry a waiting reservation after publishing it so a concurrent release cannot miss the request; synchronize completion with release/cancellation to prevent duplicate charging.

Ready-input Collect and plan properties

  • Collect and MappingCollect observe every live direct child, reuse pending-future listeners, and keep the selected child stable during output backpressure. Operator methods run on the driver thread; completion callbacks only report readiness or failure.
  • Refresh readiness after null/empty results, propagate child failures and cancellation, close all inputs, and account for the retained memory of all live children. MappingCollect uses the actual selected child's mapping.
  • Clear invalid pre-grouping above unordered Collect/Union so aggregation uses HashAggregation. Preserve valid streaming aggregation in ordered branches, restore partition sorting when required, and prevent sort elimination across unordered fan-in.
  • Derive TopK input-order flags from actual physical inputs and preserve runtime filter identities. MergeSort/Join are covered by dependency validation only; their consumption algorithms and progress mechanisms are outside this change. Any-ready consumption for TableTopK and CountMerge is also outside this change.

Design and detailed validation commands: exchange fan-in design.

Validation

  • 118 targeted unit tests across 14 classes passed, with no failures or skipped tests. Coverage includes a deterministic free-before-enqueue race, shared-budget contention and cleanup, Collect with 100/1000 children, concurrent readiness, failure/close handling, and grouping/ordering/runtime-filter properties.
  • Full-reactor English compilation/tests and Chinese test-compile -DskipTests -P with-zh-locale passed.
  • Rebuilt and installed the branch's node artifacts, then ran all 7 IoTDBCollectFanInIT tests in both TableSimpleIT and TableClusterIT (three DataNodes). Both runs passed. SQL coverage includes aggregation/DISTINCT, Union, nested windows, Fill, TopK, tree/table exchange paths, and join dependencies.
  • Spotless, Checkstyle, license checks, and git diff --check passed. Maven validation used offline mode with rebuilt local artifacts.

This PR has:

  • been self-reviewed, including concurrent reservation, release, cancellation, and readiness paths.
  • added documentation for modified behaviors.
  • added comments explaining non-obvious concurrency and ownership decisions.
  • added or updated unit tests.
  • added integration tests.
  • been tested in a test IoTDB cluster.
Key changed/added classes

MemoryPool, MemorySide, LocalExecutionPlanContext, exchange handles and SharedTsBlockQueue; AnyChildBlocked, CollectOperator, MappingCollectOperator; TableDistributedPlanGenerator, ParallelizeGrouping, and SortElimination.

@ColinLeeo ColinLeeo closed this Oct 14, 2026
@ColinLeeo
ColinLeeo deleted the codex/exchange-fanin-budget branch October 14, 2026 07:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant