Skip to content

feat(writer): limit open partitions in FanoutWriter - #3382

Draft
manuzhang wants to merge 2 commits into
apache:mainfrom
manuzhang:codex/issue-1744-max-open-partitions
Draft

manuzhang wants to merge 2 commits into
apache:mainfrom
manuzhang:codex/issue-1744-max-open-partitions

Conversation

@manuzhang

@manuzhang manuzhang commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes #1744.

What changes are included in this PR?

High partition cardinality can keep too many fanout writers open. Add an opt-in positive limit through a constructor or write.datafusion.fanout.max-open-partitions, closing the least recently used partition to make room. The limit counts active writers, not bytes; interleaved input can still produce many small files and accumulate metadata, with a warning on the first eviction.

Failed or cancelled eviction aborts the fanout writer, so subsequent writes and final close cannot silently commit incomplete output. Successful writes determine recency, and the built-in rolling writer shares its filename generator across builds to keep reopened partition files distinct.

Are these changes tested?

107 writer-related unit tests pass, including interrupted/failed eviction, cancellation after completed eviction, build/write failures, LRU limits, and real Parquet readback using a per-instance filename counter. Table-property tests cover defaults, positive limits, and invalid values. 85 doctests pass (14 existing ignored).

Formatting, core all-target/all-feature Clippy, Cargo Machete, locked workspace dependency validation, and license/notice checks pass. Generated API snapshots add only the table-property key/getter and property-based constructor.

AI Disclosure

Codex assisted with implementation and tests. Review the abort-and-rewrite contract after eviction failure or cancellation, the active-writer limit's memory/file-size tradeoffs, and the requirement that custom builders produce distinct paths on repeated builds of the same partition. This branch has no DataFusion sink; callers consume the property through the new constructor.

manuzhang and others added 2 commits October 9, 2026 23:26
Co-authored-by: Codex <codex@openai.com>
Abort fanout writes and final close after failed or cancelled eviction. Share filename generators across rolling writer builds, update recency only after successful writes, and expose the limit through table properties.

Co-authored-by: Codex <codex@openai.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add max_open_partitions in FanoutWriter to help manage memory

1 participant