Repository navigation
Conversation
Co-authored-by: Codex <codex@openai.com>
Abort fanout writes and final close after failed or cancelled eviction. Share filename generators across rolling writer builds, update recency only after successful writes, and expose the limit through table properties. Co-authored-by: Codex <codex@openai.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #1744.
What changes are included in this PR?
High partition cardinality can keep too many fanout writers open. Add an opt-in positive limit through a constructor or
write.datafusion.fanout.max-open-partitions, closing the least recently used partition to make room. The limit counts active writers, not bytes; interleaved input can still produce many small files and accumulate metadata, with a warning on the first eviction.Failed or cancelled eviction aborts the fanout writer, so subsequent writes and final close cannot silently commit incomplete output. Successful writes determine recency, and the built-in rolling writer shares its filename generator across builds to keep reopened partition files distinct.
Are these changes tested?
107 writer-related unit tests pass, including interrupted/failed eviction, cancellation after completed eviction, build/write failures, LRU limits, and real Parquet readback using a per-instance filename counter. Table-property tests cover defaults, positive limits, and invalid values. 85 doctests pass (14 existing ignored).
Formatting, core all-target/all-feature Clippy, Cargo Machete, locked workspace dependency validation, and license/notice checks pass. Generated API snapshots add only the table-property key/getter and property-based constructor.
AI Disclosure
Codex assisted with implementation and tests. Review the abort-and-rewrite contract after eviction failure or cancellation, the active-writer limit's memory/file-size tradeoffs, and the requirement that custom builders produce distinct paths on repeated builds of the same partition. This branch has no DataFusion sink; callers consume the property through the new constructor.