Repository navigation
fix(arrow): load equality deletes on fields nested in structs - #3453
Draft
JeonDaehong wants to merge 2 commits into
Draft
JeonDaehong wants to merge 2 commits into
JeonDaehong wants to merge 2 commits into
Conversation
Equality ids may name a field nested in a struct, which the spec allows and EqualityDeleteWriter writes. Loading such a delete file failed because schema evolution only maps top-level field ids, so the delete file is now read by the top-level fields that hold the equality ids. The delete predicate also used the leaf name (`f`), which binds to a top-level column of the same name or fails; it now uses the full name (`s.f`).
Contributor
Author
|
@mbutrovich could you take a look when you have time? This fixes loading equality deletes on nested fields (#3450); applying them on scan also needs #2906, which you've been reviewing. |
This was referenced Oct 15, 2026
JeonDaehong
marked this pull request as draft
October 16, 2026 06:23
Contributor
Author
|
Converted to draft to stay within the new limit of 3 open PRs (#3465). The change is still ready for review; I will mark it ready again when one of my other PRs lands. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
What changes are included in this PR?
Equality ids may name a field nested in a struct (the spec allows it and
EqualityDeleteWriterwrites it), but loading such a delete file failed withfield not found:f), which fails to bind or binds to a top-level column with the same name. It now uses the full name (s.f).test_delete_file_loader_parse_equality_deletesexpected the leaf namesafor a nested field, so its expected string changes tos.sa.Applying the delete to data rows also needs predicates on nested leaves in the row filter, which #2906 adds. With #2906 merged locally, an end-to-end scan deletes the expected rows (including rows whose parent struct is null). I'll add that end-to-end test once #2906 lands.
cc @brgr-s, as discussed in #2961.
Are these changes tested?
Yes.
test_nested_field_equality_delete_predicatecovers loading and the full-name predicate (the table also has a top-level column with the same leaf name).cargo test -p iceberg --lib, clippy and fmt are clean.AI Disclosure
I used Claude Code to find this, write the change and run the tests. I reviewed it before opening the PR.