Repository navigation
fix(manifest-list): read V1 file counts written with spec field names - #3447
Draft
kacper-li-airspace-intelligence wants to merge 6 commits into
Draft
kacper-li-airspace-intelligence wants to merge 6 commits into
kacper-li-airspace-intelligence wants to merge 6 commits into
Conversation
Resolve the parse_with_version conflict by reading entries with into_deser_iter and the Resolved adapter from apache#3354.
Cover round trips, empty lists, truncated input and V1 read as V2 as separate unit tests, and drop the allocation-budget integration test.
Drop the round-trip and V1-as-V2 tests, which duplicate test_parse_manifest_list_v* and test_manifest_list_writer_v1_as_v2. Set a non-empty key_metadata and null contains_nan and upper_bound on existing entries to keep that coverage.
Java and the spec name the V1 count fields added_files_count, existing_files_count and deleted_files_count. Add them as serde aliases on ManifestFileV1, as ManifestFileV2 already has.
2 of 15 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
What changes are included in this PR?
Stacked on #3224; only the last commit is new.
ManifestFileV1only knowsadded_data_files_count,existing_data_files_countanddeleted_data_files_count. Java and the spec useadded_files_count,existing_files_countanddeleted_files_count, so these counts in V1 lists written by Java parse asNone. Add the serde aliasesManifestFileV2already has. This relies on #3224 reading V1 against the writer schema; resolution againstMANIFEST_LIST_AVRO_SCHEMA_V1filled the old names with null before serde saw the record.Are these changes tested?
test_parse_manifest_list_v1_with_spec_field_nameswrites a V1 list with the spec field names and parses it. It fails without the aliases.AI Disclosure
AI assisted with the implementation, test and description.