Repository navigation
fix(client-contract): keep vectors servable and judge every column by same_value - #34
Merged
Merged
Conversation
… same_value The nested-cells vector's flags column holds one item type, the validator reports a vector column Arrow cannot build, and served_table judges flat columns by same_value as it does nested ones.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every reading-order vector is now a table a server could serve, and the validator judges every column by one equality rule.
The SDKs replay each vector by writing it to Parquet and reading it back, as a client reads a dataset. In
nested-cells-compare-by-valuetheflagscolumn held[true]in one row and[1]in another, which no Arrow list column can hold, so no server could serve it and Python could not replay it.flagsnow holds[true]and[false], and_unorderablereports any member columnpa.arraycannot build (mixed list items, a struct field of mixed types). The bool-against-number case stays in the unit tests of both reference implementations, since no vector can carry it.served_tablejudged flat columns with Arrowunique(), which counts 0.0 and -0.0, and two NaN bit patterns, as two values, whilesame_valuecounts each pair as one. A vector and its Parquet file could therefore group columns differently. Every column, flat or nested, is now judged bysame_valuein one pass, the ruletable_from_rowsuses.Testing:
test_a_vector_and_its_parquet_file_group_columns_alike(signed zeros, NaN payloads) fails ondev; the two new malformed-vector cases are reported.uv run tools/validate_client_contract.pypasses (125 scenarios) and the tools tests pass (586).