Repository navigation
fix: do not treat float min==max Parquet statistics as constant columns - #26494
charan-rathore wants to merge 2 commits into
Conversation
…Parquet float min/max statistics exclude NaN (the spec requires it), so min == max cannot prove a floating-point column is constant. Treating it as one silently replaces NaN rows with the statistic value on plain SELECT and on COPY TO. Skip Float16/Float32/Float64 in the constant column substitution and add a regression test. Fixes apache#26490. Signed-off-by: Charan Rathore <180254320+charan-rathore@users.noreply.github.com>
raghavyadav01
left a comment
There was a problem hiding this comment.
Good catch. The COPY case makes this worse than it first looks — it's not just a bad read, it writes the wrong values into new files.
One thing on the test: it asserts against hand-built ColumnStatistics rather than a real Parquet file containing NaN. That covers the new branch, but it wouldn't catch the original bug returning by another route. Is there a sqllogictest that reads an actual file with NaNs in it? That's the one that would fail loudly if this optimization gets re-enabled somewhere else later.
Also, does data_type here ever see a float nested inside a struct or list, or do only primitive leaf columns reach this point?
|
@raghavyadav01 fair point, the test only pins the new branch. I'll add a sqllogictest that reads a real Parquet file with NaN once I can run it, or happy for a maintainer to say if they want it in this PR. On nested types: |
Which issue does this PR close?
SELECTof a Parquet float column returns wrong values: NaN rows are replaced by the statistics constant when min == max #26490.Rationale for this change
Parquet writers leave NaN out of floating-point min/max statistics (the spec requires it). So a column chunk holding one distinct finite value plus some NaNs has statistics
min == maxandnull_count == 0, although its values are not constant. The constant-column substitution in the Parquet opener then treats the column as constant and returns the statistic for every NaN row: a plainSELECT *silently loses NaNs, andCOPY ... TOwrites the wrong values into the new files. pyarrow, DuckDB and Polars read the same files correctly (see #26490 for a reproducer).What changes are included in this PR?
constant_value_from_statsindatafusion-datasource-parquetno longer derives a constant column frommin == maxstatistics forFloat16/Float32/Float64columns, since float min/max cannot prove the column is constant when NaN is excluded from the statistics. All-null columns (proven bynull_count == num_rows) are still treated as constant null, and non-float types are unchanged.What is the testing strategy for this PR?
New unit test
extract_constant_columns_float_nan_unsafeindatafusion/datasource-parquet/src/opener/mod.rscovers Float64 and Float32 columns whose statistics look constant (min == max,null_count == 0) and asserts no constant is substituted, while the existingextract_constant_columns_non_null/extract_constant_columns_all_nulltests pin the unchanged behavior for other types. The new test fails on current main (it returns the bogus constant) and passes with this change. Fulldatafusion-datasource-parquetlib suite locally: 290 passed, 0 failed (with theparquet-testingsubmodule checked out).Are there any user-facing changes?
Bug fix only:
SELECTandCOPY ... TOon affected Parquet files now return the real NaN values instead of the statistics constant. No API changes.AI disclosure: this change was prepared with AI assistance under human direction. The fix and the test were run locally as described above (test confirmed failing on main and passing with the fix; full crate suite green).