Skip to content

fix: do not treat float min==max Parquet statistics as constant columns - #26494

Open
charan-rathore wants to merge 2 commits into
apache:mainfrom
charan-rathore:fix-parquet-float-constant-stats
Open

charan-rathore wants to merge 2 commits into
apache:mainfrom
charan-rathore:fix-parquet-float-constant-stats

Conversation

@charan-rathore

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

Parquet writers leave NaN out of floating-point min/max statistics (the spec requires it). So a column chunk holding one distinct finite value plus some NaNs has statistics min == max and null_count == 0, although its values are not constant. The constant-column substitution in the Parquet opener then treats the column as constant and returns the statistic for every NaN row: a plain SELECT * silently loses NaNs, and COPY ... TO writes the wrong values into the new files. pyarrow, DuckDB and Polars read the same files correctly (see #26490 for a reproducer).

What changes are included in this PR?

constant_value_from_stats in datafusion-datasource-parquet no longer derives a constant column from min == max statistics for Float16/Float32/Float64 columns, since float min/max cannot prove the column is constant when NaN is excluded from the statistics. All-null columns (proven by null_count == num_rows) are still treated as constant null, and non-float types are unchanged.

What is the testing strategy for this PR?

New unit test extract_constant_columns_float_nan_unsafe in datafusion/datasource-parquet/src/opener/mod.rs covers Float64 and Float32 columns whose statistics look constant (min == max, null_count == 0) and asserts no constant is substituted, while the existing extract_constant_columns_non_null / extract_constant_columns_all_null tests pin the unchanged behavior for other types. The new test fails on current main (it returns the bogus constant) and passes with this change. Full datafusion-datasource-parquet lib suite locally: 290 passed, 0 failed (with the parquet-testing submodule checked out).

Are there any user-facing changes?

Bug fix only: SELECT and COPY ... TO on affected Parquet files now return the real NaN values instead of the statistics constant. No API changes.


AI disclosure: this change was prepared with AI assistance under human direction. The fix and the test were run locally as described above (test confirmed failing on main and passing with the fix; full crate suite green).

…Parquet float min/max statistics exclude NaN (the spec requires it), so min == max cannot prove a floating-point column is constant. Treating it as one silently replaces NaN rows with the statistic value on plain SELECT and on COPY TO. Skip Float16/Float32/Float64 in the constant column substitution and add a regression test. Fixes apache#26490.  Signed-off-by: Charan Rathore <180254320+charan-rathore@users.noreply.github.com>
@github-actions github-actions Bot added the datasource Changes to the datasource crate label Oct 19, 2026

@raghavyadav01 raghavyadav01 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. The COPY case makes this worse than it first looks — it's not just a bad read, it writes the wrong values into new files.

One thing on the test: it asserts against hand-built ColumnStatistics rather than a real Parquet file containing NaN. That covers the new branch, but it wouldn't catch the original bug returning by another route. Is there a sqllogictest that reads an actual file with NaNs in it? That's the one that would fail loudly if this optimization gets re-enabled somewhere else later.

Also, does data_type here ever see a float nested inside a struct or list, or do only primitive leaf columns reach this point?

@charan-rathore

Copy link
Copy Markdown
Author

@raghavyadav01 fair point, the test only pins the new branch. I'll add a sqllogictest that reads a real Parquet file with NaN once I can run it, or happy for a maintainer to say if they want it in this PR.

On nested types: constant_columns_from_stats loops over the file schema's top-level fields and the float check is on that field's own type, so a float inside a struct or list never counts as a float column here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

datasource Changes to the datasource crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Plain SELECT of a Parquet float column returns wrong values: NaN rows are replaced by the statistics constant when min == max

2 participants