Skip to content

quantile()/median() don't skip NaN, returning a too-high quantile #302

Description

@vahid-ahmadi

MicroSeries.quantile (and therefore median) has no skipna handling. NaN values sort to the end of np.argsort and still contribute their weight to the cumulative distribution, so the inverse-CDF cutoff is pushed upward and the returned quantile is too high:

import numpy as np
from microdf import MicroSeries

MicroSeries([1.0, np.nan, 3.0], weights=[1, 1, 1]).median()   # -> 3.0
MicroSeries([1.0, 3.0], weights=[1, 1]).median()              # -> 1.0

The second line is the reference: dropping the NaN row should give the same answer as never having had it. Instead the NaN row's weight makes up a third of the cumulative distribution, moving the 0.5 cutoff past the second value and onto the largest one.

(Correction to the first version of this issue: I originally wrote "expected 2.0" by comparing with pd.Series.median. That's wrong — pandas interpolates, this method is documented as inverse CDF / survey::svyquantile, which returns an observed value. The correct post-fix answer here is 1.0, and the bug is the NaN weight inflating the CDF, not the absence of interpolation.)

Every neighbouring statistic already takes skipna: mean (#269), count (#292), var/std (#290). quantile/median are the remaining inconsistency, and unlike the others they return a wrong number rather than propagating NaN.

Suggested fix

  • Add skipna: bool = True to quantile and median.
  • With skipna=True, drop NaN rows (and their weights) before sorting — the same place the existing zero-weight filter from Skip zero-weight rows in MicroSeries.quantile #287 runs.
  • With skipna=False, return NaN when any value is NaN, matching mean/var.

Found while reviewing the repo against pandas 3.0.5 / Python 3.13.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions