You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
MicroSeries.quantile (and therefore median) has no skipna handling. NaN values sort to the end of np.argsort and still contribute their weight to the cumulative distribution, so the inverse-CDF cutoff is pushed upward and the returned quantile is too high:
The second line is the reference: dropping the NaN row should give the same answer as never having had it. Instead the NaN row's weight makes up a third of the cumulative distribution, moving the 0.5 cutoff past the second value and onto the largest one.
(Correction to the first version of this issue: I originally wrote "expected 2.0" by comparing with pd.Series.median. That's wrong — pandas interpolates, this method is documented as inverse CDF / survey::svyquantile, which returns an observed value. The correct post-fix answer here is 1.0, and the bug is the NaN weight inflating the CDF, not the absence of interpolation.)
Every neighbouring statistic already takes skipna: mean (#269), count (#292), var/std (#290). quantile/median are the remaining inconsistency, and unlike the others they return a wrong number rather than propagating NaN.
MicroSeries.quantile(and thereforemedian) has noskipnahandling. NaN values sort to the end ofnp.argsortand still contribute their weight to the cumulative distribution, so the inverse-CDF cutoff is pushed upward and the returned quantile is too high:The second line is the reference: dropping the NaN row should give the same answer as never having had it. Instead the NaN row's weight makes up a third of the cumulative distribution, moving the 0.5 cutoff past the second value and onto the largest one.
(Correction to the first version of this issue: I originally wrote "expected 2.0" by comparing with
pd.Series.median. That's wrong — pandas interpolates, this method is documented as inverse CDF /survey::svyquantile, which returns an observed value. The correct post-fix answer here is 1.0, and the bug is the NaN weight inflating the CDF, not the absence of interpolation.)Every neighbouring statistic already takes
skipna:mean(#269),count(#292),var/std(#290).quantile/medianare the remaining inconsistency, and unlike the others they return a wrong number rather than propagating NaN.Suggested fix
skipna: bool = Truetoquantileandmedian.skipna=True, drop NaN rows (and their weights) before sorting — the same place the existing zero-weight filter from Skip zero-weight rows in MicroSeries.quantile #287 runs.skipna=False, return NaN when any value is NaN, matchingmean/var.Found while reviewing the repo against pandas 3.0.5 / Python 3.13.