Repository navigation
Commit 0d49309
address review feedback on dataframe_checks.py (#989)
* address review feedback on dataframe_checks.py
Follow-up to FBruzzesi's review on PR #965:
- Clarify docstrings for check_X, check_y, check_X_y in terms of which
dataframe libraries are safe to pass in (pandas, polars, PyArrow, modin,
cuDF), instead of narwhals-specific "eager" terminology.
- Use narwhals' IntoDataFrameT instead of IntoDataFrame for check_X and
check_X_y, since both return the same concrete dataframe type they
receive.
- Fix a null/NaN detection bug: in polars, is_null() does not catch an
explicit float("nan") value (only None counts as null), so check_y and
_check_contains_na could silently miss NaNs in polars data. Now also
check is_nan() for numeric columns/series, keeping numpy for the
finite/inf checks since it benchmarks as fast or faster there.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* inline null/nan checks to match FBruzzesi's suggested one-liner
Collapses the has_na/has_null/has_nan accumulator variables into a single
short-circuiting if-condition, as suggested in review. This also avoids an
unnecessary is_nan() call when is_null() already found a null value.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* speed up numeric column detection in _check_contains_na
The schema-based list comprehension rebuilt narwhals' full column
schema on every single-column access, making it scale roughly
quadratically with column count on pandas (benchmarked up to ~500x
slower than necessary at 200 columns). Switch to the pandas fast-path /
narwhals-selector pattern already used in variable_handling
(find_numerical_variables, check_numerical_variables) for the same
"which of these columns are numeric" problem.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>1 parent 3995a26 commit 0d49309
2 files changed
Lines changed: 85 additions & 12 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
5 | | - | |
| 5 | + | |
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| 9 | + | |
9 | 10 | | |
10 | | - | |
| 11 | + | |
11 | 12 | | |
12 | 13 | | |
13 | 14 | | |
14 | | - | |
| 15 | + | |
15 | 16 | | |
16 | 17 | | |
17 | 18 | | |
18 | 19 | | |
19 | 20 | | |
20 | 21 | | |
21 | | - | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
22 | 26 | | |
23 | 27 | | |
24 | 28 | | |
| |||
63 | 67 | | |
64 | 68 | | |
65 | 69 | | |
66 | | - | |
67 | | - | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
68 | 75 | | |
69 | 76 | | |
70 | 77 | | |
| |||
84 | 91 | | |
85 | 92 | | |
86 | 93 | | |
87 | | - | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
88 | 97 | | |
89 | 98 | | |
90 | 99 | | |
| |||
95 | 104 | | |
96 | 105 | | |
97 | 106 | | |
98 | | - | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
99 | 111 | | |
100 | 112 | | |
101 | 113 | | |
| |||
109 | 121 | | |
110 | 122 | | |
111 | 123 | | |
112 | | - | |
| 124 | + | |
113 | 125 | | |
114 | 126 | | |
115 | | - | |
| 127 | + | |
116 | 128 | | |
117 | 129 | | |
118 | 130 | | |
119 | 131 | | |
120 | 132 | | |
121 | 133 | | |
122 | | - | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
123 | 138 | | |
124 | 139 | | |
125 | 140 | | |
| |||
213 | 228 | | |
214 | 229 | | |
215 | 230 | | |
216 | | - | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
217 | 239 | | |
218 | 240 | | |
219 | 241 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
132 | 132 | | |
133 | 133 | | |
134 | 134 | | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
135 | 147 | | |
136 | 148 | | |
137 | 149 | | |
| |||
190 | 202 | | |
191 | 203 | | |
192 | 204 | | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
193 | 217 | | |
194 | 218 | | |
195 | 219 | | |
| |||
372 | 396 | | |
373 | 397 | | |
374 | 398 | | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
375 | 426 | | |
376 | 427 | | |
377 | 428 | | |
| |||
0 commit comments