Repository navigation
Commit 441ed3e
Migrate DecisionTreeDiscretiser to narwhals, add polars support (#1042)
* Migrate DecisionTreeDiscretiser to narwhals, add polars support
DecisionTreeDiscretiser now accepts pandas or polars input via narwhals,
extending BaseNumericalTransformer directly (independent of the
BaseDiscretiser migration). Never imports pandas; confirmed the module
loads with pandas import blocked.
Merge (single narwhals codepath, one is_pandas branch only at the final
column reassembly) over split (branching at every column-selection call
site): benchmarked at 10k/50k/100k rows x 1/2/10 cols, full fit+transform
time is dominated by GridSearchCV tree training (10-1500ms) vs plumbing
(~0.05-1.4ms per call, <1% of total even where a hand-branched pandas
path was ~2x faster on the isolated plumbing microbenchmark). Merge also
avoids the one-column-at-a-time write pattern that caused pandas
fragmentation warnings in the DecisionTreeFeatures migration.
Added optional n_jobs (default None = sequential, unchanged behaviour),
parallelizing the per-variable tree fits with joblib threads, mirroring
DecisionTreeFeatures. Benchmarked: net loss on small workloads (2 vars,
small grid: 0.6-0.8x), real win once there's enough work (2-50 vars with
a larger grid: 1.4-2.3x). Verified n_jobs=2 produces identical trees and
predictions to n_jobs=None.
Bug found and fixed (introduced by the base-transformer narwhals
migration, not present pre-migration): check_X used to always copy its
pandas input; the narwhals-based check_X no longer does, so the old
transform()'s in-place `X[feature] = ...` assignments would have mutated
the caller's original dataframe. Rewrote transform() to batch every
replacement column and apply them in one non-mutating `.assign()`
(pandas) / `.with_columns()` (polars) call instead, which also sidesteps
polars' immutability and avoids per-column pandas fragmentation.
Reimplemented pandas.cut's binning (bin_number/boundaries outputs)
without importing pandas: np.digitize for bin assignment, and a
from-scratch port of pandas' internal `_round_frac`/`_infer_precision`
label-rounding algorithm (rounds each edge, bumping precision globally
if that would collide two edges) so boundary labels are byte-for-byte
identical to the old pd.cut output. Verified against pandas.cut directly
across 500 randomized threshold/precision/value trials with zero
mismatches, in addition to the existing hardcoded-value tests passing
unmodified.
Tests rewritten to one parametrized test per behavior over
make_df in [pd.DataFrame, pl.DataFrame], replacing the pandas-only
df_normal_dist/df_discretise fixtures with local data dicts (matching
the DecisionTreeFeatures precedent, since those shared fixtures are
still pandas-only). Fixed test_non_fitted_error, which was instantiating
EqualWidthDiscretiser instead of DecisionTreeDiscretiser (a pre-existing
copy-paste bug, confirmed present on main before this migration).
tests/test_discretisation full suite: 123 passed (was 108 pre-migration,
+15 from parametrization), same 5 pre-existing check_estimator failures
(numpy-array input rejected by narwhals check_X, unrelated to this file,
confirmed identical on the pre-migration baseline). flake8 and mypy
clean. sphinx -W build produces only the pre-existing linkcode_resolve
warning (confirmed identical on baseline).
Docs: added "With polars" and "Training trees in parallel" sections,
verified against real output (network available this session, so the
existing fetch_openml house-prices example was re-run and confirmed
still accurate). The two `binner_dict_` boundary/bin_number code blocks
now display floats as plain numbers as before; current numpy's list
repr actually renders them as np.float64(...), a numpy-version-only
cosmetic drift present across the whole docs tree and not caused by
this migration, left as-is and noted here instead.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Use shared backend test fixtures and helpers in DecisionTreeDiscretiser tests
Replace the file-local _normal_dist_data/_discretise_data/_unique_sorted
helpers with the shared test structure: data_normal_dist fixture, y built
with make_series on the backend under test, isinstance(X, make_df) plus
to_dict() checks, and pytest.raises(match=re.escape(msg)). Add a test
passing the target as a list and as a numpy array, which must give the same
result as a Series.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Use frame_to_dict after the shared helper rename in #1045
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Group init tests and match errors in DecisionTreeDiscretiser tests, check bin_output type
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Move DecisionTreeDiscretiser helper functions into the class as private methods
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Simplify the n_jobs note in the DecisionTreeDiscretiser user guide
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Use the scikit-learn wording for n_jobs in DecisionTreeDiscretiser
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Call is_pandas_dataframe in the condition instead of storing it
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Shorten comment in DecisionTreeDiscretiser.transform
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Reorder DecisionTreeDiscretiser tests to the test file convention
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>1 parent 9a875cf commit 441ed3e
3 files changed
Lines changed: 416 additions & 169 deletions
File tree
- docs/user_guide/discretisation
- feature_engine/discretisation
- tests/test_discretisation
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
443 | 443 | | |
444 | 444 | | |
445 | 445 | | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
446 | 528 | | |
447 | 529 | | |
448 | 530 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
| 6 | + | |
| 7 | + | |
6 | 8 | | |
7 | | - | |
| 9 | + | |
| 10 | + | |
8 | 11 | | |
9 | 12 | | |
10 | 13 | | |
| |||
115 | 118 | | |
116 | 119 | | |
117 | 120 | | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
118 | 126 | | |
119 | 127 | | |
120 | 128 | | |
| |||
163 | 171 | | |
164 | 172 | | |
165 | 173 | | |
| 174 | + | |
166 | 175 | | |
167 | | - | |
168 | | - | |
| 176 | + | |
| 177 | + | |
169 | 178 | | |
170 | 179 | | |
171 | 180 | | |
172 | 181 | | |
173 | 182 | | |
174 | 183 | | |
175 | 184 | | |
| 185 | + | |
176 | 186 | | |
177 | 187 | | |
178 | | - | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
179 | 206 | | |
180 | 207 | | |
181 | 208 | | |
| |||
189 | 216 | | |
190 | 217 | | |
191 | 218 | | |
| 219 | + | |
192 | 220 | | |
193 | 221 | | |
194 | | - | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
195 | 227 | | |
196 | 228 | | |
197 | 229 | | |
| |||
223 | 255 | | |
224 | 256 | | |
225 | 257 | | |
| 258 | + | |
226 | 259 | | |
227 | 260 | | |
228 | | - | |
| 261 | + | |
229 | 262 | | |
230 | 263 | | |
231 | 264 | | |
232 | 265 | | |
233 | 266 | | |
234 | 267 | | |
235 | 268 | | |
236 | | - | |
| 269 | + | |
237 | 270 | | |
238 | 271 | | |
239 | 272 | | |
240 | | - | |
| 273 | + | |
241 | 274 | | |
242 | 275 | | |
243 | 276 | | |
| |||
259 | 292 | | |
260 | 293 | | |
261 | 294 | | |
262 | | - | |
263 | | - | |
| 295 | + | |
| 296 | + | |
264 | 297 | | |
265 | | - | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
266 | 301 | | |
267 | | - | |
268 | | - | |
269 | | - | |
270 | | - | |
271 | | - | |
272 | | - | |
273 | | - | |
274 | | - | |
275 | | - | |
276 | | - | |
277 | | - | |
278 | | - | |
279 | | - | |
280 | | - | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
281 | 307 | | |
282 | 308 | | |
283 | 309 | | |
| |||
296 | 322 | | |
297 | 323 | | |
298 | 324 | | |
299 | | - | |
| 325 | + | |
300 | 326 | | |
301 | 327 | | |
302 | 328 | | |
303 | 329 | | |
304 | 330 | | |
305 | 331 | | |
306 | | - | |
| 332 | + | |
307 | 333 | | |
308 | 334 | | |
309 | 335 | | |
310 | 336 | | |
311 | | - | |
| 337 | + | |
312 | 338 | | |
313 | 339 | | |
314 | | - | |
315 | 340 | | |
316 | 341 | | |
317 | 342 | | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
318 | 349 | | |
319 | 350 | | |
320 | | - | |
321 | | - | |
322 | | - | |
323 | | - | |
324 | | - | |
325 | | - | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
326 | 354 | | |
327 | | - | |
328 | | - | |
329 | | - | |
330 | | - | |
331 | | - | |
332 | | - | |
333 | | - | |
334 | | - | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
335 | 359 | | |
336 | 360 | | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
337 | 364 | | |
338 | | - | |
339 | | - | |
340 | | - | |
341 | | - | |
342 | | - | |
343 | | - | |
344 | | - | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
345 | 370 | | |
346 | 371 | | |
347 | 372 | | |
348 | | - | |
349 | | - | |
350 | | - | |
351 | | - | |
352 | | - | |
353 | | - | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
354 | 385 | | |
355 | 386 | | |
356 | 387 | | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
357 | 401 | | |
358 | 402 | | |
359 | 403 | | |
| |||
363 | 407 | | |
364 | 408 | | |
365 | 409 | | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
0 commit comments