Skip to content

Commit 9e4e99b

Browse files
solegalliclaude
andauthored
Migrate WoEEncoder to narwhals, add polars support (#1032)
* Migrate WoEEncoder to narwhals, add polars support fit() splits by backend: pandas keeps _calculate_woe()'s existing two-groupby implementation unchanged (it's directly unit-tested for that exact pandas-Series-with-category-index contract); polars/other narwhals backends use one group_by() instead of two, deriving the negative-class count as the complement of the positive-class count per category - benchmarked competitive with, and often faster than, pandas-native at 50k-100k rows. Zero-count-per-class fill_value handling preserved exactly. Bug fix: _check_fit_input() previously assumed y was always a pandas Series (y.nunique()/y.min()/y.max()), breaking on a numpy y (e.g. a plain list/array-like target, which sklearn's check_X_y machinery converts via column_or_1d). Wrapped numpy y into a narwhals Series aligned to X's backend; for pandas specifically, also had to line the wrapped Series up with X's actual index, since _calculate_woe()'s y.groupby(X[var]) aligns by index and a mismatched default RangeIndex silently drops every row instead of raising, leaving encoder_dict_ empty. Fixes test_encoders_when_x_pandas_y_numpy's WoEEncoder case (was failing on the unmigrated file, confirmed pre-existing). Verified: 44/44 own tests, full encoding suite 342 passed/16 failed (was 17 pre-existing on the narwhals-encoding-base baseline - one less here since this branch's own numpy-y bug is now fixed, rest confirmed unrelated), flake8 and mypy clean, sphinx -W build clean (only the pre-existing unrelated linkcode_resolve warning). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Adapt WoEEncoder to narwhals-returning check_X check_X_y now returns a narwhals frame. In _check_fit_input, bind that to nw_X and keep the original native X: the nwd.is_pandas_dataframe(X) check, the native_y.index = X.index alignment and the returned X all need native input, and fit()'s pandas _calculate_woe fast path and nwd checks are then unchanged (X stays native so no rehydration is needed). Take the y-series backend from nw_X.implementation instead of re-wrapping X. In transform(), bind _check_transform_input_and_state to nw_X, keep native X for _check_contains_na, and pass nw_X to _encode (which now expects narwhals). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Use shared backend test fixtures and helpers in WoEEncoder tests Replace the file-local data dicts and assert_df_equal/_none_to_nan helpers with the shared test structure: make_df and data_enc* fixtures, y built with make_series on the backend under test, isinstance(X, make_df) plus to_dict() checks, and pytest.raises/warns(match=re.escape(msg)). Add a test passing the target as a list and as a numpy array, which take a different code path than a Series. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use frame_to_dict after the shared helper rename in #1045 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use add_target_to_X in WoEEncoder, group init tests, match errors Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Compute WoE with narwhals in _calculate_woe, shared by WoEEncoder and SelectByInformationValue Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Replace zero counts by 0.5 in WoE, remove fill_value, add variables_with_zero_counts_ Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
1 parent f83c278 commit 9e4e99b

5 files changed

Lines changed: 434 additions & 460 deletions

File tree

‎docs/user_guide/encoding/WoEEncoder.rst‎

Lines changed: 79 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -112,8 +112,10 @@ This occurs when a category shows only 1 of the possible values of the target (e
112112
always takes 1 or 0). In practice, this happens mostly when a category has a low frequency
113113
in the dataset, that is, when only very few observations show that category.
114114

115-
To overcome this limitation, consider using a variable transformation method to group
116-
those categories together, for example by using feature-engine's :class:`RareLabelEncoder()`.
115+
A common way to obtain a WoE for these categories is to replace the zero count by 0.5,
116+
which is what :class:`WoEEncoder()` does. Still, WoE values calculated from very few
117+
observations are unreliable, so consider grouping infrequent categories first, for
118+
example with feature-engine's :class:`RareLabelEncoder()`.
117119

118120
Taking into account the above considerations, conducting a detailed exploratory data
119121
analysis (EDA) is essential as part of the data science and model-building process.
@@ -162,9 +164,46 @@ with feature-engine's imputers.
162164

163165
:class:`WoEEncoder()` will ignore unseen categories by default, in which case, they will
164166
be replaced by np.nan after the encoding. You have the option to make the encoder raise
165-
an error instead, by setting `unseen='raise'`. You can also replace unseen categories
166-
by an arbitrary value you need to define in `fill_value`, although we do not recommend
167-
this option because it may lead to unpredictable results.
167+
an error instead, by setting `unseen='raise'`.
168+
169+
Categories with no positive or no negative cases
170+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
171+
172+
.. attention::
173+
174+
**New in version 2.0:** :class:`WoEEncoder()` used to raise an error when a category
175+
had no positive or no negative cases, unless you set the parameter `fill_value`.
176+
`fill_value` was removed. The encoder now replaces zero counts by 0.5 and lists the
177+
affected variables in the attribute `variables_with_zero_counts_`.
178+
179+
When a category has no positive or no negative cases in the training set,
180+
:class:`WoEEncoder()` replaces the zero count by 0.5 to calculate the WoE, and stores the
181+
names of the affected variables in `variables_with_zero_counts_`. In the following
182+
example, the category red has only positive cases:
183+
184+
.. code:: python
185+
186+
import pandas as pd
187+
from feature_engine.encoding import WoEEncoder
188+
189+
X = pd.DataFrame(
190+
{"colour": ["blue", "blue", "blue", "red", "red", "green", "green", "green"]}
191+
)
192+
y = pd.Series([1, 0, 1, 1, 1, 0, 1, 0])
193+
194+
woe = WoEEncoder()
195+
woe.fit(X, y)
196+
197+
print(woe.encoder_dict_)
198+
print(woe.variables_with_zero_counts_)
199+
200+
There are 5 positive and 3 negative cases. Red has 2 positive cases and no negative
201+
cases, so its WoE is log((2 / 5) / (0.5 / 3)) = 0.88:
202+
203+
.. code:: python
204+
205+
{'colour': {'blue': 0.1823215567939548, 'green': -1.203972804325936, 'red': 0.8754687373539001}}
206+
['colour']
168207
169208
Python example
170209
--------------
@@ -280,6 +319,41 @@ variable values:
280319
686 -0.584173 female 22.000000 0 0 7.7250 -0.357528 0.012075
281320
282321
322+
With polars
323+
~~~~~~~~~~~
324+
325+
:class:`WoEEncoder()` also works with polars dataframes:
326+
327+
.. code:: python
328+
329+
import polars as pl
330+
from feature_engine.encoding import WoEEncoder
331+
332+
X = pl.DataFrame(dict(x1 = [1,2,3,4,5], x2 = ["b", "b", "b", "a", "a"]))
333+
y = pl.Series([0,1,1,1,0])
334+
335+
woe = WoEEncoder()
336+
woe.fit(X, y)
337+
woe.transform(X)
338+
339+
We see the resulting dataframe below:
340+
341+
.. code:: text
342+
343+
shape: (5, 2)
344+
┌─────┬───────────┐
345+
│ x1 ┆ x2 │
346+
│ --- ┆ --- │
347+
│ i64 ┆ f64 │
348+
╞═════╪═══════════╡
349+
│ 1 ┆ 0.287682 │
350+
│ 2 ┆ 0.287682 │
351+
│ 3 ┆ 0.287682 │
352+
│ 4 ┆ -0.405465 │
353+
│ 5 ┆ -0.405465 │
354+
└─────┴───────────┘
355+
356+
283357
WoE in categorical and numerical variables
284358
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
285359

0 commit comments

Comments
 (0)