Skip to content

Commit 35fe69e

Browse files
solegalliclaude
andauthored
Migrate MatchCategories to narwhals, add polars support (#1064)
* Migrate MatchCategories to narwhals, add polars support MatchCategories now accepts pandas and polars dataframes. With pandas it keeps casting to the category dtype; with polars it casts to Enum with the categories learned in fit. Unseen categories become missing values in both. Also fixes the warning and error message when several integer-named pandas columns get missing values (it raised a TypeError). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Mark polars output blocks in the user guide as text Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Remove the transformer printout after fit() from the MatchCategories docstring Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
1 parent c8ae828 commit 35fe69e

3 files changed

Lines changed: 521 additions & 120 deletions

File tree

‎docs/user_guide/preprocessing/MatchCategories.rst‎

Lines changed: 146 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,8 @@ MatchCategories
66
===============
77

88
:class:`MatchCategories()` ensures that categorical variables are encoded as pandas
9-
'categorical' dtype instead of generic python 'object' or other dtypes.
9+
'categorical' dtype, or polars 'Enum' dtype, instead of generic python 'object',
10+
string or other dtypes.
1011

1112
Under the hood, 'categorical' dtype is a representation that maps each
1213
category to an integer, thus providing a more memory-efficient object
@@ -79,10 +80,10 @@ Here are the mappings learnt for each categorical variable:
7980

8081
.. code:: python
8182
82-
{'pclass': Int64Index([1, 2, 3], dtype='int64'),
83-
'sex': Index(['female', 'male'], dtype='object'),
84-
'cabin': Index(['A', 'B', 'C', 'D', 'E', 'F', 'M', 'T'], dtype='object'),
85-
'embarked': Index(['C', 'Missing', 'Q', 'S'], dtype='object')}
83+
{'pclass': Index([1, 2, 3], dtype='int64'),
84+
'sex': Index(['female', 'male'], dtype='str'),
85+
'cabin': Index(['A', 'B', 'C', 'D', 'E', 'F', 'M', 'T'], dtype='str'),
86+
'embarked': Index(['C', 'Missing', 'Q', 'S'], dtype='str')}
8687
8788
To see why this matters, let's compare the order in which the categories of `embarked`
8889
appear in the raw train and test sets. This is the order in the train set:
@@ -95,7 +96,9 @@ We obtain the following order:
9596

9697
.. code:: python
9798
98-
array(['S', 'C', 'Missing', 'Q'], dtype=object)
99+
<StringArray>
100+
['S', 'C', 'Missing', 'Q']
101+
Length: 4, dtype: str
99102
100103
And this is the order in the test set:
101104

@@ -107,7 +110,9 @@ Which is different from the train set:
107110

108111
.. code:: python
109112
110-
array(['Q', 'S', 'C'], dtype=object)
113+
<StringArray>
114+
['Q', 'S', 'C']
115+
Length: 3, dtype: str
111116
112117
The categories appear in a different order in each set. If we transform the dataframes
113118
using the same `match_categories` object, categorical variables will be converted to a
@@ -122,7 +127,7 @@ Which is:
122127

123128
.. code:: python
124129
125-
Index(['C', 'Missing', 'Q', 'S'], dtype='object')
130+
Index(['C', 'Missing', 'Q', 'S'], dtype='str')
126131
127132
And this is the order we now obtain for the test set:
128133

@@ -134,11 +139,10 @@ The 2 sets now show exactly the same category order:
134139

135140
.. code:: python
136141
137-
Index(['C', 'Missing', 'Q', 'S'], dtype='object')
142+
Index(['C', 'Missing', 'Q', 'S'], dtype='str')
138143
139144
If some category was not present in the training data, it will not be mapped
140-
to any integer and will thus not get encoded. This behaviour can be modified through the
141-
parameter `errors`. Let's illustrate this with the `cabin` variable. These are the
145+
to any integer and will become a missing value instead. Let's illustrate this with the `cabin` variable. These are the
142146
categories present in the train set:
143147

144148
.. code:: python
@@ -149,7 +153,9 @@ We obtain the following categories:
149153

150154
.. code:: python
151155
152-
array(['B', 'C', 'E', 'D', 'A', 'M', 'T', 'F'], dtype=object)
156+
<StringArray>
157+
['B', 'C', 'E', 'D', 'A', 'M', 'T', 'F']
158+
Length: 8, dtype: str
153159
154160
And these are the categories present in the test set, which include a category, 'G',
155161
that was not seen during training:
@@ -162,7 +168,9 @@ We obtain the following categories, including the unseen 'G':
162168

163169
.. code:: python
164170
165-
array(['M', 'F', 'E', 'G'], dtype=object)
171+
<StringArray>
172+
['M', 'F', 'E', 'G']
173+
Length: 4, dtype: str
166174
167175
After transforming the train set, we obtain the same categories as before, now correctly
168176
typed as 'category' dtype:
@@ -176,7 +184,7 @@ Which are:
176184
.. code:: python
177185
178186
['B', 'C', 'E', 'D', 'A', 'M', 'T', 'F']
179-
Categories (8, object): ['A', 'B', 'C', 'D', 'E', 'F', 'M', 'T']
187+
Categories (8, str): ['A', 'B', 'C', 'D', 'E', 'F', 'M', 'T']
180188
181189
But when we transform the test set, the unseen category 'G' is not mapped to any integer,
182190
and becomes a missing value instead:
@@ -190,7 +198,130 @@ We see that 'G' has been replaced by a missing value:
190198
.. code:: python
191199
192200
['M', 'F', 'E', NaN]
193-
Categories (8, object): ['A', 'B', 'C', 'D', 'E', 'F', 'M', 'T']
201+
Categories (8, str): ['A', 'B', 'C', 'D', 'E', 'F', 'M', 'T']
202+
203+
Because we set `missing_values="ignore"`, :class:`MatchCategories()` warns us that
204+
missing values were introduced:
205+
206+
.. code:: python
207+
208+
UserWarning: During the encoding, NaN values were introduced in the feature(s) cabin.
209+
210+
With the default `missing_values="raise"`, :class:`MatchCategories()` raises an error
211+
instead, both when the data contains missing values and when unseen categories would
212+
introduce them.
213+
214+
With polars
215+
^^^^^^^^^^^
216+
217+
:class:`MatchCategories()` also works with polars dataframes. In polars, the variables
218+
are cast to the 'Enum' dtype, which, like pandas 'categorical', holds a fixed list of
219+
categories. Let's create a toy train set and test set:
220+
221+
.. code:: python
222+
223+
import polars as pl
224+
from feature_engine.preprocessing import MatchCategories
225+
226+
train = pl.DataFrame({
227+
"city": ["London", "Paris", "Madrid", "Paris"],
228+
"rooms": [2, 3, 1, 3],
229+
})
230+
test = pl.DataFrame({
231+
"city": ["Madrid", "Rome", "London", "Paris"],
232+
"rooms": [1, 2, 4, 3],
233+
})
234+
235+
We fit :class:`MatchCategories()` to the train set:
236+
237+
.. code:: python
238+
239+
match_categories = MatchCategories(missing_values="ignore")
240+
match_categories.fit(train)
241+
242+
match_categories.category_dict_
243+
244+
With polars, the categories are stored in lists:
245+
246+
.. code:: python
247+
248+
{'city': ['London', 'Madrid', 'Paris']}
249+
250+
Now we transform the test set:
251+
252+
.. code:: python
253+
254+
test_t = match_categories.transform(test)
255+
test_t
256+
257+
The variable `city` is now an 'Enum', and the unseen category 'Rome' became a missing
258+
value:
259+
260+
.. code:: text
261+
262+
shape: (4, 2)
263+
┌────────┬───────┐
264+
│ city ┆ rooms │
265+
│ --- ┆ --- │
266+
│ enum ┆ i64 │
267+
╞════════╪═══════╡
268+
│ Madrid ┆ 1 │
269+
│ null ┆ 2 │
270+
│ London ┆ 4 │
271+
│ Paris ┆ 3 │
272+
└────────┴───────┘
273+
274+
We can check the categories in the schema:
275+
276+
.. code:: python
277+
278+
test_t.schema
279+
280+
The categories are the ones learned from the train set:
281+
282+
.. code:: python
283+
284+
Schema({'city': Enum(categories=['London', 'Madrid', 'Paris']), 'rooms': Int64})
285+
286+
The polars 'Enum' dtype only takes strings. Hence, if we cast numerical variables by
287+
setting `ignore_format=True`, their values become strings. The categories are sorted
288+
in numerical order:
289+
290+
.. code:: python
291+
292+
match_categories = MatchCategories(
293+
variables=["rooms"], ignore_format=True, missing_values="ignore"
294+
)
295+
match_categories.fit(train)
296+
297+
match_categories.category_dict_
298+
299+
We see the categories of `rooms` as strings:
300+
301+
.. code:: python
302+
303+
{'rooms': ['1', '2', '3']}
304+
305+
And these are the values after the transformation, where the unseen value 4 became a
306+
missing value:
307+
308+
.. code:: python
309+
310+
match_categories.transform(test)
311+
312+
.. code:: text
313+
314+
shape: (4, 2)
315+
┌────────┬───────┐
316+
│ city ┆ rooms │
317+
│ --- ┆ --- │
318+
│ str ┆ enum │
319+
╞════════╪═══════╡
320+
│ Madrid ┆ 1 │
321+
│ Rome ┆ 2 │
322+
│ London ┆ null │
323+
│ Paris ┆ 3 │
324+
└────────┴───────┘
194325
195326
196327
When to use the transformer

0 commit comments

Comments
 (0)