Skip to content

Commit b87b773

Browse files
solegalliclaude
andcommitted
Migrate MatchCategories to narwhals, add polars support
MatchCategories now accepts pandas and polars dataframes. With pandas it keeps casting to the category dtype; with polars it casts to Enum with the categories learned in fit. Unseen categories become missing values in both. Also fixes the warning and error message when several integer-named pandas columns get missing values (it raised a TypeError). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 7c22dfa commit b87b773

3 files changed

Lines changed: 522 additions & 120 deletions

File tree

‎docs/user_guide/preprocessing/MatchCategories.rst‎

Lines changed: 146 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,8 @@ MatchCategories
66
===============
77

88
:class:`MatchCategories()` ensures that categorical variables are encoded as pandas
9-
'categorical' dtype instead of generic python 'object' or other dtypes.
9+
'categorical' dtype, or polars 'Enum' dtype, instead of generic python 'object',
10+
string or other dtypes.
1011

1112
Under the hood, 'categorical' dtype is a representation that maps each
1213
category to an integer, thus providing a more memory-efficient object
@@ -79,10 +80,10 @@ Here are the mappings learnt for each categorical variable:
7980

8081
.. code:: python
8182
82-
{'pclass': Int64Index([1, 2, 3], dtype='int64'),
83-
'sex': Index(['female', 'male'], dtype='object'),
84-
'cabin': Index(['A', 'B', 'C', 'D', 'E', 'F', 'M', 'T'], dtype='object'),
85-
'embarked': Index(['C', 'Missing', 'Q', 'S'], dtype='object')}
83+
{'pclass': Index([1, 2, 3], dtype='int64'),
84+
'sex': Index(['female', 'male'], dtype='str'),
85+
'cabin': Index(['A', 'B', 'C', 'D', 'E', 'F', 'M', 'T'], dtype='str'),
86+
'embarked': Index(['C', 'Missing', 'Q', 'S'], dtype='str')}
8687
8788
To see why this matters, let's compare the order in which the categories of `embarked`
8889
appear in the raw train and test sets. This is the order in the train set:
@@ -95,7 +96,9 @@ We obtain the following order:
9596

9697
.. code:: python
9798
98-
array(['S', 'C', 'Missing', 'Q'], dtype=object)
99+
<StringArray>
100+
['S', 'C', 'Missing', 'Q']
101+
Length: 4, dtype: str
99102
100103
And this is the order in the test set:
101104

@@ -107,7 +110,9 @@ Which is different from the train set:
107110

108111
.. code:: python
109112
110-
array(['Q', 'S', 'C'], dtype=object)
113+
<StringArray>
114+
['Q', 'S', 'C']
115+
Length: 3, dtype: str
111116
112117
The categories appear in a different order in each set. If we transform the dataframes
113118
using the same `match_categories` object, categorical variables will be converted to a
@@ -122,7 +127,7 @@ Which is:
122127

123128
.. code:: python
124129
125-
Index(['C', 'Missing', 'Q', 'S'], dtype='object')
130+
Index(['C', 'Missing', 'Q', 'S'], dtype='str')
126131
127132
And this is the order we now obtain for the test set:
128133

@@ -134,11 +139,10 @@ The 2 sets now show exactly the same category order:
134139

135140
.. code:: python
136141
137-
Index(['C', 'Missing', 'Q', 'S'], dtype='object')
142+
Index(['C', 'Missing', 'Q', 'S'], dtype='str')
138143
139144
If some category was not present in the training data, it will not be mapped
140-
to any integer and will thus not get encoded. This behaviour can be modified through the
141-
parameter `errors`. Let's illustrate this with the `cabin` variable. These are the
145+
to any integer and will become a missing value instead. Let's illustrate this with the `cabin` variable. These are the
142146
categories present in the train set:
143147

144148
.. code:: python
@@ -149,7 +153,9 @@ We obtain the following categories:
149153

150154
.. code:: python
151155
152-
array(['B', 'C', 'E', 'D', 'A', 'M', 'T', 'F'], dtype=object)
156+
<StringArray>
157+
['B', 'C', 'E', 'D', 'A', 'M', 'T', 'F']
158+
Length: 8, dtype: str
153159
154160
And these are the categories present in the test set, which include a category, 'G',
155161
that was not seen during training:
@@ -162,7 +168,9 @@ We obtain the following categories, including the unseen 'G':
162168

163169
.. code:: python
164170
165-
array(['M', 'F', 'E', 'G'], dtype=object)
171+
<StringArray>
172+
['M', 'F', 'E', 'G']
173+
Length: 4, dtype: str
166174
167175
After transforming the train set, we obtain the same categories as before, now correctly
168176
typed as 'category' dtype:
@@ -176,7 +184,7 @@ Which are:
176184
.. code:: python
177185
178186
['B', 'C', 'E', 'D', 'A', 'M', 'T', 'F']
179-
Categories (8, object): ['A', 'B', 'C', 'D', 'E', 'F', 'M', 'T']
187+
Categories (8, str): ['A', 'B', 'C', 'D', 'E', 'F', 'M', 'T']
180188
181189
But when we transform the test set, the unseen category 'G' is not mapped to any integer,
182190
and becomes a missing value instead:
@@ -190,7 +198,130 @@ We see that 'G' has been replaced by a missing value:
190198
.. code:: python
191199
192200
['M', 'F', 'E', NaN]
193-
Categories (8, object): ['A', 'B', 'C', 'D', 'E', 'F', 'M', 'T']
201+
Categories (8, str): ['A', 'B', 'C', 'D', 'E', 'F', 'M', 'T']
202+
203+
Because we set `missing_values="ignore"`, :class:`MatchCategories()` warns us that
204+
missing values were introduced:
205+
206+
.. code:: python
207+
208+
UserWarning: During the encoding, NaN values were introduced in the feature(s) cabin.
209+
210+
With the default `missing_values="raise"`, :class:`MatchCategories()` raises an error
211+
instead, both when the data contains missing values and when unseen categories would
212+
introduce them.
213+
214+
With polars
215+
^^^^^^^^^^^
216+
217+
:class:`MatchCategories()` also works with polars dataframes. In polars, the variables
218+
are cast to the 'Enum' dtype, which, like pandas 'categorical', holds a fixed list of
219+
categories. Let's create a toy train set and test set:
220+
221+
.. code:: python
222+
223+
import polars as pl
224+
from feature_engine.preprocessing import MatchCategories
225+
226+
train = pl.DataFrame({
227+
"city": ["London", "Paris", "Madrid", "Paris"],
228+
"rooms": [2, 3, 1, 3],
229+
})
230+
test = pl.DataFrame({
231+
"city": ["Madrid", "Rome", "London", "Paris"],
232+
"rooms": [1, 2, 4, 3],
233+
})
234+
235+
We fit :class:`MatchCategories()` to the train set:
236+
237+
.. code:: python
238+
239+
match_categories = MatchCategories(missing_values="ignore")
240+
match_categories.fit(train)
241+
242+
match_categories.category_dict_
243+
244+
With polars, the categories are stored in lists:
245+
246+
.. code:: python
247+
248+
{'city': ['London', 'Madrid', 'Paris']}
249+
250+
Now we transform the test set:
251+
252+
.. code:: python
253+
254+
test_t = match_categories.transform(test)
255+
test_t
256+
257+
The variable `city` is now an 'Enum', and the unseen category 'Rome' became a missing
258+
value:
259+
260+
.. code:: python
261+
262+
shape: (4, 2)
263+
┌────────┬───────┐
264+
│ city ┆ rooms │
265+
│ --- ┆ --- │
266+
│ enum ┆ i64 │
267+
╞════════╪═══════╡
268+
│ Madrid ┆ 1 │
269+
│ null ┆ 2 │
270+
│ London ┆ 4 │
271+
│ Paris ┆ 3 │
272+
└────────┴───────┘
273+
274+
We can check the categories in the schema:
275+
276+
.. code:: python
277+
278+
test_t.schema
279+
280+
The categories are the ones learned from the train set:
281+
282+
.. code:: python
283+
284+
Schema({'city': Enum(categories=['London', 'Madrid', 'Paris']), 'rooms': Int64})
285+
286+
The polars 'Enum' dtype only takes strings. Hence, if we cast numerical variables by
287+
setting `ignore_format=True`, their values become strings. The categories are sorted
288+
in numerical order:
289+
290+
.. code:: python
291+
292+
match_categories = MatchCategories(
293+
variables=["rooms"], ignore_format=True, missing_values="ignore"
294+
)
295+
match_categories.fit(train)
296+
297+
match_categories.category_dict_
298+
299+
We see the categories of `rooms` as strings:
300+
301+
.. code:: python
302+
303+
{'rooms': ['1', '2', '3']}
304+
305+
And these are the values after the transformation, where the unseen value 4 became a
306+
missing value:
307+
308+
.. code:: python
309+
310+
match_categories.transform(test)
311+
312+
.. code:: python
313+
314+
shape: (4, 2)
315+
┌────────┬───────┐
316+
│ city ┆ rooms │
317+
│ --- ┆ --- │
318+
│ str ┆ enum │
319+
╞════════╪═══════╡
320+
│ Madrid ┆ 1 │
321+
│ Rome ┆ 2 │
322+
│ London ┆ null │
323+
│ Paris ┆ 3 │
324+
└────────┴───────┘
194325
195326
196327
When to use the transformer

0 commit comments

Comments
 (0)