@@ -6,7 +6,8 @@ MatchCategories
66===============
77
88:class: `MatchCategories() ` ensures that categorical variables are encoded as pandas
9- 'categorical' dtype instead of generic python 'object' or other dtypes.
9+ 'categorical' dtype, or polars 'Enum' dtype, instead of generic python 'object',
10+ string or other dtypes.
1011
1112Under the hood, 'categorical' dtype is a representation that maps each
1213category to an integer, thus providing a more memory-efficient object
@@ -79,10 +80,10 @@ Here are the mappings learnt for each categorical variable:
7980
8081.. code :: python
8182
82- {' pclass' : Int64Index ([1 , 2 , 3 ], dtype = ' int64' ),
83- ' sex' : Index([' female' , ' male' ], dtype = ' object ' ),
84- ' cabin' : Index([' A' , ' B' , ' C' , ' D' , ' E' , ' F' , ' M' , ' T' ], dtype = ' object ' ),
85- ' embarked' : Index([' C' , ' Missing' , ' Q' , ' S' ], dtype = ' object ' )}
83+ {' pclass' : Index ([1 , 2 , 3 ], dtype = ' int64' ),
84+ ' sex' : Index([' female' , ' male' ], dtype = ' str ' ),
85+ ' cabin' : Index([' A' , ' B' , ' C' , ' D' , ' E' , ' F' , ' M' , ' T' ], dtype = ' str ' ),
86+ ' embarked' : Index([' C' , ' Missing' , ' Q' , ' S' ], dtype = ' str ' )}
8687
8788 To see why this matters, let's compare the order in which the categories of `embarked `
8889appear in the raw train and test sets. This is the order in the train set:
@@ -95,7 +96,9 @@ We obtain the following order:
9596
9697.. code :: python
9798
98- array([' S' , ' C' , ' Missing' , ' Q' ], dtype = object )
99+ < StringArray>
100+ [' S' , ' C' , ' Missing' , ' Q' ]
101+ Length: 4 , dtype: str
99102
100103 And this is the order in the test set:
101104
@@ -107,7 +110,9 @@ Which is different from the train set:
107110
108111.. code :: python
109112
110- array([' Q' , ' S' , ' C' ], dtype = object )
113+ < StringArray>
114+ [' Q' , ' S' , ' C' ]
115+ Length: 3 , dtype: str
111116
112117 The categories appear in a different order in each set. If we transform the dataframes
113118using the same `match_categories ` object, categorical variables will be converted to a
@@ -122,7 +127,7 @@ Which is:
122127
123128.. code :: python
124129
125- Index([' C' , ' Missing' , ' Q' , ' S' ], dtype = ' object ' )
130+ Index([' C' , ' Missing' , ' Q' , ' S' ], dtype = ' str ' )
126131
127132 And this is the order we now obtain for the test set:
128133
@@ -134,11 +139,10 @@ The 2 sets now show exactly the same category order:
134139
135140.. code :: python
136141
137- Index([' C' , ' Missing' , ' Q' , ' S' ], dtype = ' object ' )
142+ Index([' C' , ' Missing' , ' Q' , ' S' ], dtype = ' str ' )
138143
139144 If some category was not present in the training data, it will not be mapped
140- to any integer and will thus not get encoded. This behaviour can be modified through the
141- parameter `errors `. Let's illustrate this with the `cabin ` variable. These are the
145+ to any integer and will become a missing value instead. Let's illustrate this with the `cabin ` variable. These are the
142146categories present in the train set:
143147
144148.. code :: python
@@ -149,7 +153,9 @@ We obtain the following categories:
149153
150154.. code :: python
151155
152- array([' B' , ' C' , ' E' , ' D' , ' A' , ' M' , ' T' , ' F' ], dtype = object )
156+ < StringArray>
157+ [' B' , ' C' , ' E' , ' D' , ' A' , ' M' , ' T' , ' F' ]
158+ Length: 8 , dtype: str
153159
154160 And these are the categories present in the test set, which include a category, 'G',
155161that was not seen during training:
@@ -162,7 +168,9 @@ We obtain the following categories, including the unseen 'G':
162168
163169.. code :: python
164170
165- array([' M' , ' F' , ' E' , ' G' ], dtype = object )
171+ < StringArray>
172+ [' M' , ' F' , ' E' , ' G' ]
173+ Length: 4 , dtype: str
166174
167175 After transforming the train set, we obtain the same categories as before, now correctly
168176typed as 'category' dtype:
@@ -176,7 +184,7 @@ Which are:
176184.. code :: python
177185
178186 [' B' , ' C' , ' E' , ' D' , ' A' , ' M' , ' T' , ' F' ]
179- Categories (8 , object ): [' A' , ' B' , ' C' , ' D' , ' E' , ' F' , ' M' , ' T' ]
187+ Categories (8 , str ): [' A' , ' B' , ' C' , ' D' , ' E' , ' F' , ' M' , ' T' ]
180188
181189 But when we transform the test set, the unseen category 'G' is not mapped to any integer,
182190and becomes a missing value instead:
@@ -190,7 +198,130 @@ We see that 'G' has been replaced by a missing value:
190198.. code :: python
191199
192200 [' M' , ' F' , ' E' , NaN]
193- Categories (8 , object ): [' A' , ' B' , ' C' , ' D' , ' E' , ' F' , ' M' , ' T' ]
201+ Categories (8 , str ): [' A' , ' B' , ' C' , ' D' , ' E' , ' F' , ' M' , ' T' ]
202+
203+ Because we set `missing_values="ignore" `, :class: `MatchCategories() ` warns us that
204+ missing values were introduced:
205+
206+ .. code :: python
207+
208+ UserWarning : During the encoding, NaN values were introduced in the feature(s) cabin.
209+
210+ With the default `missing_values="raise" `, :class: `MatchCategories() ` raises an error
211+ instead, both when the data contains missing values and when unseen categories would
212+ introduce them.
213+
214+ With polars
215+ ^^^^^^^^^^^
216+
217+ :class: `MatchCategories() ` also works with polars dataframes. In polars, the variables
218+ are cast to the 'Enum' dtype, which, like pandas 'categorical', holds a fixed list of
219+ categories. Let's create a toy train set and test set:
220+
221+ .. code :: python
222+
223+ import polars as pl
224+ from feature_engine.preprocessing import MatchCategories
225+
226+ train = pl.DataFrame({
227+ " city" : [" London" , " Paris" , " Madrid" , " Paris" ],
228+ " rooms" : [2 , 3 , 1 , 3 ],
229+ })
230+ test = pl.DataFrame({
231+ " city" : [" Madrid" , " Rome" , " London" , " Paris" ],
232+ " rooms" : [1 , 2 , 4 , 3 ],
233+ })
234+
235+ We fit :class: `MatchCategories() ` to the train set:
236+
237+ .. code :: python
238+
239+ match_categories = MatchCategories(missing_values = " ignore" )
240+ match_categories.fit(train)
241+
242+ match_categories.category_dict_
243+
244+ With polars, the categories are stored in lists:
245+
246+ .. code :: python
247+
248+ {' city' : [' London' , ' Madrid' , ' Paris' ]}
249+
250+ Now we transform the test set:
251+
252+ .. code :: python
253+
254+ test_t = match_categories.transform(test)
255+ test_t
256+
257+ The variable `city ` is now an 'Enum', and the unseen category 'Rome' became a missing
258+ value:
259+
260+ .. code :: text
261+
262+ shape: (4, 2)
263+ ┌────────┬───────┐
264+ │ city ┆ rooms │
265+ │ --- ┆ --- │
266+ │ enum ┆ i64 │
267+ ╞════════╪═══════╡
268+ │ Madrid ┆ 1 │
269+ │ null ┆ 2 │
270+ │ London ┆ 4 │
271+ │ Paris ┆ 3 │
272+ └────────┴───────┘
273+
274+ We can check the categories in the schema:
275+
276+ .. code :: python
277+
278+ test_t.schema
279+
280+ The categories are the ones learned from the train set:
281+
282+ .. code :: python
283+
284+ Schema({' city' : Enum(categories = [' London' , ' Madrid' , ' Paris' ]), ' rooms' : Int64})
285+
286+ The polars 'Enum' dtype only takes strings. Hence, if we cast numerical variables by
287+ setting `ignore_format=True `, their values become strings. The categories are sorted
288+ in numerical order:
289+
290+ .. code :: python
291+
292+ match_categories = MatchCategories(
293+ variables = [" rooms" ], ignore_format = True , missing_values = " ignore"
294+ )
295+ match_categories.fit(train)
296+
297+ match_categories.category_dict_
298+
299+ We see the categories of `rooms ` as strings:
300+
301+ .. code :: python
302+
303+ {' rooms' : [' 1' , ' 2' , ' 3' ]}
304+
305+ And these are the values after the transformation, where the unseen value 4 became a
306+ missing value:
307+
308+ .. code :: python
309+
310+ match_categories.transform(test)
311+
312+ .. code :: text
313+
314+ shape: (4, 2)
315+ ┌────────┬───────┐
316+ │ city ┆ rooms │
317+ │ --- ┆ --- │
318+ │ str ┆ enum │
319+ ╞════════╪═══════╡
320+ │ Madrid ┆ 1 │
321+ │ Rome ┆ 2 │
322+ │ London ┆ null │
323+ │ Paris ┆ 3 │
324+ └────────┴───────┘
194325
195326
196327 When to use the transformer
0 commit comments