From bb9d673c9d3beda7a6777085ff26161e070de112 Mon Sep 17 00:00:00 2001 From: Soledad Galli Date: Wed, 26 Aug 2026 12:50:11 +0200 Subject: [PATCH 1/2] Migrate MeanEncoder.fit() to narwhals, add polars support fit() computes, per variable, the mean of y per category (and, with smoothing="auto", the target variance per category), blended with the overall target mean via a weight that increases with category count. transform() and inverse_transform() already came dataframe-agnostic for free from CategoricalMethodsMixin (base_encoder.py, merged separately). Benchmarked a pure-narwhals fit() (group_by/agg for count+mean(+var)) against pandas-native (value_counts + groupby) at 10k-100k rows x 1-10 cols x 5-50 categories: narwhals-on-pandas ran ~1.5x-2.9x slower, worst at the most common shape (1-2 columns, 50k-100k rows), crossing the ~1.7x real-loss threshold; narwhals-on-polars was competitive to faster than pandas-native throughout. Per the benchmark-driven merge-vs-split rule, and matching what the OrdinalEncoder sibling migration found for the same y-groupby-by-category shape of fit(), this splits on `is_pandas = nwd.is_pandas_dataframe(X)`: pandas keeps a close variant of its original value_counts/groupby code, while polars (and other narwhals backends) goes through group_by()/agg(). Bug fixed (pre-existing, confirmed against the unmodified file): the old fit() always called `y.groupby(X[var])`, which raises AttributeError whenever y is a numpy array rather than a Series - e.g. list/array-like y input, which sklearn's check_X_y machinery converts to numpy. This is the exact same bug the OrdinalEncoder sibling found and fixed in its own fit(). Confirmed failing against the unmodified file (tests/test_encoding/test_mean_encoder.py:: test_inverse_transform_when_no_unseen, ::test_inverse_transform_when_ ignore_unseen, ::test_inverse_transform_when_encode_unseen, plus test_check_estimator_encoders.py::test_encoders_when_x_pandas_y_numpy [encoder1] for MeanEncoder) and now passing. Fixed on the pandas branch by pairing X[var] with y via `.assign()` when y isn't a Series (aligns a numpy y positionally, matching how `y.groupby(X[var])` aligned a Series y by index), and on the narwhals branch via `nw.new_series` for a numpy y. Unlike OrdinalEncoder, no cross-backend tie-break fix was needed: MeanEncoder's encoder_dict_ is a category-to-target-mean mapping (a dict), not a rank-ordered list, so backend-dependent group order doesn't affect the result - verified pandas and polars produce identical dicts across smoothing=0.0/100/ "auto" and all three `unseen` settings. Rewrote every test in test_mean_encoder.py as one @pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) case per behavior (41 tests, up from 20), using a narwhals-based, NaN-aware comparison helper; y is passed as a plain list in most tests, which also exercises the numpy-y bug fix on every parametrized case. test_variables_cast_as_category stays pandas-only - it exercises pandas Categorical dtype, which polars has no direct equivalent for. Verified: tests/test_encoding/test_mean_encoder.py 41 passed (was 20, 3 failing). tests/test_encoding full suite: 345 passed, 13 failed - same failing test IDs as the unmodified base minus the 4 MeanEncoder- specific ones fixed here (unmodified base: 17 failed/326 passed); remaining 13 are pre-existing and unrelated (numpy-X rejection per the narwhals check_X() contract, affecting every encoder; OrdinalEncoder's and WoEEncoder's own unmigrated fit() bugs on other in-progress branches). flake8 and mypy clean. Module imports with pandas blocked (verified in isolation from unmigrated sibling modules in the encoding package, which still import pandas on this per-file migration branch). sphinx -W build clean (only the pre-existing linkcode_resolve warning). Verified the class docstring example and every code example in docs/user_guide/encoding/MeanEncoder.rst that doesn't require the Titanic dataset against real output, and added a "With polars" section verified the same way; the Titanic-dataset examples could not be re-run in this sandbox (no network access to openml.org) but are untouched by this change. Downstream consumers (feature_engine/_prediction/base_predictor.py and target_mean_selection.py, which construct MeanEncoder internally) verified via their test suites: 66 passed. Co-Authored-By: Claude Sonnet 5 --- docs/user_guide/encoding/MeanEncoder.rst | 56 +++ feature_engine/encoding/mean_encoding.py | 107 ++++- tests/test_encoding/test_mean_encoder.py | 529 +++++++++++++---------- 3 files changed, 451 insertions(+), 241 deletions(-) diff --git a/docs/user_guide/encoding/MeanEncoder.rst b/docs/user_guide/encoding/MeanEncoder.rst index 086eef00f..2419e8b74 100644 --- a/docs/user_guide/encoding/MeanEncoder.rst +++ b/docs/user_guide/encoding/MeanEncoder.rst @@ -364,6 +364,62 @@ After encoding the features we can use the data sets to train machine learning a encoded variable. Hence, this encoding method is suitable for predictive modelling that uses models that are sensitive to the size of the feature space. +With polars +~~~~~~~~~~~ + +:class:`MeanEncoder()` works the same way with a polars dataframe. Let's create a toy dataset: + +.. code:: python + + import polars as pl + from feature_engine.encoding import MeanEncoder + + X = pl.DataFrame({ + "city": ["London", "Manchester", "Liverpool", "London", "Manchester", "Liverpool"], + "price": [500, 300, 250, 520, 310, 260], + }) + y = pl.Series("target", [1, 0, 0, 1, 0, 1]) + +Let's set up :class:`MeanEncoder()` to encode `city` with the target mean, and fit it to the data: + +.. code:: python + + encoder = MeanEncoder(variables=["city"]) + encoder.fit(X, y) + + encoder.encoder_dict_ + +We see the resulting mappings from category to target mean: + +.. code:: python + + {'city': {'London': 1.0, 'Liverpool': 0.5, 'Manchester': 0.0}} + +Now let's transform the data: + +.. code:: python + + encoder.transform(X) + +We obtain a polars dataframe with the categories in `city` replaced by the target mean: + +.. code:: text + + shape: (6, 2) + ┌──────┬───────┐ + │ city ┆ price │ + │ --- ┆ --- │ + │ f64 ┆ i64 │ + ╞══════╪═══════╡ + │ 1.0 ┆ 500 │ + │ 0.0 ┆ 300 │ + │ 0.5 ┆ 250 │ + │ 1.0 ┆ 520 │ + │ 0.0 ┆ 310 │ + │ 0.5 ┆ 260 │ + └──────┴───────┘ + + Additional resources -------------------- diff --git a/feature_engine/encoding/mean_encoding.py b/feature_engine/encoding/mean_encoding.py index 145aa6394..b43f915cf 100644 --- a/feature_engine/encoding/mean_encoding.py +++ b/feature_engine/encoding/mean_encoding.py @@ -2,7 +2,9 @@ # License: BSD 3 clause from typing import List, Union -import pandas as pd +import narwhals as nw +import narwhals.dependencies as nwd +from narwhals.typing import IntoDataFrame, IntoSeries from feature_engine._check_init_parameters.check_init_input_params import ( _check_return_empty_is_bool, @@ -203,17 +205,17 @@ def __init__( check_parameter_unseen(unseen, ["ignore", "raise", "encode"]) self.unseen = unseen - def fit(self, X: pd.DataFrame, y: pd.Series): + def fit(self, X: IntoDataFrame, y: IntoSeries): """ Learn the mean value of the target for each category of the variable. Parameters ---------- - X: pandas dataframe of shape = [n_samples, n_features] + X: dataframe of shape = [n_samples, n_features] The training input samples. Can be the entire dataframe, not just the variables to be encoded. - y: pandas series + y: Series The target. """ @@ -223,44 +225,105 @@ def fit(self, X: pd.DataFrame, y: pd.Series): self.encoder_dict_ = {} - y_prior = y.mean() + # benchmarked at 10k-100k rows x 1-10 cols x 5-50 categories: a pure + # narwhals fit() ran ~1.5x-2.9x slower than pandas-native here, worse + # at low column counts (the common case), so pandas keeps its native + # groupby/value_counts fast path and only polars (and other + # backends) go through narwhals. + is_pandas = nwd.is_pandas_dataframe(X) - if self.unseen == "encode": - self._unseen = y_prior + if is_pandas is True: + y_prior = y.mean() + + if self.unseen == "encode": + self._unseen = y_prior - if self.smoothing == "auto": - y_var = y.var(ddof=0) - for var in variables_: if self.smoothing == "auto": - damping = y.groupby(X[var]).var(ddof=0) / y_var + y_var = y.var(ddof=0) + + for var in variables_: + # y may be a Series (aligned with X by index, per check_X_y) + # or a numpy array (list/array-like y goes through sklearn's + # column_or_1d, which has no .groupby()) - pair the latter + # with X[var] positionally via assign() instead. + if nwd.is_pandas_series(y): + target, group_keys = y, X[var] + else: + target_name = "__feature_engine_mean_target__" + paired = X[[var]].assign(**{target_name: y}) + target, group_keys = paired[target_name], paired[var] + + if self.smoothing == "auto": + damping = target.groupby(group_keys).var(ddof=0) / y_var + else: + damping = self.smoothing + counts = X[var].value_counts() + counts.index = counts.index.infer_objects() + _lambda = counts / (counts + damping) + self.encoder_dict_[var] = ( + _lambda * target.groupby(group_keys, observed=False).mean() + + (1.0 - _lambda) * y_prior + ).to_dict() + else: + nw_X = nw.from_native(X, eager_only=True) + target_name = "__feature_engine_mean_target__" + if nwd.is_into_series(y): + y_nw = nw.from_native(y, series_only=True).alias(target_name) else: - damping = self.smoothing - counts = X[var].value_counts() - counts.index = counts.index.infer_objects() - _lambda = counts / (counts + damping) - self.encoder_dict_[var] = ( - _lambda * y.groupby(X[var], observed=False).mean() - + (1.0 - _lambda) * y_prior - ).to_dict() + y_nw = nw.new_series( + name=target_name, values=y, backend=nw_X.implementation + ) + nw_Xy = nw_X.with_columns(y_nw) + + y_prior = y_nw.mean() + + if self.unseen == "encode": + self._unseen = y_prior + + if self.smoothing == "auto": + y_var = y_nw.var(ddof=0) + + for var in variables_: + aggs = [ + nw.col(target_name).mean().alias("__mean__"), + nw.col(target_name).len().alias("__count__"), + ] + if self.smoothing == "auto": + aggs.append(nw.col(target_name).var(ddof=0).alias("__var__")) + grouped = nw_Xy.group_by(var, drop_null_keys=True).agg(*aggs) + stats = grouped.to_dict(as_series=False) + + mapping = {} + for i, cat in enumerate(stats[var]): + n_i = stats["__count__"][i] + if self.smoothing == "auto": + damping = stats["__var__"][i] / y_var + else: + damping = self.smoothing + _lambda = n_i / (n_i + damping) + mapping[cat] = ( + _lambda * stats["__mean__"][i] + (1.0 - _lambda) * y_prior + ) + self.encoder_dict_[var] = mapping # assign underscore parameters at the end in case code above fails self.variables_ = variables_ self._get_feature_names_in(X) return self - def inverse_transform(self, X: pd.DataFrame) -> pd.DataFrame: + def inverse_transform(self, X: IntoDataFrame) -> IntoDataFrame: """Convert the encoded variable back to the original values. Note that if unseen was set to 'encode', then this method is not implemented. Parameters ---------- - X: pandas dataframe of shape = [n_samples, n_features]. + X: dataframe of shape = [n_samples, n_features]. The transformed dataframe. Returns ------- - X_tr: pandas dataframe of shape = [n_samples, n_features]. + X_tr: dataframe of shape = [n_samples, n_features]. The un-transformed dataframe, with the categorical variables containing the original values. """ diff --git a/tests/test_encoding/test_mean_encoder.py b/tests/test_encoding/test_mean_encoder.py index a13d0e5bf..7e5cb3a4c 100644 --- a/tests/test_encoding/test_mean_encoder.py +++ b/tests/test_encoding/test_mean_encoder.py @@ -1,4 +1,8 @@ +import math + +import narwhals as nw import pandas as pd +import polars as pl import pytest from numpy import nan from sklearn.exceptions import NotFittedError @@ -6,6 +10,37 @@ from feature_engine.encoding import MeanEncoder +def _to_backend(df: pd.DataFrame, make_df): + """Rebuild a pandas fixture dataframe on the requested backend. + + Swaps float NaN for None in string columns - polars (unlike pandas) + rejects a float NaN mixed into an otherwise-string column. + """ + data = {} + for col in df.columns: + values = df[col].tolist() + if any(isinstance(v, str) for v in values): + values = [ + None if isinstance(v, float) and math.isnan(v) else v for v in values + ] + data[col] = values + return make_df(data) + + +def _assert_values(X, expected: dict) -> None: + """NaN-aware, backend-agnostic comparison of a dataframe's contents.""" + result = nw.from_native(X, eager_only=True).to_dict(as_series=False) + assert list(result.keys()) == list(expected.keys()) + for col, exp_values in expected.items(): + got_values = result[col] + assert len(got_values) == len(exp_values) + for got, exp in zip(got_values, exp_values): + if isinstance(exp, float) and math.isnan(exp): + assert got is None or (isinstance(got, float) and math.isnan(got)) + else: + assert got == pytest.approx(exp) + + # test init params @pytest.mark.parametrize("params", [("raise", True, "auto"), ("ignore", False, 1)]) def test_init_param_assignment(params): @@ -32,36 +67,42 @@ def test_raises_error_when_not_allowed_smoothing_param_in_init(smoothing): # fit and transform -def test_user_enters_1_variable(df_enc): +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_user_enters_1_variable(df_enc, make_df): # test case 1: 1 variable + X = _to_backend(df_enc[["var_A", "var_B"]], make_df) + y = df_enc["target"].tolist() + encoder = MeanEncoder(variables=["var_A"]) - encoder.fit(df_enc[["var_A", "var_B"]], df_enc["target"]) - X = encoder.transform(df_enc[["var_A", "var_B"]]) + encoder.fit(X, y) + Xt = encoder.transform(X) # expected output - transf_df = df_enc.copy() - transf_df["var_A"] = [ - 0.3333333333333333, - 0.3333333333333333, - 0.3333333333333333, - 0.3333333333333333, - 0.3333333333333333, - 0.3333333333333333, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.5, - 0.5, - 0.5, - 0.5, - ] + expected = { + "var_A": [ + 0.3333333333333333, + 0.3333333333333333, + 0.3333333333333333, + 0.3333333333333333, + 0.3333333333333333, + 0.3333333333333333, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.5, + 0.5, + 0.5, + 0.5, + ], + "var_B": df_enc["var_B"].tolist(), + } # test init params assert encoder.variables == ["var_A"] @@ -72,61 +113,66 @@ def test_user_enters_1_variable(df_enc): } assert encoder.n_features_in_ == 2 # test transform output - pd.testing.assert_frame_equal(X, transf_df[["var_A", "var_B"]]) + _assert_values(Xt, expected) -def test_automatically_find_variables(df_enc): +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_automatically_find_variables(df_enc, make_df): # test case 2: automatically select variables + X = _to_backend(df_enc[["var_A", "var_B"]], make_df) + y = df_enc["target"].tolist() + encoder = MeanEncoder(variables=None) - encoder.fit(df_enc[["var_A", "var_B"]], df_enc["target"]) - X = encoder.transform(df_enc[["var_A", "var_B"]]) + encoder.fit(X, y) + Xt = encoder.transform(X) # expected output - transf_df = df_enc.copy() - transf_df["var_A"] = [ - 0.3333333333333333, - 0.3333333333333333, - 0.3333333333333333, - 0.3333333333333333, - 0.3333333333333333, - 0.3333333333333333, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.5, - 0.5, - 0.5, - 0.5, - ] - transf_df["var_B"] = [ - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.3333333333333333, - 0.3333333333333333, - 0.3333333333333333, - 0.3333333333333333, - 0.3333333333333333, - 0.3333333333333333, - 0.5, - 0.5, - 0.5, - 0.5, - ] + expected = { + "var_A": [ + 0.3333333333333333, + 0.3333333333333333, + 0.3333333333333333, + 0.3333333333333333, + 0.3333333333333333, + 0.3333333333333333, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.5, + 0.5, + 0.5, + 0.5, + ], + "var_B": [ + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.3333333333333333, + 0.3333333333333333, + 0.3333333333333333, + 0.3333333333333333, + 0.3333333333333333, + 0.3333333333333333, + 0.5, + 0.5, + 0.5, + 0.5, + ], + } # test init params assert encoder.variables is None @@ -138,50 +184,49 @@ def test_automatically_find_variables(df_enc): } assert encoder.n_features_in_ == 2 # test transform output - pd.testing.assert_frame_equal(X, transf_df[["var_A", "var_B"]]) + _assert_values(Xt, expected) -def test_encoding_when_nan_in_fit_df(df_enc): - df = df_enc.copy() - df.loc[len(df)] = [nan, nan, 0] +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_encoding_when_nan_in_fit_df(df_enc, make_df): + data = { + "var_A": df_enc["var_A"].tolist() + [None], + "var_B": df_enc["var_B"].tolist() + [None], + "target": df_enc["target"].tolist() + [0], + } + df = make_df(data) encoder = MeanEncoder(missing_values="ignore") - encoder.fit(df[["var_A", "var_B"]], df["target"]) + encoder.fit(df[["var_A", "var_B"]], data["target"]) - X = encoder.transform( - pd.DataFrame( - { - "var_A": ["A", nan], - "var_B": ["A", nan], - } - ) - ) + Xt = encoder.transform(make_df({"var_A": ["A", None], "var_B": ["A", None]})) - # transform params - pd.testing.assert_frame_equal( - X, - pd.DataFrame( - { - "var_A": [0.3333333333333333, nan], - "var_B": [0.2, nan], - } - ), + _assert_values( + Xt, + { + "var_A": [0.3333333333333333, nan], + "var_B": [0.2, nan], + }, ) +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) def test_warning_if_transform_df_contains_categories_not_present_in_fit_df( - df_enc, df_enc_rare + df_enc, df_enc_rare, make_df ): # test case 4: when dataset to be transformed contains categories not present # in training dataset + X = _to_backend(df_enc[["var_A", "var_B"]], make_df) + y = df_enc["target"].tolist() + X_rare = _to_backend(df_enc_rare[["var_A", "var_B"]], make_df) msg = "During the encoding, NaN values were introduced in the feature(s) var_A." # check for warning when rare_labels equals 'ignore' with pytest.warns(UserWarning) as record: encoder = MeanEncoder(unseen="ignore") - encoder.fit(df_enc[["var_A", "var_B"]], df_enc["target"]) - encoder.transform(df_enc_rare[["var_A", "var_B"]]) + encoder.fit(X, y) + encoder.transform(X_rare) # check that at least one warning was raised (Pandas 3 may emit additional # deprecation warnings) @@ -190,20 +235,24 @@ def test_warning_if_transform_df_contains_categories_not_present_in_fit_df( assert any(r.message.args[0] == msg for r in record) # check for error when rare_labels equals 'raise' - with pytest.raises(ValueError) as record: + with pytest.raises(ValueError) as record2: encoder = MeanEncoder(unseen="raise") - encoder.fit(df_enc[["var_A", "var_B"]], df_enc["target"]) - encoder.transform(df_enc_rare[["var_A", "var_B"]]) + encoder.fit(X, y) + encoder.transform(X_rare) # check that the error message matches - assert str(record.value) == msg + assert str(record2.value) == msg -def test_fit_raises_error_if_df_contains_na(df_enc_na): +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_fit_raises_error_if_df_contains_na(df_enc_na, make_df): # test case 4: when dataset contains na, fit method + X = _to_backend(df_enc_na[["var_A", "var_B"]], make_df) + y = df_enc_na["target"].tolist() + encoder = MeanEncoder() with pytest.raises(ValueError) as record: - encoder.fit(df_enc_na[["var_A", "var_B"]], df_enc_na["target"]) + encoder.fit(X, y) msg = ( "Some of the variables in the dataset contain NaN. Check and " "remove those before using this transformer or set the parameter " @@ -212,12 +261,17 @@ def test_fit_raises_error_if_df_contains_na(df_enc_na): assert str(record.value) == msg -def test_transform_raises_error_if_df_contains_na(df_enc, df_enc_na): +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_transform_raises_error_if_df_contains_na(df_enc, df_enc_na, make_df): # test case 4: when dataset contains na, transform method + X = _to_backend(df_enc[["var_A", "var_B"]], make_df) + y = df_enc["target"].tolist() + X_na = _to_backend(df_enc_na[["var_A", "var_B"]], make_df) + encoder = MeanEncoder() - encoder.fit(df_enc[["var_A", "var_B"]], df_enc["target"]) + encoder.fit(X, y) with pytest.raises(ValueError) as record: - encoder.transform(df_enc_na[["var_A", "var_B"]]) + encoder.transform(X_na) msg = ( "Some of the variables in the dataset contain NaN. Check and " "remove those before using this transformer or set the parameter " @@ -226,36 +280,42 @@ def test_transform_raises_error_if_df_contains_na(df_enc, df_enc_na): assert str(record.value) == msg -def test_user_enters_1_variable_ignore_format(df_enc_numeric): +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_user_enters_1_variable_ignore_format(df_enc_numeric, make_df): # test case 1: 1 variable + X = _to_backend(df_enc_numeric[["var_A", "var_B"]], make_df) + y = df_enc_numeric["target"].tolist() + encoder = MeanEncoder(variables=["var_A"], ignore_format=True) - encoder.fit(df_enc_numeric[["var_A", "var_B"]], df_enc_numeric["target"]) - X = encoder.transform(df_enc_numeric[["var_A", "var_B"]]) + encoder.fit(X, y) + Xt = encoder.transform(X) # expected output - transf_df = df_enc_numeric.copy() - transf_df["var_A"] = [ - 0.3333333333333333, - 0.3333333333333333, - 0.3333333333333333, - 0.3333333333333333, - 0.3333333333333333, - 0.3333333333333333, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.5, - 0.5, - 0.5, - 0.5, - ] + expected = { + "var_A": [ + 0.3333333333333333, + 0.3333333333333333, + 0.3333333333333333, + 0.3333333333333333, + 0.3333333333333333, + 0.3333333333333333, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.5, + 0.5, + 0.5, + 0.5, + ], + "var_B": df_enc_numeric["var_B"].tolist(), + } # test init params assert encoder.variables == ["var_A"] @@ -264,61 +324,66 @@ def test_user_enters_1_variable_ignore_format(df_enc_numeric): assert encoder.encoder_dict_ == {"var_A": {1: 0.3333333333333333, 2: 0.2, 3: 0.5}} assert encoder.n_features_in_ == 2 # test transform output - pd.testing.assert_frame_equal(X, transf_df[["var_A", "var_B"]]) + _assert_values(Xt, expected) -def test_automatically_find_variables_ignore_format(df_enc_numeric): +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_automatically_find_variables_ignore_format(df_enc_numeric, make_df): # test case 2: automatically select variables + X = _to_backend(df_enc_numeric[["var_A", "var_B"]], make_df) + y = df_enc_numeric["target"].tolist() + encoder = MeanEncoder(variables=None, ignore_format=True) - encoder.fit(df_enc_numeric[["var_A", "var_B"]], df_enc_numeric["target"]) - X = encoder.transform(df_enc_numeric[["var_A", "var_B"]]) + encoder.fit(X, y) + Xt = encoder.transform(X) # expected output - transf_df = df_enc_numeric.copy() - transf_df["var_A"] = [ - 0.3333333333333333, - 0.3333333333333333, - 0.3333333333333333, - 0.3333333333333333, - 0.3333333333333333, - 0.3333333333333333, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.5, - 0.5, - 0.5, - 0.5, - ] - transf_df["var_B"] = [ - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.2, - 0.3333333333333333, - 0.3333333333333333, - 0.3333333333333333, - 0.3333333333333333, - 0.3333333333333333, - 0.3333333333333333, - 0.5, - 0.5, - 0.5, - 0.5, - ] + expected = { + "var_A": [ + 0.3333333333333333, + 0.3333333333333333, + 0.3333333333333333, + 0.3333333333333333, + 0.3333333333333333, + 0.3333333333333333, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.5, + 0.5, + 0.5, + 0.5, + ], + "var_B": [ + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.3333333333333333, + 0.3333333333333333, + 0.3333333333333333, + 0.3333333333333333, + 0.3333333333333333, + 0.3333333333333333, + 0.5, + 0.5, + 0.5, + 0.5, + ], + } # test init params assert encoder.variables is None @@ -330,10 +395,12 @@ def test_automatically_find_variables_ignore_format(df_enc_numeric): } assert encoder.n_features_in_ == 2 # test transform output - pd.testing.assert_frame_equal(X, transf_df[["var_A", "var_B"]]) + _assert_values(Xt, expected) def test_variables_cast_as_category(df_enc_category_dtypes): + # pandas-only: exercises pandas Categorical dtype, which polars has no + # direct equivalent for. df = df_enc_category_dtypes.copy() encoder = MeanEncoder(variables=["var_A"]) encoder.fit(df[["var_A", "var_B"]], df["target"]) @@ -368,13 +435,16 @@ def test_variables_cast_as_category(df_enc_category_dtypes): assert X["var_A"].dtypes.name == "float64" -def test_auto_smoothing(df_enc): +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_auto_smoothing(df_enc, make_df): + X = _to_backend(df_enc[["var_A", "var_B"]], make_df) + y = df_enc["target"].tolist() + encoder = MeanEncoder(smoothing="auto") - encoder.fit(df_enc[["var_A", "var_B"]], df_enc["target"]) - X = encoder.transform(df_enc[["var_A", "var_B"]]) + encoder.fit(X, y) + Xt = encoder.transform(X) # expected output - transf_df = df_enc.copy() var_A_dict = { "A": 0.328335832083958, "B": 0.20707964601769913, @@ -385,8 +455,10 @@ def test_auto_smoothing(df_enc): "B": 0.328335832083958, "C": 0.4541284403669725, } - transf_df["var_A"] = transf_df["var_A"].map(var_A_dict) - transf_df["var_B"] = transf_df["var_B"].map(var_B_dict) + expected = { + "var_A": [var_A_dict[v] for v in df_enc["var_A"]], + "var_B": [var_B_dict[v] for v in df_enc["var_B"]], + } # test init params assert encoder.variables is None @@ -398,16 +470,19 @@ def test_auto_smoothing(df_enc): } assert encoder.n_features_in_ == 2 # test transform output - pd.testing.assert_frame_equal(X, transf_df[["var_A", "var_B"]]) + _assert_values(Xt, expected) -def test_value_smoothing(df_enc): +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_value_smoothing(df_enc, make_df): + X = _to_backend(df_enc[["var_A", "var_B"]], make_df) + y = df_enc["target"].tolist() + encoder = MeanEncoder(smoothing=100) - encoder.fit(df_enc[["var_A", "var_B"]], df_enc["target"]) - X = encoder.transform(df_enc[["var_A", "var_B"]]) + encoder.fit(X, y) + Xt = encoder.transform(X) # expected output - transf_df = df_enc.copy() var_A_dict = { "A": 0.3018867924528302, "B": 0.2909090909090909, @@ -418,8 +493,10 @@ def test_value_smoothing(df_enc): "B": 0.3018867924528302, "C": 0.30769230769230765, } - transf_df["var_A"] = transf_df["var_A"].map(var_A_dict) - transf_df["var_B"] = transf_df["var_B"].map(var_B_dict) + expected = { + "var_A": [var_A_dict[v] for v in df_enc["var_A"]], + "var_B": [var_B_dict[v] for v in df_enc["var_B"]], + } # test init params assert encoder.variables is None @@ -431,40 +508,53 @@ def test_value_smoothing(df_enc): } assert encoder.n_features_in_ == 2 # test transform output - pd.testing.assert_frame_equal(X, transf_df[["var_A", "var_B"]]) + _assert_values(Xt, expected) -def test_encoding_new_categories(df_enc): - df_unseen = pd.DataFrame({"var_A": ["D"], "var_B": ["D"]}) +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_encoding_new_categories(df_enc, make_df): + X = _to_backend(df_enc[["var_A", "var_B"]], make_df) + y = df_enc["target"].tolist() + df_unseen = make_df({"var_A": ["D"], "var_B": ["D"]}) + encoder = MeanEncoder(unseen="encode") - encoder.fit(df_enc[["var_A", "var_B"]], df_enc["target"]) + encoder.fit(X, y) df_transformed = encoder.transform(df_unseen) - assert (df_transformed == df_enc["target"].mean()).all(axis=None) + _assert_values( + df_transformed, + {"var_A": [df_enc["target"].mean()], "var_B": [df_enc["target"].mean()]}, + ) -def test_inverse_transform_when_no_unseen(): - df = pd.DataFrame({"words": ["dog", "dog", "cat", "cat", "cat", "bird"]}) +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_inverse_transform_when_no_unseen(make_df): + df = make_df({"words": ["dog", "dog", "cat", "cat", "cat", "bird"]}) y = [1, 0, 1, 0, 1, 0] enc = MeanEncoder() enc.fit(df, y) dft = enc.transform(df) - pd.testing.assert_frame_equal(enc.inverse_transform(dft), df) + expected = {"words": ["dog", "dog", "cat", "cat", "cat", "bird"]} + _assert_values(enc.inverse_transform(dft), expected) -def test_inverse_transform_when_ignore_unseen(): - df1 = pd.DataFrame({"words": ["dog", "dog", "cat", "cat", "cat", "bird"]}) - df2 = pd.DataFrame({"words": ["dog", "dog", "cat", "cat", "cat", "frog"]}) - df3 = pd.DataFrame({"words": ["dog", "dog", "cat", "cat", "cat", nan]}) +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_inverse_transform_when_ignore_unseen(make_df): + df1 = make_df({"words": ["dog", "dog", "cat", "cat", "cat", "bird"]}) + df2 = make_df({"words": ["dog", "dog", "cat", "cat", "cat", "frog"]}) y = [1, 0, 1, 0, 1, 0] enc = MeanEncoder(unseen="ignore") enc.fit(df1, y) dft = enc.transform(df2) - pd.testing.assert_frame_equal(enc.inverse_transform(dft), df3) + _assert_values( + enc.inverse_transform(dft), + {"words": ["dog", "dog", "cat", "cat", "cat", nan]}, + ) -def test_inverse_transform_when_encode_unseen(): - df1 = pd.DataFrame({"words": ["dog", "dog", "cat", "cat", "cat", "bird"]}) - df2 = pd.DataFrame({"words": ["dog", "dog", "cat", "cat", "cat", "frog"]}) +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_inverse_transform_when_encode_unseen(make_df): + df1 = make_df({"words": ["dog", "dog", "cat", "cat", "cat", "bird"]}) + df2 = make_df({"words": ["dog", "dog", "cat", "cat", "cat", "frog"]}) y = [1, 0, 1, 0, 1, 0] enc = MeanEncoder(unseen="encode") enc.fit(df1, y) @@ -478,8 +568,9 @@ def test_inverse_transform_when_encode_unseen(): assert str(record.value) == msg -def test_inverse_transform_raises_non_fitted_error(): - df1 = pd.DataFrame({"words": ["dog", "dog", "cat", "cat", "cat", "bird"]}) +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_inverse_transform_raises_non_fitted_error(make_df): + df1 = make_df({"words": ["dog", "dog", "cat", "cat", "cat", "bird"]}) y = [1, 0, 1, 0, 1, 0] enc = MeanEncoder() @@ -487,11 +578,11 @@ def test_inverse_transform_raises_non_fitted_error(): with pytest.raises(NotFittedError): enc.inverse_transform(df1) - df1.loc[len(df1) - 1] = nan + df1_na = make_df({"words": ["dog", "dog", "cat", "cat", "cat", None]}) with pytest.raises(ValueError): - enc.fit(df1, y) + enc.fit(df1_na, y) # Test when fit is not called prior to transform. with pytest.raises(NotFittedError): - enc.inverse_transform(df1) + enc.inverse_transform(df1_na) From d9fbf784f3965e96333daab0a02e63494fd21b22 Mon Sep 17 00:00:00 2001 From: Soledad Galli Date: Mon, 31 Aug 2026 00:46:48 +0200 Subject: [PATCH 2/2] Adapt MeanEncoder to narwhals-returning check_X check_X_y now returns a narwhals frame, so bind that to nw_X and keep the original native X for _check_or_select_variables, _check_na, _get_feature_names_in and the nwd.is_pandas_dataframe(X) fast-path check (those helpers still expect native input, matching the CategoricalImputer migration on narwhals-migration). The pandas value_counts/groupby fast path is unchanged - X stays native so no rehydration is needed. The narwhals branch reuses nw_X from check_X_y instead of nw.from_native(X). Co-Authored-By: Claude Sonnet 5 --- feature_engine/encoding/mean_encoding.py | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/feature_engine/encoding/mean_encoding.py b/feature_engine/encoding/mean_encoding.py index b43f915cf..a95de4f43 100644 --- a/feature_engine/encoding/mean_encoding.py +++ b/feature_engine/encoding/mean_encoding.py @@ -219,7 +219,7 @@ def fit(self, X: IntoDataFrame, y: IntoSeries): The target. """ - X, y = check_X_y(X, y) + nw_X, y = check_X_y(X, y) variables_ = self._check_or_select_variables(X) self._check_na(X, variables_) @@ -230,9 +230,7 @@ def fit(self, X: IntoDataFrame, y: IntoSeries): # at low column counts (the common case), so pandas keeps its native # groupby/value_counts fast path and only polars (and other # backends) go through narwhals. - is_pandas = nwd.is_pandas_dataframe(X) - - if is_pandas is True: + if nwd.is_pandas_dataframe(X): y_prior = y.mean() if self.unseen == "encode": @@ -265,7 +263,6 @@ def fit(self, X: IntoDataFrame, y: IntoSeries): + (1.0 - _lambda) * y_prior ).to_dict() else: - nw_X = nw.from_native(X, eager_only=True) target_name = "__feature_engine_mean_target__" if nwd.is_into_series(y): y_nw = nw.from_native(y, series_only=True).alias(target_name)