refactor variable handling module for narwahls (#978)

* update variable handling module for narwahls

* creating own datetime parser

* Improve readability of narwhals date/type-check helpers, add missing tests

Replace the double-parse-with-disagreeing-defaults trick in
_looks_like_date_string with a direct call to dateutil's parser()._parse(),
which exposes which date/time fields were actually found in a string without
needing to approximate it - this also drops the now-unneeded sentinel
default datetimes and the defensive str() coercion at its call site. Make
truthiness checks and compound boolean returns explicit throughout the
module, and restore the pre-narwhals function names that PR #978 had
prefixed with _nw_ for no continuing reason.

Rename test_fe_type_checks.py to test_variable_type_checks.py to match the
module it tests, add docstrings, and add coverage for _looks_like_date_string
and _is_categories_num, the two functions that previously had no direct
tests.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Replace per-column schema access with bulk narwhals selectors for speed

nw_X.schema is not cached - every access re-derives the full schema from
the underlying native dataframe, so checking dtype-based conditions
(is_numeric(), native Date/Datetime, categorical/enum/string) one column
at a time inside a loop was quadratic instead of linear. Replace each such
loop with a single nw_df.select(<selector>).columns call converted to a
set, then a plain membership test per column - confirmed old vs new give
identical results, and measured 8x-120x speedups depending on backend and
column count. Also use by_dtype(Date, Datetime) to bulk-detect native
datetime columns in one pass, only falling back to the expensive
per-value _is_categorical_and_is_datetime check for columns that aren't
already known to be numeric or natively datetime. Drop the now-unused
_is_date_or_datetime import from both files.

Simplify _looks_like_date_string's comment to link directly to the pandas
source it mirrors, and instantiate dateutil's parser() per call instead of
reusing a module-level instance.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* refactor find variables:

* refactor check_variables

* final refactor of find and check variables

* finalise migration of variable handling module

* update user guide

* Trim backend-difference notes from docs, revert datetime.py out of scope

Removes the trailing pandas/polars note blocks from the check/find
categorical and datetime variable docs, keeping them focused on the
walkthrough. Reverts feature_engine/datetime/datetime.py to main - the
DatetimeFeatures index-datetime fix needed there for the narwhals
migration belongs in a separate datetime-module PR.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

Author: solegalli

AGENTS.md

@@ -0,0 +1,55 @@
+# AGENTS.md
+
+Conventions for working in this repo. Optimize for readability and speed, 
+in that order of how you decide, but don't ship a slow default when a 
+fast one is free.
+
+## Inputs
+
+Feature-engine transformers take dataframes (pandas, polars, or any other
+narwhals-supported backend) as input, not numpy arrays. Don't add
+handling for array input.
+
+## Booleans and control flow
+
+- Compare booleans explicitly: `if x is True:` / `if x is False:`, never
+  `if x:` / `if not x:`.
+- Check container emptiness with `len(x) == 0`, never `if not x:`.
+- `isinstance(...)` checks and `in`/`not in` membership tests are already
+  explicit — leave them as-is, this rule isn't about those.
+
+## Comments
+
+Max 2 lines. Only explain a non-obvious WHY (a hidden constraint, a subtle
+backend difference, a workaround) — never describe WHAT the code does.
+
+## Don't anticipate errors
+
+Don't add error handling or validation for scenarios that can't happen. If
+unsure whether something can happen, check it (grep, run a quick repro) or
+ask — don't guess and defensively code around it.
+
+## Redundant lists/sets
+
+- Narwhals' `.columns` is already `list[str]` — don't wrap it in `list()`.
+- pandas' `.columns` is an `Index`, not a list — `list()` is required there
+  (an `Index == list` comparison is elementwise, not a clean bool).
+
+## Verify before applying
+
+Benchmark before claiming a speedup, and diff old-vs-new output across
+realistic and edge cases (empty/all-NaN, both backends, both dtype
+branches) before trusting a rewrite — logic mistakes here are easy to make
+and easy to miss without an actual comparison.
+
+## Tests
+
+- `pytest.raises(ExceptionType, match=msg)`, never
+  `with pytest.raises() as record: ... assert str(record.value) == msg`.
+
+## API changes
+
+- New parameters default to preserve current behavior.
+- When adding a parameter to a function called from multiple sites (or a
+  shared private helper), thread it through every call site, not just the
+  one you're looking at.

docs/user_guide/variable_handling/check_all_variables.rst

@@ -38,8 +38,8 @@ Now, we create the dataset:
     X["cat_var1"] = ["Hello"] * 1000
     X["cat_var2"] = ["Bye"] * 1000
 
-    X["date1"] = pd.date_range("2020-02-24", periods=1000, freq="T")
-    X["date2"] = pd.date_range("2021-09-29", periods=1000, freq="H")
+    X["date1"] = pd.date_range("2020-02-24", periods=1000, freq="min")
+    X["date2"] = pd.date_range("2021-09-29", periods=1000, freq="h")
     X["date3"] = ["2020-02-24"] * 1000
 
     print(X.head())
@@ -89,4 +89,48 @@ Below we see the error message:
 
 .. code:: python
 
-    KeyError: 'Some of the variables are not in the dataframe.'
+    KeyError: 'Some of the variables are not in the dataframe.'
+
+With polars
+-----------
+
+:class:`check_all_variables()` works in the same way with a polars dataframe.
+Let's create an equivalent toy dataset:
+
+.. code:: python
+
+    import polars as pl
+    from datetime import datetime, timedelta
+    from sklearn.datasets import make_classification
+    from feature_engine.variable_handling import check_all_variables
+
+    X, y = make_classification(
+        n_samples=1000,
+        n_features=4,
+        n_redundant=1,
+        n_clusters_per_class=1,
+        weights=[0.50],
+        class_sep=2,
+        random_state=1,
+    )
+
+    colnames = [f"num_var_{i+1}" for i in range(4)]
+    X = pl.DataFrame(X, schema=colnames)
+
+    X = X.with_columns(
+        pl.lit("Hello").alias("cat_var1"),
+        pl.lit("Bye").alias("cat_var2"),
+        pl.Series("date1", [datetime(2020, 2, 24) + timedelta(minutes=i) for i in range(1000)]),
+        pl.Series("date2", [datetime(2021, 9, 29) + timedelta(hours=i) for i in range(1000)]),
+        pl.lit("2020-02-24").alias("date3"),
+    )
+
+    checked_vars = check_all_variables(X, ["num_var_1", "cat_var1", "date1"])
+
+    checked_vars
+
+The output is the list of variable names passed to the function:
+
+.. code:: python
+
+    ['num_var_1', 'cat_var1', 'date1']

docs/user_guide/variable_handling/check_categorical_variables.rst

@@ -38,8 +38,8 @@ Now, we create the dataset:
     X["cat_var1"] = ["Hello"] * 1000
     X["cat_var2"] = ["Bye"] * 1000
 
-    X["date1"] = pd.date_range("2020-02-24", periods=1000, freq="T")
-    X["date2"] = pd.date_range("2021-09-29", periods=1000, freq="H")
+    X["date1"] = pd.date_range("2020-02-24", periods=1000, freq="min")
+    X["date2"] = pd.date_range("2021-09-29", periods=1000, freq="h")
     X["date3"] = ["2020-02-24"] * 1000
 
     print(X.head())
@@ -89,4 +89,49 @@ Below we see the error message:
 .. code:: python
 
     TypeError: Some of the variables are not categorical. Please cast them as object
-    or categorical before using this transformer.
+    or categorical before using this transformer.
+
+With polars
+-----------
+
+:class:`check_categorical_variables()` works in the same way with a polars dataframe.
+Let's create an equivalent toy dataset:
+
+.. code:: python
+
+    import polars as pl
+    from datetime import datetime, timedelta
+    from sklearn.datasets import make_classification
+    from feature_engine.variable_handling import check_categorical_variables
+
+    X, y = make_classification(
+        n_samples=1000,
+        n_features=4,
+        n_redundant=1,
+        n_clusters_per_class=1,
+        weights=[0.50],
+        class_sep=2,
+        random_state=1,
+    )
+
+    colnames = [f"num_var_{i+1}" for i in range(4)]
+    X = pl.DataFrame(X, schema=colnames)
+
+    X = X.with_columns(
+        pl.lit("Hello").alias("cat_var1"),
+        pl.lit("Bye").alias("cat_var2"),
+        pl.Series("date1", [datetime(2020, 2, 24) + timedelta(minutes=i) for i in range(1000)]),
+        pl.Series("date2", [datetime(2021, 9, 29) + timedelta(hours=i) for i in range(1000)]),
+        pl.lit("2020-02-24").alias("date3"),
+    )
+
+    var_cat = check_categorical_variables(X, ["cat_var1", "date3"])
+
+    var_cat
+
+Both variables are of type string and hence, will be in the resulting list:
+
+.. code:: python
+
+    ['cat_var1', 'date3']
+

docs/user_guide/variable_handling/check_datetime_variables.rst

@@ -38,8 +38,8 @@ Now, we create the dataset:
     X["cat_var1"] = ["Hello"] * 1000
     X["cat_var2"] = ["Bye"] * 1000
 
-    X["date1"] = pd.date_range("2020-02-24", periods=1000, freq="T")
-    X["date2"] = pd.date_range("2021-09-29", periods=1000, freq="H")
+    X["date1"] = pd.date_range("2020-02-24", periods=1000, freq="min")
+    X["date2"] = pd.date_range("2021-09-29", periods=1000, freq="h")
     X["date3"] = ["2020-02-24"] * 1000
 
     print(X.head())
@@ -92,3 +92,49 @@ Below the error message:
 .. code:: python
 
     TypeError: Some of the variables are not or cannot be parsed as datetime.
+
+With polars
+-----------
+
+:class:`check_datetime_variables()` works in the same way with a polars dataframe.
+Let's create an equivalent toy dataset:
+
+.. code:: python
+
+    import polars as pl
+    from datetime import datetime, timedelta
+    from sklearn.datasets import make_classification
+    from feature_engine.variable_handling import check_datetime_variables
+
+    X, y = make_classification(
+        n_samples=1000,
+        n_features=4,
+        n_redundant=1,
+        n_clusters_per_class=1,
+        weights=[0.50],
+        class_sep=2,
+        random_state=1,
+    )
+
+    colnames = [f"num_var_{i+1}" for i in range(4)]
+    X = pl.DataFrame(X, schema=colnames)
+
+    X = X.with_columns(
+        pl.lit("Hello").alias("cat_var1"),
+        pl.lit("Bye").alias("cat_var2"),
+        pl.Series("date1", [datetime(2020, 2, 24) + timedelta(minutes=i) for i in range(1000)]),
+        pl.Series("date2", [datetime(2021, 9, 29) + timedelta(hours=i) for i in range(1000)]),
+        pl.lit("2020-02-24").alias("date3"),
+    )
+
+    var_date = check_datetime_variables(X, ["date2", "date3"])
+
+    var_date
+
+In this case, both variables, if they can be parsed as datetime, will be in the
+resulting list:
+
+.. code:: python
+
+    ['date2', 'date3']
+

docs/user_guide/variable_handling/check_numerical_variables.rst

@@ -25,7 +25,7 @@ Now, we create the dataset:
         "City": ["London", "Manchester", "Liverpool", "Bristol"],
         "Age": [20, 21, 19, 18],
         "Marks": [0.9, 0.8, 0.7, 0.6],
-        "dob": pd.date_range("2020-02-24", periods=4, freq="T"),
+        "dob": pd.date_range("2020-02-24", periods=4, freq="min"),
     })
 
     print(df.head())
@@ -67,3 +67,32 @@ Below we see the error message:
 
     TypeError: Some of the variables are not numerical. Please cast them as numerical
     before using this transformer.
+
+With polars
+-----------
+
+:class:`check_numerical_variables()` works in the same way with a polars dataframe:
+
+.. code:: python
+
+    import polars as pl
+    from datetime import datetime
+    from feature_engine.variable_handling import check_numerical_variables
+
+    df = pl.DataFrame({
+        "Name": ["tom", "nick", "krish", "jack"],
+        "City": ["London", "Manchester", "Liverpool", "Bristol"],
+        "Age": [20, 21, 19, 18],
+        "Marks": [0.9, 0.8, 0.7, 0.6],
+        "dob": [datetime(2020, 2, 24, 0, i) for i in range(4)],
+    })
+
+    var_num = check_numerical_variables(df, ['Age', 'Marks'])
+
+    var_num
+
+If the variables are numerical, the function returns their names in a list:
+
+.. code:: python
+
+    ['Age', 'Marks']

docs/user_guide/variable_handling/find_all_variables.rst

@@ -125,4 +125,68 @@ However, this command returns an empty list:
         X[[ 'date1', 'date2', 'date3']],
         exclude_datetime=True,
         return_empty=True,
-    )
+    )
+
+With polars
+-----------
+
+:class:`find_all_variables()` works in the same way with a polars dataframe.
+Let's create an equivalent toy dataset:
+
+.. code:: python
+
+    import polars as pl
+    from datetime import datetime, timedelta
+    from sklearn.datasets import make_classification
+    from feature_engine.variable_handling import find_all_variables
+
+    X, y = make_classification(
+        n_samples=1000,
+        n_features=4,
+        n_redundant=1,
+        n_clusters_per_class=1,
+        weights=[0.50],
+        class_sep=2,
+        random_state=1,
+    )
+
+    colnames = [f"num_var_{i+1}" for i in range(4)]
+    X = pl.DataFrame(X, schema=colnames)
+
+    X = X.with_columns(
+        pl.lit("Hello").alias("cat_var1"),
+        pl.lit("Bye").alias("cat_var2"),
+        pl.Series("date1", [datetime(2020, 2, 24) + timedelta(minutes=i) for i in range(1000)]),
+        pl.Series("date2", [datetime(2021, 9, 29) + timedelta(hours=i) for i in range(1000)]),
+        pl.lit("2020-02-24").alias("date3"),
+    )
+
+    vars_all = find_all_variables(X)
+
+    vars_all
+
+We see the variable names in the list below:
+
+.. code:: python
+
+    ['num_var_1',
+     'num_var_2',
+     'num_var_3',
+     'num_var_4',
+     'cat_var1',
+     'cat_var2',
+     'date1',
+     'date2',
+     'date3']
+
+And, as with pandas, we can exclude the datetime variables:
+
+.. code:: python
+
+    vars_all = find_all_variables(X, exclude_datetime=True)
+
+    vars_all
+
+.. code:: python
+
+    ['num_var_1', 'num_var_2', 'num_var_3', 'num_var_4', 'cat_var1', 'cat_var2']