feat: add optional resampler= kwarg to AutoML.fit for per-fold class-imbalance handling (#1200)

[Imran Ahamed (immu4989)]

Why are these changes needed?

Implements the option (3) design agreed in #1200: an optional, off-by-default resampler= hook for users who need imbalanced-learn-style resampling inside FLAML's evaluation loop without altering validation partitions.

The benchmark in #1200 found that per-fold plus final SMOTE did not materially outperform applying SMOTE once before AutoML.fit() on the tested datasets. This PR therefore presents resampler= as a convenience rather than a guaranteed quality improvement, and retains pre-applied SMOTE as a valid alternative.

What does this PR do?

• Adds a keyword-only resampler= argument to AutoML.fit.
• Requires a callable, sklearn.base.clone-compatible fit_resample(X, y) implementation.
• Clones and applies the resampler to CV and holdout training partitions while leaving validation partitions unchanged.
• Applies a fresh clone during final and retrain fitting so tuned and deployed models use consistent class distributions.
• Rejects combinations with sample_weight, including estimator-specific settings, because resampling breaks row alignment.
• Rejects combinations with ensemble because sklearn stacking performs internal cross-validation and pre-resampling can leak synthetic samples across its folds.
• Keeps imbalanced-learn optional: deterministic local tests cover core behavior, with a separate SMOTE integration test when the package is installed.
• Updates the production-deployment guide with usage, compatibility constraints, benchmark context, and the pre-applied-SMOTE alternative.

Related issue number

Closes #1200.

Verification

• pytest test/automl/test_resampler.py -q — 13 passed.
• pytest test/automl/test_multiclass.py -k ensemble -q — 2 passed.
• pre-commit run --files flaml/automl/automl.py flaml/automl/ml.py test/automl/test_resampler.py website/docs/Use-Cases/Production-Deployment.md — all hooks passed.

Checks

• I've used pre-commit to lint the changes in this PR (note the same in integrated in our CI checks).
• I've included any doc changes needed for https://microsoft.github.io/FLAML/. See https://microsoft.github.io/FLAML/docs/Contribute#documentation to build and test documentation locally.
• I've added tests (if relevant) corresponding to the changes introduced in this PR.
• I've made sure all auto checks have passed.

[Copilot]

Pull request overview

This PR adds an opt-in resampler= hook to AutoML.fit to support per-fold resampling (e.g., SMOTE-style) inside FLAML’s CV/holdout evaluation loop, avoiding synthesized-sample leakage into validation folds as requested in issue #1200.

Changes:

• Add resampler=None parameter to AutoML.fit, validate inputs, and store the resampler on the task for downstream access.
• Apply per-fold resampling in get_val_loss by cloning the provided resampler and calling fit_resample(X_train, y_train) before estimator.fit(...).
• Add a new test module to exercise the new API and its validation behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

File	Description
flaml/automl/automl.py	Introduces resampler argument and performs early validation / task attachment.
flaml/automl/ml.py	Applies the resampler per fold/call (clone + fit_resample) just before estimator training.
test/automl/test_resampler.py	Adds tests for the new resampler= behavior and validation.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Suppressed comments (4)

flaml/automl/automl.py:2175

• The user-facing Production-Deployment guide still says applying SMOTE upstream is the current recommendation and describes resampler= as a future integration (website/docs/Use-Cases/Production-Deployment.md:229). After this API ships, that guidance is stale and continues directing users toward the leakage-prone workaround; update the guide in this PR.

                by default. See issue #1200 for the design discussion and benchmarks.

flaml/automl/automl.py:2357

• hasattr accepts a non-callable attribute, so a cloneable object with fit_resample = None passes entry validation and fails mid-fold with an unrelated 'NoneType' object is not callable error. Check callability to preserve the documented fail-fast behavior.

            if not hasattr(resampler, "fit_resample"):

flaml/automl/automl.py:2350

• This validation checks only the argument passed directly to fit(), but fit_kwargs_by_estimator is resolved from AutoML(...) settings at line 2385. Thus AutoML(fit_kwargs_by_estimator={"lgbm": {"sample_weight": ...}}).fit(..., resampler=...) bypasses the error and reaches estimator fitting with weights shorter than the resampled data. Resolve the effective settings before checking them.

This issue also appears on line 2357 of the same file.

            weight_sources = [fit_kwargs] + list((fit_kwargs_by_estimator or {}).values())
            if any("sample_weight" in kw for kw in weight_sources):

flaml/automl/automl.py:2175

• The public contract is inaccurate: a plain object exposing fit_resample is rejected unless sklearn.clone-compatible, while the implementation also applies the sampler to holdout evaluation rather than only CV folds. Document both constraints/paths so callers can select a compatible object and understand holdout behavior.

This issue also appears on line 2175 of the same file.

            resampler: object, default=None | An imbalanced-learn-compatible resampler
                (any object exposing `fit_resample(X, y) -> (X, y)`, such as
                `imblearn.over_sampling.SMOTE`). When set, the resampler is cloned and
                applied to each cross-validation fold's training partition before the
                estimator is fitted — validation partitions are left at the raw class

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

[Imran Ahamed (immu4989)]

I dug into the red build matrix. This does not appear to be caused by the resampler changes.

The new resampler tests pass in CI (12 passed, with the optional SMOTE integration test skipped), and the pre-commit and docs checks are green. The failures are all in existing time-series tests for naive/seasonal-naive models, with:

TypeError: initialization_method must be a string

Current main is failing with the same 7-of-8 matrix pattern and the same tests: https://github.com/microsoft/FLAML/actions/runs/33228511089

The dependency change appears to be statsmodels: the last passing main run used 0.14.6, while both the current main run and this PR installed 0.15.0. Last passing main run for comparison: https://github.com/microsoft/FLAML/actions/runs/33037515139

I’ll leave the final auto-check checkbox unchecked until the upstream statsmodels compatibility issue is resolved and this PR can be rerun.