Skip to content

Commit 3353bd5

Browse files
authored
Record why scikit-learn is a real dependency, not a mistaken one (pytorch#22269)
## Summary This started as a removal and became a comment fix, because the removal was wrong. The reading was that `scikit-learn>=1.7.1` could never work: coremltools 9.0 accepts scikit-learn only up to 1.5.1, so `_HAS_SKLEARN` is always False and coremltools reports its scikit-learn support disabled on every install. That is all true, and the conclusion drawn from it was still wrong. **coremltools does not reach scikit-learn only through that flag.** `quantization_utils.py` imports `sklearn.cluster.KMeans` at the point of use, and that is the *default* clustering path, taken unless the weight is a single column of at least ten thousand float16 values. So palettization needs scikit-learn whatever `_HAS_SKLEARN` says, and the `CODEBOOK_WEIGHT_ONLY` recipe this repo exposes fails without it: ``` ModuleNotFoundError: scikit-learn is required for k-means quantization. ``` The version ceiling and the palettization path are simply unrelated. The ceiling gates coremltools' converter for scikit-learn's *own* model types, which nothing here uses. Palettization never checks the version. ## So what changed Nothing functional. The dependency stays exactly as it is. What was wrong was the comment. It said nothing here imports scikit-learn, which is true but misleading, and it did not explain why an apparently unsatisfiable pin is correct. Anyone auditing dependencies would reach the same wrong conclusion I did, and might act on it. The comment now records the real reason, and the test tying this entry to coremltools says why they cannot be separated. ## Test plan Measured with coremltools 9.0 on macOS arm64: | environment | result | | --- | --- | | scikit-learn 1.9.0 present | `_HAS_SKLEARN` False, palettization works, lut shape `(16, 1)` | | scikit-learn absent | `ModuleNotFoundError` from `quantization_utils.py` | Which confirms both halves: the converter really is disabled, and palettization is unaffected by that. `.ci/scripts/tests/test_coreml_markers.py` passes, 167 tests.
1 parent 80f7005 commit 3353bd5

2 files changed

Lines changed: 21 additions & 9 deletions

File tree

‎.ci/scripts/tests/test_coreml_markers.py‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -221,10 +221,12 @@ def test_conftest_matches_the_coremltools_marker(system, machine, sys_platform,
221221
@pytest.mark.parametrize("system,machine,sys_platform", PLATFORMS)
222222
@pytest.mark.parametrize("python", PYTHONS)
223223
def test_scikit_learn_follows_coremltools(system, machine, sys_platform, python):
224-
# scikit-learn is there for coremltools' palettization, so it is only useful where
225-
# coremltools is. Nothing else in the wheel imports scikit-learn itself; scipy, which
226-
# used to arrive as its transitive dependency, is declared in requirements-examples.txt
227-
# for the code that does import it.
224+
# coremltools needs scikit-learn to palettize weights, so it is only useful where
225+
# coremltools is. Nothing here imports scikit-learn directly, and coremltools does not
226+
# reach it through the _HAS_SKLEARN flag either: quantization_utils.py imports
227+
# sklearn.cluster.KMeans at the point of use, on the default clustering path. So the
228+
# CODEBOOK_WEIGHT_ONLY recipe fails without this entry even though _HAS_SKLEARN is False,
229+
# and the two must stay tied together.
228230
assert _marker_says_installed(
229231
_base_dependency("scikit-learn"), system, machine, sys_platform, python
230232
) == _marker_says_installed(

‎setup.py‎

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -696,11 +696,21 @@ def _base_dependencies() -> List[str]:
696696
# with coremltools 9.0 on a Linux aarch64 machine. Keep this in sync with the condition
697697
# in conftest.py; .ci/scripts/tests/test_coreml_markers.py checks that the two agree.
698698
"coremltools==9.0; (platform_system == 'Darwin' or (platform_system == 'Linux' and platform_machine == 'x86_64')) and python_version < '3.14'",
699-
# coremltools uses scikit-learn for palettization, so it follows coremltools. Without a
700-
# marker it also installed where coremltools does not, most visibly on Windows, and
701-
# brought scipy with it. Nothing in this repository imports scikit-learn from the Core
702-
# ML backend; the example scripts that do import it, and the scripts that import scipy,
703-
# now name both in requirements-examples.txt rather than relying on this entry.
699+
# coremltools needs scikit-learn to palettize weights, so it follows coremltools.
700+
# Without a marker it also installed where coremltools does not, most visibly on
701+
# Windows, and brought scipy with it.
702+
#
703+
# Nothing here imports scikit-learn directly. coremltools does, and not through the
704+
# _HAS_SKLEARN flag it sets: quantization_utils.py imports sklearn.cluster.KMeans at the
705+
# point of use, and that is the default clustering path, taken unless the weight is a
706+
# single column of at least ten thousand float16 values. So the CODEBOOK_WEIGHT_ONLY
707+
# recipe raises ModuleNotFoundError without this, whatever _HAS_SKLEARN says.
708+
#
709+
# The floor is above the version coremltools accepts for its own scikit-learn model
710+
# converter, which is why that converter reports itself disabled. That is a separate
711+
# feature nothing here uses, and palettization does not check the version, so the two
712+
# are unrelated. Measured with coremltools 9.0 and scikit-learn 1.9.0: the converter is
713+
# off and palettization works.
704714
"scikit-learn>=1.7.1; (platform_system == 'Darwin' or (platform_system == 'Linux' and platform_machine == 'x86_64')) and python_version < '3.14'",
705715
"hydra-core>=1.3.0",
706716
"omegaconf>=2.3.0",

0 commit comments

Comments
 (0)