Repository navigation
Commit ea3641d
authored
Skip torchtune on Python 3.14 in the examples requirements (pytorch#21832)
## The problem
Every Python 3.14 wheel job fails, on all four platforms, before it
builds anything. It dies in the
pre-build step that installs `requirements-examples.txt`.
Here is the chain:
```
requirements-examples.txt
-> torchtune declares "pyarrow<21.0.0"
-> no pyarrow below 21 publishes a cp314 wheel
-> pip downloads pyarrow-20.0.0.tar.gz and compiles it
-> the source build needs a prebuilt Arrow C++ library, which is absent
-> pip returns non-zero and the whole job dies
```
From a failing log:
```
Collecting pyarrow<21.0.0 (from torchtune ...)
Downloading pyarrow-20.0.0.tar.gz
...
ERROR: Failed building wheel for pyarrow
error: failed-wheel-build-for-install
```
Python 3.10 through 3.13 are fine because pyarrow 20.0.0 ships wheels
for those versions, so they
download a binary and never compile anything. Only 3.14 falls through to
the source build.
The first pyarrow release with a cp314 wheel is 22.0.0, which is above
torchtune's cap, so there is
no version in the allowed range that works on 3.14.
## The fix
One environment marker, so torchtune is skipped on Python 3.14:
```
torchtune @ git+https://github.com/pytorch/torchtune.git@3c1872ac... ; python_version < "3.14"
```
The pyarrow cap belongs to torchtune, so it cannot be relaxed from here.
Once torchtune is out of
the picture on 3.14, the only remaining constraint is `pyarrow>=15.0.0`
from `datasets`, and pip
picks a cp314 wheel.
This is the same approach already used for `coremltools` in `setup.py`,
which likewise has no cp314
support.
## What this does and does not fix
Being precise, because this change is a prerequisite rather than the
whole story.
It fixes the pre-build step on all four platforms. With that step
passing, the aarch64 Linux and
Windows 3.14 jobs run to completion.
It does not by itself make the Linux x86_64 and macOS 3.14 jobs green.
Once the pre-build step
passes, those two go on to run a Core ML case in their wheel smoke test:
```
.ci/scripts/wheel/test_linux.py backend=Backend.CoreMlExportOnly
.ci/scripts/wheel/test_macos.py backend=Backend.CoreMlExportAndTest
```
The wheel deliberately omits `coremltools` on 3.14 (`setup.py`, the
marker mentioned above), and
neither test script has a version guard, so the Core ML case fails with
`ModuleNotFoundError: No module named 'coremltools'`. That needs a
separate, small change to skip
the Core ML case on 3.14, which is a CI test change rather than a
requirements change and so is
kept out of this commit.
A developer install on 3.14 is also worth one note. It previously failed
at the pyarrow source
build and now succeeds, but `executorch.extension.llm.modules` imports
torchtune eagerly, so that
one subsystem is unavailable on 3.14. torchtune cannot be installed on
3.14 by any means today, so
no version of this change keeps those modules working.
## Testing
Verified:
- Confirmed against the package index that no pyarrow below 21.0.0 has a
cp314 file, and that
22.0.0 is the first release that does.
- Reproduced the failure in a real Python 3.14 environment, matching the
error chain above.
- Confirmed that simply providing cmake is not enough: with cmake
working, the source build then
fails at `Could not find a package configuration file provided by
"Arrow"`.
- With this change, installing `requirements-examples.txt` on Python
3.14 succeeds and resolves
pyarrow to a cp314 wheel, and `datasets`, `timm`, `torchsr` and
`transformers` all import.
- On Python 3.13 the same file still installs torchtune with pyarrow
20.0.0 from a wheel, so
behaviour below 3.14 is unchanged.
Not verified:
- A full wheel build or its smoke tests on Python 3.14.1 parent 813145d commit ea3641d
1 file changed
Lines changed: 4 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
7 | 10 | | |
0 commit comments