Skip to content

Commit 730b77a

Browse files
authored
Build PyTorch wheel via PEP 517 without poisoning the CI image (pytorch#21685)
Forward fix for pytorch#21562. pytorch#21562 moved the PyTorch wheel build from `setup.py bdist_wheel` to `python -m build --wheel --no-isolation`. That direction is correct and is kept here. The problem was that it installed the PEP 517 build requirements permanently into the image's conda environment, and two different things in that install changed how every later build in the image behaves. Both broke main. ### 1. All 17 RISC-V jobs failed scikit-build-core ships a setuptools plugin that registers itself as the `build_ext` command class. Once it is present in the environment, any later `pip install --no-build-isolation` picks it up. The tokenizers build is exactly that, so its CMake configure started running through scikit-build-core, which turns on C++20 module dependency scanning. The image compiler cannot produce the scan output: ``` [299/312] Generating CXX dyndep file CMakeFiles/pytorch_tokenizers_cpp.dir/CXX.dd FAILED: CMakeFiles/pytorch_tokenizers_cpp.dir/src/python_bindings.cpp.o cc1plus: error: to generate dependencies you must specify either '-M' or '-MM' ``` ### 2. The wheel was built without LAPACK The same install also put a pip `cmake` wheel into the environment, which lands on PATH ahead of the image's own cmake. The two are the same version but not interchangeable: the pip wheel does not search the conda environment, so `FindMKL` stopped finding the MKL libraries that are installed there and PyTorch was configured with MKL and LAPACK off. 12 tests in `extension/llm/modules/test/test_turboquant_kv_cache.py` then failed with: ``` RuntimeError: Calling torch.geqrf on a CPU tensor requires compiling PyTorch with LAPACK. ``` ### Fix Install the build requirements into a throwaway venv instead of into the image, and invoke the build through that venv's interpreter by absolute path. The venv is created with `--system-site-packages`, so PyTorch still builds against the image's numpy and toolchain and still reuses sccache. The venv is deleted once the wheel is built, so neither the scikit-build-core plugin nor the pip cmake survives into the image, and the image's own cmake stays in front. ### Evidence From the docker image build log, PyTorch's configure summary: | | pytorch#21562, as merged | this PR | | --- | --- | --- | | cmake actually used | `.../site-packages/cmake/data/bin/cmake` | `/opt/conda/envs/py_3.10/bin/cmake` | | `USE_MKL` | `OFF` | `ON` | | `USE_LAPACK` | `0` | `1` | CI on this PR: - `test-riscv`: 17 of 17 green. On main it is 17 of 17 red. - `test-lora-multimethod-linux`: green. On main it is red. ### Note on the jobs that did not finish `unittest / linux`, `unittest-editable / linux` and `test-lora-linux` were all cancelled at 1 hour 31 minutes, which is the 90 minute job cap. Any PR that touches `.ci/docker/` forces a full image rebuild that consumes roughly 30 minutes of that budget, so these three cannot complete on a PR of this shape. That is also why the LAPACK problem was not visible on pytorch#21562 before it merged. The `USE_LAPACK` line in the image build log above is the direct evidence for that half of the fix, since it is read at configure time rather than at test time. ### Relationship to the revert This branch contains two commits, a revert of pytorch#21562 followed by the corrected version. The net diff against main is a single file, `.ci/docker/common/install_pytorch.sh`, and squash merge collapses them into one commit. This supersedes pytorch#21684, which is a straight revert of pytorch#21562. If this lands, pytorch#21684 should be closed without merging. Do not merge pytorch#21684 after this one, since it would restore the previous file wholesale and undo this fix.
1 parent 48741ac commit 730b77a

1 file changed

Lines changed: 19 additions & 8 deletions

File tree

‎.ci/docker/common/install_pytorch.sh‎

Lines changed: 19 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -53,14 +53,25 @@ install_pytorch_and_domains() {
5353
# through scikit-build-core (PEP 517). Build the wheel with the standard
5454
# frontend and keep build isolation off, so PyTorch builds against this
5555
# environment's numpy and toolchain (avoids an ABI mismatch) and reuses
56-
# sccache. With isolation off the frontend does not fetch the PEP 517 build
57-
# requirements, so install the ones not already in the image here. Keep in
58-
# sync with pytorch/pyproject.toml [build-system].requires.
59-
# cmake stays below 4 because the wheel lands on PATH ahead of the image's
60-
# cmake 3.x, so every later build in the image would jump a major version.
61-
conda_run pip install build "scikit-build-core>=1.0" "setuptools>=77.0.0,<82" \
62-
"cmake>=3.27,<4" ninja "packaging>=24.2" "typing-extensions>=4.10.0" pyyaml six
63-
conda_run python -m build --wheel --no-isolation
56+
# sccache.
57+
#
58+
# With isolation off the frontend does not fetch the PEP 517 build
59+
# requirements, so install them here. They go into a throwaway venv rather
60+
# than into the image, because scikit-build-core registers a setuptools
61+
# build_ext plugin: left in the image it hijacks every later
62+
# "pip install --no-build-isolation" and turns on C++20 module scanning that
63+
# the image compiler cannot satisfy. The venv inherits the image's
64+
# site-packages, so PyTorch still builds against the same numpy.
65+
#
66+
# Keep the list in sync with pytorch/pyproject.toml [build-system].requires.
67+
local build_venv=/tmp/pytorch-build-venv
68+
rm -rf "${build_venv}"
69+
conda_run python -m venv --system-site-packages "${build_venv}"
70+
conda_run "${build_venv}/bin/pip" install build "scikit-build-core>=1.0" \
71+
"setuptools>=77.0.0,<82" "cmake>=3.27,<4" ninja "packaging>=24.2" \
72+
"typing-extensions>=4.10.0" pyyaml six
73+
conda_run "${build_venv}/bin/python" -m build --wheel --no-isolation
74+
rm -rf "${build_venv}"
6475
pip_install "$(echo dist/*.whl)"
6576

6677
# Grab the pinned audio and vision commits from PyTorch

0 commit comments

Comments
 (0)