Ship the CUDA runtime in the Windows CUDA wheel (pytorch#23123)

Builds on pytorch#23121 (merged), which ships linkable libraries in the Windows
wheel. Stacked on
pytorch#23405, which makes the CUDA backend fall back to cudaMalloc on devices
without memory
pools: the Windows GPU CI runner's A10G is one, so this PR's end-to-end
runs need it.

A Windows user can run a CUDA program but cannot lower one: lowering
compiles the model
with a toolchain only the Linux side has, and the CUDA Windows CI job
already exports on
Linux and runs on Windows for exactly this reason. What was missing is a
wheel that gives
that Windows user the runtime. This adds a Windows CUDA wheel carrying
the CUDA delegate
as C++ libraries, the same components the Linux CUDA wheel ships
(pytorch#21645, pytorch#21668), and
nothing CUDA in the Python module, which on Windows has no CUDA program
to lower. The
export (AOT) side is unchanged and identical to the CPU wheel's; only
the runtime links CUDA.

lib/executorch_backend_cuda.dll + .lib the delegate
executorch::backend_cuda
lib/executorch_extension_cuda.dll + .lib allocator + streams
executorch::extension_cuda
backends/cuda/aoti_cuda_shims.dll the AOTI shim layer (runtime DLL of
the delegate)
data/lib/aoti_cuda_shims.lib already shipped as the lowering stub; also
the
shim layer's import library here
extension/pybindings/_C.pyd unchanged: no CUDA dependency, no
CudaBackend

A C++ application links it exactly as on Linux:

find_package(executorch REQUIRED COMPONENTS kernels_optimized
backend_cuda)
target_link_libraries(app PRIVATE executorch::runtime
executorch::kernels_optimized
                                      executorch::backend_cuda)
# plus the $<TARGET_RUNTIME_DLLS:app> copy from the CPU PR, which now
also brings the
    # CUDA extension and the shim layer

The wheel's DLLs import no CUDA runtime DLL: CMake links cudart.lib into
them on Windows,
and that reaches the driver (nvcuda.dll). The compiled model is
different: the library
AOTInductor builds into model.pte imports cudart64_13.dll, which comes
from the CUDA
Toolkit's bin directory on PATH, as the guide says. PyPI's
nvidia-cuda-runtime has win_amd64
builds that carry that DLL, but a C++ program does not search
site-packages, so declaring it
would not make it loadable; the wheel declares no NVIDIA package for
Windows.

## How

- Generator. The CUDA toolkit's Visual Studio integration fails compiler
identification
under the ClangCL toolset on some toolkit and Visual Studio pairs
(MSB4023 in its targets
file), and the MSVC toolset cannot compile the Python extension's
sources. A Windows CUDA
build therefore uses Ninja Multi-Config with clang-cl, the compiler the
CPU wheel uses,
and nvcc with cl.exe as its host compiler, in a Visual Studio developer
environment that
has ninja (ninja, clang-cl and cl all on PATH, which is how the wheel
build runs).
Elsewhere, such as a plain shell, the build keeps the Visual Studio
generator it always
used. CMAKE_CUDA_COMPILER is passed only when CUDACXX is unset, so
options a user put in
CUDACXX (such as -allow-unsupported-compiler) still apply. The toolkit
is the one
install_utils already reports, so what compiles is the train packaging
declares.
- CUDA 13.2 and newer. Their CCCL stops with an #error under cl.exe's
traditional
preprocessor, so the CUDA sources pass -Xcompiler=/Zc:preprocessor on
Windows (13.0
  accepts it too).
- flatc / flatcc byproducts and imported locations share one host
executable suffix chosen
by CMAKE_HOST_WIN32. CMAKE_EXECUTABLE_SUFFIX is the target's: a Zephyr
toolchain sets
.elf, and naming the byproduct after it left Ninja with no rule for the
host tool.
- The shims resolve the runtime from executorch.dll rather than a static
core. (The PAL's
strong symbols are in pytorch#23121; C++20 for the delegate and shims comes
from pytorch#23135.)
- The shims keep WINDOWS_EXPORT_ALL_SYMBOLS OFF in a static build, as
upstream sets it,
and turn it ON in the shared layout: the delegate DLL imports the stream
guard functions
(get/set/peek/clearCurrentCUDAStream) from the shim DLL, and those carry
no
AOTI_SHIM_EXPORT, so the annotations alone would not export them. The
allocator is not
in either DLL: main moved it into extension_cuda (pytorch#23447), a shared
library the
  delegate and the shims both import it from, so the process has one.
- pytorch#23121 keeps a Windows CUDA build static; this drops that guard, so
the pybind preset
now builds the shared layout with CUDA too, and adds the CUDA import
libraries to the
  Windows packaging list.
- The root CMake file no longer links the CUDA delegate into the Python
extension on
  Windows; extension/cuda finds the CUDA toolkit itself, as on main.
- The package config adds the CUDA extension (allocator and stream
helpers) and the shim
  layer to the delegate's runtime
DLL set, and reports EXECUTORCH_RUNTIME_DLLS_EXTRA for the pre-3.28
route, since the
  shim layer ships outside lib/.

## CI

- build-wheels-cuda-windows.yml mirrors the Linux CUDA workflow on the
Windows wheel
builder and reuses filter_cuda_matrix.py. pre_build_script.sh drops the
blanket Windows
CUDA=OFF (a CPU row is still forced off by the generic row
classification), resolves
TORCH_CUDA_ARCH_LIST from cuda_arch_list.sh, and writes it to
BUILD_ENV_FILE as well as
GITHUB_ENV: the reusable workflow sources that file after the hook, and
its own list had
replaced the row's (sm_75 in, sm_89 out). It also installs ninja for the
generator.
- End-to-end, per published CUDA train, the route a Windows user has,
modeled on
cuda-windows.yml. The trains are read from filter_cuda_matrix.py's
SUPPORTED_CUDA_VERSIONS
(today cu132 and cu134; main stopped publishing cu130 in pytorch#23380), the
same list the release
  rows are built from, so adding or
dropping a train there changes what is tested. The end-to-end jobs take
their trains from
the build job's own filtered matrix, so a pull request without the
ciflow/binaries/all
label tests the one train it builds, and a train the generator dropped
is not tested
against a wheel that does not exist. (install_utils.py's list also has
12.6 for
source builds, but 12.6 is no longer a published wheel train.) A Linux
GPU job exports a
Windows-target program with that train's torch (`test_cuda_windows.py
--export`); a small
job bundles it with the wheel the build job published for that train
(windows_job.yml
downloads one artifact and gives its script no token for a second); and
a Windows GPU job
installs that published wheel and runs test_cuda_windows.py with the
program
(test_cuda_windows_e2e.ps1), so CI runs a CUDA program through the
artifact users install
rather than a wheel rebuilt on the GPU runner. The export and end-to-end
jobs check out the
commit the build job builds (on a pull request, its merge commit), so
the wheel, the
program and the scripts that test them match. Each train's CUDA (on
Linux for the export,
and on Windows for the smoke test's cuobjdump and the model's CUDA
runtime) is assembled
  from NVIDIA's checksummed redistributable archives by
install_cuda_redist.py, which needs no installer or admin rights and
fails on a train it
has no release for. Each train's Windows job waits only for its own
bundle, so one train
failing does not skip the others, and like the export it does not run
for a
pull request from a fork. The Windows GPU runners come without an NVIDIA
driver (nvcuda.dll is missing and cudaGetDeviceCount returns
cudaErrorInsufficientDriver),
so the job first installs the same data-center driver, from the same
place, that
pytorch/pytorch's Windows CUDA smoke tests install (driver_update.bat,
580.88); cu132 and
cu134 run on it through CUDA's minor-version compatibility, as PyTorch's
own wheels do. The
  wheel build's own smoke test has no GPU program and says so.

## Tests

test_cuda_windows.py, the Windows counterpart of test_cuda_linux.py:

- the CUDA DLLs and import libraries ship;
- _C depends on no CUDA DLL and registers no CUDA backend;
- no shipped CUDA DLL imports cudart64_*.dll (only the driver), and no
Windows NVIDIA
  requirement is declared;
- every DLL with device code carries exactly the row's GPUs, both
directions, and the
newest also as PTX; the expected list comes from
executorch_cuda_arch_list in
cuda_arch_list.sh, called the way the pre-build hook calls it and keyed
by the wheel's
  +cuXYZ tag, not from the TORCH_CUDA_ARCH_LIST the build consumed;
- a C++ application linking executorch::backend_cuda gets the three DLLs
copied beside it
  and sees CudaBackend registered;
- a program lowered on Linux for Windows runs through the C++ SDK and
matches eager, after
checking it was exported with the same CUDA train as the wheel (every
CUDA 13 train loads
  cudart64_13.dll, so a mismatch would not fail on its own);
- then what a CPU Windows row checks: test_shared_libraries.py (with the
three CUDA
ownership rows), test_cpp_sdk.py and the MobileNetV3/XNNPACK model run,
including the Arm
Cortex-M Python module, which builds under Ninja too: CMSIS-NN's
test-only shared library
took the static library's name, so its import library collided with it
on Windows, and
  backends/cortex_m now gives the import library its own name.

## Test plan

On Windows 11 x64 with an RTX 5080 (sm_120, driver 595.95), VS 2022
BuildTools, CPython 3.12,
and WSL Ubuntu for the export, once per CUDA train. 13.2 and 13.4 are
the published trains
and the ones CI covers; 13.0 is no longer published (pytorch#23380) but still
builds from source, so
it was run too:

| train | Windows nvcc | WSL export torch | wheel | test_cuda_windows.py
| Linux-exported program |
|---|---|---|---|---|---|
| 13.0 | 13.0.88 (installer) | 2.14.1+cu130 | executorch-1.6.0+cu130 |
52 checks pass | runs on the GPU, maxdiff 2.4e-7 |
| 13.2 | 13.2.86 (redist) | 2.14.1+cu132 | executorch-1.6.0+cu132 | 52
checks pass | runs on the GPU, maxdiff 2.4e-7 |
| 13.4 | 13.4.59 (redist) | 2.14.0.dev20260810+cu134 |
executorch-1.6.0+cu134 | 52 checks pass | runs on the GPU, maxdiff
2.4e-7 |

- Each wheel built with TORCH_CUDA_ARCH_LIST from cuda_arch_list.sh:
device code for exactly
sm_80/86/89/90/100/120, plus sm_120 PTX. 13.4 runs on a driver that
reports 13.2, through
  CUDA's minor-version compatibility.
- Before the /Zc:preprocessor change, 13.2 and 13.4 failed to compile
the .cu shims with
  CCCL's "traditional preprocessor" #error; 13.0 was unaffected.
- Negative: the cu134 wheel pointed at the 13.0-exported program fails
the train check.
- The export used `test_cuda_windows.py --export` from this branch in
WSL, with mingw-w64 and
the Windows CUDA runtime of the same train (13.0 from
install_cuda_windows_cross_compile.sh,
  13.2/13.4 from the redist archives, the steps the guide now gives).
- CI, before the rebase onto main: the end-to-end jobs passed for both
published trains on
the Windows A10G runner, with a Linux-exported program matching eager
(maxdiff 1.3e-7 for
  13.2, 2.4e-7 for 13.4).
- After rebasing onto main with the allocator moved into extension_cuda
(pytorch#23447), the cu132
wheel was rebuilt and passes all 55 checks, including the new ownership
row: a single CUDA
allocator and stream helpers owned by executorch_extension_cuda.dll
across the 15 shipped
  libraries. The rows above were measured before that rebase.
- Linux: the CMake changes are behind WIN32 / MSVC, or pick the same
suffix there (empty on a
  Linux host); setup.py's new path runs only on Windows.

Author: Gasoonjia

.ci/scripts/wheel/install_cuda_redist.py

@@ -0,0 +1,129 @@
+#!/usr/bin/env python
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+"""Assemble part of a CUDA toolkit from NVIDIA's redistributable component archives.
+
+The Windows CUDA wheel end-to-end jobs cover several CUDA trains while the CI images carry
+one toolkit, and NVIDIA publishes no full Windows installer for every release. The redist
+archives exist for every release on both platforms, need no installer or admin rights, and
+are checksummed in a JSON index. Merging the components a job needs into one directory
+gives the layout nvcc, CMake's FindCUDAToolkit and torch's cross-compile step expect.
+
+    install_cuda_redist.py --train 13.4 --platform linux-x86_64 --dest /opt/cuda-13.4
+    install_cuda_redist.py --train 13.4 --platform windows-x86_64 --dest DIR --components cuda_cudart
+"""
+
+import argparse
+import hashlib
+import json
+import shutil
+import sys
+import tarfile
+import tempfile
+import urllib.request
+import zipfile
+from pathlib import Path
+
+_REDIST = "https://developer.download.nvidia.com/compute/cuda/redist"
+
+# The patch release each train resolves to: one entry per train in filter_cuda_matrix.py's
+# SUPPORTED_CUDA_VERSIONS, which the end-to-end jobs take their matrix from, and 13.0, which
+# is no longer published but still builds from source and is what local tests of it use. A
+# train missing here fails the job at argument parsing rather than testing a different
+# release.
+_RELEASES = {"13.0": "13.0.2", "13.2": "13.2.2", "13.4": "13.4.1"}
+
+# What a build of the CUDA backend needs. "a|b" names one component a release publishes
+# under either name (CCCL was renamed between releases).
+_BUILD_COMPONENTS = [
+    "cuda_nvcc",
+    "cuda_cudart",
+    "cuda_crt",
+    "cccl|cuda_cccl",
+    "libnvvm",
+    "cuda_cuobjdump",
+    "cuda_nvrtc",
+    "cuda_profiler_api",
+    "cuda_nvtx",
+    # rand.cu includes curand_kernel.h.
+    "libcurand",
+]
+
+
+def _fetch(url: str, destination: Path) -> None:
+    with urllib.request.urlopen(url, timeout=300) as response, open(
+        destination, "wb"
+    ) as handle:
+        shutil.copyfileobj(response, handle)
+
+
+def _extract(archive: Path, into: Path) -> Path:
+    if archive.suffix == ".zip":
+        with zipfile.ZipFile(archive) as bundle:
+            bundle.extractall(into)
+    else:
+        with tarfile.open(archive) as bundle:
+            if hasattr(tarfile, "fully_trusted_filter"):
+                bundle.extractall(into, filter="fully_trusted")
+            else:
+                bundle.extractall(into)
+    # Each archive holds one top-level directory named after itself.
+    (top,) = [entry for entry in into.iterdir() if entry.is_dir()]
+    return top
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
+    parser.add_argument("--train", required=True, choices=sorted(_RELEASES))
+    parser.add_argument(
+        "--platform", required=True, choices=["linux-x86_64", "windows-x86_64"]
+    )
+    parser.add_argument("--dest", required=True, type=Path)
+    parser.add_argument("--components", nargs="+", default=_BUILD_COMPONENTS)
+    arguments = parser.parse_args()
+
+    release = _RELEASES[arguments.train]
+    with urllib.request.urlopen(
+        f"{_REDIST}/redistrib_{release}.json", timeout=120
+    ) as response:
+        index = json.load(response)
+    arguments.dest.mkdir(parents=True, exist_ok=True)
+    with tempfile.TemporaryDirectory() as work:
+        work = Path(work)
+        for choice in arguments.components:
+            names = [
+                name
+                for name in choice.split("|")
+                if arguments.platform in index.get(name, {})
+            ]
+            if not names:
+                sys.exit(
+                    f"CUDA {release} publishes none of {choice} for {arguments.platform}"
+                )
+            entry = index[names[0]][arguments.platform]
+            archive = work / Path(entry["relative_path"]).name
+            _fetch(f"{_REDIST}/{entry['relative_path']}", archive)
+            digest = hashlib.sha256(archive.read_bytes()).hexdigest()
+            if digest != entry["sha256"]:
+                sys.exit(f"checksum mismatch for {archive.name}")
+            staging = work / "staging"
+            top = _extract(archive, staging)
+            shutil.copytree(top, arguments.dest, symlinks=True, dirs_exist_ok=True)
+            shutil.rmtree(staging)
+            archive.unlink()
+            print(
+                f"merged {names[0]} {index[names[0]]['version']} into {arguments.dest}"
+            )
+    # The Linux archives keep libraries in lib/, while nvcc's link step and CMake's
+    # FindCUDAToolkit look in lib64/, which is where an installed toolkit has them.
+    lib, lib64 = arguments.dest / "lib", arguments.dest / "lib64"
+    if arguments.platform == "linux-x86_64" and lib.is_dir() and not lib64.exists():
+        lib64.symlink_to("lib", target_is_directory=True)
+
+
+if __name__ == "__main__":
+    main()

.ci/scripts/wheel/pre_build_script.sh

@@ -116,6 +116,23 @@ else
     export CMAKE_ARGS="${CMAKE_ARGS:-} -DEXECUTORCH_BUILD_CUDA=ON"
     echo "CMAKE_ARGS=${CMAKE_ARGS}" >> "${GITHUB_ENV}"
     echo "row '${CU_VERSION:-${DESIRED_CUDA:-}}' is a CUDA row, requiring the CUDA build"
+    # The Linux rows take the row's GPU list from envvar_cuda_linux.sh. Windows has no env-var
+    # script slot, so the same list is resolved here; without it the build compiles device
+    # code only for whatever GPU the builder has. The reusable workflow writes its own list
+    # into BUILD_ENV_FILE, which the build step sources after this hook and which would
+    # otherwise win, so the row's list is appended there as well as to GITHUB_ENV.
+    # Also needed for the ninja the Windows CUDA build generates with.
+    if [[ $UNAME_S == *"MINGW"* || $UNAME_S == *"MSYS"* ]]; then
+        source "${GITHUB_WORKSPACE}/${REPOSITORY}/.ci/scripts/wheel/cuda_arch_list.sh"
+        TORCH_CUDA_ARCH_LIST="$(EXECUTORCH_BUILD_CUDA=1 executorch_cuda_arch_list)"
+        export TORCH_CUDA_ARCH_LIST
+        echo "TORCH_CUDA_ARCH_LIST=${TORCH_CUDA_ARCH_LIST}" >> "${GITHUB_ENV}"
+        if [[ -n "${BUILD_ENV_FILE:-}" ]]; then
+            echo "export TORCH_CUDA_ARCH_LIST='${TORCH_CUDA_ARCH_LIST}'" >> "${BUILD_ENV_FILE}"
+        fi
+        echo "building device code for: ${TORCH_CUDA_ARCH_LIST}"
+        pip install --quiet ninja
+    fi
 fi
 
 # On Windows, enable symlinks and re-checkout the current revision to create
@@ -125,14 +142,6 @@ if [[ $UNAME_S == *"MINGW"* || $UNAME_S == *"MSYS"* ]]; then
     git config core.symlinks true
     git checkout -f HEAD
 
-    # Windows wheels are CPU-only (build-wheels-windows.yml sets
-    # with-cuda: disabled), but the Windows CI image ships a CUDA toolkit on
-    # PATH, which makes setup.py auto-enable EXECUTORCH_BUILD_CUDA. That bakes a
-    # CUDA _C into the CPU wheel, which then fails its DLL load in the
-    # smoke test ("DLL load failed while importing _C"). Force a
-    # CPU-only build.
-    export CMAKE_ARGS="${CMAKE_ARGS:-} -DEXECUTORCH_BUILD_CUDA=OFF"
-    echo "CMAKE_ARGS=${CMAKE_ARGS}" >> "${GITHUB_ENV}"
 fi
 
 # Manually install build requirements because `python setup.py bdist_wheel` does
@@ -202,7 +211,8 @@ if [[ "${EXECUTORCH_BUILD_VULKAN:-0}" != "0" \
     echo "glslc installed: $(command -v glslc)"
   fi
 else
-  # This is the Vulkan equivalent of the Windows CUDA force-off above (#20527).
+  # Off unless asked for: a builder with the Vulkan SDK would otherwise turn it on
+  # for a wheel that does not ship it, as a CUDA toolkit once did for Windows (#20527).
   export CMAKE_ARGS="${CMAKE_ARGS:-} -DEXECUTORCH_BUILD_VULKAN=OFF"
   echo "CMAKE_ARGS=${CMAKE_ARGS}" >> "${GITHUB_ENV}"
 fi

.ci/scripts/wheel/test_cpp_sdk.py

@@ -1741,11 +1741,13 @@ def test_pre_3_28_route_builds_a_consumer_through_variables(work_dir: Path) -> N
         "target_link_libraries(consumer PRIVATE ${EXECUTORCH_LIBRARIES})\n"
         "set_target_properties(consumer PROPERTIES CXX_STANDARD ${EXECUTORCH_CXX_STANDARD})\n"
         # No imported targets on this route, so no TARGET_RUNTIME_DLLS either. A Windows
-        # consumer copies the DLLs from the directory the package reports.
+        # consumer copies the DLLs from the directory the package reports, plus any the
+        # package lists separately because they ship elsewhere.
         "if(WIN32)\n"
         '  file(GLOB _dlls "${EXECUTORCH_RUNTIME_LIBRARY_DIR}/*.dll")\n'
         "  add_custom_command(TARGET consumer POST_BUILD COMMAND ${CMAKE_COMMAND} -E "
-        "copy_if_different ${_dlls} $<TARGET_FILE_DIR:consumer>)\n"
+        "copy_if_different ${_dlls} ${EXECUTORCH_RUNTIME_DLLS_EXTRA} "
+        "$<TARGET_FILE_DIR:consumer>)\n"
         "endif()\n"
     )
     build_dir = work_dir / "pre-328-build"