Repository navigation
Build Python 3.15 Windows tokenizer dependencies before wheel smoke tests - #23940
Closed
JacobSzwejbka wants to merge 2 commits into
Closed
JacobSzwejbka wants to merge 2 commits into
JacobSzwejbka wants to merge 2 commits into
Conversation
No cp315 Windows pytorch-tokenizers wheel is published, so pip falls back to the old 1.0.1 source distribution in the plain smoke shell and cannot identify ClangCL. Build the pinned source first through the Visual Studio environment helper using the already installed build dependencies. Authored with OpenAI Codex.
Build the pinned tokenizer source from a temporary short path. MSBuild file tracking in CMake compiler probes fails on the nested source build path even inside the Visual Studio environment. Authored with OpenAI Codex.
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/23940
Note: Links to docs will display an error until the docs builds have been completed. ⏳ No Failures, 220 PendingAs of commit 55cdbdc with merge base cbcc5ef ( UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
JacobSzwejbka
added a commit
that referenced
this pull request
Oct 17, 2026
## Summary Combine the remaining Python 3.15 CUDA and Windows wheel fixes, including the Windows tokenizer changes previously proposed in #23940. CUDA 13.2 installs request torchaudio 2.11.0, which has no Python 3.15 wheel. Select the available 2.11.1 release for Python 3.15. Preserve the CUDA 13.4 package selection and source-pinned installs. Move the PyPI extra index setting from the CPU Linux environment script into the shared wheel pre-build hook. Export it for the hook and persist it through `GITHUB_ENV` so later CUDA Linux and Windows smoke installs can resolve dependencies such as `expecttest` while using PyTorch's index. Preserve existing extra indices. PyPI has no cp315 Windows pytorch-tokenizers wheel, so pip falls back to an older source distribution whose isolated build cannot find the compiler in the smoke shell. Install the pinned tokenizer source through the existing Visual Studio environment helper with build isolation disabled. Copy it to a short temporary directory to keep CMake compiler probes below the Windows path limit. Dereference submodule symlinks during the copy because MSYS cannot recreate them on every Windows runner, and remove the copy after installation. This provisions the tokenizer dependency for CI. PyPI still needs an upstream cp315 Windows tokenizer wheel for installation without building that dependency from source. ## Validation The 13 installer regression tests pass on Python 3.14 and 3.15. Lintrunner, bash syntax validation, and git diff checks pass. A source-copy check confirms that the two symlinks which failed on Windows CUDA become regular files with identical contents. Earlier checks verified torchaudio wheel availability and metadata, preservation of extra indices through the hook, and actual pip resolution of `expecttest` against the PyTorch indices with PyPI enabled. The earlier full Python 3.15 wheel builds and smoke tests passed for Linux x86_64 CUDA 13.2 and 13.4 (https://github.com/pytorch/executorch/actions/runs/38936913096), Linux aarch64 CUDA 13.2 and 13.4 (https://github.com/pytorch/executorch/actions/runs/38936920119), and Windows CPU (https://github.com/pytorch/executorch/actions/runs/38961222041). Both Windows CUDA configurations then failed copying submodule symlinks before compilation (https://github.com/pytorch/executorch/actions/runs/38964126110); this combined revision includes the copy fix. Fresh validation of the combined revision is running for all seven configurations. Windows CPU: https://github.com/pytorch/executorch/actions/runs/38968122117. Windows CUDA 13.2 and 13.4: https://github.com/pytorch/executorch/actions/runs/38968126439. Linux x86_64 CUDA 13.2 and 13.4: https://github.com/pytorch/executorch/actions/runs/38968132148. Linux aarch64 CUDA 13.2 and 13.4: https://github.com/pytorch/executorch/actions/runs/38968137580. These are pending results, not passes. The validation branch has identical source fixes and temporarily restricts the wheel matrix and its guard to Python 3.15. It omits additional Windows CUDA program export jobs while retaining the wheel build and smoke suite. Those validation-only edits are not included in this PR. Authored with OpenAI Codex.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #23932. Provision the pinned tokenizer source for Python 3.15 Windows wheel jobs before the reusable smoke installer runs. PyPI has no cp315 Windows pytorch-tokenizers wheel, so pip falls back to the old 1.0.1 source distribution and its isolated CMake build cannot identify the ClangCL compiler in the smoke shell.
Build the pinned source through the existing Visual Studio environment helper with build isolation disabled, using the build dependencies installed by the wheel hook. Copy the tokenizer source into a temporary short directory because MSBuild file tracking in CMake compiler probes exceeds the Windows path limit under the nested checkout path. Remove the copy after installation.
This fixes CI dependency provisioning. PyPI still needs an upstream cp315 Windows tokenizer wheel for installation without building that dependency from source.
Validation
The full Windows CPU Python 3.15 wheel build, dependency install, and smoke tests pass: https://github.com/pytorch/executorch/actions/runs/38961222041/job/117176579233. The smoke suite checks clean imports, shipped shared libraries, a standalone C++ SDK consumer, and model execution. The wheel artifact upload also passes.
Both Windows CUDA 3.15 configurations are being validated on the same source fixes: https://github.com/pytorch/executorch/actions/runs/38964126110. The validation branch restricts the matrix and its guard to Python 3.15 and omits additional Windows CUDA program export jobs; those validation-only changes are not part of this PR. Bash syntax validation, lintrunner, source comparison, and git diff checks pass.
Authored with OpenAI Codex.