ci: bump torch ceiling to <2.14.0 and fix torch-tensorrt/DeepSpeed fallout - #21891
Open
bhimrazy wants to merge 7 commits into
Open
ci: bump torch ceiling to <2.14.0 and fix torch-tensorrt/DeepSpeed fallout#21891bhimrazy wants to merge 7 commits into
bhimrazy wants to merge 7 commits into
Conversation
bhimrazy
requested review from
ethanwharris,
justusschock and
tchaton
as code owners
August 9, 2026 09:38
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #21891 +/- ##
=======================================
Coverage 87% 87%
=======================================
Files 270 270
Lines 23998 23998
=======================================
Hits 20838 20838
Misses 3160 3160 |
bhimrazy
force-pushed
the
ci/bump-torch-tensorrt-ceiling
branch
from
August 9, 2026 13:50
a8fab23 to
f6745fe
Compare
bhimrazy
changed the title
bhimrazy
force-pushed
the
ci/bump-torch-tensorrt-ceiling
branch
from
August 9, 2026 13:52
f6745fe to
de301ec
Compare
Follow-up to Lightning-AI#21851, which capped torch at <2.13.0 because torch-tensorrt had no stable build past it yet. torch-tensorrt 2.13.0 (released 2026-07-28) now requires torch>=2.13.0,<2.14.0, so the cap can move up.
CI on torch-tensorrt 2.13 failed: ValueError: Requested using the TS frontend but the TS frontend is not available in this build of Torch-TensorRT The TorchScript frontend has been deprecated since torch-tensorrt 2.6 and is no longer bundled in some prebuilt wheels (e.g. 2.13+), independent of the torch version. The existing torch==2.9 skip only covered a different failure mode (a crash), so add a runtime check against torch_tensorrt.ENABLED_FEATURES.torchscript_frontend and skip ir="ts" cases whenever the installed build doesn't have it.
Corrects the previous commit's rationale. The TorchScript frontend is not removed in torch-tensorrt 2.13 -- the PyPI wheel still ships libtorchtrt.so and the release notes even include TS bug fixes. What changed is packaging: the 2.13.0+cu130 wheel on download.pytorch.org is pure Python (608 KB, zero .so files), while 2.12.1+cu130 bundled libtorchtrt.so. GPU CI installs from that index via --find-links, so it gets a build without the frontend. Feature detection via ENABLED_FEATURES.torchscript_frontend stays (it is the portable check -- has_torchscript_frontend() only exists in newer releases), but the two skip conditions are now separate marks so the report names the one that actually applies instead of a misleading combined message.
…peed The call comes from DeepSpeed's ZeRO-3 checkpointing; the API appears nowhere in Lightning's source or tests, so there is nothing to migrate on our side.
The 2.13.0+cu130 wheel on download.pytorch.org is pure Python (608 KB, zero .so files); 2.12.1+cu130 bundled libtorchtrt.so. Since a local version sorts above the plain release, the find-links entry always won and the TorchScript frontend went missing. PyPI's 2.13.0 ships the libraries and is likewise a CUDA 13 build.
bhimrazy
force-pushed
the
ci/bump-torch-tensorrt-ceiling
branch
from
August 9, 2026 13:52
de301ec to
ae297fe
Compare
torch 2.13 renamed the `*_tensor` collectives to `*_single`, and DeepSpeed's backend calls both `all_gather_into_tensor` and `reduce_scatter_tensor` from deepspeed/comm/torch.py, so filtering one message at a time is whack-a-mole. Filter on the module instead, matching the existing ddp_comm_hooks entry. Bumping DeepSpeed would not help: 0.19.4 still binds torch.distributed.reduce_scatter_tensor via hasattr and has no migration to the new names.
The earlier module-based filter (`deepspeed.comm.torch`) never matched: `typing_extensions.deprecated()` uses stacklevel=2, which attributes the warning to whoever called the deprecated function, and every collective is wrapped by torch's own `c10d_logger`, so that's always the immediate caller - not DeepSpeed. A module filter scoped to `torch.distributed.c10d_logger` would work but also swallow any *other* collective deprecation raised through that same wrapper, which is broader than intended. Match the two known messages instead, same as the other entries in this list. Verified locally against torch 2.13 (CPU-only) with a fake DeepSpeed-shaped caller.
bhimrazy
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Raises the
torchceiling to<2.14.0— a follow-up to #21851, which capped it at<2.13.0only becausetorch-tensorrthad no build past it. That build shipped on 2026-07-28.Moving past 2.13 surfaced two unrelated CI failures, both fixed here.
TorchScript frontend missing
torch-tensorrt's TS frontend lives in the compiledlibtorchtrt.so.2.13.0+cu130wheel from the PyTorch index ships without it (608 KB, zero.sofiles, vs. 4 MB for2.12.1+cu130).+cu130always outranks a plain PyPI release, sofind-linkskept winning that resolution — even though PyPI's2.13.0does bundle the libraries.find-linksfortorch-tensorrtso it resolves from PyPI instead. Tests also now feature-detect viaENABLED_FEATURES.torchscript_frontendand skip gracefully on any build that lacks it.Torch 2.13 collective deprecations
all_gather_into_tensorandreduce_scatter_tensorin favor of their*_singlecounterparts;error::FutureWarningturned that into a test failure.PR review
Anyone in the community is free to review the PR once the tests have passed.