Skip to content

Commit e61d3d3

Browse files
authored
Publish the CUDA trains PyTorch still offers, skip the ones it drops (pytorch#22652)
## Problem The nightly CUDA wheel build stopped publishing anything after PyTorch removed its CUDA 12.6 nightlies. ExecuTorch still listed `cu126` as a version to publish, and the release check treated a listed-but-absent train as a hard error, so it failed the whole job at the matrix-filter step. That blocked `cu130` and `cu132` from publishing too, even though their builds were fine. The result: no CUDA nightly wheel at all for several days, while the CPU wheel kept publishing normally. ## Fix Make the check degrade per train instead of all-or-nothing. A version listed in the policy is published only when the shared matrix generator still offers it. When PyTorch stops shipping a train, its rows simply do not appear, the release skips it with a note, and the other trains publish as usual. If PyTorch ships that train again later, it returns on its own with no code change here. The release still fails loudly in the two cases that are real problems: - an empty matrix, which would otherwise read as a green check for a build that never happened - a train that is offered but comes through missing some Python versions, which would ship that train incomplete Example, with PyTorch no longer offering cu126: ``` before: filter exits 1, publishes nothing after: filter exits 0, publishes cu130 and cu132, skips cu126 with a note ``` ## Test plan Extended the filter unit tests. A fully absent train is now asserted to be skipped while the others publish, plus a case for the exact cu126 drop. The incomplete-train and empty-matrix cases are asserted to still exit nonzero. Full suite: 21 passed, 2 subtests. Also ran the filter against today's real matrix (cu130 and cu132 only) and confirmed it exits 0 and publishes both.
1 parent ad2de69 commit e61d3d3

2 files changed

Lines changed: 61 additions & 28 deletions

File tree

‎.ci/scripts/tests/test_filter_cuda_matrix.py‎

Lines changed: 38 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -191,33 +191,58 @@ def test_unparseable_matrix_exits_nonzero(self):
191191
FILTER.main(argv)
192192
self.assertNotEqual(raised.exception.code, 0)
193193

194-
def test_absent_train_exits_nonzero(self):
195-
# A supported train the generator offers nothing for would publish no wheel at all.
194+
def test_absent_train_is_skipped_not_fatal(self):
195+
# A supported train the generator offers nothing for is one PyTorch stopped shipping. The
196+
# release skips it and publishes the rest, so one dropped train cannot take the others down.
196197
#
197-
# Patching the supported list rather than deleting rows, because deleting every row for one
198-
# train also creates missing combinations, so both gates fire and the test cannot tell which
199-
# one it exercised. Adding an extra supported train makes it absent while every offered
200-
# combination stays complete.
201-
# These two gates cannot be separated by input: any matrix leaving a train absent also
202-
# leaves every combination for that train missing, so the later gate always catches what the
203-
# earlier one would. Measured. So each gate gets its own case, and the case asserts on the
204-
# message rather than only on a nonzero exit, which is the only way to tell them apart.
198+
# Offering every train but the last leaves that train absent while every offered combination
199+
# stays complete, which is exactly the shape of an upstream drop.
205200
offered = FILTER.SUPPORTED_CUDA_VERSIONS[:-1]
201+
dropped = FILTER.SUPPORTED_CUDA_VERSIONS[-1]
206202
matrix = {
207203
"include": [
208204
{"python_version": python, "desired_cuda": cuda}
209205
for python in FILTER.SUPPORTED_PYTHON_VERSIONS
210206
for cuda in offered
211207
]
212208
}
213-
message = self._exit_message(matrix)
214-
self.assertIn("publish no wheel for that CUDA version", message)
209+
emitted = _emitted(_run(matrix))
210+
published = sorted({row["desired_cuda"] for row in emitted["include"]})
211+
self.assertEqual(published, sorted(offered))
212+
self.assertNotIn(dropped, published)
213+
214+
def test_dropped_train_still_publishes_the_others(self):
215+
# The exact upstream drop this resilience is for: PyTorch stops shipping cu126, the generator
216+
# offers only cu130 and cu132, and the release must still publish those two rather than fail
217+
# because cu126 is gone. Skips the case cleanly if the policy no longer lists cu126.
218+
if "cu126" not in FILTER.SUPPORTED_CUDA_VERSIONS:
219+
self.skipTest("cu126 is not a published train")
220+
survivors = [c for c in FILTER.SUPPORTED_CUDA_VERSIONS if c != "cu126"]
221+
matrix = {
222+
"include": [
223+
{"python_version": python, "desired_cuda": cuda}
224+
for python in FILTER.SUPPORTED_PYTHON_VERSIONS
225+
for cuda in survivors
226+
]
227+
}
228+
emitted = _emitted(_run(matrix))
229+
published = sorted({row["desired_cuda"] for row in emitted["include"]})
230+
self.assertEqual(published, sorted(survivors))
231+
self.assertNotIn("cu126", published)
232+
# Every survivor keeps all its pythons, so what publishes is complete, just narrower.
233+
self.assertEqual(
234+
len(emitted["include"]),
235+
len(survivors) * len(FILTER.SUPPORTED_PYTHON_VERSIONS),
236+
)
215237

216238
def test_missing_combination_exits_nonzero(self):
239+
# A train that IS offered but missing one python is a real break, not an upstream drop: the
240+
# release would ship that train incomplete. Deleting one row from a full matrix leaves its
241+
# train present, so this exercises the incomplete-train gate rather than the skip above.
217242
matrix = _full_matrix()
218243
del matrix["include"][0]
219244
message = self._exit_message(matrix)
220-
self.assertIn("combination(s) produced no row", message)
245+
self.assertIn("incomplete train", message)
221246

222247
def test_jetpack_not_published_exits_nonzero(self):
223248
# Refused explicitly rather than allowed to fall through to an empty result, so the reason a

‎.github/scripts/filter_cuda_matrix.py‎

Lines changed: 23 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@
3939
# not on that list is rejected whether or not it appears here.
4040
DISABLED_PYTHON_VERSIONS: List[str] = ["3.13t", "3.14t", "3.15", "3.15t"]
4141

42-
# CUDA versions to publish.
42+
# CUDA versions to publish, when the generator offers them.
4343
#
4444
# Chosen so that every consumer row can find a matching wheel rather than by what is
4545
# convenient to verify. A delegate built against one of these has to be able to depend on an
@@ -50,6 +50,12 @@
5050
# cu130 the generator's stable choice, and the default for accelerator consumers
5151
# cu132 the newest, which consumers building against a current TensorRT need
5252
#
53+
# A version listed here is published only when the shared generator still offers it. When
54+
# PyTorch stops shipping a CUDA train, its rows simply do not appear and the release skips it,
55+
# rather than failing the whole build. So a train PyTorch drops (as it did with cu126) costs
56+
# only that train, and a train PyTorch restores returns here with no edit. The release still
57+
# fails if a train that IS offered comes through incomplete, which is a real build break.
58+
#
5359
# cu132 is included because omitting it would leave a published consumer row with no
5460
# ExecuTorch wheel to pair with. It is executable on a device one minor behind, since CUDA
5561
# minor versions are compatible, so a cu132 wheel has been run end to end on a CUDA 13.0
@@ -198,31 +204,33 @@ def main(argv: List[str]) -> None:
198204
# blind to a python that disappeared from every supported train. The generator lives in another
199205
# repository and its axes move independently of what this policy promises to publish.
200206
built = {(item["python_version"], item["desired_cuda"]) for item in items}
201-
# A train that produced no row at all is missing for every python, so reporting it per python
202-
# would read as a python problem. Named on its own instead, and first, because the per-pair
203-
# report below would otherwise bury it.
204-
absent_trains = sorted(
205-
set(SUPPORTED_CUDA_VERSIONS) - {cuda for _, cuda in built}
206-
)
207+
built_trains = {cuda for _, cuda in built}
208+
# A train the generator offered nothing for is one PyTorch stopped shipping, not a build
209+
# break here. Skip it and publish the rest, so one dropped train cannot take the others
210+
# down with it. When PyTorch dropped CUDA 12.6, failing here also blocked cu130 and cu132
211+
# from publishing, which is the opposite of what a consumer needs. The train returns on its
212+
# own if PyTorch ships it again, with no edit here.
213+
absent_trains = sorted(set(SUPPORTED_CUDA_VERSIONS) - built_trains)
207214
if absent_trains:
208215
print(
209-
f"this policy publishes {SUPPORTED_CUDA_VERSIONS}, but the generator offered no row "
210-
f"this filter could keep for {absent_trains}, so a release would publish no wheel for "
211-
"that CUDA version at all",
216+
f"the generator offered no row for {absent_trains}, so they are skipped this run; "
217+
f"publishing {sorted(built_trains)}",
212218
file=sys.stderr,
213219
)
214-
sys.exit(1)
220+
# A train that IS offered but missing some python versions is a real break, not an upstream
221+
# drop: the release would ship an incomplete train, fewer wheels than promised for a version
222+
# that is otherwise present. Checked only against the trains actually offered, so a fully
223+
# absent train handled above does not also trip this and read as a python problem.
215224
missing = sorted(
216225
f"{python}/{cuda}"
217226
for python in SUPPORTED_PYTHON_VERSIONS
218-
for cuda in SUPPORTED_CUDA_VERSIONS
227+
for cuda in built_trains
219228
if (python, cuda) not in built
220229
)
221230
if missing:
222231
print(
223-
f"this policy publishes {SUPPORTED_CUDA_VERSIONS} for each of "
224-
f"{SUPPORTED_PYTHON_VERSIONS}, but {len(missing)} combination(s) produced no row, so a "
225-
f"release would publish no wheel for them: {missing}",
232+
f"a published train is missing some of {SUPPORTED_PYTHON_VERSIONS}, so a release "
233+
f"would ship an incomplete train: {missing}",
226234
file=sys.stderr,
227235
)
228236
sys.exit(1)

0 commit comments

Comments
 (0)