Skip to content

Commit 549ba1c

Browse files
authored
Add targeted gfx1100 Voxtral CI (pytorch#22001)
Run the existing packed-W4/BF16 streaming canary on the scarce RDNA runner only for direct Voxtral and ROCm execution-path changes or manual dispatches. Share the CI script with gfx950 while validating the expected architecture and wave size on each runner.
1 parent 6a8eae9 commit 549ba1c

4 files changed

Lines changed: 93 additions & 20 deletions

File tree

‎.ci/scripts/test-rocm-voxtral.sh‎

Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,15 +9,22 @@ set -euo pipefail
99

1010
ROCM_VERSION="${ROCM_VERSION:-7.1}"
1111
ROCM_PATH="${ROCM_PATH:-/opt/rocm}"
12+
EXPECTED_ROCM_ARCH="${EXPECTED_ROCM_ARCH:-gfx950}"
13+
EXPECTED_WARP_SIZE="${EXPECTED_WARP_SIZE:-64}"
1214
PYTORCH_ROCM_INDEX="${PYTORCH_ROCM_INDEX:-https://download.pytorch.org/whl/test/rocm${ROCM_VERSION}}"
1315
TORCHAO_ROCM_WHEEL_BASE="${TORCHAO_ROCM_WHEEL_BASE:-https://download.pytorch.org/whl/nightly/rocm${ROCM_VERSION}}"
1416
VOXTRAL_CI_TMP_ROOT="${RUNNER_TEMP:-/tmp}"
15-
mkdir -p "${VOXTRAL_CI_TMP_ROOT}" 2>/dev/null || VOXTRAL_CI_TMP_ROOT=/tmp
17+
if ! mkdir -p "${VOXTRAL_CI_TMP_ROOT}" 2>/dev/null ||
18+
[[ ! -w "${VOXTRAL_CI_TMP_ROOT}" ]]; then
19+
VOXTRAL_CI_TMP_ROOT=/tmp
20+
fi
1621
VOXTRAL_CI_TMPDIR="$(mktemp -d "${VOXTRAL_CI_TMP_ROOT}/executorch-rocm-voxtral.XXXXXX")"
1722
trap 'rm -rf "${VOXTRAL_CI_TMPDIR}"' EXIT
1823
VOXTRAL_START_SECONDS="${SECONDS}"
1924

2025
export ROCM_PATH
26+
export EXPECTED_ROCM_ARCH
27+
export EXPECTED_WARP_SIZE
2128
export HIP_VISIBLE_DEVICES=0
2229
export CUDA_VISIBLE_DEVICES=0
2330
export HF_HOME="${VOXTRAL_CI_TMP_ROOT}/hf-cache"
@@ -55,6 +62,8 @@ python -m pip install torchcodec==0.11.0 \
5562
--extra-index-url https://download.pytorch.org/whl/test/cpu
5663

5764
python - <<'PY'
65+
import os
66+
5867
import torch
5968
import torchao
6069
@@ -65,8 +74,12 @@ assert "+rocm" in torchao.__version__, "TorchAO is not a ROCm build"
6574
6675
device = torch.cuda.get_device_properties(0)
6776
arch = device.gcnArchName.split(":", 1)[0]
68-
assert arch == "gfx950", f"Expected gfx950, got {device.gcnArchName}"
69-
assert device.warp_size == 64, f"Expected wave64, got {device.warp_size}"
77+
expected_arch = os.environ["EXPECTED_ROCM_ARCH"]
78+
expected_warp_size = int(os.environ["EXPECTED_WARP_SIZE"])
79+
assert arch == expected_arch, f"Expected {expected_arch}, got {device.gcnArchName}"
80+
assert device.warp_size == expected_warp_size, (
81+
f"Expected wave{expected_warp_size}, got {device.warp_size}"
82+
)
7083
print(
7184
torch.__version__,
7285
torchao.__version__,

‎.github/workflows/rocm.yml‎

Lines changed: 67 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Test ExecuTorch AOTI ROCm support.
2-
# Use gfx950 for functional coverage and reserve scarce gfx1100 RDNA runners
3-
# for a future focused performance job.
2+
# Use gfx950 for routine functional coverage and reserve scarce gfx1100 RDNA
3+
# runners for changes that directly affect the Voxtral ROCm path.
44

55
name: Test ROCm AOTI
66

@@ -20,6 +20,7 @@ on:
2020
- .ci/scripts/test-rocm-voxtral.sh
2121
- CMakeLists.txt
2222
- CMakePresets.json
23+
- Makefile
2324
- install_requirements.py
2425
- torch_pin.py
2526
- backends/aoti/**
@@ -102,19 +103,25 @@ jobs:
102103
runs-on: ubuntu-latest
103104
outputs:
104105
run-voxtral: ${{ steps.decide.outputs.run-voxtral }}
106+
run-gfx1100: ${{ steps.decide.outputs.run-gfx1100 }}
105107
steps:
106108
- name: Select Voxtral runs
107109
id: decide
108110
env:
109111
CHANGED_FILES: ${{ needs.changed-files.outputs.changed-files }}
112+
EVENT_NAME: ${{ github.event_name }}
110113
FULL_RUN: ${{ needs.run-decision.outputs.is-full-run }}
111114
run: |
112115
set -efu
113116
RUN_VOXTRAL=false
117+
RUN_GFX1100=false
114118
115-
if [[ "$FULL_RUN" == "true" ]]; then
119+
if [[ "$CHANGED_FILES" == "*" || "$FULL_RUN" == "true" ]]; then
116120
RUN_VOXTRAL=true
117121
fi
122+
if [[ "$EVENT_NAME" == "workflow_dispatch" ]]; then
123+
RUN_GFX1100=true
124+
fi
118125
119126
for path in $CHANGED_FILES; do
120127
case "$path" in
@@ -124,19 +131,29 @@ jobs:
124131
examples/models/voxtral_realtime/* | \
125132
backends/cuda/aoti_packed_int4_tensor.py | \
126133
backends/cuda/triton/kernels/int4_matmul.py | \
134+
backends/cuda/triton/replacement_pass.py | \
127135
.ci/scripts/test-rocm-voxtral.sh | \
136+
.github/workflows/rocm.yml)
137+
RUN_GFX1100=true
138+
RUN_VOXTRAL=true
139+
;;
140+
Makefile | \
141+
CMakeLists.txt | \
142+
CMakePresets.json | \
143+
backends/aoti/* | \
144+
backends/cuda/* | \
145+
extension/cuda/* | \
128146
.github/workflows/_ci-run-decision.yml | \
129147
.github/workflows/_get-changed-files.yml | \
130148
install_requirements.py | \
131-
torch_pin.py | \
132-
.github/workflows/rocm.yml)
149+
torch_pin.py)
133150
RUN_VOXTRAL=true
134-
break
135151
;;
136152
esac
137153
done
138154
139155
echo "run-voxtral=$RUN_VOXTRAL" >> "$GITHUB_OUTPUT"
156+
echo "run-gfx1100=$RUN_GFX1100" >> "$GITHUB_OUTPUT"
140157
141158
unittest-rocm-gfx950:
142159
name: unittest-rocm-gfx950-rocm${{ matrix.rocm-version }}
@@ -199,4 +216,47 @@ jobs:
199216
eval "$(conda shell.bash hook)"
200217
conda create -n executorch-rocm-ci python=3.10 -y
201218
conda activate executorch-rocm-ci
202-
ROCM_VERSION=${{ matrix.rocm-version }} bash .ci/scripts/test-rocm-voxtral.sh
219+
EXPECTED_ROCM_ARCH=gfx950 EXPECTED_WARP_SIZE=64 \
220+
ROCM_VERSION=${{ matrix.rocm-version }} \
221+
bash .ci/scripts/test-rocm-voxtral.sh
222+
223+
# This scarce RDNA runner is limited to manual runs and direct changes to the
224+
# Voxtral ROCm execution path; it does not participate in broad sampling.
225+
test-voxtral-realtime-rocm-gfx1100:
226+
name: test-voxtral-realtime-rocm-gfx1100-rocm${{ matrix.rocm-version }}
227+
needs: [voxtral-run-decision]
228+
if: |
229+
needs.voxtral-run-decision.outputs.run-gfx1100 == 'true' &&
230+
(github.event_name != 'pull_request' ||
231+
github.event.pull_request.head.repo.full_name == github.repository)
232+
concurrency:
233+
group: rocm-gfx1100-${{ github.event.pull_request.number || github.ref_name }}
234+
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
235+
uses: pytorch/test-infra/.github/workflows/linux_job_v2.yml@main
236+
permissions:
237+
id-token: write
238+
contents: read
239+
secrets: inherit
240+
strategy:
241+
fail-fast: false
242+
matrix:
243+
rocm-version: ["7.1"]
244+
with:
245+
timeout: 180
246+
no-sudo: true
247+
secrets-env: EXECUTORCH_HF_TOKEN
248+
runner: linux.rocm.gpu.gfx1100
249+
gpu-arch-type: rocm
250+
gpu-arch-version: ${{ matrix.rocm-version }}
251+
use-custom-docker-registry: false
252+
docker-image: pytorch/manylinux2_28-builder:rocm${{ matrix.rocm-version }}
253+
submodules: recursive
254+
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
255+
script: |
256+
set -eux
257+
eval "$(conda shell.bash hook)"
258+
conda create -n executorch-rocm-ci python=3.10 -y
259+
conda activate executorch-rocm-ci
260+
EXPECTED_ROCM_ARCH=gfx1100 EXPECTED_WARP_SIZE=32 \
261+
ROCM_VERSION=${{ matrix.rocm-version }} \
262+
bash .ci/scripts/test-rocm-voxtral.sh

‎backends/cuda/rocm.md‎

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,8 @@ GPUs as device type `cuda`, so names such as `CudaBackend` and
66

77
`EXECUTORCH_BUILD_ROCM` is off by default, is never auto-enabled, and is
88
mutually exclusive with `EXECUTORCH_BUILD_CUDA`. It requires
9-
`EXECUTORCH_BUILD_EXTENSION_TENSOR`. Execution has been validated only on
10-
MI300X (`gfx942`); CI is configured to exercise `gfx950`.
9+
`EXECUTORCH_BUILD_EXTENSION_TENSOR`. Execution has been validated manually on
10+
MI300X (`gfx942`); CI canaries exercise gfx950 and gfx1100.
1111

1212
## Requirements and limitations
1313

@@ -19,10 +19,9 @@ MI300X (`gfx942`); CI is configured to exercise `gfx950`.
1919
as `executor_runner`.
2020
- Installed CMake consumers must be able to find the HIP package.
2121
- Voxtral Realtime has an explicit ROCm workflow. Other model runners do not
22-
claim ROCm support. The gfx950 CI job covers the backend build, native
23-
runtime, AOTI export and execution, and Triton W4 tests. Sampled CI is
24-
configured to run the streaming packed-W4/BF16 Voxtral path; offline and
25-
performance validation remain manual.
22+
claim ROCm support. gfx950 provides routine coverage, while the scarce
23+
gfx1100 runner is limited to direct Voxtral/ROCm changes and manual runs.
24+
Both run streaming packed-W4/BF16; offline validation remains manual.
2625

2726
## Build
2827

‎examples/models/voxtral_realtime/README.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -160,10 +160,11 @@ python export_voxtral_rt.py \
160160

161161
#### ROCm export examples
162162

163-
ROCm support is experimental and currently validated only on MI300X (`gfx942`).
164-
It is never enabled automatically. Use a ROCm PyTorch build with its matching
165-
Triton AMD backend; do not run `install_executorch.sh`, because its dependency
166-
setup can replace ROCm PyTorch with a CPU build.
163+
ROCm support is experimental. Manual validation currently covers MI300X
164+
(`gfx942`); CI canaries exercise `gfx950` and `gfx1100`. ROCm is never enabled
165+
automatically. Use a ROCm PyTorch build with its matching Triton AMD backend;
166+
do not run `install_executorch.sh`, because its dependency setup can replace
167+
ROCm PyTorch with a CPU build.
167168

168169
The validated ROCm configurations use BF16 and optionally packed `4w` linear
169170
weights with an `8w` embedding. The exporter rejects other ROCm dtype and

0 commit comments

Comments
 (0)