Repository navigation
Commit 7256479
committed
[ET-VK][CI] Add end-to-end quantized LLM coverage
Summary:
## Rationale
Add the seven LLM recipes used by internal GPU CI to the existing OSS NVIDIA
workflow: Qwen3-0.6B, five Llama3.2-1B configurations, and Phi4 Mini. SwiftShader
cannot exercise their required quantized compute features.
Use CUDA's shared model export/build/run pattern. Select pinned public HF
checkpoints, prepare them with existing model utilities, invoke the existing
LLM exporter, check called Vulkan delegates and required operator families,
and validate greedy native output. Quantized Llama embeddings also retain the
770 MiB artifact limit.
## Review order
Read vulkan.yml and the shared export/run scripts first, then the HF/Llama
preparation and PTE checker. Continue with the Release Vulkan Make/CMake path
and installed-package Boost dependency, then the Vulkan CI regression suites
and their CPU CI wiring. The shell orchestration suite was restored to Python
and Vulkan CI tests are grouped under .ci/scripts/tests/vulkan.
| Area | Before | After |
|---|---|---|
| OSS Vulkan GPU CI | Model/operator coverage without an LLM answer gate | Seven pinned LLM export/inference cells |
| Shared scripts | CUDA/Metal/XNNPACK dispatch | Also supports the selected Vulkan recipes |
| Llama preparation | Local Unsloth LoRA conversion wrapper | Full HF conversion through TorchTune and optional parameter output |
| Native build | No llama-vulkan Make target | Release Vulkan installation plus the existing Llama runner build |
| Installed package | Boost target references could be unresolved | Discover Boost only when the exported Vulkan interface needs it |
Authored with OpenAI Codex.
Test Plan:
## Observed Results
Results below were recorded before updating the mainline base. The final shell
refactor preserved all seven export command traces, metadata, prompts, and
expected answers. The seven-device-case sweep preceded the final structural
refactors; Qwen was freshly exported and run after existing-exporter reuse.
| Correctness validation | Environment | Observed result |
|---|---|---|
| Vulkan CI Python suites, Llama converter tests, existing CUDA workflow tests | Isolated current-pin Python environment | 47 passed, 0 failed |
| CMake package producer/consumer suite | macOS, real configure/build/install | 3 passed, 0 failed |
| Restored orchestration suite after UFMT | macOS | 25 passed, included in the Python coverage above |
| Seven before/after export recipe traces and metadata | Real shared scripts with command stubs | All seven identical |
| Renamed PTE checker | Actual saved Qwen PTE | Passed; called VulkanBackend and required families found |
| Seven earlier shared native LLM cases | Qwen: S24/Adreno 750; Llama/Phi: Pixel 9 Pro XL/Mali-G715 | Seven passed expected answers and native exit checks |
| Fresh Qwen export plus shared native execution after exporter reuse | S24/Adreno 750 | Passed |
| make llama-vulkan | Isolated macOS build, Boost enabled; CoreML/ASR disabled locally | Passed |
| Lintrunner on edited files and new/moved CI tests | Local | Passed |
| Shell syntax, whitespace, complete-patch reverse check | Local | Passed |
| New NVIDIA workflow execution | NVIDIA A10G | Not run; no accessible NVIDIA validation host |
| Performance validation | Result |
|---|---|
| Operator performance tests | Not run; no operator implementation change |
| End-to-end latency/memory A/B against parent/public ancestor | Not run; this adds correctness coverage, with no performance claim |
## Reproduction Guide
### Checkout and development dependencies
Install Git and Conda (Miniforge installers are available at
https://github.com/conda-forge/miniforge/releases). On Linux install a C++
toolchain; on macOS install Xcode command-line tools with xcode-select --install.
For NVIDIA inference, install a compatible NVIDIA Vulkan driver. The existing
setup script selects and checks the hardware ICD.
```bash
git clone --recurse-submodules https://github.com/pytorch/executorch.git
cd executorch
git checkout ssjia/vulkan-llm-ci
git submodule update --init --recursive
conda create -y -n etvk-ci python=3.11
conda activate etvk-ci
conda install -y -c conda-forge cmake ninja glslc rust
CMAKE_ARGS="-DEXECUTORCH_BUILD_VULKAN=ON -DEXECUTORCH_BUILD_KERNELS_LLM_AOT=ON" \
PYTHON_EXECUTABLE=python ./install_executorch.sh
python -m pip install pytest pytest-xdist
```
On Linux, run the Vulkan dependency setup before installation. Select real-gpu
for NVIDIA runtime testing, or the default setup for software-driver host
preparation/export-only testing:
```bash
source .ci/scripts/setup-vulkan-linux-deps.sh real-gpu
```
### CPU regression tests
These create their own fixtures; no model downloads are needed. The Python
orchestration tests use command stubs, the conversion tests use real tensors,
and the CMake suite builds real small package producers/consumers.
```bash
python -m pytest -q .ci/scripts/tests/vulkan \
.ci/scripts/tests/test_cuda_workflow.py \
examples/models/llama/tests/test_convert_weights.py
bash .ci/scripts/tests/vulkan/test_vulkan_cmake_package.sh
```
### Generate all model artifacts
The shared script downloads pinned public checkpoints, copies their tokenizers,
creates exact prompt/answer files, and checks required Vulkan lowering and size.
This generation stage does not require NVIDIA inference.
```bash
ET_ARTIFACT_DIR="$PWD/ci-repro/models"
mkdir -p "$ET_ARTIFACT_DIR"
while read -r model quant; do
name=$(printf '%s' "$model" | tr / _)
bash .ci/scripts/export_model_artifact.sh vulkan "$model" "$quant" \
"$ET_ARTIFACT_DIR/$name/$quant"
done <<'CASES'
Qwen/Qwen3-0.6B quantized-8da4w-fp16-embedding
unsloth/Llama-3.2-1B-Instruct quantized-8da4w
unsloth/Llama-3.2-1B-Instruct quantized-8da4w-fp16
unsloth/Llama-3.2-1B-Instruct quantized-8da4w-embedding
unsloth/Llama-3.2-1B-Instruct quantized-8da4w-fp16-embedding
unsloth/Llama-3.2-1B-Instruct quantized-int4-weight-only
microsoft/Phi-4-mini-instruct quantized-8da4w-embedding
CASES
```
### Linux/NVIDIA shared end-to-end path
Use the same generated directories with RUN_EXPORT=0, or regenerate and run in
one step with RUN_EXPORT=1. The helper builds make llama-vulkan and checks native
exit status and the expected answer in stdout. Repeat for the other pairs above.
The GitHub matrix runs these pairs as separate jobs.
```bash
source .ci/scripts/setup-vulkan-linux-deps.sh real-gpu
RUN_EXPORT=1 bash .ci/scripts/test_model_e2e.sh vulkan \
Qwen/Qwen3-0.6B quantized-8da4w-fp16-embedding \
"$ET_ARTIFACT_DIR/Qwen_Qwen3-0.6B/quantized-8da4w-fp16-embedding"
```
### Android native correctness reproduction
Install Android Studio from https://developer.android.com/studio. In SDK
Manager install Android SDK Platform-Tools and NDK 29.0.13846066. Set
ET_ANDROID_SDK to the SDK Location shown by SDK Manager and ET_DEVICE to the
serial reported by adb devices. Enable USB debugging and authorize the host.
Use S24 for Qwen and Pixel 9 Pro XL for the larger Llama/Phi cases, matching the
recorded device sweep.
Build a static Android runtime/runner from this checkout:
```bash
ET_NDK="$ET_ANDROID_SDK/ndk/29.0.13846066"
ET_ANDROID_BUILD="$PWD/ci-repro/android-core"
ET_ANDROID_INSTALL="$PWD/ci-repro/android-install"
ET_ANDROID_RUNNER="$PWD/ci-repro/android-runner"
cmake -S . -B "$ET_ANDROID_BUILD" -G Ninja \
-DCMAKE_TOOLCHAIN_FILE="$ET_NDK/build/cmake/android.toolchain.cmake" \
-DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-26 \
-DANDROID_STL=c++_static -DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX="$ET_ANDROID_INSTALL" -DPYTHON_EXECUTABLE="$(command -v python)" \
-DEXECUTORCH_BUILD_VULKAN=ON -DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON -DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON -DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM_RUNNER=ON \
-DEXECUTORCH_BUILD_KERNELS_LLM=ON -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON -DEXECUTORCH_BUILD_XNNPACK=ON
cmake --build "$ET_ANDROID_BUILD" --target install --parallel 4
cmake -S examples/models/llama -B "$ET_ANDROID_RUNNER" -G Ninja \
-DCMAKE_TOOLCHAIN_FILE="$ET_NDK/build/cmake/android.toolchain.cmake" \
-DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-26 \
-DANDROID_STL=c++_static -DCMAKE_BUILD_TYPE=Release \
-DCMAKE_FIND_ROOT_PATH="$ET_ANDROID_INSTALL" -DPYTHON_EXECUTABLE="$(command -v python)"
cmake --build "$ET_ANDROID_RUNNER" --target llama_main --parallel 4
```
Stage and run one generated artifact; repeat with the other directories/device
as appropriate. Modern adb preserves the remote native exit code. Keep stdout
and stderr separate so diagnostic text cannot satisfy the answer check.
```bash
ET_MODEL_DIR="$ET_ARTIFACT_DIR/Qwen_Qwen3-0.6B/quantized-8da4w-fp16-embedding"
ET_REMOTE=/data/local/tmp/etvk_llm_ci
adb -s "$ET_DEVICE" shell mkdir -p "$ET_REMOTE"
adb -s "$ET_DEVICE" push "$ET_ANDROID_RUNNER/llama_main" "$ET_REMOTE/llama_main"
adb -s "$ET_DEVICE" shell chmod +x "$ET_REMOTE/llama_main"
for file in model.pte tokenizer.json prompt.txt; do
adb -s "$ET_DEVICE" push "$ET_MODEL_DIR/$file" "$ET_REMOTE/$file"
done
adb -s "$ET_DEVICE" shell "$ET_REMOTE/llama_main" \
--model_path "$ET_REMOTE/model.pte" --tokenizer_path "$ET_REMOTE/tokenizer.json" \
--prompt_file "$ET_REMOTE/prompt.txt" --temperature 0 --max_new_tokens 50 \
> "$ET_MODEL_DIR/runner.stdout" 2> "$ET_MODEL_DIR/runner.stderr"
grep -Fiw -- "$(cat "$ET_MODEL_DIR/expected_output.txt")" "$ET_MODEL_DIR/runner.stdout"
```
No performance tests were run. There are no latency regression thresholds to
reproduce in this change.
Reviewers:1 parent ad1c24d commit 7256479
15 files changed
Lines changed: 1301 additions & 28 deletions
File tree
- .ci/scripts
- tests/vulkan
- .github/workflows
- backends/vulkan/test
- examples/models/llama
- tests
- tools/cmake
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
8 | | - | |
| 8 | + | |
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
13 | 13 | | |
14 | | - | |
| 14 | + | |
15 | 15 | | |
16 | 16 | | |
17 | | - | |
| 17 | + | |
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
29 | 32 | | |
30 | 33 | | |
31 | 34 | | |
32 | 35 | | |
33 | 36 | | |
34 | | - | |
| 37 | + | |
35 | 38 | | |
36 | | - | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
37 | 43 | | |
38 | 44 | | |
39 | 45 | | |
| |||
85 | 91 | | |
86 | 92 | | |
87 | 93 | | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
88 | 97 | | |
89 | 98 | | |
90 | 99 | | |
| |||
118 | 127 | | |
119 | 128 | | |
120 | 129 | | |
121 | | - | |
| 130 | + | |
122 | 131 | | |
123 | 132 | | |
124 | 133 | | |
125 | | - | |
| 134 | + | |
126 | 135 | | |
127 | 136 | | |
128 | 137 | | |
129 | 138 | | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
130 | 142 | | |
131 | 143 | | |
132 | 144 | | |
| |||
168 | 180 | | |
169 | 181 | | |
170 | 182 | | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
171 | 218 | | |
172 | 219 | | |
173 | 220 | | |
| |||
232 | 279 | | |
233 | 280 | | |
234 | 281 | | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
235 | 286 | | |
236 | 287 | | |
237 | 288 | | |
| |||
250 | 301 | | |
251 | 302 | | |
252 | 303 | | |
| 304 | + | |
253 | 305 | | |
254 | 306 | | |
255 | 307 | | |
| |||
259 | 311 | | |
260 | 312 | | |
261 | 313 | | |
262 | | - | |
263 | | - | |
| 314 | + | |
| 315 | + | |
264 | 316 | | |
265 | 317 | | |
266 | 318 | | |
267 | 319 | | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
268 | 333 | | |
269 | 334 | | |
270 | 335 | | |
| |||
295 | 360 | | |
296 | 361 | | |
297 | 362 | | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
298 | 439 | | |
299 | 440 | | |
300 | 441 | | |
| |||
0 commit comments