Repository navigation
[ET-VK][CI] Add end-to-end quantized LLM coverage - #23708
Merged
Merged
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/23708
Note: Links to docs will display an error until the docs builds have been completed. ❌ 1 Cancelled Job, 1 Unclassified FailureAs of commit 8c8c62c with merge base a70907c ( UNCLASSIFIED FAILURE - DrCI could not classify the following job because the workflow did not run on the merge base. The failure may be pre-existing on trunk or introduced by this PR:
CANCELLED JOB - The following job was cancelled. Please retry:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
This PR needs a
|
SS-JIA
added a commit
that referenced
this pull request
Oct 12, 2026
Summary: ## Rationale Add the seven LLM recipes used by internal GPU CI to the existing OSS NVIDIA workflow: Qwen3-0.6B, five Llama3.2-1B configurations, and Phi4 Mini. SwiftShader cannot exercise their required quantized compute features. Use CUDA's shared model export/build/run pattern. Select pinned public HF checkpoints, prepare them with existing model utilities, invoke the existing LLM exporter, check called Vulkan delegates and required operator families, and validate greedy native output. Quantized Llama embeddings also retain the 770 MiB artifact limit. Override HF_HUB_CACHE together with HF_HOME so the GPU runner's read-only shared-cache mount cannot intercept uncached downloads. A regression executes the actual workflow cache setup and acquires a real Hugging Face file lock. ## Review order Read vulkan.yml and the shared export/run scripts first, then the HF/Llama preparation and PTE checker. Continue with the Release Vulkan Make/CMake path and installed-package Boost dependency, then the Vulkan CI regression suites and their CPU CI wiring. The shell orchestration suite was restored to Python and Vulkan CI tests are grouped under .ci/scripts/tests/vulkan. | Area | Before | After | |---|---|---| | OSS Vulkan GPU CI | Model/operator coverage without an LLM answer gate | Seven pinned LLM export/inference cells | | Shared scripts | CUDA/Metal/XNNPACK dispatch | Also supports the selected Vulkan recipes | | Llama preparation | Local Unsloth LoRA conversion wrapper | Full HF conversion through TorchTune and optional parameter output | | Native build | No llama-vulkan Make target | Release Vulkan installation plus the existing Llama runner build | | Installed package | Boost target references could be unresolved | Discover Boost only when the exported Vulkan interface needs it | Authored with OpenAI Codex. Test Plan: ## Observed Results Local Python regressions were rechecked after the HF cache fix. Initial NVIDIA CI passed Qwen and Phi, but all five Llama jobs failed in snapshot downloading because HF_HUB_CACHE still pointed at a read-only mount. The new regression reproduced that cache-selection failure and passed after the explicit override. See PR #23708 for subsequent per-head CI results. The final shell refactor preserved all seven export command traces, metadata, prompts, and expected answers. The seven-device-case sweep preceded the final structural refactors; Qwen was freshly exported and run after exporter reuse. | Correctness validation | Environment | Observed result | |---|---|---| | Vulkan CI Python suites, Llama converter tests, existing CUDA workflow tests | Isolated current-pin Python environment | 48 passed, 0 failed | | CMake package producer/consumer suite | macOS, real configure/build/install | 3 passed, 0 failed | | Restored orchestration suite after UFMT | macOS | 26 passed, included in the Python coverage above | | Seven before/after export recipe traces and metadata | Real shared scripts with command stubs | All seven identical | | Renamed PTE checker | Actual saved Qwen PTE | Passed; called VulkanBackend and required families found | | Seven earlier shared native LLM cases | Qwen: S24/Adreno 750; Llama/Phi: Pixel 9 Pro XL/Mali-G715 | Seven passed expected answers and native exit checks | | Fresh Qwen export plus shared native execution after exporter reuse | S24/Adreno 750 | Passed | | make llama-vulkan | Isolated macOS build, Boost enabled; CoreML/ASR disabled locally | Passed | | Lintrunner on edited files and new/moved CI tests | Local | Passed | | Shell syntax, whitespace, complete-patch reverse check | Local | Passed | | Real HF cache lock regression | macOS, inherited read-only cache | Failed before fix; passed afterward | | Initial NVIDIA workflow execution | NVIDIA A10G | Qwen/Phi passed; five Llama downloads failed in the read-only cache; latest results tracked on PR #23708 | | Performance validation | Result | |---|---| | Operator performance tests | Not run; no operator implementation change | | End-to-end latency/memory A/B against parent/public ancestor | Not run; this adds correctness coverage, with no performance claim | ## Reproduction Guide ### Checkout and development dependencies Install Git and Conda (Miniforge installers are available at https://github.com/conda-forge/miniforge/releases). On Linux install a C++ toolchain; on macOS install Xcode command-line tools with xcode-select --install. For NVIDIA inference, install a compatible NVIDIA Vulkan driver. The existing setup script selects and checks the hardware ICD. ```bash git clone --recurse-submodules https://github.com/pytorch/executorch.git cd executorch git checkout ssjia/vulkan-llm-ci git submodule update --init --recursive conda create -y -n etvk-ci python=3.11 conda activate etvk-ci conda install -y -c conda-forge cmake ninja glslc rust CMAKE_ARGS="-DEXECUTORCH_BUILD_VULKAN=ON -DEXECUTORCH_BUILD_KERNELS_LLM_AOT=ON" \ PYTHON_EXECUTABLE=python ./install_executorch.sh python -m pip install pytest pytest-xdist ``` On Linux, run the Vulkan dependency setup before installation. Select real-gpu for NVIDIA runtime testing, or the default setup for software-driver host preparation/export-only testing: ```bash source .ci/scripts/setup-vulkan-linux-deps.sh real-gpu ``` ### CPU regression tests These create their own fixtures; no model downloads are needed. The Python orchestration tests use command stubs, the conversion tests use real tensors, and the CMake suite builds real small package producers/consumers. The workflow-cache regression also resolves the actual HF cache constant and attempts a real lock, rather than replacing the downloader with a stub. ```bash python -m pytest -q .ci/scripts/tests/vulkan \ .ci/scripts/tests/test_cuda_workflow.py \ examples/models/llama/tests/test_convert_weights.py bash .ci/scripts/tests/vulkan/test_vulkan_cmake_package.sh ``` ### Generate all model artifacts The shared script downloads pinned public checkpoints, copies their tokenizers, creates exact prompt/answer files, and checks required Vulkan lowering and size. This generation stage does not require NVIDIA inference. ```bash ET_ARTIFACT_DIR="$PWD/ci-repro/models" mkdir -p "$ET_ARTIFACT_DIR" while read -r model quant; do name=$(printf '%s' "$model" | tr / _) bash .ci/scripts/export_model_artifact.sh vulkan "$model" "$quant" \ "$ET_ARTIFACT_DIR/$name/$quant" done <<'CASES' Qwen/Qwen3-0.6B quantized-8da4w-fp16-embedding unsloth/Llama-3.2-1B-Instruct quantized-8da4w unsloth/Llama-3.2-1B-Instruct quantized-8da4w-fp16 unsloth/Llama-3.2-1B-Instruct quantized-8da4w-embedding unsloth/Llama-3.2-1B-Instruct quantized-8da4w-fp16-embedding unsloth/Llama-3.2-1B-Instruct quantized-int4-weight-only microsoft/Phi-4-mini-instruct quantized-8da4w-embedding CASES ``` ### Linux/NVIDIA shared end-to-end path Use the same generated directories with RUN_EXPORT=0, or regenerate and run in one step with RUN_EXPORT=1. The helper builds make llama-vulkan and checks native exit status and the expected answer in stdout. Repeat for the other pairs above. The GitHub matrix runs these pairs as separate jobs. ```bash source .ci/scripts/setup-vulkan-linux-deps.sh real-gpu RUN_EXPORT=1 bash .ci/scripts/test_model_e2e.sh vulkan \ Qwen/Qwen3-0.6B quantized-8da4w-fp16-embedding \ "$ET_ARTIFACT_DIR/Qwen_Qwen3-0.6B/quantized-8da4w-fp16-embedding" ``` ### Android native correctness reproduction Install Android Studio from https://developer.android.com/studio. In SDK Manager install Android SDK Platform-Tools and NDK 29.0.13846066. Set ET_ANDROID_SDK to the SDK Location shown by SDK Manager and ET_DEVICE to the serial reported by adb devices. Enable USB debugging and authorize the host. Use S24 for Qwen and Pixel 9 Pro XL for the larger Llama/Phi cases, matching the recorded device sweep. Build a static Android runtime/runner from this checkout: ```bash ET_NDK="$ET_ANDROID_SDK/ndk/29.0.13846066" ET_ANDROID_BUILD="$PWD/ci-repro/android-core" ET_ANDROID_INSTALL="$PWD/ci-repro/android-install" ET_ANDROID_RUNNER="$PWD/ci-repro/android-runner" cmake -S . -B "$ET_ANDROID_BUILD" -G Ninja \ -DCMAKE_TOOLCHAIN_FILE="$ET_NDK/build/cmake/android.toolchain.cmake" \ -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-26 \ -DANDROID_STL=c++_static -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_INSTALL_PREFIX="$ET_ANDROID_INSTALL" -DPYTHON_EXECUTABLE="$(command -v python)" \ -DEXECUTORCH_BUILD_VULKAN=ON -DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \ -DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON -DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \ -DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON -DEXECUTORCH_BUILD_EXTENSION_LLM=ON \ -DEXECUTORCH_BUILD_EXTENSION_LLM_RUNNER=ON \ -DEXECUTORCH_BUILD_KERNELS_LLM=ON -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \ -DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON -DEXECUTORCH_BUILD_XNNPACK=ON cmake --build "$ET_ANDROID_BUILD" --target install --parallel 4 cmake -S examples/models/llama -B "$ET_ANDROID_RUNNER" -G Ninja \ -DCMAKE_TOOLCHAIN_FILE="$ET_NDK/build/cmake/android.toolchain.cmake" \ -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-26 \ -DANDROID_STL=c++_static -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_FIND_ROOT_PATH="$ET_ANDROID_INSTALL" -DPYTHON_EXECUTABLE="$(command -v python)" cmake --build "$ET_ANDROID_RUNNER" --target llama_main --parallel 4 ``` Stage and run one generated artifact; repeat with the other directories/device as appropriate. Modern adb preserves the remote native exit code. Keep stdout and stderr separate so diagnostic text cannot satisfy the answer check. ```bash ET_MODEL_DIR="$ET_ARTIFACT_DIR/Qwen_Qwen3-0.6B/quantized-8da4w-fp16-embedding" ET_REMOTE=/data/local/tmp/etvk_llm_ci adb -s "$ET_DEVICE" shell mkdir -p "$ET_REMOTE" adb -s "$ET_DEVICE" push "$ET_ANDROID_RUNNER/llama_main" "$ET_REMOTE/llama_main" adb -s "$ET_DEVICE" shell chmod +x "$ET_REMOTE/llama_main" for file in model.pte tokenizer.json prompt.txt; do adb -s "$ET_DEVICE" push "$ET_MODEL_DIR/$file" "$ET_REMOTE/$file" done adb -s "$ET_DEVICE" shell "$ET_REMOTE/llama_main" \ --model_path "$ET_REMOTE/model.pte" --tokenizer_path "$ET_REMOTE/tokenizer.json" \ --prompt_file "$ET_REMOTE/prompt.txt" --temperature 0 --max_new_tokens 50 \ > "$ET_MODEL_DIR/runner.stdout" 2> "$ET_MODEL_DIR/runner.stderr" grep -Fiw -- "$(cat "$ET_MODEL_DIR/expected_output.txt")" "$ET_MODEL_DIR/runner.stdout" ``` No performance tests were run. There are no latency regression thresholds to reproduce in this change. Reviewers:
SS-JIA
force-pushed
the
ssjia/vulkan-llm-ci
branch
from
October 12, 2026 16:46
7256479 to
fd3cbce
Compare
SS-JIA
commented
Oct 12, 2026
SS-JIA
commented
Oct 12, 2026
SS-JIA
commented
Oct 12, 2026
SS-JIA
added a commit
that referenced
this pull request
Oct 12, 2026
Summary: ## Rationale Add the seven LLM recipes used by internal GPU CI to the existing OSS NVIDIA workflow: Qwen3-0.6B, five Llama3.2-1B configurations, and Phi4 Mini. SwiftShader cannot exercise their required quantized compute features. Use CUDA's shared model export/build/run pattern. Select pinned public HF checkpoints, prepare them with existing model utilities, invoke the existing LLM exporter, check called Vulkan delegates and required operator families, and validate greedy native output. Quantized Llama embeddings also retain the 770 MiB artifact limit. Override HF_HUB_CACHE together with HF_HOME so the GPU runner's read-only shared-cache mount cannot intercept uncached downloads. A regression executes the actual workflow cache setup and acquires a real Hugging Face file lock. ## Review order Read vulkan.yml and the shared export/run scripts first, then the HF/Llama preparation and PTE checker. Continue with the Release Vulkan Make/CMake path and installed-package Boost dependency, then the Vulkan CI regression suites and their CPU CI wiring. The GPU matrix owns end-to-end harness coverage; focused artifact/API guards and the real-cache regression live together under .ci/scripts/tests/vulkan. | Area | Before | After | |---|---|---| | OSS Vulkan GPU CI | Model/operator coverage without an LLM answer gate | Seven pinned LLM export/inference cells | | Shared scripts | CUDA/Metal/XNNPACK dispatch | Also supports the selected Vulkan recipes | | Llama preparation | Local Unsloth LoRA conversion wrapper | Full HF conversion through TorchTune and optional parameter output | | Native build | No llama-vulkan Make target | Release Vulkan installation plus the existing Llama runner build | | Installed package | Boost target references could be unresolved | Discover Boost only when the exported Vulkan interface needs it | Authored with OpenAI Codex. Test Plan: ## Observed Results Local Python regressions were rechecked after removing the broad mocked harness suite. The GPU model matrix covers real export/build/inference failures; the focused cache regression forces a missing cache write against read-only runner settings, which fully warmed model caches can otherwise mask. The prior amended head passed all seven NVIDIA LLM jobs, the complete Vulkan workflow, and all six Linux/macOS unit jobs. Production scripts and model recipes are unchanged by the test-only cleanup. The initial Llama cache failure and its fix remain recorded below; see PR #23708 for latest-head CI results. | Correctness validation | Environment | Observed result | |---|---|---| | Vulkan CI Python suites, Llama converter tests, existing CUDA workflow tests | Isolated current-pin Python environment | 23 passed, 0 failed | | CMake package producer/consumer suite | macOS, real configure/build/install | 3 passed, 0 failed | | Seven before/after export recipe traces and metadata | Real shared scripts with command stubs | All seven identical | | Renamed PTE checker | Actual saved Qwen PTE | Passed; called VulkanBackend and required families found | | Seven earlier shared native LLM cases | Qwen: S24/Adreno 750; Llama/Phi: Pixel 9 Pro XL/Mali-G715 | Seven passed expected answers and native exit checks | | Fresh Qwen export plus shared native execution after exporter reuse | S24/Adreno 750 | Passed | | make llama-vulkan | Isolated macOS build, Boost enabled; CoreML/ASR disabled locally | Passed | | Lintrunner on edited files and new/moved CI tests | Local | Passed | | Shell syntax, whitespace, complete-patch reverse check | Local | Passed | | Real HF cache lock regression | macOS, inherited read-only cache | Failed before fix; passed afterward | | NVIDIA workflow after the cache fix, before test-only cleanup | NVIDIA A10G | All seven LLM jobs, existing NVIDIA operator/model job, and Windows Vulkan build passed | | Linux/macOS unit jobs before test-only cleanup | Release/default/editable | All six passed | | Performance validation | Result | |---|---| | Operator performance tests | Not run; no operator implementation change | | End-to-end latency/memory A/B against parent/public ancestor | Not run; this adds correctness coverage, with no performance claim | ## Reproduction Guide ### Checkout and development dependencies Install Git and Conda (Miniforge installers are available at https://github.com/conda-forge/miniforge/releases). On Linux install a C++ toolchain; on macOS install Xcode command-line tools with xcode-select --install. For NVIDIA inference, install a compatible NVIDIA Vulkan driver. The existing setup script selects and checks the hardware ICD. ```bash git clone --recurse-submodules https://github.com/pytorch/executorch.git cd executorch git checkout ssjia/vulkan-llm-ci git submodule update --init --recursive conda create -y -n etvk-ci python=3.11 conda activate etvk-ci conda install -y -c conda-forge cmake ninja glslc rust CMAKE_ARGS="-DEXECUTORCH_BUILD_VULKAN=ON -DEXECUTORCH_BUILD_KERNELS_LLM_AOT=ON" \ PYTHON_EXECUTABLE=python ./install_executorch.sh python -m pip install pytest pytest-xdist ``` On Linux, run the Vulkan dependency setup before installation. Select real-gpu for NVIDIA runtime testing, or the default setup for software-driver host preparation/export-only testing: ```bash source .ci/scripts/setup-vulkan-linux-deps.sh real-gpu ``` ### CPU regression tests These create their own fixtures; no model downloads are needed. Export guards use synthetic program objects, conversion tests use real tensors, and the CMake suite builds real small package producers/consumers. The workflow-cache regression also resolves the actual HF cache constant and attempts a real lock, rather than replacing the downloader with a stub. ```bash python -m pytest -q .ci/scripts/tests/vulkan \ .ci/scripts/tests/test_cuda_workflow.py \ examples/models/llama/tests/test_convert_weights.py bash .ci/scripts/tests/vulkan/test_vulkan_cmake_package.sh ``` ### Generate all model artifacts The shared script downloads pinned public checkpoints, copies their tokenizers, creates exact prompt/answer files, and checks required Vulkan lowering and size. This generation stage does not require NVIDIA inference. ```bash ET_ARTIFACT_DIR="$PWD/ci-repro/models" mkdir -p "$ET_ARTIFACT_DIR" while read -r model quant; do name=$(printf '%s' "$model" | tr / _) bash .ci/scripts/export_model_artifact.sh vulkan "$model" "$quant" \ "$ET_ARTIFACT_DIR/$name/$quant" done <<'CASES' Qwen/Qwen3-0.6B quantized-8da4w-fp16-embedding unsloth/Llama-3.2-1B-Instruct quantized-8da4w unsloth/Llama-3.2-1B-Instruct quantized-8da4w-fp16 unsloth/Llama-3.2-1B-Instruct quantized-8da4w-embedding unsloth/Llama-3.2-1B-Instruct quantized-8da4w-fp16-embedding unsloth/Llama-3.2-1B-Instruct quantized-int4-weight-only microsoft/Phi-4-mini-instruct quantized-8da4w-embedding CASES ``` ### Linux/NVIDIA shared end-to-end path Use the same generated directories with RUN_EXPORT=0, or regenerate and run in one step with RUN_EXPORT=1. The helper builds make llama-vulkan and checks native exit status and the expected answer in stdout. Repeat for the other pairs above. The GitHub matrix runs these pairs as separate jobs. ```bash source .ci/scripts/setup-vulkan-linux-deps.sh real-gpu RUN_EXPORT=1 bash .ci/scripts/test_model_e2e.sh vulkan \ Qwen/Qwen3-0.6B quantized-8da4w-fp16-embedding \ "$ET_ARTIFACT_DIR/Qwen_Qwen3-0.6B/quantized-8da4w-fp16-embedding" ``` ### Android native correctness reproduction Install Android Studio from https://developer.android.com/studio. In SDK Manager install Android SDK Platform-Tools and NDK 29.0.13846066. Set ET_ANDROID_SDK to the SDK Location shown by SDK Manager and ET_DEVICE to the serial reported by adb devices. Enable USB debugging and authorize the host. Use S24 for Qwen and Pixel 9 Pro XL for the larger Llama/Phi cases, matching the recorded device sweep. Build a static Android runtime/runner from this checkout: ```bash ET_NDK="$ET_ANDROID_SDK/ndk/29.0.13846066" ET_ANDROID_BUILD="$PWD/ci-repro/android-core" ET_ANDROID_INSTALL="$PWD/ci-repro/android-install" ET_ANDROID_RUNNER="$PWD/ci-repro/android-runner" cmake -S . -B "$ET_ANDROID_BUILD" -G Ninja \ -DCMAKE_TOOLCHAIN_FILE="$ET_NDK/build/cmake/android.toolchain.cmake" \ -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-26 \ -DANDROID_STL=c++_static -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_INSTALL_PREFIX="$ET_ANDROID_INSTALL" -DPYTHON_EXECUTABLE="$(command -v python)" \ -DEXECUTORCH_BUILD_VULKAN=ON -DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \ -DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON -DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \ -DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON -DEXECUTORCH_BUILD_EXTENSION_LLM=ON \ -DEXECUTORCH_BUILD_EXTENSION_LLM_RUNNER=ON \ -DEXECUTORCH_BUILD_KERNELS_LLM=ON -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \ -DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON -DEXECUTORCH_BUILD_XNNPACK=ON cmake --build "$ET_ANDROID_BUILD" --target install --parallel 4 cmake -S examples/models/llama -B "$ET_ANDROID_RUNNER" -G Ninja \ -DCMAKE_TOOLCHAIN_FILE="$ET_NDK/build/cmake/android.toolchain.cmake" \ -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-26 \ -DANDROID_STL=c++_static -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_FIND_ROOT_PATH="$ET_ANDROID_INSTALL" -DPYTHON_EXECUTABLE="$(command -v python)" cmake --build "$ET_ANDROID_RUNNER" --target llama_main --parallel 4 ``` Stage and run one generated artifact; repeat with the other directories/device as appropriate. Modern adb preserves the remote native exit code. Keep stdout and stderr separate so diagnostic text cannot satisfy the answer check. ```bash ET_MODEL_DIR="$ET_ARTIFACT_DIR/Qwen_Qwen3-0.6B/quantized-8da4w-fp16-embedding" ET_REMOTE=/data/local/tmp/etvk_llm_ci adb -s "$ET_DEVICE" shell mkdir -p "$ET_REMOTE" adb -s "$ET_DEVICE" push "$ET_ANDROID_RUNNER/llama_main" "$ET_REMOTE/llama_main" adb -s "$ET_DEVICE" shell chmod +x "$ET_REMOTE/llama_main" for file in model.pte tokenizer.json prompt.txt; do adb -s "$ET_DEVICE" push "$ET_MODEL_DIR/$file" "$ET_REMOTE/$file" done adb -s "$ET_DEVICE" shell "$ET_REMOTE/llama_main" \ --model_path "$ET_REMOTE/model.pte" --tokenizer_path "$ET_REMOTE/tokenizer.json" \ --prompt_file "$ET_REMOTE/prompt.txt" --temperature 0 --max_new_tokens 50 \ > "$ET_MODEL_DIR/runner.stdout" 2> "$ET_MODEL_DIR/runner.stderr" grep -Fiw -- "$(cat "$ET_MODEL_DIR/expected_output.txt")" "$ET_MODEL_DIR/runner.stdout" ``` No performance tests were run. There are no latency regression thresholds to reproduce in this change. Reviewers:
SS-JIA
force-pushed
the
ssjia/vulkan-llm-ci
branch
from
October 12, 2026 19:25
fd3cbce to
e4819f3
Compare
SS-JIA
added a commit
that referenced
this pull request
Oct 12, 2026
Summary: ## Rationale Add the seven LLM recipes used by internal GPU CI to the existing OSS NVIDIA workflow: Qwen3-0.6B, five Llama3.2-1B configurations, and Phi4 Mini. SwiftShader cannot exercise their required quantized compute features. Use CUDA's shared model export/build/run pattern. Select pinned public HF checkpoints, prepare them with existing model utilities, invoke the existing LLM exporter, check called Vulkan delegates and required operator families, and validate greedy native output. Quantized Llama embeddings also retain the 770 MiB artifact limit. Override HF_HUB_CACHE together with HF_HOME so the GPU runner's read-only shared-cache mount cannot intercept uncached downloads. A regression executes the actual workflow cache setup and acquires a real Hugging Face file lock. The HF converter is named convert_hf_to_meta. Its CLI exposes hf-to-meta, unsloth-to-meta, and load-and-convert-unsloth-to-meta. The downloader exposes snapshot and download-and-convert; the latter accepts a MODULE:FUNCTION callback. Library conversion/cache APIs retain their existing behavior. ## Review order Read vulkan.yml and the shared export/run scripts first, then the HF/Llama preparation and PTE checker. Continue with the Release Vulkan Make/CMake path and installed-package Boost dependency, then the Vulkan CI regression suites and their CPU CI wiring. The GPU matrix owns end-to-end harness coverage; focused artifact/API guards and the real-cache regression live together under .ci/scripts/tests/vulkan. | Area | Before | After | |---|---|---| | OSS Vulkan GPU CI | Model/operator coverage without an LLM answer gate | Seven pinned LLM export/inference cells | | Shared scripts | CUDA/Metal/XNNPACK dispatch | Also supports the selected Vulkan recipes | | Llama preparation | Local Unsloth LoRA conversion wrapper | Full HF conversion through TorchTune and optional parameter output; all functions exposed through CLI | | Native build | No llama-vulkan Make target | Release Vulkan installation plus the existing Llama runner build | | Installed package | Boost target references could be unresolved | Discover Boost only when the exported Vulkan interface needs it | Authored with OpenAI Codex. Test Plan: ## Observed Results Local Python regressions pass after exposing all converter/downloader functions through CLI subcommands and updating the shared CI callers. The GPU matrix covers real export/build/inference; the cache regression forces a new lock write. The earlier cache-fix head passed all seven NVIDIA LLM jobs, the complete Vulkan workflow, and all six Linux/macOS unit jobs. Fresh CI for the CLI amendment is pending; see PR #23708 for the latest result. | Correctness validation | Environment | Observed result | |---|---|---| | Vulkan CI Python suites, Llama converter tests, existing CUDA workflow tests | Isolated current-pin Python environment | 28 passed, 0 failed | | CMake package producer/consumer suite | macOS, real configure/build/install | 3 passed, 0 failed | | Seven before/after export recipe traces and metadata | Real shared scripts with command stubs | All seven identical | | Renamed PTE checker | Actual saved Qwen PTE | Passed; called VulkanBackend and required families found | | Seven earlier shared native LLM cases | Qwen: S24/Adreno 750; Llama/Phi: Pixel 9 Pro XL/Mali-G715 | Seven passed expected answers and native exit checks | | Fresh Qwen export plus shared native execution after exporter reuse | S24/Adreno 750 | Passed | | make llama-vulkan | Isolated macOS build, Boost enabled; CoreML/ASR disabled locally | Passed | | Lintrunner on edited files and new/moved CI tests | Local | Passed | | Shell syntax, whitespace, complete-patch reverse check | Local | Passed | | Real HF cache lock regression | macOS, inherited read-only cache | Failed before fix; passed afterward | | NVIDIA workflow after the cache fix, before test-only cleanup | NVIDIA A10G | All seven LLM jobs, existing NVIDIA operator/model job, and Windows Vulkan build passed | | Linux/macOS unit jobs before test-only cleanup | Release/default/editable | All six passed | | Performance validation | Result | |---|---| | Operator performance tests | Not run; no operator implementation change | | End-to-end latency/memory A/B against parent/public ancestor | Not run; this adds correctness coverage, with no performance claim | ## Reproduction Guide ### Checkout and development dependencies Install Git and Conda (Miniforge installers are available at https://github.com/conda-forge/miniforge/releases). On Linux install a C++ toolchain; on macOS install Xcode command-line tools with xcode-select --install. For NVIDIA inference, install a compatible NVIDIA Vulkan driver. The existing setup script selects and checks the hardware ICD. ```bash git clone --recurse-submodules https://github.com/pytorch/executorch.git cd executorch git checkout ssjia/vulkan-llm-ci git submodule update --init --recursive conda create -y -n etvk-ci python=3.11 conda activate etvk-ci conda install -y -c conda-forge cmake ninja glslc rust CMAKE_ARGS="-DEXECUTORCH_BUILD_VULKAN=ON -DEXECUTORCH_BUILD_KERNELS_LLM_AOT=ON" \ PYTHON_EXECUTABLE=python ./install_executorch.sh python -m pip install pytest pytest-xdist ``` On Linux, run the Vulkan dependency setup before installation. Select real-gpu for NVIDIA runtime testing, or the default setup for software-driver host preparation/export-only testing: ```bash source .ci/scripts/setup-vulkan-linux-deps.sh real-gpu ``` ### CPU regression tests These create their own fixtures; no model downloads are needed. Export guards use synthetic program objects, conversion tests use real tensors, and the CMake suite builds real small package producers/consumers. The workflow-cache regression also resolves the actual HF cache constant and attempts a real lock, rather than replacing the downloader with a stub. ```bash python -m pytest -q .ci/scripts/tests/vulkan \ .ci/scripts/tests/test_cuda_workflow.py \ examples/models/llama/tests/test_convert_weights.py bash .ci/scripts/tests/vulkan/test_vulkan_cmake_package.sh ``` ### Generate all model artifacts The shared script downloads pinned public checkpoints, copies their tokenizers, creates exact prompt/answer files, and checks required Vulkan lowering and size. This generation stage does not require NVIDIA inference. ```bash ET_ARTIFACT_DIR="$PWD/ci-repro/models" mkdir -p "$ET_ARTIFACT_DIR" while read -r model quant; do name=$(printf '%s' "$model" | tr / _) bash .ci/scripts/export_model_artifact.sh vulkan "$model" "$quant" \ "$ET_ARTIFACT_DIR/$name/$quant" done <<'CASES' Qwen/Qwen3-0.6B quantized-8da4w-fp16-embedding unsloth/Llama-3.2-1B-Instruct quantized-8da4w unsloth/Llama-3.2-1B-Instruct quantized-8da4w-fp16 unsloth/Llama-3.2-1B-Instruct quantized-8da4w-embedding unsloth/Llama-3.2-1B-Instruct quantized-8da4w-fp16-embedding unsloth/Llama-3.2-1B-Instruct quantized-int4-weight-only microsoft/Phi-4-mini-instruct quantized-8da4w-embedding CASES ``` ### Linux/NVIDIA shared end-to-end path Use the same generated directories with RUN_EXPORT=0, or regenerate and run in one step with RUN_EXPORT=1. The helper builds make llama-vulkan and checks native exit status and the expected answer in stdout. Repeat for the other pairs above. The GitHub matrix runs these pairs as separate jobs. ```bash source .ci/scripts/setup-vulkan-linux-deps.sh real-gpu RUN_EXPORT=1 bash .ci/scripts/test_model_e2e.sh vulkan \ Qwen/Qwen3-0.6B quantized-8da4w-fp16-embedding \ "$ET_ARTIFACT_DIR/Qwen_Qwen3-0.6B/quantized-8da4w-fp16-embedding" ``` ### Android native correctness reproduction Install Android Studio from https://developer.android.com/studio. In SDK Manager install Android SDK Platform-Tools and NDK 29.0.13846066. Set ET_ANDROID_SDK to the SDK Location shown by SDK Manager and ET_DEVICE to the serial reported by adb devices. Enable USB debugging and authorize the host. Use S24 for Qwen and Pixel 9 Pro XL for the larger Llama/Phi cases, matching the recorded device sweep. Build a static Android runtime/runner from this checkout: ```bash ET_NDK="$ET_ANDROID_SDK/ndk/29.0.13846066" ET_ANDROID_BUILD="$PWD/ci-repro/android-core" ET_ANDROID_INSTALL="$PWD/ci-repro/android-install" ET_ANDROID_RUNNER="$PWD/ci-repro/android-runner" cmake -S . -B "$ET_ANDROID_BUILD" -G Ninja \ -DCMAKE_TOOLCHAIN_FILE="$ET_NDK/build/cmake/android.toolchain.cmake" \ -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-26 \ -DANDROID_STL=c++_static -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_INSTALL_PREFIX="$ET_ANDROID_INSTALL" -DPYTHON_EXECUTABLE="$(command -v python)" \ -DEXECUTORCH_BUILD_VULKAN=ON -DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \ -DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON -DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \ -DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON -DEXECUTORCH_BUILD_EXTENSION_LLM=ON \ -DEXECUTORCH_BUILD_EXTENSION_LLM_RUNNER=ON \ -DEXECUTORCH_BUILD_KERNELS_LLM=ON -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \ -DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON -DEXECUTORCH_BUILD_XNNPACK=ON cmake --build "$ET_ANDROID_BUILD" --target install --parallel 4 cmake -S examples/models/llama -B "$ET_ANDROID_RUNNER" -G Ninja \ -DCMAKE_TOOLCHAIN_FILE="$ET_NDK/build/cmake/android.toolchain.cmake" \ -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-26 \ -DANDROID_STL=c++_static -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_FIND_ROOT_PATH="$ET_ANDROID_INSTALL" -DPYTHON_EXECUTABLE="$(command -v python)" cmake --build "$ET_ANDROID_RUNNER" --target llama_main --parallel 4 ``` Stage and run one generated artifact; repeat with the other directories/device as appropriate. Modern adb preserves the remote native exit code. Keep stdout and stderr separate so diagnostic text cannot satisfy the answer check. ```bash ET_MODEL_DIR="$ET_ARTIFACT_DIR/Qwen_Qwen3-0.6B/quantized-8da4w-fp16-embedding" ET_REMOTE=/data/local/tmp/etvk_llm_ci adb -s "$ET_DEVICE" shell mkdir -p "$ET_REMOTE" adb -s "$ET_DEVICE" push "$ET_ANDROID_RUNNER/llama_main" "$ET_REMOTE/llama_main" adb -s "$ET_DEVICE" shell chmod +x "$ET_REMOTE/llama_main" for file in model.pte tokenizer.json prompt.txt; do adb -s "$ET_DEVICE" push "$ET_MODEL_DIR/$file" "$ET_REMOTE/$file" done adb -s "$ET_DEVICE" shell "$ET_REMOTE/llama_main" \ --model_path "$ET_REMOTE/model.pte" --tokenizer_path "$ET_REMOTE/tokenizer.json" \ --prompt_file "$ET_REMOTE/prompt.txt" --temperature 0 --max_new_tokens 50 \ > "$ET_MODEL_DIR/runner.stdout" 2> "$ET_MODEL_DIR/runner.stderr" grep -Fiw -- "$(cat "$ET_MODEL_DIR/expected_output.txt")" "$ET_MODEL_DIR/runner.stdout" ``` No performance tests were run. There are no latency regression thresholds to reproduce in this change. Reviewers:
SS-JIA
force-pushed
the
ssjia/vulkan-llm-ci
branch
from
October 12, 2026 19:43
e4819f3 to
871588e
Compare
Contributor
|
Each 7 configuration repeat installation and setup work. Is it possible to just install once and test 7 models on that? You can probably use a dependent flow (first flow that builds the llm runner uploads the binary, and another job that pulls in a binary and sequentially runs those 7 models). We're getting some pushback recently around CI costs |
SS-JIA
added a commit
that referenced
this pull request
Oct 14, 2026
Summary: ## Rationale Add the seven LLM recipes used by internal GPU CI to the existing OSS NVIDIA workflow: Qwen3-0.6B, five Llama3.2-1B configurations, and Phi4 Mini. SwiftShader cannot exercise their required quantized compute features. Use CUDA's shared model export/build/run pattern. Select pinned public HF checkpoints, prepare them with existing model utilities, invoke the existing LLM exporter, check called Vulkan delegates and required operator families, and validate greedy native output. Quantized Llama embeddings also retain the 770 MiB artifact limit. Override HF_HUB_CACHE together with HF_HOME so the GPU runner's read-only shared-cache mount cannot intercept uncached downloads. A regression executes the actual workflow cache setup and acquires a real Hugging Face file lock. The HF converter is named convert_hf_to_meta. Its CLI exposes hf-to-meta, unsloth-to-meta, and load-and-convert-unsloth-to-meta. The downloader exposes snapshot and download-and-convert; the latter accepts a MODULE:FUNCTION callback. Library conversion/cache APIs retain their existing behavior. ## Review order Read vulkan.yml and the shared export/run scripts first, then the HF/Llama preparation and PTE checker. Continue with the Release Vulkan Make/CMake path and installed-package Boost dependency, then the Vulkan CI regression suites and their CPU CI wiring. The GPU matrix owns end-to-end harness coverage; focused artifact/API guards and the real-cache regression live together under .ci/scripts/tests/vulkan. | Area | Before | After | |---|---|---| | OSS Vulkan GPU CI | Model/operator coverage without an LLM answer gate | Seven pinned LLM export/inference cells | | Shared scripts | CUDA/Metal/XNNPACK dispatch | Also supports the selected Vulkan recipes | | Llama preparation | Local Unsloth LoRA conversion wrapper | Full HF conversion through TorchTune and optional parameter output; all functions exposed through CLI | | Native build | No llama-vulkan Make target | Release Vulkan installation plus the existing Llama runner build | | Installed package | Boost target references could be unresolved | Discover Boost only when the exported Vulkan interface needs it | Authored with OpenAI Codex. Test Plan: ## Observed Results Local Python regressions pass after exposing all converter/downloader functions through CLI subcommands and updating the shared CI callers. The GPU matrix covers real export/build/inference; the cache regression forces a new lock write. The earlier cache-fix head passed all seven NVIDIA LLM jobs, the complete Vulkan workflow, and all six Linux/macOS unit jobs. Fresh CI for the CLI amendment is pending; see PR #23708 for the latest result. | Correctness validation | Environment | Observed result | |---|---|---| | Vulkan CI Python suites, Llama converter tests, existing CUDA workflow tests | Isolated current-pin Python environment | 28 passed, 0 failed | | CMake package producer/consumer suite | macOS, real configure/build/install | 3 passed, 0 failed | | Seven before/after export recipe traces and metadata | Real shared scripts with command stubs | All seven identical | | Renamed PTE checker | Actual saved Qwen PTE | Passed; called VulkanBackend and required families found | | Seven earlier shared native LLM cases | Qwen: S24/Adreno 750; Llama/Phi: Pixel 9 Pro XL/Mali-G715 | Seven passed expected answers and native exit checks | | Fresh Qwen export plus shared native execution after exporter reuse | S24/Adreno 750 | Passed | | make llama-vulkan | Isolated macOS build, Boost enabled; CoreML/ASR disabled locally | Passed | | Lintrunner on edited files and new/moved CI tests | Local | Passed | | Shell syntax, whitespace, complete-patch reverse check | Local | Passed | | Real HF cache lock regression | macOS, inherited read-only cache | Failed before fix; passed afterward | | NVIDIA workflow after the cache fix, before test-only cleanup | NVIDIA A10G | All seven LLM jobs, existing NVIDIA operator/model job, and Windows Vulkan build passed | | Linux/macOS unit jobs before test-only cleanup | Release/default/editable | All six passed | | Performance validation | Result | |---|---| | Operator performance tests | Not run; no operator implementation change | | End-to-end latency/memory A/B against parent/public ancestor | Not run; this adds correctness coverage, with no performance claim | ## Reproduction Guide ### Checkout and development dependencies Install Git and Conda (Miniforge installers are available at https://github.com/conda-forge/miniforge/releases). On Linux install a C++ toolchain; on macOS install Xcode command-line tools with xcode-select --install. For NVIDIA inference, install a compatible NVIDIA Vulkan driver. The existing setup script selects and checks the hardware ICD. ```bash git clone --recurse-submodules https://github.com/pytorch/executorch.git cd executorch git checkout ssjia/vulkan-llm-ci git submodule update --init --recursive conda create -y -n etvk-ci python=3.11 conda activate etvk-ci conda install -y -c conda-forge cmake ninja glslc rust CMAKE_ARGS="-DEXECUTORCH_BUILD_VULKAN=ON -DEXECUTORCH_BUILD_KERNELS_LLM_AOT=ON" \ PYTHON_EXECUTABLE=python ./install_executorch.sh python -m pip install pytest pytest-xdist ``` On Linux, run the Vulkan dependency setup before installation. Select real-gpu for NVIDIA runtime testing, or the default setup for software-driver host preparation/export-only testing: ```bash source .ci/scripts/setup-vulkan-linux-deps.sh real-gpu ``` ### CPU regression tests These create their own fixtures; no model downloads are needed. Export guards use synthetic program objects, conversion tests use real tensors, and the CMake suite builds real small package producers/consumers. The workflow-cache regression also resolves the actual HF cache constant and attempts a real lock, rather than replacing the downloader with a stub. ```bash python -m pytest -q .ci/scripts/tests/vulkan \ .ci/scripts/tests/test_cuda_workflow.py \ examples/models/llama/tests/test_convert_weights.py bash .ci/scripts/tests/vulkan/test_vulkan_cmake_package.sh ``` ### Generate all model artifacts The shared script downloads pinned public checkpoints, copies their tokenizers, creates exact prompt/answer files, and checks required Vulkan lowering and size. This generation stage does not require NVIDIA inference. ```bash ET_ARTIFACT_DIR="$PWD/ci-repro/models" mkdir -p "$ET_ARTIFACT_DIR" while read -r model quant; do name=$(printf '%s' "$model" | tr / _) bash .ci/scripts/export_model_artifact.sh vulkan "$model" "$quant" \ "$ET_ARTIFACT_DIR/$name/$quant" done <<'CASES' Qwen/Qwen3-0.6B quantized-8da4w-fp16-embedding unsloth/Llama-3.2-1B-Instruct quantized-8da4w unsloth/Llama-3.2-1B-Instruct quantized-8da4w-fp16 unsloth/Llama-3.2-1B-Instruct quantized-8da4w-embedding unsloth/Llama-3.2-1B-Instruct quantized-8da4w-fp16-embedding unsloth/Llama-3.2-1B-Instruct quantized-int4-weight-only microsoft/Phi-4-mini-instruct quantized-8da4w-embedding CASES ``` ### Linux/NVIDIA shared end-to-end path Use the same generated directories with RUN_EXPORT=0, or regenerate and run in one step with RUN_EXPORT=1. The helper builds make llama-vulkan and checks native exit status and the expected answer in stdout. Repeat for the other pairs above. The GitHub matrix runs these pairs as separate jobs. ```bash source .ci/scripts/setup-vulkan-linux-deps.sh real-gpu RUN_EXPORT=1 bash .ci/scripts/test_model_e2e.sh vulkan \ Qwen/Qwen3-0.6B quantized-8da4w-fp16-embedding \ "$ET_ARTIFACT_DIR/Qwen_Qwen3-0.6B/quantized-8da4w-fp16-embedding" ``` ### Android native correctness reproduction Install Android Studio from https://developer.android.com/studio. In SDK Manager install Android SDK Platform-Tools and NDK 29.0.13846066. Set ET_ANDROID_SDK to the SDK Location shown by SDK Manager and ET_DEVICE to the serial reported by adb devices. Enable USB debugging and authorize the host. Use S24 for Qwen and Pixel 9 Pro XL for the larger Llama/Phi cases, matching the recorded device sweep. Build a static Android runtime/runner from this checkout: ```bash ET_NDK="$ET_ANDROID_SDK/ndk/29.0.13846066" ET_ANDROID_BUILD="$PWD/ci-repro/android-core" ET_ANDROID_INSTALL="$PWD/ci-repro/android-install" ET_ANDROID_RUNNER="$PWD/ci-repro/android-runner" cmake -S . -B "$ET_ANDROID_BUILD" -G Ninja \ -DCMAKE_TOOLCHAIN_FILE="$ET_NDK/build/cmake/android.toolchain.cmake" \ -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-26 \ -DANDROID_STL=c++_static -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_INSTALL_PREFIX="$ET_ANDROID_INSTALL" -DPYTHON_EXECUTABLE="$(command -v python)" \ -DEXECUTORCH_BUILD_VULKAN=ON -DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \ -DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON -DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \ -DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON -DEXECUTORCH_BUILD_EXTENSION_LLM=ON \ -DEXECUTORCH_BUILD_EXTENSION_LLM_RUNNER=ON \ -DEXECUTORCH_BUILD_KERNELS_LLM=ON -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \ -DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON -DEXECUTORCH_BUILD_XNNPACK=ON cmake --build "$ET_ANDROID_BUILD" --target install --parallel 4 cmake -S examples/models/llama -B "$ET_ANDROID_RUNNER" -G Ninja \ -DCMAKE_TOOLCHAIN_FILE="$ET_NDK/build/cmake/android.toolchain.cmake" \ -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-26 \ -DANDROID_STL=c++_static -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_FIND_ROOT_PATH="$ET_ANDROID_INSTALL" -DPYTHON_EXECUTABLE="$(command -v python)" cmake --build "$ET_ANDROID_RUNNER" --target llama_main --parallel 4 ``` Stage and run one generated artifact; repeat with the other directories/device as appropriate. Modern adb preserves the remote native exit code. Keep stdout and stderr separate so diagnostic text cannot satisfy the answer check. ```bash ET_MODEL_DIR="$ET_ARTIFACT_DIR/Qwen_Qwen3-0.6B/quantized-8da4w-fp16-embedding" ET_REMOTE=/data/local/tmp/etvk_llm_ci adb -s "$ET_DEVICE" shell mkdir -p "$ET_REMOTE" adb -s "$ET_DEVICE" push "$ET_ANDROID_RUNNER/llama_main" "$ET_REMOTE/llama_main" adb -s "$ET_DEVICE" shell chmod +x "$ET_REMOTE/llama_main" for file in model.pte tokenizer.json prompt.txt; do adb -s "$ET_DEVICE" push "$ET_MODEL_DIR/$file" "$ET_REMOTE/$file" done adb -s "$ET_DEVICE" shell "$ET_REMOTE/llama_main" \ --model_path "$ET_REMOTE/model.pte" --tokenizer_path "$ET_REMOTE/tokenizer.json" \ --prompt_file "$ET_REMOTE/prompt.txt" --temperature 0 --max_new_tokens 50 \ > "$ET_MODEL_DIR/runner.stdout" 2> "$ET_MODEL_DIR/runner.stderr" grep -Fiw -- "$(cat "$ET_MODEL_DIR/expected_output.txt")" "$ET_MODEL_DIR/runner.stdout" ``` No performance tests were run. There are no latency regression thresholds to reproduce in this change. Reviewers:
SS-JIA
force-pushed
the
ssjia/vulkan-llm-ci
branch
from
October 14, 2026 18:34
871588e to
2e21a03
Compare
Contributor
Author
|
ack. Will rewrite the workflow to minimize cost. |
SS-JIA
force-pushed
the
ssjia/vulkan-llm-ci
branch
from
October 14, 2026 19:00
2e21a03 to
b5b1cc3
Compare
Contributor
Author
|
@mergennachin Consolidated the seven configurations into one GPU job. It installs dependencies and builds the runner once, then exports/tests all seven cases sequentially through the shared scripts. Per-case logs and results are retained; individual failures do not skip later cases, and any failed case fails the job. All seven cases passed in the new CI run. |
mergennachin
approved these changes
Oct 14, 2026
Summary: ## Rationale Add the seven LLM recipes used by internal GPU CI to the existing OSS NVIDIA workflow: Qwen3-0.6B, five Llama3.2-1B configurations, and Phi4 Mini. SwiftShader cannot exercise their required quantized compute features. Use CUDA's shared model export/build/run pattern. Select pinned public HF checkpoints, prepare them with existing model utilities, invoke the existing LLM exporter, check called Vulkan delegates and required operator families, and validate greedy native output. Quantized Llama embeddings also retain the 770 MiB artifact limit. Run the seven recipes sequentially in one NVIDIA job to avoid repeating setup and compilation. Build llama-vulkan once and pass the existing runner override to each shared test invocation. Continue after individual failures, report an aggregate failure, and retain per-configuration logs/results without model PTEs. Override HF_HUB_CACHE together with HF_HOME so the GPU runner's read-only shared-cache mount cannot intercept uncached downloads. A regression executes the actual workflow cache setup and acquires a real Hugging Face file lock. The HF converter is named convert_hf_to_meta. Its CLI exposes hf-to-meta, unsloth-to-meta, and load-and-convert-unsloth-to-meta. The downloader exposes snapshot and download-and-convert; the latter accepts a MODULE:FUNCTION callback. Library conversion/cache APIs retain their existing behavior. ## Review order Read vulkan.yml and the shared export/run scripts first, then the HF/Llama preparation and PTE checker. Continue with the Release Vulkan Make/CMake path and installed-package Boost dependency, then the Vulkan CI regression suites and their CPU CI wiring. The GPU job owns end-to-end harness coverage; focused artifact/API guards and the real-cache regression live together under .ci/scripts/tests/vulkan. | Area | Before | After | |---|---|---| | OSS Vulkan GPU CI | Model/operator coverage without an LLM answer gate | One setup/build job runs all seven pinned recipes sequentially | | Shared scripts | CUDA/Metal/XNNPACK dispatch | Also supports the selected Vulkan recipes | | Llama preparation | Local Unsloth LoRA conversion wrapper | Full HF conversion through TorchTune and optional parameter output; all functions exposed through CLI | | Native build | No llama-vulkan Make target | Release Vulkan installation plus the existing Llama runner build | | Installed package | Boost target references could be unresolved | Discover Boost only when the exported Vulkan interface needs it | Authored with OpenAI Codex. Test Plan: ## Observed Results The 28 Python regression checks pass after consolidating the GPU workflow. Temporary orchestration validation passed five cases: success, export failure, native failure, build failure, and setup failure. It executes the actual workflow shell and shared runner script, stubbing external downloads/build/native compute. The seven recipe pairs match the former matrix exactly. Shell syntax and edited file Lintrunner pass. The new single-job NVIDIA run is pending; earlier seven-job GPU results are historical. | Correctness validation | Environment | Observed result | |---|---|---| | Vulkan CI Python suites, Llama converter tests, existing CUDA workflow tests | Isolated current-pin Python environment | 28 passed, 0 failed | | Temporary sequential orchestration | Actual shell/shared script, external commands stubbed | 5 passed; failures propagate and later recipes execute | | CMake package producer/consumer suite | macOS, real configure/build/install | 3 passed, 0 failed | | Seven before/after export recipe traces and metadata | Real shared scripts with command stubs | All seven identical | | Renamed PTE checker | Actual saved Qwen PTE | Passed; called VulkanBackend and required families found | | Seven earlier shared native LLM cases | Qwen: S24/Adreno 750; Llama/Phi: Pixel 9 Pro XL/Mali-G715 | Seven passed expected answers and native exit checks | | Fresh Qwen export plus shared native execution after exporter reuse | S24/Adreno 750 | Passed | | make llama-vulkan | Isolated macOS build, Boost enabled; CoreML/ASR disabled locally | Passed | | Lintrunner on edited files and new/moved CI tests | Local | Passed | | Shell syntax, whitespace, complete-patch reverse check | Local | Passed | | Real HF cache lock regression | macOS, inherited read-only cache | Failed before fix; passed afterward | | NVIDIA workflow after the cache fix, before test-only cleanup | NVIDIA A10G | All seven LLM jobs, existing NVIDIA operator/model job, and Windows Vulkan build passed | | Linux/macOS unit jobs before test-only cleanup | Release/default/editable | All six passed | | Performance validation | Result | |---|---| | Operator performance tests | Not run; no operator implementation change | | End-to-end latency/memory A/B against parent/public ancestor | Not run; this adds correctness coverage, with no performance claim | ## Reproduction Guide ### Checkout and development dependencies Install Git and Conda (Miniforge installers are available at https://github.com/conda-forge/miniforge/releases). On Linux install a C++ toolchain; on macOS install Xcode command-line tools with xcode-select --install. For NVIDIA inference, install a compatible NVIDIA Vulkan driver. The existing setup script selects and checks the hardware ICD. ```bash git clone --recurse-submodules https://github.com/pytorch/executorch.git cd executorch git checkout ssjia/vulkan-llm-ci git submodule update --init --recursive conda create -y -n etvk-ci python=3.11 conda activate etvk-ci conda install -y -c conda-forge cmake ninja glslc rust CMAKE_ARGS="-DEXECUTORCH_BUILD_VULKAN=ON -DEXECUTORCH_BUILD_KERNELS_LLM_AOT=ON" \ PYTHON_EXECUTABLE=python ./install_executorch.sh python -m pip install pytest pytest-xdist ``` On Linux, run the Vulkan dependency setup before installation. Select real-gpu for NVIDIA runtime testing, or the default setup for software-driver host preparation/export-only testing: ```bash source .ci/scripts/setup-vulkan-linux-deps.sh real-gpu ``` ### CPU regression tests These create their own fixtures; no model downloads are needed. Export guards use synthetic program objects, conversion tests use real tensors, and the CMake suite builds real small package producers/consumers. The workflow-cache regression also resolves the actual HF cache constant and attempts a real lock, rather than replacing the downloader with a stub. ```bash python -m pytest -q .ci/scripts/tests/vulkan \ .ci/scripts/tests/test_cuda_workflow.py \ examples/models/llama/tests/test_convert_weights.py bash .ci/scripts/tests/vulkan/test_vulkan_cmake_package.sh ``` ### Generate all model artifacts The shared script downloads pinned public checkpoints, copies their tokenizers, creates exact prompt/answer files, and checks required Vulkan lowering and size. This generation stage does not require NVIDIA inference. ```bash ET_ARTIFACT_DIR="$PWD/ci-repro/models" mkdir -p "$ET_ARTIFACT_DIR" while read -r model quant; do name=$(printf '%s' "$model" | tr / _) bash .ci/scripts/export_model_artifact.sh vulkan "$model" "$quant" \ "$ET_ARTIFACT_DIR/$name/$quant" done <<'CASES' Qwen/Qwen3-0.6B quantized-8da4w-fp16-embedding unsloth/Llama-3.2-1B-Instruct quantized-8da4w unsloth/Llama-3.2-1B-Instruct quantized-8da4w-fp16 unsloth/Llama-3.2-1B-Instruct quantized-8da4w-embedding unsloth/Llama-3.2-1B-Instruct quantized-8da4w-fp16-embedding unsloth/Llama-3.2-1B-Instruct quantized-int4-weight-only microsoft/Phi-4-mini-instruct quantized-8da4w-embedding CASES ``` ### Linux/NVIDIA single-job reproduction Use a Linux NVIDIA Vulkan host with the development environment above. Extract and execute the actual workflow shell, which performs common setup/build once, then runs all seven cases. It writes results.tsv and per-case logs to the artifact directory and deletes each PTE after testing. ```bash export RUNNER_TEMP="$PWD/ci-repro/scratch" export RUNNER_ARTIFACT_DIR="$PWD/ci-repro/results" mkdir -p "$RUNNER_TEMP" "$RUNNER_ARTIFACT_DIR" python - <<'PY' from pathlib import Path import yaml workflow = yaml.safe_load(Path('.github/workflows/vulkan.yml').read_text()) Path('ci-repro/vulkan-llm-job.sh').write_text( workflow['jobs']['test-model-vulkan-e2e']['with']['script'] ) PY bash ci-repro/vulkan-llm-job.sh cat "$RUNNER_ARTIFACT_DIR/results.tsv" ``` ### Reproduce local orchestration validation These checks execute the actual workflow shell and shared test script. The fixture substitutes environment setup, package installation, model download, compilation, and native GPU computation. Actual NVIDIA correctness remains covered by the CI job above. ```bash export ET_VALIDATION_DIR="$PWD/ci-repro/orchestration" mkdir -p "$ET_VALIDATION_DIR" cat > "$ET_VALIDATION_DIR/expected-cases.json" <<'JSON' [ [ "Qwen/Qwen3-0.6B", "quantized-8da4w-fp16-embedding" ], [ "unsloth/Llama-3.2-1B-Instruct", "quantized-8da4w" ], [ "unsloth/Llama-3.2-1B-Instruct", "quantized-8da4w-fp16" ], [ "unsloth/Llama-3.2-1B-Instruct", "quantized-8da4w-embedding" ], [ "unsloth/Llama-3.2-1B-Instruct", "quantized-8da4w-fp16-embedding" ], [ "unsloth/Llama-3.2-1B-Instruct", "quantized-int4-weight-only" ], [ "microsoft/Phi-4-mini-instruct", "quantized-8da4w-embedding" ] ] JSON cat > "$ET_VALIDATION_DIR/validate_job.py" <<'PY' import json import os from pathlib import Path import shutil import subprocess import tempfile import yaml REPO = Path.cwd() OUTPUT = Path(os.environ['ET_VALIDATION_DIR']) job = yaml.safe_load((REPO / '.github/workflows/vulkan.yml').read_text())['jobs']['test-model-vulkan-e2e'] assert 'strategy' not in job script = job['with']['script'] expected = json.loads((OUTPUT / 'expected-cases.json').read_text()) cases = [line.split() for line in script.split("done <<'MODELS'\n", 1)[1].split('\nMODELS', 1)[0].splitlines()] assert cases == expected def executable(path, contents): path.parent.mkdir(parents=True, exist_ok=True) path.write_text('#!/bin/bash\nset -e\n' + contents) path.chmod(0o755) observations = [] for failure in ('none', 'export', 'native', 'build', 'setup'): with tempfile.TemporaryDirectory(prefix='vulkan-sequential-') as directory: root = Path(directory) workspace = root / 'workspace' workspace.mkdir() scripts = workspace / '.ci/scripts' scripts.mkdir(parents=True) shutil.copyfile(REPO / '.ci/scripts/test_model_e2e.sh', scripts / 'test_model_e2e.sh') tools = root / 'bin' tools.mkdir() trace = root / 'trace' scratch = root / 'scratch' scratch.mkdir() artifacts = root / 'artifacts' artifacts.mkdir() executable(scripts / 'setup-vulkan-linux-deps.sh', ''' printf 'setup\n' >> "$TRACE" if [ "$FAILURE" = setup ]; then return 11; fi ''') executable(workspace / 'install_executorch.sh', ''' printf 'install\n' >> "$TRACE" ''') executable(tools / 'pip', 'printf "pip\\n" >> "$TRACE"\n') executable(tools / 'python', 'printf "[]\\n"\n') executable(tools / 'vulkaninfo', 'printf "GPU probe\\n"\n') executable(tools / 'uname', 'printf "Linux\\n"\n') executable(tools / 'make', ''' printf 'make %s\n' "$*" >> "$TRACE" if [ "$FAILURE" = build ]; then exit 13; fi mkdir -p cmake-out/examples/models/llama cat > cmake-out/examples/models/llama/llama_main <<'RUNNER' #!/bin/bash set -e printf 'native %s\n' "$*" >> "$TRACE" if [ "$FAILURE" = native ] && [[ "$*" == *"$FAIL_CASE"* ]]; then echo 'native failure' >&2 exit 7 fi echo 'The capital of France is Paris' RUNNER chmod +x cmake-out/examples/models/llama/llama_main ''') executable(scripts / 'export_model_artifact.sh', ''' printf 'export %s %s\n' "$2" "$3" >> "$TRACE" if [ "$FAILURE" = export ] && [[ "$4" == *"$FAIL_CASE"* ]]; then echo 'export failure' >&2 exit 9 fi printf 'pte' > "$4/model.pte" printf '{}' > "$4/tokenizer.json" printf '{}' > "$4/metadata.json" printf 'prompt' > "$4/prompt.txt" printf 'The capital of France is Paris\n' > "$4/expected_output.txt" ''') result = subprocess.run( ['bash', '-c', script], cwd=workspace, env={**os.environ, 'PATH': str(tools) + ':' + os.environ['PATH'], 'RUNNER_TEMP': str(scratch), 'RUNNER_ARTIFACT_DIR': str(artifacts), 'TRACE': str(trace), 'FAILURE': failure, 'FAIL_CASE': 'unsloth_Llama-3.2-1B-Instruct-quantized-8da4w-embedding'}, capture_output=True, text=True, ) (OUTPUT / f'{failure}.log').write_text(result.stdout + result.stderr) actions = trace.read_text().splitlines() assert actions.count('setup') == 1 if failure == 'setup': assert result.returncode == 11 assert actions == ['setup'] else: assert actions.count('install') == 1 assert actions.count('make llama-vulkan') == 1 if failure == 'build': assert result.returncode == 13 assert not any(a.startswith('export') for a in actions) else: assert result.returncode == (0 if failure == 'none' else 1) assert [a.split()[1:] for a in actions if a.startswith('export ')] == expected native = [a for a in actions if a.startswith('native ')] assert len(native) == (6 if failure == 'export' else 7) results = (artifacts / 'results.tsv').read_text().splitlines()[1:] assert [r.split('\t')[:2] for r in results] == expected statuses = [r.split('\t')[2] for r in results] assert statuses.count('FAIL') == (0 if failure == 'none' else 1) for model, quant in expected: case_artifacts = artifacts / (model.replace('/', '_') + '-' + quant) assert (case_artifacts / 'test.log').is_file() if failure != 'export' or quant != 'quantized-8da4w-embedding' or model.startswith('microsoft'): assert (case_artifacts / 'metadata.json').is_file() assert not list(artifacts.rglob('*.pte')) assert not list(scratch.rglob('*.pte')) observations.append({'scenario': failure, 'exit_code': result.returncode, 'passed': True}) print(f'{failure}: passed, exit {result.returncode}') (OUTPUT / 'local-job-validation.json').write_text(json.dumps(observations, indent=2) + '\n') PY python "$ET_VALIDATION_DIR/validate_job.py" ``` ### Android native correctness reproduction Install Android Studio from https://developer.android.com/studio. In SDK Manager install Android SDK Platform-Tools and NDK 29.0.13846066. Set ET_ANDROID_SDK to the SDK Location shown by SDK Manager and ET_DEVICE to the serial reported by adb devices. Enable USB debugging and authorize the host. Use S24 for Qwen and Pixel 9 Pro XL for the larger Llama/Phi cases, matching the recorded device sweep. Build a static Android runtime/runner from this checkout: ```bash ET_NDK="$ET_ANDROID_SDK/ndk/29.0.13846066" ET_ANDROID_BUILD="$PWD/ci-repro/android-core" ET_ANDROID_INSTALL="$PWD/ci-repro/android-install" ET_ANDROID_RUNNER="$PWD/ci-repro/android-runner" cmake -S . -B "$ET_ANDROID_BUILD" -G Ninja \ -DCMAKE_TOOLCHAIN_FILE="$ET_NDK/build/cmake/android.toolchain.cmake" \ -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-26 \ -DANDROID_STL=c++_static -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_INSTALL_PREFIX="$ET_ANDROID_INSTALL" -DPYTHON_EXECUTABLE="$(command -v python)" \ -DEXECUTORCH_BUILD_VULKAN=ON -DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \ -DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON -DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \ -DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON -DEXECUTORCH_BUILD_EXTENSION_LLM=ON \ -DEXECUTORCH_BUILD_EXTENSION_LLM_RUNNER=ON \ -DEXECUTORCH_BUILD_KERNELS_LLM=ON -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \ -DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON -DEXECUTORCH_BUILD_XNNPACK=ON cmake --build "$ET_ANDROID_BUILD" --target install --parallel 4 cmake -S examples/models/llama -B "$ET_ANDROID_RUNNER" -G Ninja \ -DCMAKE_TOOLCHAIN_FILE="$ET_NDK/build/cmake/android.toolchain.cmake" \ -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-26 \ -DANDROID_STL=c++_static -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_FIND_ROOT_PATH="$ET_ANDROID_INSTALL" -DPYTHON_EXECUTABLE="$(command -v python)" cmake --build "$ET_ANDROID_RUNNER" --target llama_main --parallel 4 ``` Stage and run one generated artifact; repeat with the other directories/device as appropriate. Modern adb preserves the remote native exit code. Keep stdout and stderr separate so diagnostic text cannot satisfy the answer check. ```bash ET_MODEL_DIR="$ET_ARTIFACT_DIR/Qwen_Qwen3-0.6B/quantized-8da4w-fp16-embedding" ET_REMOTE=/data/local/tmp/etvk_llm_ci adb -s "$ET_DEVICE" shell mkdir -p "$ET_REMOTE" adb -s "$ET_DEVICE" push "$ET_ANDROID_RUNNER/llama_main" "$ET_REMOTE/llama_main" adb -s "$ET_DEVICE" shell chmod +x "$ET_REMOTE/llama_main" for file in model.pte tokenizer.json prompt.txt; do adb -s "$ET_DEVICE" push "$ET_MODEL_DIR/$file" "$ET_REMOTE/$file" done adb -s "$ET_DEVICE" shell "$ET_REMOTE/llama_main" \ --model_path "$ET_REMOTE/model.pte" --tokenizer_path "$ET_REMOTE/tokenizer.json" \ --prompt_file "$ET_REMOTE/prompt.txt" --temperature 0 --max_new_tokens 50 \ > "$ET_MODEL_DIR/runner.stdout" 2> "$ET_MODEL_DIR/runner.stderr" grep -Fiw -- "$(cat "$ET_MODEL_DIR/expected_output.txt")" "$ET_MODEL_DIR/runner.stdout" ``` No performance tests were run. There are no latency regression thresholds to reproduce in this change. Reviewers:
SS-JIA
force-pushed
the
ssjia/vulkan-llm-ci
branch
from
October 16, 2026 17:23
b5b1cc3 to
8c8c62c
Compare
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add the seven internal Vulkan LLM correctness recipes to the existing OSS NVIDIA CI: Qwen3-0.6B, five Llama3.2-1B configurations, and Phi4 Mini. One GPU job installs dependencies and builds the runner once, then tests all seven configurations sequentially through CUDA's shared export/build/run pattern and uses pinned public HF checkpoints. Llama's public checkpoint is checksum verified.
The shared exporter prepares weights through existing utilities, invokes the existing generic LLM exporter, checks called Vulkan delegates and required operator families, and applies the 770 MiB limit to Llama's two quantized-embedding cases. The native runner checks greedy output against the expected Paris answer. All recipes use group size 64, context 2048, and a 50-token cap.
Authored with OpenAI Codex.
Review guide
Rebased onto latest main on October 16 without conflicts. Git range-diff confirms the PR patch is unchanged. Fresh rebased-head CI is pending; validation below predates this rebase.
One sequential GPU job now reuses setup and the runner build. Before the October 16 rebase, the single-job NVIDIA run passed all seven configurations; earlier seven-job results below are historical.
convert_weights.pybelow is the Llama utility;utils.pyis the Vulkan test utility.CMakePresets.jsonis the root file. Both Vulkan CI suites live under.ci/scripts/tests/vulkan/.Walkthrough
1. vulkan.yml
test-model-vulkan-e2eruns one A10G job. It installs dependencies and buildsmake llama-vulkanonce, then invokes the shared test script for all seven configurations sequentially usingEXECUTORCH_E2E_RUNNER. The CUDA container provides the environment; Vulkan executes inference. Public pinned weights need no new HF secret.The workflow and job filters now cover the Vulkan CI test directory. The LLM job also runs daily at 02:30 UTC, on manual runs, and according to the existing sampled/path-filtered push policy.
collect_artifacts()retains configuration, operator counts, prompts and logs in separate per-case directories. Common build/package logs andresults.tsvsit at the artifact root. Individual failures are recorded while remaining cases run; any failed case makes the job fail. Each PTE is deleted after testing to bound scratch usage.Verified cache fix: The first run inherited
HF_HUB_CACHE=/mnt/hf_cache/hubeven after changingHF_HOME, so all five Llama downloads failed on cache locks. The amended commit explicitly setsHF_HUB_CACHEunder writableHF_HOME. The real-lock regression passes, and all seven amended-head LLM jobs now pass on NVIDIA.2. export_model_artifact.sh
Start with the normal HF model case statement. Qwen keeps its existing generic configuration and adds its Vulkan recipe inside a backend condition. Llama and Phi have their own model cases and backend restrictions. Their revisions, prompts, converters, checksum, and EOS/BOS settings remain the validated values.
The normal quantization case handles the new Vulkan variants. The later Vulkan export block copies the pinned tokenizer, prepares a temporary checkpoint, and invokes the existing generic LLM exporter. It removes stale model.pte before export and calls
verify_vulkan_llm_pte()afterward. Explicit metadata defaults prevent inherited shell variables from changing a recipe.Review focus: Model selection precedes backend export work. The seven recipe arguments, metadata, prompts, and expected answers match the previous implementation exactly.
3. hf_download.py
download_hf_snapshot()returns a specified public revision's raw Safetensors/JSON snapshot. Its optional checksum is streamed and recorded before rejecting a mismatch. ThesnapshotCLI subcommand returns the snapshot path to the shell flow.The older
download_and_convert_hf_checkpoint()downloads the default revision, converts it, and returns a converted .pth cache entry keyed by repository name. The separate snapshot API provides reproducible revision/config/tokenizer selection without that converted-cache shortcut.snapshot_download()uses Hugging Face environment defaults for its cache.HF_HUB_CACHEtakes precedence overHF_HOME; the workflow now redirects both variables before importing the downloader.4. convert_weights.py
The existing local conversion functions handled Unsloth LoRA adapters. The new
convert_hf_to_meta()wrapper uses existing TorchTune conversion for full HF Llama weights, including Q/K RoPE ordering. It preserves tied embedding storage and optionally emits model parameters from the pinned HF configuration.The conversion algorithm already existed in TorchTune; this wrapper and CLI connect it to the existing ExecuTorch exporter.
params_outputsupplies the dimensions and RoPE settings that exporter expects.5. utils.py
verify_vulkan_llm_pte()is the renamed artifact checker. It examines called delegates in forward, decodes their Vulkan payloads, records operator counts, and checks required quantized-linear/SDPA/optional embedding families plus the applicable size limit.run_and_check_output()andlower_module_and_test_output()defer their native runtime imports until execution. The checker remains usable without those Python runtime bindings.Review focus: Family presence is the gate; full delegation and exact layer/operator counts are not enforced.
6. test_model_e2e.sh
Qwen/Llama/Phi share the existing LLM model case. Its Vulkan condition loads the pinned prompt/tokenizer/expected answer and resets irrelevant inherited runner/image state. Other models are rejected for Vulkan.
run_and_validate()contains one Vulkan invocation/answer block. It uses quoted native arguments, separate stdout/stderr, a native exit check, and literal case-insensitive answer matching with word boundaries. The original generic execution and answer checks follow unchanged. A prebuilt-runner override still skips setup/build.The shared script keeps its generic name because the other backend workflows call it. The broad mocked harness suite was removed; real end-to-end execution is covered by the sequential model job.
7–9. CMakePresets.json, Makefile, executorch-config.cmake
The root configure/build/workflow presets enable a Release Vulkan LLM runtime.
make llama-vulkaninstalls that runtime and reuses the existing Llama runner workflow.The installed package discovers Boost stacktrace components only when the exported Vulkan target references them. This fixes downstream imported-target failures without requiring Boost for packages that omit it. These build/package changes are unchanged by the latest refactor.
10. test_export_vulkan_llm.py
Sixteen tests cover download/hash failures, runtime-free imports, called versus unused/metadata-only delegates, operator families, weight-only rules, size limits, and real cache locking. Synthetic deserialization fixtures isolate artifact-guard behavior.
The retained cache regression executes the actual workflow cache setup, imports the real Hugging Face cache constant, and acquires a real file lock against an inherited read-only runner cache. A fully warmed model cache can mask this configuration bug in GPU testing; the regression forces a new cache write. The broad mocked export/runner orchestration suite was removed.
11. test_convert_weights.py
Six tests use actual Safetensors and torch.save/load to verify Q/K permutation, tied tensor storage, generated params, and all three conversion CLI commands. This general Llama conversion suite remains in the model's test directory.
12. test_vulkan_cmake_package.sh
Three real configure/build/install/consumer-build cases cover Vulkan with Boost, Vulkan without Boost, and no Vulkan. A synthetic FindBoost checks the requested components/version. Its repository-root lookup now accounts for the deeper directory.
13–14. unittest-linux-cmake.sh and unittest-macos-cmake.sh
Both scripts retain one explicit CMake shell-suite invocation; macOS uses the existing conda wrapper. The old E2E shell-suite calls are removed. Pytest already recursively collects the new Vulkan Python directory, so those suites need no extra calls.
CLI review changes
convert_hf_to_meta()names the conversion direction explicitly. The converter CLI exposes every library function:hf-to-metaconvert_hf_to_meta()unsloth-to-metaunsloth_to_meta()load-and-convert-unsloth-to-metaload_and_convert_unsloth_to_meta()The downloader CLI exposes
snapshotanddownload-and-convert. The latter accepts--converter MODULE:FUNCTION, preserving the existing callback API and converted-checkpoint cache. Both commands print only the returned path to stdout; conversion progress goes to stderr. CI usessnapshot, thenhf-to-metafor Llama; Qwen/Phi conversion interfaces are unchanged.Five new tests exercise actual CLI parsing, tensor conversion, parameter output, checksum metadata, and converted-cache reuse. Only network retrieval is stubbed in downloader tests.
Validation before the October 14 rebase
The CLI amendment before rebase passed all seven real NVIDIA LLM jobs, the complete Vulkan workflow, and all six Linux/macOS release/default/editable unit jobs. No additional fix was needed after this amendment.
CI results before the October 14 rebase
Vulkan workflow passed. Each LLM job completed export, delegation/operator/size checks, native build, and expected-answer validation.
Earlier Android validation passed on S24/Adreno 750 for Qwen and Pixel 9 Pro XL/Mali-G715 for Llama/Phi. The initial CI cache failure was fixed by setting writable HF_HUB_CACHE; that real-lock regression remains. No performance comparison was run.
All three inline review comments have individual replies. The reviewer has since resolved all three threads; this agent did not resolve them. Final checks found no new inline comments. Evidence: cli-review-final-run.json, cli-review-final-pr.json, cli-review-validation.json, and cli-review-replies.json.
Previously recorded CI results belong to the linked earlier runs. Fresh rebased-head CI is pending.
Colleague review: CI cost
mergennachin’s comment asks to avoid repeating setup/build work. The selected implementation uses one job: common installation and runner build, followed by all seven configurations in sequence. All recipe pairs, export checks, expected answers, and Llama size limits remain unchanged. The shared export/run scripts need no changes.
Sequential workflow validation before the October 16 rebase
Temporary orchestration checks remain session artifacts rather than adding a broad harness suite to the repository. Full reproduction is in the commit Test Plan. Earlier seven-job green CI remains historical.
Reproduction
Linux/NVIDIA single-job reproduction
Use a Linux NVIDIA Vulkan host with the development environment above. Extract
and execute the actual workflow shell, which performs common setup/build once,
then runs all seven cases. It writes results.tsv and per-case logs to the artifact
directory and deletes each PTE after testing.
Single-job results before the October 16 rebase
Vulkan workflow passed. Its single LLM job produced seven PASS rows in results.tsv. Each case has separate configuration, operator counts, export/runner logs and native output; common build.log is retained. The test logs confirm each case reuses the common runner build.
All three Linux unit jobs passed. Unchanged macOS unit jobs remain queued; they are not reported as passing. Local orchestration checks passed five scenarios and the existing Python suite passed 28 tests. No new harness test file was added.
Current-head merge readiness
Approved and mergeable. All five required checks pass. All Vulkan coverage and six Linux/macOS unit jobs pass; the current-head LLM artifact confirms all seven configurations passed.
Readiness uses repository-required checks and validation for this PR. Other non-required CI is still running. Two non-required issues remain: an unavailable pinned TorchAO wheel in macOS model setup, and a QNN imports setup timeout. Neither dependency/setup path was changed by this PR. The QNN Windows corrupt SDK download passed after a targeted retry. Retrying the QNN imports cancellation is deferred until its parent workflow finishes; GitHub currently refuses that retry.
No source changes were needed during this monitoring turn. The PR remains open for the user to merge.