Skip to content

Commit 460c688

Browse files
authored
Arm backend: Enable Vulkan validation layer (pytorch#22841)
## Summary Enable the Khronos Vulkan Validation Layer for the Arm VGF/VKML test paths. The goal of this change is to detect invalid Vulkan API usage during VGF execution and make validation errors visible as CI failures, while keeping the production VGF runtime unchanged. ## What this change does * Enables `VK_LAYER_KHRONOS_validation` for VGF/VKML tests. * Configures the validation layer through the Vulkan loader environment rather than modifying `VkInstanceCreateInfo` in the backend. * Ensures the validation layer is placed before the Arm ML emulation layers. * Makes the validation-layer manifest visible when the ML SDK overrides `VK_LAYER_PATH`. * Restricts validation output to errors so warnings do not fail otherwise valid tests. * Adds validation to both: * Python-based VKML execution through `runner_utils.py`. * Direct command paths such as `examples/arm/run.sh` and the VGF OOTB tests. * Adds a wrapper for direct commands that converts Vulkan validation errors into a non-zero exit status. * Preserves validation errors produced by earlier Vulkan subprocesses when a test launches multiple Vulkan processes. * Keeps validation opt-in for generic pytest/Buck runs so environments without the Vulkan SDK or validation layer can still run the normal Arm test suite. * Adds integration tests verifying that the validation layer is enabled and that validation failures are propagated correctly. Validation can be controlled using: ```text EXECUTORCH_VGF_VULKAN_VALIDATION ``` The dedicated VGF/VKML test entry points enable it, while explicitly setting it to `0`, `false`, `off`, or `no` disables validation for debugging. ## Known validation errors There are currently several validation errors produced by known issues in the VKML / Vulkan emulation-layer dependencies. These VUIDs are temporarily filtered so that enabling Vulkan validation does not make the VGF CI permanently fail on already-known dependency bugs. The filtered errors currently include issues related to: * BF16 shader capability advertisement. * Tensor-view usage validation. * Descriptor layouts generated for duplicated custom-shader inputs. The corresponding VUID filters are intentionally narrow so that any new or unexpected Vulkan validation error still fails the test. These filters are temporary and should be removed individually once the corresponding bugs are fixed in the emulation-layer dependencies. ## Why Without the validation layer, invalid Vulkan API usage can remain unnoticed when running against the emulation layers. This change makes Vulkan API correctness part of the VGF/VKML CI signal and should help catch regressions closer to where they are introduced. cc @SS-JIA @manuelcandales @digantdesai @cbilgin @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell @rascani Signed-off-by: Elena Zhelezina <elena.zhelezina@arm.com>
1 parent 8a33ec8 commit 460c688

8 files changed

Lines changed: 901 additions & 10 deletions

File tree

‎.github/workflows/pull.yml‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -854,6 +854,46 @@ jobs:
854854
# Test test_arm_backend.sh with test
855855
backends/arm/test/test_arm_backend.sh "${ARM_TEST}"
856856
857+
test-arm-backend-vkml:
858+
name: test-arm-backend-vkml
859+
uses: pytorch/test-infra/.github/workflows/linux_job_v2.yml@main
860+
permissions:
861+
id-token: write
862+
contents: read
863+
strategy:
864+
matrix:
865+
include:
866+
- test_arm_backend: test_pytest_ops_vkml
867+
- test_arm_backend: test_ootb_tests_vgf
868+
fail-fast: false
869+
with:
870+
runner: linux.2xlarge.memory
871+
docker-image: ci-image:executorch-ubuntu-24.04-arm-sdk
872+
submodules: 'recursive'
873+
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
874+
timeout: 120
875+
script: |
876+
CONDA_ENV=$(conda env list --json | jq -r ".envs | .[-1]")
877+
conda activate "${CONDA_ENV}"
878+
879+
source .ci/scripts/utils.sh
880+
install_executorch "--use-pt-pinned-commit"
881+
882+
.ci/scripts/setup-arm-baremetal-tools.sh \
883+
--disable-ethos-u-deps \
884+
--enable-mlsdk-deps \
885+
--install-mlsdk-deps-with-pip
886+
887+
sudo sysctl fs.inotify.max_user_watches=1048576
888+
889+
ARM_TEST=${{ matrix.test_arm_backend }}
890+
891+
if [[ -n "${RUNNER_TEST_RESULTS_DIR:-}" ]]; then
892+
export PYTEST_ADDOPTS="--junit-xml=${RUNNER_TEST_RESULTS_DIR}/${ARM_TEST}.xml ${PYTEST_ADDOPTS:-}"
893+
fi
894+
895+
backends/arm/test/test_arm_backend.sh "${ARM_TEST}"
896+
857897
test-arm-backend-public-api-backward-compatibility:
858898
needs: docker-image
859899
name: test-arm-backend-public-api-backward-compatibility

‎backends/arm/test/conftest.py‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -157,6 +157,39 @@ def pytest_sessionfinish(session, exitstatus):
157157
# ==== Pytest fixtures =====
158158

159159

160+
@pytest.fixture(autouse=True)
161+
def enable_vulkan_validation_for_vgf_tests(request, monkeypatch) -> None:
162+
"""Configure Vulkan validation for VGF tests only when explicitly requested.
163+
164+
Dedicated VGF/VKML test entry points set EXECUTORCH_VGF_VULKAN_VALIDATION
165+
when validation is required. Generic pytest/Buck runs leave it unset, so
166+
validation-specific integration tests can skip cleanly on hosts without the
167+
Vulkan SDK or Khronos validation layer.
168+
169+
"""
170+
if "vgf" not in request.node.nodeid.lower():
171+
return
172+
173+
from executorch.backends.arm.test import runner_utils
174+
175+
if not runner_utils._vulkan_validation_requested():
176+
return
177+
178+
configured_env = runner_utils._enable_vulkan_validation(dict(os.environ))
179+
for variable in (
180+
"VK_LAYER_PATH",
181+
"VK_ADD_LAYER_PATH",
182+
"VK_INSTANCE_LAYERS",
183+
"VK_KHRONOS_VALIDATION_REPORT_FLAGS",
184+
"VK_KHRONOS_VALIDATION_LOG_FILENAME",
185+
"VK_KHRONOS_VALIDATION_DEBUG_ACTION",
186+
runner_utils.VULKAN_VALIDATION_MESSAGE_FILTER_ENV,
187+
):
188+
value = configured_env.get(variable)
189+
if value is not None:
190+
monkeypatch.setenv(variable, value)
191+
192+
160193
@pytest.fixture(autouse=True)
161194
def set_random_seed(request):
162195
"""Control random numbers in Arm test suite. Default behavior is to use a

‎backends/arm/test/misc/test_runner_utils.py‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44
# LICENSE file in the root directory of this source tree.
55

66
import json
7+
import os
78
from pathlib import Path
89
from types import SimpleNamespace
910
from typing import Any, cast
@@ -249,3 +250,74 @@ def fake_tensor(node):
249250
monkeypatch.setattr(runner_utils, "get_first_fake_tensor", fake_tensor)
250251

251252
assert not runner_utils.user_inputs_need_shape_inference(cast(Any, program))
253+
254+
255+
def test_enable_vulkan_validation_puts_validation_first_and_deduplicates(
256+
monkeypatch,
257+
tmp_path: Path,
258+
) -> None:
259+
monkeypatch.setenv(runner_utils.VULKAN_VALIDATION_ENV, "1")
260+
261+
vulkan_sdk = tmp_path / "vulkan-sdk"
262+
validation_layer_dir = vulkan_sdk / "share/vulkan/explicit_layer.d"
263+
validation_layer_dir.mkdir(parents=True)
264+
(validation_layer_dir / "VkLayer_khronos_validation.json").write_text(
265+
"{}", encoding="utf-8"
266+
)
267+
268+
emulation_layer_dir = tmp_path / "emulation-layers"
269+
emulation_layer_dir.mkdir()
270+
emulation_layers = ["VK_LAYER_ARM_tensor", "VK_LAYER_ARM_graph"]
271+
env = {
272+
# Include a trailing path separator to cover setup_path.sh's CI form.
273+
"VULKAN_SDK": f"{vulkan_sdk}{os.path.pathsep}",
274+
"VK_LAYER_PATH": str(emulation_layer_dir),
275+
"VK_ADD_LAYER_PATH": str(emulation_layer_dir),
276+
"VK_INSTANCE_LAYERS": os.path.pathsep.join(
277+
[
278+
emulation_layers[0],
279+
runner_utils.VULKAN_VALIDATION_LAYER,
280+
emulation_layers[1],
281+
runner_utils.VULKAN_VALIDATION_LAYER,
282+
]
283+
),
284+
}
285+
286+
result = runner_utils._enable_vulkan_validation(env)
287+
layers = result["VK_INSTANCE_LAYERS"].split(os.path.pathsep)
288+
289+
assert layers == [
290+
runner_utils.VULKAN_VALIDATION_LAYER,
291+
*emulation_layers,
292+
]
293+
assert result["VK_LAYER_PATH"].split(os.path.pathsep) == [
294+
str(validation_layer_dir),
295+
str(emulation_layer_dir),
296+
]
297+
assert result["VK_ADD_LAYER_PATH"].split(os.path.pathsep) == [
298+
str(validation_layer_dir),
299+
str(emulation_layer_dir),
300+
]
301+
assert result["VK_KHRONOS_VALIDATION_REPORT_FLAGS"] == "error"
302+
303+
304+
def test_add_known_vkml_validation_filters_uses_platform_separator() -> None:
305+
custom_vuid = "VUID-Test-existing-filter-00001"
306+
first_known_vuid = runner_utils.KNOWN_VKML_VALIDATION_VUIDS[0]
307+
env = {
308+
runner_utils.VULKAN_VALIDATION_MESSAGE_FILTER_ENV: os.path.pathsep.join(
309+
[custom_vuid, first_known_vuid]
310+
)
311+
}
312+
313+
runner_utils._add_known_vkml_validation_filters(env)
314+
315+
filters = env[runner_utils.VULKAN_VALIDATION_MESSAGE_FILTER_ENV].split(
316+
os.path.pathsep
317+
)
318+
319+
assert filters == [
320+
custom_vuid,
321+
*runner_utils.KNOWN_VKML_VALIDATION_VUIDS,
322+
]
323+
assert filters.count(first_known_vuid) == 1

0 commit comments

Comments
 (0)