Skip to content

Commit 5030bdd

Browse files
authored
Run a model through the OpenVINO delegate in the wheel tests (pytorch#21988)
The wheel tests assert that the OpenVINO delegate library ships, that it is the only owner of its symbols, and that it registers itself. None of that proves it computes. A delegate can register and then refuse the graph or return wrong numbers, and a registration check reports success either way. Export a small model through the OpenVINO partitioner, run it, and compare against eager. On a runner with the export stack installed this is the check that fails when the delegate stops working; elsewhere it skips and prints the missing dependency rather than passing quietly. The guard wraps the whole import chain rather than the OpenVINO runtime alone. Measured on a default install: with openvino present and nncf absent, importing the partitioner still fails, because it reaches the quantizer which requires nncf. A guard that looked only for openvino would have skipped for a reason it did not name, or failed on an import it could not satisfy. Verified with the export stack installed: the delegate loads its runtime and the output matches eager exactly, maximum absolute difference 0.000e+00. Verified again with nncf removed: the check skips and names nncf as the missing piece.
1 parent 98cf058 commit 5030bdd

2 files changed

Lines changed: 70 additions & 0 deletions

File tree

‎.ci/scripts/wheel/test_base.py‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,8 +8,10 @@
88
import os
99
import subprocess
1010
import sys
11+
import tempfile
1112
from dataclasses import dataclass
1213
from functools import cache
14+
from pathlib import Path
1315
from typing import List
1416

1517

@@ -41,6 +43,70 @@ class ModelTest:
4143
backend: Backend
4244

4345

46+
def test_a_model_runs_through_the_openvino_delegate() -> None:
47+
"""Export a model to the OpenVINO delegate and run it, comparing against eager.
48+
49+
The checks elsewhere prove the delegate library ships, owns its symbols and registers
50+
itself. None of that proves it computes: a delegate can register and then produce wrong
51+
numbers or refuse the graph, and a registration check reports success either way.
52+
53+
The OpenVINO runtime is not part of the wheel. It comes from the openvino extra, so a
54+
default install cannot run this and says so rather than passing quietly. Where the runtime
55+
is present the export and the comparison are the point of the check, and a failure there
56+
is a real one.
57+
"""
58+
import torch
59+
from executorch.exir import to_edge_transform_and_lower
60+
from executorch.exir.backend.compile_spec_schema import CompileSpec
61+
from executorch.runtime import Runtime
62+
63+
# One guard around the whole chain rather than around openvino alone. Importing the
64+
# partitioner reaches the quantizer, which requires nncf, so a check that only looked for
65+
# openvino would still fail on the import it cannot satisfy. Measured on a default install
66+
# with openvino present and nncf absent.
67+
try:
68+
from executorch.backends.openvino.partitioner import OpenvinoPartitioner
69+
except ImportError as error:
70+
print(
71+
f"SKIP: the OpenVINO export stack is not installed, so the delegate cannot "
72+
f"execute here ({error}). The wheel ships the adapter only; the runtime and the "
73+
f"quantizer dependencies come from backends/openvino/requirements.txt. This row "
74+
f"still verifies the shipped library, its owner and its registration."
75+
)
76+
return
77+
78+
class Add(torch.nn.Module):
79+
def forward(self, x, y):
80+
return x + y
81+
82+
example = (torch.ones(2, 2), torch.ones(2, 2) * 2)
83+
eager = Add()(*example)
84+
85+
exported = torch.export.export(Add(), example)
86+
lowered = to_edge_transform_and_lower(
87+
exported,
88+
partitioner=[OpenvinoPartitioner([CompileSpec("device", b"CPU")])],
89+
)
90+
program = lowered.to_executorch()
91+
92+
with tempfile.TemporaryDirectory() as work_dir:
93+
pte = Path(work_dir) / "add_openvino.pte"
94+
pte.write_bytes(program.buffer)
95+
96+
method = Runtime.get().load_program(pte).load_method("forward")
97+
actual = method.execute(list(example))[0]
98+
99+
difference = (actual - eager).abs().max().item()
100+
assert difference == 0.0, (
101+
f"the OpenVINO delegate ran but its output does not match eager: maximum absolute "
102+
f"difference {difference:.3e}. The delegate computed something other than the model."
103+
)
104+
print(
105+
f"\u2713 a model runs through the OpenVINO delegate and matches eager "
106+
f"\u2014 max abs diff {difference:.3e}"
107+
)
108+
109+
44110
def test_cmsis_nn_install():
45111
import executorch.backends.cortex_m.library.cmsis_nn as cmsis_nn
46112

‎.ci/scripts/wheel/test_linux.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,10 @@
4646

4747
test_base.test_cmsis_nn_install()
4848

49+
# Registration above proves the delegate loaded, not that it computes. This runs a
50+
# model through it where the OpenVINO runtime is available and says why when it is not.
51+
test_base.test_a_model_runs_through_the_openvino_delegate()
52+
4953
# The wheel ships the runtime, the kernels, the delegate, the thread
5054
# pool and the profiler as separate shared libraries now, so check that
5155
# each has exactly one owner and that all of them are loadable.

0 commit comments

Comments
 (0)