Skip to content

Commit f2d148b

Browse files
authored
Check the wheel is usable with only the dependencies it declares (pytorch#22273)
## Summary The wheel checks all run inside the build checkout, where every development requirement is installed and the source tree is on the path. So a module that imports a package the wheel never declares passes there, and then fails for a user who only ran `pip install executorch`. Two such gaps shipped. One was in the import path of XNNPACK, the default delegate: ``` >>> import executorch.backends.xnnpack.partition.xnnpack_partitioner ModuleNotFoundError: No module named 'torchao' ``` Asking "does this import here" cannot catch that, because in a build environment the answer is always yes. ## What this does instead It computes the transitive closure of what the wheel declares plus `torch`, hides every *other* installed distribution behind an import hook, and then imports. A module that quietly relied on a development requirement now fails the way it fails for a user, and the traceback names both the missing package and the chain that wanted it. Four entry points are covered, all documented and platform independent: `exir`, `export`, the runtime bindings, and the XNNPACK partitioner. A delegate whose availability depends on a separate SDK, an extra, or the host architecture is deliberately excluded, since its absence is a legitimate state and asserting on it would fail for the wrong reason. The same check also asserts the installed metadata claims exactly one top level name, read from the install rather than the wheel file because that is what any tool consuming it sees. ## Test plan Run against an installed nightly that predates the `torchao` fix, it reports exactly that bug and nothing else: ``` 2 of 4 modules are not usable from a clean install executorch.export: No module named 'torchao' (not declared by the executorch wheel) executorch.backends.xnnpack…: No module named 'torchao' (not declared by the executorch wheel) ``` with the full chain down to `backends/xnnpack/utils/utils.py:23: from torchao.quantization.pt2e.utils import ...`. Treating `torchao` as declared, which is what `main` does now, **all four imports pass**. So the check fails on the real defect and passes once it is fixed. One detail worth noting: the dependency closure is computed to a fixed point rather than a fixed depth. A depth-limited walk reported `six` as undeclared, which was a false positive, since it arrives three levels down through `pandas` and `python-dateutil`. On that same nightly the top level metadata assertion also fires, which is the separate defect fixed in pytorch#22267.
1 parent dbddba4 commit f2d148b

5 files changed

Lines changed: 295 additions & 1 deletion

File tree

Lines changed: 269 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,269 @@
1+
#!/usr/bin/env python
2+
# Copyright (c) Meta Platforms, Inc. and affiliates.
3+
# All rights reserved.
4+
#
5+
# This source code is licensed under the BSD-style license found in the
6+
# LICENSE file in the root directory of this source tree.
7+
8+
"""Checks the wheel is usable with only the dependencies it declares.
9+
10+
Why this exists, and why the other wheel checks cannot cover it: they run inside the build
11+
checkout, which has every development requirement installed and the source tree on the path.
12+
A module that imports a package the wheel never declares still passes there, and then fails
13+
for a user who only ran `pip install executorch`. Two such gaps shipped, one of them in the
14+
import path of XNNPACK, the default delegate.
15+
16+
The check asks a different question from "does this import here", which is always yes in a
17+
build environment. It hides every distribution the wheel does not declare, then imports. So an
18+
import that quietly relied on a development requirement fails here the way it fails for a
19+
user, and names the package it wanted.
20+
"""
21+
22+
import json
23+
import os
24+
import subprocess
25+
import sys
26+
from pathlib import Path
27+
from typing import Dict, List, Set
28+
29+
# Modules a user reaches for after `pip install executorch`. Each must import with only the
30+
# wheel's declared dependencies present.
31+
#
32+
# Limited to entry points that are documented and platform independent. A delegate whose
33+
# availability depends on a separate SDK, an extra, or the host architecture is deliberately
34+
# absent, because its absence is a legitimate state and asserting on it would fail for the
35+
# wrong reason. What this list asserts is that nothing on it needs an undeclared package.
36+
REQUIRED_IMPORTS: List[str] = [
37+
# The core export path.
38+
"executorch.exir",
39+
"executorch.export",
40+
# The runtime bindings, which are what load and run a .pte.
41+
"executorch.extension.pybindings.portable_lib",
42+
# XNNPACK is the default delegate and ships on every platform.
43+
"executorch.backends.xnnpack.partition.xnnpack_partitioner",
44+
]
45+
46+
# Distributions that are legitimately present without being declared.
47+
#
48+
# torch, because the wheel deliberately does not declare it: a consumer brings the build
49+
# matching their platform and accelerator. The rest are what torch itself requires, so they are
50+
# guaranteed alongside it.
51+
ASSUMED_PRESENT: Set[str] = {"torch", "executorch"}
52+
53+
54+
def _normalise(name: str) -> str:
55+
return name.lower().replace("-", "_").replace(".", "_")
56+
57+
58+
def _requirements_of(distribution: str) -> Set[str]:
59+
import importlib.metadata as metadata
60+
61+
from packaging.requirements import Requirement
62+
63+
try:
64+
raw_requirements = metadata.requires(distribution) or []
65+
except metadata.PackageNotFoundError:
66+
return set()
67+
68+
names = set()
69+
for raw in raw_requirements:
70+
requirement = Requirement(raw)
71+
# An extra is opt in, so a module that needs one is not a clean install failure. Its
72+
# own absence is what this check is happy to see.
73+
if requirement.marker is not None and "extra" in str(requirement.marker):
74+
continue
75+
names.add(_normalise(requirement.name))
76+
return names
77+
78+
79+
def _allowed_distributions() -> Set[str]:
80+
"""Every distribution a clean install is entitled to import.
81+
82+
The wheel's own declared requirements and torch, then the full transitive closure of both,
83+
because pip installs a dependency's dependencies too. Computed to a fixed point rather than
84+
a fixed depth: `six` arrives three levels down, through pandas and python-dateutil, and a
85+
depth limited walk reported it as undeclared.
86+
"""
87+
allowed = {_normalise(name) for name in ASSUMED_PRESENT}
88+
allowed |= _requirements_of("executorch")
89+
allowed |= _requirements_of("torch")
90+
91+
pending = set(allowed)
92+
while pending:
93+
allowed |= pending
94+
discovered = set()
95+
for name in pending:
96+
discovered |= _requirements_of(name)
97+
pending = discovered - allowed
98+
return allowed
99+
100+
101+
def _import_names(distribution: str) -> Set[str]:
102+
"""The import names a distribution provides.
103+
104+
Read from the installed file list rather than top_level.txt, because most distributions
105+
built by a modern backend no longer write that file, and the ones that matter here are
106+
exactly the ones whose import name differs from their project name. `protobuf` provides
107+
`google`, `scikit-learn` provides `sklearn`, `attrs` provides `attr`, and `torch` also
108+
provides `functorch` and `torchgen`. Guessing from the project name would leave all of
109+
those unrecognised, and a name left unrecognised on the permitted side gets hidden, which
110+
fails a correct wheel.
111+
"""
112+
import importlib.metadata as metadata
113+
114+
try:
115+
files = metadata.distribution(distribution).files or []
116+
except metadata.PackageNotFoundError:
117+
return set()
118+
119+
names = set()
120+
for entry in files:
121+
path = str(entry)
122+
# Outside the site directory, or not importable.
123+
if path.startswith("..") or path.startswith("__pycache__"):
124+
continue
125+
head = path.split("/")[0]
126+
if head.endswith((".dist-info", ".egg-info", ".data")):
127+
continue
128+
if head.endswith(".py"):
129+
names.add(head[: -len(".py")])
130+
elif head.endswith((".so", ".dylib", ".pyd")):
131+
# An extension module carries a platform tag, as in _foo.cpython-312-darwin.so.
132+
names.add(head.split(".")[0])
133+
elif "." not in head:
134+
names.add(head)
135+
return names
136+
137+
138+
def _blocked_modules(allowed: Set[str]) -> Set[str]:
139+
"""Import names in this environment that a clean install would not have.
140+
141+
Built by elimination rather than by listing what to hide, then narrowed, because a wrong
142+
entry here fails a correct wheel rather than catching anything.
143+
"""
144+
import importlib.metadata as metadata
145+
146+
allowed_modules = set()
147+
for name in allowed:
148+
allowed_modules |= _import_names(name)
149+
allowed_modules.add(name)
150+
151+
blocked = set()
152+
for distribution in metadata.distributions():
153+
name = distribution.metadata["Name"]
154+
if not name or _normalise(name) in allowed:
155+
continue
156+
blocked |= _import_names(name) or {_normalise(name)}
157+
158+
# Never hide a name a permitted distribution also provides. Two distributions can share a
159+
# namespace package, `google` being the common one, and blocking the shared name would
160+
# break the permitted one.
161+
blocked -= allowed_modules
162+
163+
# Never hide anything the standard library provides, whatever a distribution claims. A
164+
# backport publishes the same import name as the module it backports, so blocking it would
165+
# remove a name that is present on a clean install anyway.
166+
blocked -= set(sys.stdlib_module_names)
167+
168+
return blocked
169+
170+
171+
def run_tests(work_dir: Path) -> None:
172+
allowed = _allowed_distributions()
173+
blocked = _blocked_modules(allowed)
174+
print(
175+
f"A clean install may import {len(allowed)} distributions; hiding "
176+
f"{len(blocked)} module names this environment adds"
177+
)
178+
179+
# Somewhere with no checkout, so `import executorch` resolves to the installed package.
180+
# The probe drops that directory from sys.path itself, so no stray file can shadow a real
181+
# module either. Done in the probe rather than with the interpreter's -P flag, which only
182+
# exists from Python 3.11 and the wheel supports 3.10.
183+
neutral_dir = work_dir / "neutral"
184+
neutral_dir.mkdir(parents=True, exist_ok=True)
185+
186+
failures: Dict[str, str] = {}
187+
for module in REQUIRED_IMPORTS:
188+
result = subprocess.run(
189+
[sys.executable, "-c", _IMPORT_PROBE],
190+
input=json.dumps({"module": module, "blocked": sorted(blocked)}),
191+
cwd=os.fspath(neutral_dir),
192+
capture_output=True,
193+
text=True,
194+
check=False,
195+
)
196+
if result.returncode != 0:
197+
failures[module] = (result.stderr or result.stdout).strip()
198+
199+
if failures:
200+
report = "\n\n".join(
201+
f"{name} failed to import:\n{err}" for name, err in failures.items()
202+
)
203+
raise AssertionError(
204+
f"{len(failures)} of {len(REQUIRED_IMPORTS)} modules are not usable from a clean "
205+
f"install, so a user who runs `pip install executorch` cannot use them. Either "
206+
f"declare the package they import, or import it lazily so the module itself still "
207+
f"loads.\n\n{report}"
208+
)
209+
210+
_check_top_level_names()
211+
print(
212+
f"All {len(REQUIRED_IMPORTS)} modules import with only declared dependencies, and the "
213+
f"metadata names only the package."
214+
)
215+
216+
217+
# Reads its arguments from stdin, blocks the named modules, then imports. A blocked module
218+
# raises ModuleNotFoundError exactly as it would be absent, so the traceback shows the import
219+
# chain that wanted it.
220+
_IMPORT_PROBE = """
221+
import importlib.abc, importlib.util, json, os, sys
222+
223+
# Drop the working directory, which `python -c` puts first. The equivalent flag, -P, is only
224+
# available from 3.11.
225+
for entry in ('', '.', os.getcwd()):
226+
while entry in sys.path:
227+
sys.path.remove(entry)
228+
229+
request = json.load(sys.stdin)
230+
blocked = set(request["blocked"])
231+
232+
233+
class _Blocker(importlib.abc.MetaPathFinder):
234+
def find_spec(self, fullname, path=None, target=None):
235+
if fullname.split(".")[0] in blocked:
236+
raise ModuleNotFoundError(
237+
f"No module named {fullname!r} (not declared by the executorch wheel)",
238+
name=fullname,
239+
)
240+
return None
241+
242+
243+
sys.meta_path.insert(0, _Blocker())
244+
importlib.import_module(request["module"])
245+
"""
246+
247+
248+
def _check_top_level_names() -> None:
249+
"""The distribution must claim exactly one top level name.
250+
251+
Read from the installed metadata rather than the wheel file, because that is what any tool
252+
consuming it sees. This once listed twenty build recipe strings beside the real name.
253+
"""
254+
import importlib.metadata as metadata
255+
256+
text = metadata.distribution("executorch").read_text("top_level.txt") or ""
257+
names = [line for line in text.splitlines() if line.strip()]
258+
assert names == ["executorch"], (
259+
f"the installed wheel claims the top level names {names}, but only 'executorch' is "
260+
f"importable, so anything reading this metadata is misled"
261+
)
262+
263+
264+
if __name__ == "__main__":
265+
import tempfile
266+
267+
with tempfile.TemporaryDirectory() as tmp:
268+
run_tests(Path(tmp))
269+
sys.exit(0)

‎.ci/scripts/wheel/test_linux.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,11 +12,17 @@
1212
from pathlib import Path
1313

1414
import test_base
15+
import test_clean_install
1516
import test_cpp_sdk
1617
import test_shared_libraries
1718
from examples.models import Backend, Model
1819

1920
if __name__ == "__main__":
21+
# Before anything else, because this is the check that fails the way a user
22+
# fails: with only the dependencies the wheel declares.
23+
with tempfile.TemporaryDirectory() as work_dir:
24+
test_clean_install.run_tests(Path(work_dir))
25+
2026
if platform.system() == "Linux":
2127
from executorch.extension.pybindings.portable_lib import (
2228
_get_registered_backend_names,

‎.ci/scripts/wheel/test_linux_aarch64.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,11 +9,17 @@
99
from pathlib import Path
1010

1111
import test_base
12+
import test_clean_install
1213
import test_cpp_sdk
1314
import test_shared_libraries
1415
from examples.models import Backend, Model
1516

1617
if __name__ == "__main__":
18+
# Before anything else, because this is the check that fails the way a user
19+
# fails: with only the dependencies the wheel declares.
20+
with tempfile.TemporaryDirectory() as work_dir:
21+
test_clean_install.run_tests(Path(work_dir))
22+
1723
# coremltools does not support linux aarch64 yet and install from the source fails on runtime
1824
# https://github.com/apple/coremltools/issues/1254
1925
# https://github.com/apple/coremltools/issues/2195

‎.ci/scripts/wheel/test_macos.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,11 +11,17 @@
1111
from pathlib import Path
1212

1313
import test_base
14+
import test_clean_install
1415
import test_cpp_sdk
1516
import test_shared_libraries
1617
from examples.models import Backend, Model
1718

1819
if __name__ == "__main__":
20+
# Before anything else, because this is the check that fails the way a user
21+
# fails: with only the dependencies the wheel declares.
22+
with tempfile.TemporaryDirectory() as work_dir:
23+
test_clean_install.run_tests(Path(work_dir))
24+
1925
test_base.test_cmsis_nn_install()
2026

2127
# The wheel ships the runtime, the kernels, the delegate, the thread pool and

‎.ci/scripts/wheel/test_windows.py‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,10 +7,12 @@
77
# LICENSE file in the root directory of this source tree.
88

99
import platform
10+
import tempfile
11+
from pathlib import Path
1012
from typing import List
1113

1214
import test_base
13-
15+
import test_clean_install
1416
import torch
1517
from executorch.backends.xnnpack.partition.xnnpack_partitioner import XnnpackPartitioner
1618
from executorch.examples.models import Backend, Model, MODEL_NAME_TO_MODEL
@@ -68,6 +70,11 @@ def run_tests(model_tests: List[ModelTest]) -> None:
6870

6971

7072
if __name__ == "__main__":
73+
# Before anything else, because this is the check that fails the way a user
74+
# fails: with only the dependencies the wheel declares.
75+
with tempfile.TemporaryDirectory() as work_dir:
76+
test_clean_install.run_tests(Path(work_dir))
77+
7178
if platform.system() == "Windows":
7279
registered = _get_registered_backend_names()
7380
# Vulkan backend is optional: only present when the wheel was built with

0 commit comments

Comments
 (0)