Skip to content

Commit dbddba4

Browse files
authored
Publish only the package name in the wheel's top_level.txt (pytorch#22267)
## Summary The pip wheel's metadata advertises twenty top level import names that do not exist. `top_level.txt` inside `executorch-*.dist-info/` is the list of names a package claims to put at the top level of your Python environment. Ours had twenty-one lines, and only the last one was real: ``` @EXECUTORCH_BuiltFile_%CMAKE_CACHE_DIR%/libexecutorch @EXECUTORCH_BuiltFile_%CMAKE_CACHE_DIR%/backends/cuda/%BUILD_TYPE%/libaoti_cuda_shims @EXECUTORCH_BuiltFile_%CMAKE_CACHE_DIR%/backends/mlx/mlx/mlx/backend/metal/kernels/mlx ... 17 more like this ... executorch ``` Those are build recipes. The `%CMAKE_CACHE_DIR%` and `%BUILD_TYPE%` parts are placeholders, substituted much later when the file is actually copied into the wheel. ## Why it happened The wheel ships prebuilt files it only copies, such as the runtime library and `flatc`. Each one is declared to setuptools as an `Extension`. That is deliberate: a non-empty `ext_modules` is how setuptools decides a wheel is platform specific rather than pure Python. Since these entries are not importable modules, each carries a synthetic name beginning with `@`, chosen so it cannot be mistaken for a module path. setuptools has no way to know that. It builds `top_level.txt` from the name of every `ext_modules` entry, so each recipe string was published as written. Nothing breaks when you import ExecuTorch, because imports resolve through `RECORD` and the real directory layout. But any tool that reads `top_level.txt`, and some packaging, vendoring and license scanners do, sees twenty names that resolve to nothing. ## The change A small `Distribution` subclass drops the synthetic names from the metadata setuptools derives. The `@`-prefix is now a named constant, so the filter and the name being filtered cannot drift apart later. ## Test plan Ran `setup.py egg_info` and read the result, in both wheel modes: | | `top_level.txt` | | --- | --- | | before, full wheel | 21 lines, 20 of them recipes | | after, full wheel | `executorch` | | after, minimal wheel | `executorch` | Added `.ci/scripts/tests/test_wheel_top_level_names.py`. It is a unit test rather than a wheel check because the behaviour is a pure function of the names on the entries, so it needs no built wheel. It asserts the published set is exactly `{executorch}`, and carries a second test that fails if the filter ever stops being necessary, so the pair cannot quietly stop checking anything. I confirmed the first test fails when the filter body is removed: ``` AssertionError: assert ['@EXECUTORCH... 'executorch'] == ['executorch'] ```
1 parent 5e46a63 commit dbddba4

2 files changed

Lines changed: 182 additions & 2 deletions

File tree

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
# Copyright (c) Meta Platforms, Inc. and affiliates.
2+
# All rights reserved.
3+
#
4+
# This source code is licensed under the BSD-style license found in the
5+
# LICENSE file in the root directory of this source tree.
6+
7+
"""Unit tests for the top level names the wheel publishes.
8+
9+
Here rather than in the wheel checks, because the behaviour under test is how setuptools
10+
turns ext_modules into metadata, which is a pure function of the names on those entries.
11+
Asserting it against a built wheel would need a full build to exercise one string filter,
12+
and would not run on a machine that had not built one.
13+
14+
The defect this covers shipped. A prebuilt file the wheel only copies is declared as an
15+
Extension, because a non-empty ext_modules is how setuptools decides a wheel is platform
16+
specific, and those entries carry a synthetic name rather than a module path. setuptools
17+
derives top_level.txt from every entry's name, so twenty recipe strings such as
18+
"@EXECUTORCH_BuiltFile_%CMAKE_CACHE_DIR%/libexecutorch" were published as top level
19+
import names on every row, leaving one correct line among them.
20+
21+
setup.py is read rather than imported. It calls setup() at module scope, so importing it
22+
under a test runner hands setup() the runner's own arguments and the session dies on an
23+
invalid command name.
24+
"""
25+
26+
import ast
27+
from pathlib import Path
28+
from typing import Dict
29+
30+
from setuptools import Distribution, Extension
31+
32+
SETUP_PY = Path(__file__).resolve().parents[3] / "setup.py"
33+
34+
35+
def _load_from_setup_py() -> Dict[str, object]:
36+
"""The distribution class and the name prefix, taken from setup.py's own source.
37+
38+
Only the two class definitions are executed, so nothing in setup.py's module level
39+
build logic runs.
40+
"""
41+
module = ast.parse(SETUP_PY.read_text())
42+
wanted = ("_BaseExtension", "_ExecuTorchDistribution")
43+
classes = [
44+
node
45+
for node in module.body
46+
if isinstance(node, ast.ClassDef) and node.name in wanted
47+
]
48+
assert {node.name for node in classes} == set(
49+
wanted
50+
), f"setup.py no longer defines {wanted}, so this test is checking nothing"
51+
52+
# _BaseExtension's body refers to names setup.py imports, and only the prefix
53+
# constant matters here, so keep the assignments and drop everything else.
54+
for node in classes:
55+
if node.name == "_BaseExtension":
56+
node.body = [
57+
statement
58+
for statement in node.body
59+
if isinstance(statement, (ast.Assign, ast.AnnAssign))
60+
]
61+
62+
namespace: Dict[str, object] = {
63+
"Distribution": Distribution,
64+
"Extension": Extension,
65+
}
66+
exec(
67+
compile(ast.Module(body=classes, type_ignores=[]), str(SETUP_PY), "exec"),
68+
namespace,
69+
)
70+
return namespace
71+
72+
73+
_NAMESPACE = _load_from_setup_py()
74+
_ExecuTorchDistribution = _NAMESPACE["_ExecuTorchDistribution"]
75+
PREFIX = _NAMESPACE["_BaseExtension"].SYNTHETIC_NAME_PREFIX
76+
77+
# The shapes the real ext_modules list produces, including the two that made the
78+
# published output confusing to read: setuptools splits a name at its first dot, so a
79+
# library file name loses its suffix, and a copy target keeps its ":dst" half when the
80+
# recipe has no dot at all.
81+
SYNTHETIC_NAMES = [
82+
f"{PREFIX}%CMAKE_CACHE_DIR%/libexecutorch",
83+
f"{PREFIX}%CMAKE_CACHE_DIR%/backends/cuda/%BUILD_TYPE%/libaoti_cuda_shims",
84+
f"{PREFIX}%CMAKE_CACHE_DIR%/backends/mlx/mlx/mlx/backend/metal/kernels/mlx",
85+
f"{PREFIX}%CMAKE_CACHE_DIR%/third-party/flatc_ep/bin/flatc:executorch/data/bin/",
86+
f"{PREFIX}tools/wheel/pip_data_bin_init.py.in:executorch/data/bin/__init__.py",
87+
]
88+
89+
# A real Python extension, which must survive the filter.
90+
REAL_NAME = "executorch.extension.pybindings._portable_lib"
91+
92+
93+
def _top_level_names(distribution_class):
94+
"""The top level names setuptools would write for a distribution.
95+
96+
Mirrors setuptools.command.egg_info.write_toplevel_names, which is what turns the
97+
names into the published file.
98+
"""
99+
distribution = distribution_class(
100+
{
101+
"name": "executorch",
102+
"packages": ["executorch"],
103+
"ext_modules": [
104+
Extension(name=name, sources=[])
105+
for name in [*SYNTHETIC_NAMES, REAL_NAME]
106+
],
107+
}
108+
)
109+
return sorted(
110+
dict.fromkeys(
111+
name.split(".", 1)[0] for name in distribution.iter_distribution_names()
112+
)
113+
)
114+
115+
116+
def test_publishes_only_the_package_name():
117+
assert _top_level_names(_ExecuTorchDistribution) == ["executorch"]
118+
119+
120+
def test_setup_is_told_to_use_the_distribution():
121+
"""The filter only takes effect if setup() is actually given the class.
122+
123+
Without this, deleting the `distclass=` argument leaves the test above green while the
124+
published metadata goes back to listing every recipe, which is the whole defect.
125+
"""
126+
module = ast.parse(SETUP_PY.read_text())
127+
setup_calls = [
128+
node
129+
for statement in module.body
130+
if isinstance(statement, ast.Expr)
131+
for node in ast.walk(statement)
132+
if isinstance(node, ast.Call)
133+
and isinstance(node.func, ast.Name)
134+
and node.func.id == "setup"
135+
]
136+
assert len(setup_calls) == 1, "expected exactly one module level setup() call"
137+
distclass = [
138+
keyword.value
139+
for keyword in setup_calls[0].keywords
140+
if keyword.arg == "distclass"
141+
]
142+
assert distclass, "setup() is not given a distclass, so the filter never runs"
143+
assert isinstance(distclass[0], ast.Name)
144+
assert distclass[0].id == "_ExecuTorchDistribution"
145+
146+
147+
def test_plain_distribution_publishes_the_recipes():
148+
"""Guards the test above, which passes trivially if the filter is unnecessary.
149+
150+
Without this, removing the filter and the synthetic names together would leave a
151+
green test that no longer checks anything.
152+
"""
153+
published = _top_level_names(Distribution)
154+
assert [name for name in published if name.startswith(PREFIX)]
155+
assert published != ["executorch"]

‎setup.py‎

Lines changed: 27 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,7 @@
7676
raise ImportError(f"Module spec has no loader for {_install_utils_path}")
7777
_spec.loader.exec_module(install_utils)
7878

79-
from setuptools import Extension, find_namespace_packages, setup
79+
from setuptools import Distribution, Extension, find_namespace_packages, setup
8080
from setuptools.command.build import build
8181
from setuptools.command.build_ext import build_ext
8282
from setuptools.command.build_py import build_py
@@ -863,6 +863,12 @@ def get_executable_name(name: str) -> str:
863863
class _BaseExtension(Extension):
864864
"""A base class that maps an abstract source to an abstract destination."""
865865

866+
# Prefix of the synthetic name a BuiltFile carries. A built file is not a Python
867+
# module, so its name is deliberately not a module path, and setuptools has to be
868+
# able to tell it apart from a real extension. See _ExecuTorchDistribution, which
869+
# keeps these names out of the metadata setuptools derives from ext_modules.
870+
SYNTHETIC_NAME_PREFIX = "@EXECUTORCH_BuiltFile_"
871+
866872
def __init__(
867873
self,
868874
src: str,
@@ -1018,7 +1024,7 @@ def __init__(
10181024
super().__init__(
10191025
src=src,
10201026
dst=dst,
1021-
name=f"@EXECUTORCH_BuiltFile_{src}:{dst}",
1027+
name=f"{_BaseExtension.SYNTHETIC_NAME_PREFIX}{src}:{dst}",
10221028
dependent_cmake_flags=dependent_cmake_flags,
10231029
)
10241030

@@ -2218,6 +2224,24 @@ def run(self): # noqa C901
22182224
build.run(self)
22192225

22202226

2227+
class _ExecuTorchDistribution(Distribution):
2228+
"""A Distribution that hides the synthetic BuiltFile names from metadata.
2229+
2230+
A prebuilt file that the wheel merely copies is declared as an Extension, because
2231+
a non-empty ext_modules is how setuptools decides a wheel is platform specific.
2232+
Those entries are not Python modules, so they carry a synthetic name rather than a
2233+
module path. setuptools does not know that: it derives top_level.txt from every
2234+
ext_modules entry's name, so each recipe string was landing in the published
2235+
metadata as a top level import name.
2236+
"""
2237+
2238+
def iter_distribution_names(self):
2239+
for name in super().iter_distribution_names():
2240+
if name.startswith(_BaseExtension.SYNTHETIC_NAME_PREFIX):
2241+
continue
2242+
yield name
2243+
2244+
22212245
setup_kwargs = {}
22222246
if _is_minimal_build():
22232247
setup_kwargs["packages"] = _minimal_packages()
@@ -2230,6 +2254,7 @@ def run(self): # noqa C901
22302254

22312255
setup(
22322256
version=Version.string(),
2257+
distclass=_ExecuTorchDistribution,
22332258
cmdclass={
22342259
"build": CustomBuild,
22352260
"build_ext": InstallerBuildExt,

0 commit comments

Comments
 (0)