Repository navigation
Commit 5410b1a
Stop shipping unusable MKL search paths in the Linux wheel (pytorch#22541)
Fixes pytorch#21611
## The problem, in plain terms
Five shared libraries in the Linux wheel search three directories that
exist on nobody's machine:
```
/lib/intel64
/lib/intel64_win
/lib/win-x64
```
Two of them name a Windows layout, in a Linux wheel.
They come from PyTorch. Its exported CMake package creates a
`caffe2::mkl` imported target with a hardcoded list of link directories,
and linking torch brings them in even though this project never asks for
MKL:
```
${MKL_ROOT}/lib ${MKL_ROOT}/lib/intel64 ${MKL_ROOT}/lib/intel64_win ${MKL_ROOT}/lib/win-x64
```
`MKL_ROOT` resolves to nothing here, so what the linker records is left
anchored at the filesystem root. Packaging copies the built libraries
out of the build tree rather than installing them, so whatever the
linker recorded ships as is. The bare `/lib` form does not survive,
because CMake filters its own implicit link directories out of the link
line, which is why three entries appear rather than four.
This only affects Linux x86_64. The aarch64 nightly records no absolute
entries at all, since MKL is not found there, and the macOS and Windows
wheels record none either.
## Why it is worth fixing
Nothing needs those directories. No shipped library names an MKL or
OpenMP runtime among its dependencies, so nothing resolves through them.
They are not merely untidy either. They sit ahead of the relative
entries packaging appends, and the loader searches in order, so a user
who happens to have a matching directory resolves a library from there
instead of from the one the wheel installed. That is the same shadowing
the release check already rejects a CUDA toolkit prefix for.
## What changed
Two small pieces.
Packaging now drops these entries, in the same place the other unusable
ones are already dropped. The match is deliberately narrow: only the
exact `/lib/<arch>` form that an empty prefix produces. A real MKL
installation spells the same arch directory below a prefix, as
`/opt/intel/mkl/lib/intel64`, and that is a directory the environment
genuinely provides, so it is kept.
The release check that rejects absolute search paths listed these three
as allowed. That is why they shipped while a check whose whole purpose
is rejecting absolute paths reported the wheel clean. It now rejects the
empty prefix form specifically, and still accepts a real installation's
prefixed directory, so the two halves agree rather than contradict each
other.
## Test plan
New unit tests in `.ci/scripts/tests/test_runtime_path_filter.py`. They
read the functions out of `setup.py`, so they exercise the code that
ships rather than a copy, and they run on every pull request through the
existing unit test job. They need no wheel build, which is what the
previous check could not manage: it could only see this after a full
build, and only on the platform that built one.
```
pytest .ci/scripts/tests/test_runtime_path_filter.py
27 passed
```
They cover both directions, because a filter that satisfies either one
alone is wrong:
- the three unresolved entries are dropped
- a real MKL installation and ordinary system directories are kept
- every relative entry on a shipped library survives, so a library can
still find its siblings
- the absolute torch directory is kept when it is a library's only route
to torch, which is what stops this becoming a blanket rule that breaks
importing
I checked by mutation that each part of the fix is load bearing:
| what I broke | tests that failed |
| --- | ---: |
| removed the packaging filter | 4 |
| removed the release check's maths rejection | 3 |
| removed its build directory branch | 2 |
| removed its catch-all rejection | 1 |
| severed the call that applies it to a shipped library | 1 |
| reordered its guards so a build directory is accepted | 1 |
| narrowed what it accepts, refusing a real install | 3 |
| made the packaging filter reject an ordinary system directory | 1 |
| added an arch to the shared constant, untaught to the check | 2 |
The release check's per-entry decision is a small module level function,
so the unit test calls the same code the wheel check runs and compares
the reason it returns. Asserting only that a path was rejected was not
enough: the check rejects every absolute path it does not recognise, so
an unknown arch passed for the wrong reason and two of its three
branches could each be deleted on their own.
I also measured the published artifacts directly, reading the recorded
search paths out of every shared library in each wheel:
| wheel | libraries with a search path | unusable entries |
| --- | ---: | ---: |
| 1.4.1 release, CPython 3.12 | 6 | 0 |
| current nightly, CPU, CPython 3.12 | 18 | 15 |
| current nightly, CUDA, CPython 3.13 | 21 | 15 |
| current nightly, CPU, aarch64 | 16 | 0 |
The 1.4.1 release predates the code changed here. Both nightly rows show
the same 15 entries across the same five libraries, which is what this
removes. Reading those same libraries' declared dependencies shows no
MKL or OpenMP runtime in any of them.
## Scope
This covers the first of the two things the issue suggests: not shipping
a runtime search path a user cannot use. It does not do the second,
which is to resolve MKL through something the wheel or its declared
dependencies own. That one belongs upstream, since the directory list is
set in PyTorch's own CMake package, and its own comment there marks it
as a hack. Nothing in this wheel links MKL, so there is no dependency
here left to redirect.
## What I did not verify
No wheel was built for this. The change is to the filter those builds
already run, and the tests exercise it directly. The link line that
emits the paths is unchanged; the entries are dropped at packaging, the
same way the CUDA toolkit and torch directories already are.
Co-authored-by: PyTorch Bot <pytorchbot@users.noreply.github.com>1 parent 89e108c commit 5410b1a
3 files changed
Lines changed: 445 additions & 68 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
0 commit comments