Skip to content

Commit e8286a1

Browse files
Merge pull request #1393 from smoparth/codex/issue-1208-prebuilt-resolvers
fix(resolver): honor configured prebuilt wheel resolvers
2 parents a7bc164 + c9cc5dd commit e8286a1

18 files changed

Lines changed: 1276 additions & 92 deletions

‎docs/concepts/bootstrapper-architecture.rst‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,12 @@ Bootstrap Phase Flow
1616

1717
Every package passes through a sequence of phases. Source packages
1818
traverse the full pipeline; prebuilt wheels skip the build phases.
19+
``pypi-prebuilt`` and ``hook-prebuilt`` use the selected source resolver to
20+
download a fresh wheel in ``PrepareSource``. The legacy ``pre_built`` setting
21+
keeps its existing wheel download path. Both routes update the local mirror
22+
and record the package as prebuilt in the graph and build order.
23+
When a local wheel is already available, bootstrap reuses it before asking
24+
the selected provider to resolve a fresh wheel.
1925

2026
.. code-block:: text
2127

‎docs/concepts/package-settings.rst‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,16 @@ cannot be combined with package-level ``download_source`` or
4242
``resolver_dist``. An explicit ``pre_built: false`` also conflicts;
4343
an omitted ``pre_built`` does not.
4444

45+
``PackageBuildInfo.pre_built`` follows the selected ``source`` resolver's
46+
``resolves_prebuilt_wheel`` capability. Without ``source``, it reads the
47+
legacy variant ``pre_built`` setting. Configured prebuilt resolvers download
48+
fresh wheels through their ``download()`` contract; legacy prebuilt packages
49+
keep the wheel downloader.
50+
51+
.. versionchanged:: 0.102.0
52+
Configured prebuilt source resolvers now select the prebuilt pipeline
53+
during bootstrap and later builds.
54+
4555
.. versionchanged:: 0.102.0
4656
Source and legacy settings mixed across package and variant levels
4757
now raise a validation error when package settings load.

‎docs/concepts/resolver-architecture.rst‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,8 @@ the cache already contains a matching version.
4343
The configured wheel cache server is not a version-resolution fallback.
4444
After a version has been selected, the bootstrapper checks the cache
4545
server separately for a previously built wheel.
46+
For prebuilt packages, the local wheel server can supply a cached wheel
47+
before a fresh resolution through the selected source provider.
4648

4749
Provider Hierarchy
4850
------------------
@@ -132,7 +134,15 @@ The separate ``step prepare-source`` invocation accepts ``--source-kind`` and
132134
``step download-source-archive --output-format json`` exposes that result's
133135
actual path and kind without changing its default path-only output.
134136
Without ``--source-kind``, it keeps the legacy archive behavior.
135-
Integrating prebuilt detection into the bootstrapper remains a separate step.
137+
Configured prebuilt resolvers select the prebuilt bootstrap path. Fresh
138+
external wheels use their ``download()`` contract and are staged in the
139+
prebuilt wheel directory. Bootstrap, build, and download-sequence use the
140+
direct wheel downloader for local mirror URLs, so they do not rerun the
141+
configured download hook. Test-mode fallback also downloads directly.
142+
When a fresh configured wheel shares a filename with an older cached wheel,
143+
the local mirror and later build replay use the fresh staged wheel.
144+
Legacy build downloads still pass through the build wheel directory before
145+
the mirror moves them to downloads.
136146
See :ref:`download_source_hook` for the return contract and compatibility rules.
137147

138148
.. versionchanged:: 0.102.0
@@ -143,7 +153,9 @@ See :ref:`download_source_hook` for the return contract and compatibility rules.
143153
``build_sdist: tarball``. PyPI sdist resolvers
144154
repack a tarball. Both configured strategies take precedence over an installed
145155
``build_sdist`` override and log a warning when it is ignored. Legacy
146-
settings continue to use the override.
156+
settings continue to use the override. Configured prebuilt profiles use
157+
their source provider after a local cache check, and local mirror URLs
158+
bypass configured download hooks during replay.
147159

148160
Version Filtering Window
149161
-------------------------

‎src/fromager/bootstrap_requirement_resolver.py‎

Lines changed: 39 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -10,9 +10,11 @@
1010
import threading
1111
import typing
1212

13+
import requests.exceptions
1314
from packaging.requirements import Requirement
1415
from packaging.utils import NormalizedName, canonicalize_name
1516
from packaging.version import Version
17+
from resolvelib.resolvers import ResolverException
1618

1719
from . import resolver, sources, wheels
1820
from .dependency_graph import DependencyGraph
@@ -171,13 +173,17 @@ def _resolve_and_extend(
171173
parent_req=parent_req,
172174
)
173175

176+
source_resolver = self.ctx.package_build_info(req).source_resolver
177+
configured_prebuilt = (
178+
source_resolver is not None and source_resolver.resolves_prebuilt_wheel
179+
)
174180
if cached_resolution:
175181
logger.debug(
176182
f"resolved from previous bootstrap: {len(cached_resolution)} version(s)"
177183
)
178184
results = cached_resolution
179-
elif pre_built:
180-
# Resolve prebuilt wheel
185+
elif pre_built and not configured_prebuilt:
186+
# Legacy prebuilt resolution also serves test-mode source-build fallback.
181187
wheel_server_urls = wheels.get_wheel_server_urls(
182188
self.ctx, req, cache_wheel_server_url=resolver.PYPI_SERVER_URL
183189
)
@@ -188,30 +194,39 @@ def _resolve_and_extend(
188194
req_type=req_type,
189195
)
190196
else:
191-
# Resolve source (sdist)
192-
provider = sources.get_source_provider(
193-
ctx=self.ctx,
194-
req=req,
195-
sdist_server_url=resolver.PYPI_SERVER_URL,
196-
req_type=req_type,
197-
)
198-
max_age_cutoff = resolver._compute_max_age_cutoff(self.ctx)
199-
age_fallback = (
200-
resolver.AgeFallback.NEWEST
201-
if self.multiple_versions
202-
else resolver.AgeFallback.ALL
203-
)
204-
results = resolver.find_all_matching_from_provider(
205-
provider,
206-
req,
207-
max_age_cutoff=max_age_cutoff,
208-
age_fallback=age_fallback,
209-
)
197+
results = []
198+
if configured_prebuilt and self.ctx.wheel_server_url:
199+
try:
200+
results = wheels.resolve_all_cached_wheels(
201+
ctx=self.ctx,
202+
req=req,
203+
cache_server_url=self.ctx.wheel_server_url,
204+
)
205+
except ResolverException:
206+
logger.debug("no matching wheel in local cache")
207+
except requests.exceptions.RequestException as err:
208+
logger.warning("could not check local wheel cache: %s", err)
210209

211210
if not results:
212-
logger.warning(
213-
"resolver returned no results (%s version mode)",
214-
"multiple" if self.multiple_versions else "single",
211+
# The configured source provider resolves either source archives
212+
# or prebuilt wheels, according to its selected capability.
213+
provider = sources.get_source_provider(
214+
ctx=self.ctx,
215+
req=req,
216+
sdist_server_url=resolver.PYPI_SERVER_URL,
217+
req_type=req_type,
218+
)
219+
max_age_cutoff = resolver._compute_max_age_cutoff(self.ctx)
220+
age_fallback = (
221+
resolver.AgeFallback.NEWEST
222+
if self.multiple_versions
223+
else resolver.AgeFallback.ALL
224+
)
225+
results = resolver.find_all_matching_from_provider(
226+
provider,
227+
req,
228+
max_age_cutoff=max_age_cutoff,
229+
age_fallback=age_fallback,
215230
)
216231

217232
if results:

‎src/fromager/bootstrapper/_bootstrapper.py‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -470,8 +470,12 @@ def _download_prebuilt(
470470
resolved_version: Version,
471471
wheel_url: str,
472472
) -> tuple[pathlib.Path, pathlib.Path]:
473-
result = _cache.bg_prepare_prebuilt(
474-
self.ctx, req, req_type, resolved_version, wheel_url
473+
result = _cache.bg_prepare_test_mode_prebuilt(
474+
self.ctx,
475+
req,
476+
req_type,
477+
resolved_version,
478+
wheel_url,
475479
)
476480
assert result.wheel_filename is not None
477481
assert result.unpack_dir is not None

‎src/fromager/bootstrapper/_cache.py‎

Lines changed: 25 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -246,10 +246,34 @@ def bg_prepare_prebuilt(
246246
resolved_version: Version,
247247
wheel_url: str,
248248
) -> PreparedSourceData:
249-
"""Background-safe prebuilt download: no Bootstrapper state accessed."""
249+
"""Download a prebuilt wheel and make it available to the local mirror."""
250250
# Thread-safe: paths include {req.name}-{resolved_version} (unique per package),
251251
# mkdir uses exist_ok=True (atomic), and update_wheel_mirror() is already locked.
252252
logger.info(f"using pre-built wheel for {req_type} requirement")
253+
wheel_filename = wheels.download_prebuilt_wheel(
254+
ctx=ctx,
255+
req=req,
256+
version=resolved_version,
257+
download_url=wheel_url,
258+
direct_download_directory=ctx.wheels_prebuilt,
259+
)
260+
unpack_dir = _create_unpack_dir(ctx.work_dir, req, resolved_version)
261+
if ctx.package_build_info(req).source_resolver is not None:
262+
server.update_wheel_mirror(ctx, preferred_prebuilt_wheel=wheel_filename)
263+
else:
264+
server.update_wheel_mirror(ctx)
265+
return PreparedSourceData(wheel_filename=wheel_filename, unpack_dir=unpack_dir)
266+
267+
268+
def bg_prepare_test_mode_prebuilt(
269+
ctx: context.WorkContext,
270+
req: Requirement,
271+
req_type: RequirementType,
272+
resolved_version: Version,
273+
wheel_url: str,
274+
) -> PreparedSourceData:
275+
"""Download a test-mode fallback wheel directly and update the mirror."""
276+
logger.info(f"using pre-built wheel for {req_type} requirement")
253277
wheel_filename = wheels.download_wheel(req, wheel_url, ctx.wheels_prebuilt)
254278
unpack_dir = _create_unpack_dir(ctx.work_dir, req, resolved_version)
255279
server.update_wheel_mirror(ctx)

‎src/fromager/commands/build.py‎

Lines changed: 35 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -362,17 +362,18 @@ def _build(
362362
if prebuilt:
363363
if not wheel_filename:
364364
logger.info("downloading prebuilt wheel")
365-
wheel_filename = wheels.download_wheel(
365+
wheel_filename = wheels.download_prebuilt_wheel(
366+
ctx=wkctx,
366367
req=req,
367-
wheel_url=source_download_url,
368-
output_directory=wkctx.wheels_build,
368+
version=resolved_version,
369+
download_url=source_download_url,
370+
direct_download_directory=wkctx.wheels_build,
369371
)
370372
else:
371373
# already downloaded prebuilt wheel
372374
use_exiting_wheel = True
373375

374-
# Run hooks for prebuilt wheels. At this point wheel_filename should
375-
# be set either from _is_wheel_built() or download_wheel().
376+
# Run hooks for both cached and freshly downloaded prebuilt wheels.
376377
hooks.run_prebuilt_wheel_hooks(
377378
ctx=wkctx,
378379
req=req,
@@ -446,11 +447,19 @@ def _build(
446447
root_logger.removeHandler(file_handler)
447448
file_handler.close()
448449

449-
server.update_wheel_mirror(wkctx)
450+
if (
451+
prebuilt
452+
and pbi.source_resolver is not None
453+
and wheel_filename.parent == wkctx.wheels_prebuilt
454+
):
455+
server.update_wheel_mirror(wkctx, preferred_prebuilt_wheel=wheel_filename)
456+
else:
457+
server.update_wheel_mirror(wkctx)
450458

451-
# After we update the wheel mirror, the built file has
452-
# moved to a new directory.
453-
wheel_filename = wkctx.wheels_downloads / wheel_filename.name
459+
# The mirror moves wheels from the build directory to downloads;
460+
# configured prebuilt wheels stay in the prebuilt directory.
461+
if wheel_filename.parent != wkctx.wheels_prebuilt:
462+
wheel_filename = wkctx.wheels_downloads / wheel_filename.name
454463

455464
return BuildSequenceEntry(
456465
name=canonicalize_name(req.name),
@@ -529,8 +538,8 @@ def _is_wheel_built(
529538
logger.info("found candidate wheel %s", url)
530539
try:
531540
wheel_basename = downloads.extract_filename_from_url(url)
532-
_, _, build_tag_from_name, wheel_tags = parse_wheel_filename(
533-
wheel_basename
541+
normalized_name, _, build_tag_from_name, wheel_tags = (
542+
parse_wheel_filename(wheel_basename)
534543
)
535544
except Exception:
536545
logger.debug("could not parse candidate wheel %s", url, exc_info=True)
@@ -567,9 +576,23 @@ def _is_wheel_built(
567576
continue
568577

569578
wheel_filename: pathlib.Path | None = None
570-
if wkctx.wheel_server_url and url.startswith(wkctx.wheel_server_url):
579+
if wheels.is_local_wheel_url(url, wkctx.wheel_server_url):
571580
logging.debug("found wheel on local server")
572581
wheel_filename = wkctx.wheels_downloads / wheel_basename
582+
mirror_wheel = wkctx.wheel_server_dir / normalized_name / wheel_basename
583+
if mirror_wheel.is_symlink():
584+
mirror_target = mirror_wheel.resolve()
585+
if (
586+
mirror_target.parent
587+
in (
588+
wkctx.wheels_downloads,
589+
wkctx.wheels_prebuilt,
590+
)
591+
and mirror_target.is_file()
592+
):
593+
# The mirror may point to a fresh prebuilt wheel while
594+
# an older copy remains in downloads.
595+
wheel_filename = mirror_target
573596
if not wheel_filename.exists():
574597
logger.info("wheel not found in local cache, preparing to download")
575598
wheel_filename = None

0 commit comments

Comments
 (0)