Skip to content

Commit 448cfef

Browse files
Merge pull request #1344 from LalatenduMohanty/feat/1331-typed-download-result
Feat/1331 typed download result
2 parents 6537698 + 5c9ed71 commit 448cfef

15 files changed

Lines changed: 618 additions & 78 deletions

‎docs/concepts/hooks-and-overrides.rst‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,12 @@ a Python module.
5656
See :doc:`/reference/hooks` for the complete list of hooks and their
5757
arguments.
5858

59+
.. versionchanged:: 0.96.0
60+
61+
Configured ``hook-sdist`` and ``hook-prebuilt`` profiles support typed
62+
download results with explicit artifact kinds. See :ref:`download_source_hook`
63+
for the contract, legacy path defaults, and a migration example.
64+
5965
Global Hooks
6066
------------
6167

‎docs/concepts/resolver-architecture.rst‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -97,7 +97,8 @@ CLI commands interact with providers through a common
9797
.. versionchanged:: 0.96.0
9898
The ``hook-sdist`` and ``hook-prebuilt`` profiles create providers through
9999
the required ``get_resolver_provider`` override hook. They do not fall back
100-
to PyPI, and hook-backed artifact downloading remains a separate feature.
100+
to PyPI. Their ``download()`` methods invoke the package's ``download_source``
101+
hook when present and return a validated ``DownloadedSource``.
101102

102103
Per-package settings in YAML can select which provider to use and
103104
configure its parameters (index URL, tag pattern, etc.). When a package
@@ -109,6 +110,14 @@ Otherwise, override plugins can replace the provider for a specific package
109110
via the ``get_resolver_provider`` hook, after which the legacy resolver
110111
settings are used.
111112

113+
Hook downloads accept explicit artifact kinds and adapt legacy paths using
114+
profile-specific defaults. If no download hook exists, the profile selects the
115+
standard sdist or wheel downloader and its destination directory. Hook errors
116+
propagate with context instead of triggering a fallback. The typed result makes
117+
the artifact kind available for source preparation; consuming it in preparation
118+
and integrating prebuilt detection into the bootstrapper are separate steps.
119+
See :ref:`download_source_hook` for the return contract and compatibility rules.
120+
112121
Version Filtering Window
113122
-------------------------
114123

‎docs/reference/config-reference.rst‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,9 @@ Source Resolver
7777
.. autoattribute:: pep517
7878
.. autoattribute:: tarball
7979

80+
.. autoclass:: DownloadedSource
81+
:members: path, kind
82+
8083
.. autoclass:: DownloadKind
8184

8285
.. autoattribute:: sdist
@@ -86,6 +89,13 @@ Source Resolver
8689
.. autoattribute:: any_source
8790
.. autoattribute:: not_available
8891

92+
.. versionchanged:: 0.96.0
93+
94+
Configured hook downloads require a concrete result kind supported by the
95+
selected profile. ``any_source`` and ``not_available`` are not accepted as
96+
result kinds. See :ref:`download_source_hook` for legacy path defaults and
97+
the typed hook contract.
98+
8999

90100
Global Settings
91101
---------------

‎docs/reference/hooks.rst‎

Lines changed: 69 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -219,6 +219,8 @@ Source hooks
219219
were removed. Define a :ref:`resolver_provider <resolver_provider_hook>` hook
220220
to resolve sources.
221221

222+
.. _download_source_hook:
223+
222224
.. autofromagerhook:: default_download_source
223225

224226
The ``download_source()`` function is responsible for downloading the
@@ -228,7 +230,73 @@ Source hooks
228230
evaluated, version of the package being downloaded, the download URL,
229231
and the output directory in which the source should be downloaded.
230232

231-
The return value should be a ``pathlib.Path`` file path to the downloaded source.
233+
For packages without ``source:`` configuration, return a ``pathlib.Path``
234+
to the downloaded source. This legacy contract is unchanged.
235+
236+
.. versionchanged:: 0.96.0
237+
238+
The ``hook-sdist`` and ``hook-prebuilt`` source profiles accept
239+
:class:`~fromager.packagesettings.DownloadedSource` with a local
240+
``pathlib.Path`` and an explicit
241+
:class:`~fromager.packagesettings.DownloadKind`:
242+
243+
* ``hook-sdist`` accepts ``sdist``, ``tarball``, or ``git_checkout``.
244+
* ``hook-prebuilt`` accepts only ``prebuilt_wheel``.
245+
246+
A bare ``Path`` remains supported for legacy hooks: ``hook-sdist``
247+
treats ``.tar.gz`` and ``.zip`` paths as ``sdist``; ``hook-prebuilt``
248+
treats ``.whl`` paths as ``prebuilt_wheel``. Bare directories and other
249+
path forms are rejected. The sdist default preserves legacy archive
250+
semantics; it does not infer the artifact kind from archive contents.
251+
Hooks needing tarball or git-checkout semantics must return a typed
252+
result with the explicit kind. ``any_source`` and ``not_available``
253+
are not valid concrete result kinds.
254+
255+
Invalid return types, malformed typed fields, and kinds incompatible
256+
with the profile raise errors identifying the package and hook.
257+
Hook exceptions are chained to a ``RuntimeError`` with that context;
258+
neither failures nor invalid results trigger a fallback download.
259+
260+
If the download hook is absent, ``hook-sdist`` downloads the candidate
261+
URL as an sdist into ``ctx.sdists_downloads``. ``hook-prebuilt`` downloads
262+
it as a wheel into ``ctx.wheels_prebuilt``. These defaults use the
263+
standard archive and wheel validation. Custom tarball URLs and git
264+
downloads require a download hook. The ``get_resolver_provider`` hook
265+
remains required for both profiles.
266+
267+
To migrate a raw-tarball hook to ``source: {provider: hook-sdist}``, return
268+
its artifact kind explicitly:
269+
270+
.. code-block:: python
271+
272+
import pathlib
273+
274+
from packaging.requirements import Requirement
275+
from packaging.version import Version
276+
277+
from fromager import downloads
278+
from fromager.context import WorkContext
279+
from fromager.packagesettings import DownloadedSource, DownloadKind
280+
281+
def download_source(
282+
ctx: WorkContext,
283+
req: Requirement,
284+
version: Version,
285+
download_url: str,
286+
sdists_downloads_dir: pathlib.Path,
287+
) -> DownloadedSource:
288+
"""Download a raw source tarball and identify its kind."""
289+
path = downloads.download_url(
290+
destination_dir=sdists_downloads_dir,
291+
url=download_url,
292+
destination_filename=f"{req.name}-{version}.tar.gz",
293+
)
294+
return DownloadedSource(path=path, kind=DownloadKind.tarball)
295+
296+
A hook that clones sources returns the checkout directory with
297+
``kind=DownloadKind.git_checkout``. Typed results are supported through
298+
the configured hook profiles; packages using legacy settings still require
299+
a bare ``Path``.
232300

233301
.. autofromagerhook:: default_prepare_source
234302

‎src/fromager/bootstrapper/_bootstrapper.py‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818

1919
from .. import (
2020
bootstrap_requirement_resolver,
21+
packagesettings,
2122
progress,
2223
sources,
2324
threading_utils,
@@ -483,14 +484,13 @@ def _download_source(
483484
source_url: str,
484485
) -> pathlib.Path:
485486
"""Download source for a package."""
486-
download_result = sources.download_source(
487+
result: packagesettings.DownloadedSource = sources.download_source(
487488
ctx=self.ctx,
488489
req=req,
489490
version=resolved_version,
490491
download_url=source_url,
491492
)
492-
source_path: pathlib.Path = download_result[0]
493-
return source_path
493+
return result.path
494494

495495
def _prepare_source(
496496
self,

‎src/fromager/bootstrapper/_prepare_source.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -43,9 +43,9 @@ def _bg_prepare_source(
4343
sdist_root_dir=unpacked / unpacked.stem,
4444
cached_wheel_filename=cached_wheel,
4545
)
46-
source_filename, _ = sources.download_source(
46+
source_filename = sources.download_source(
4747
ctx=ctx, req=req, version=resolved_version, download_url=source_url
48-
)
48+
).path
4949
sdist_root_dir = sources.prepare_source(
5050
ctx=ctx, req=req, source_filename=source_filename, version=resolved_version
5151
)

‎src/fromager/commands/build.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -384,12 +384,12 @@ def _build(
384384
# If we get here and still don't have a wheel filename, then we need to
385385
# build the wheel.
386386
if not wheel_filename:
387-
source_filename, _ = sources.download_source(
387+
source_filename = sources.download_source(
388388
ctx=wkctx,
389389
req=req,
390390
version=resolved_version,
391391
download_url=source_download_url,
392-
)
392+
).path
393393
logger.debug(
394394
"saved sdist of version %s from %s to %s",
395395
resolved_version,

‎src/fromager/commands/step.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -50,12 +50,12 @@ def download_source_archive(
5050
source_url, version = sources.resolve_source(
5151
ctx=wkctx, req=req, sdist_server_url=sdist_server_url
5252
)
53-
filename, _ = sources.download_source(
53+
filename = sources.download_source(
5454
ctx=wkctx,
5555
req=req,
5656
version=version,
5757
download_url=source_url,
58-
)
58+
).path
5959
requirement_ctxvar.reset(token)
6060
print(filename)
6161

‎src/fromager/packagesettings/__init__.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
from ._pbi import PackageBuildInfo
1818
from ._resolver import (
1919
BuildSDist,
20+
DownloadedSource,
2021
DownloadKind,
2122
GitHubTagCloneResolver,
2223
GitHubTagDownloadResolver,
@@ -58,6 +59,7 @@
5859
"BuildSDist",
5960
"DownloadKind",
6061
"DownloadSource",
62+
"DownloadedSource",
6163
"EnvKey",
6264
"EnvVars",
6365
"ExternalCommands",

0 commit comments

Comments
 (0)