For more complex customization requirements than are supported by the configuration file, create an override plugin.
Plugins are registered using entry
points
so they can be discovered and loaded at runtime. In pyproject.toml,
configure the entry point in the
project.entry-points."fromager.project_overrides" namespace to
link the :ref:`canonical distribution name <canonical-distribution-names>`
to an importable module.
[project.entry-points."fromager.project_overrides"]
flit_core = "package_plugins.flit_core"
pyarrow = "package_plugins.pyarrow"
torch = "package_plugins.torch"
triton = "package_plugins.triton"The plugins are treated as providing overriding implementations of functions with default implementations, so it is only necessary to implement the functions needed to make it possible to build the package.
.. py:currentmodule:: fromager.packagesettings
.. autofromagerhook:: default_update_extra_environ
The ``update_extra_environ`` can modify the extra environment variables
from settings file with dynamic values. The hook must update the
``extra_environ`` dict in-place.
The hook is called multiple times during a build. The ``version``
argument is *None* for ``get_build_backend_dependencies`` and
``get_build_sdist_dependencies``. For ``get_install_dependencies_of_sdist``,
``build_sdist``, and ``build_wheel``, the ``version`` argument
contains the resolved version.
.. versionadded:: 0.60
.. py:currentmodule:: fromager.dependencies
.. autofromagerhook:: default_get_build_system_dependencies
The ``get_build_system_dependencies()`` function should return the PEP 517
build dependencies for a package.
The arguments are the ``WorkContext``, the ``Requirement`` being evaluated,
and the ``Path`` to the root of the source tree.
The return value is an iterable of requirement specification strings for
build system dependencies for the package. The caller is responsible for
evaluating the requirements with the current build environment settings to
determine if they are actually needed.
.. autofromagerhook:: default_get_build_backend_dependencies
The ``get_build_backend_dependencies()`` function should return the PEP
517 build dependencies for a package.
The arguments are the ``WorkContext``, the ``Requirement`` being
evaluated, and the `Path` to the root of the source tree.
The return value is an iterable of requirement specification strings
for build backend dependencies for the package. The caller is
responsible for evaluating the requirements with the current build
environment settings to determine if they are actually needed.
.. autofromagerhook:: default_get_build_sdist_dependencies
The ``get_build_sdist_dependencies()`` function should return the PEP 517
dependencies for building the source distribution for a package.
The return value is an iterable of requirement specification strings
for build backend dependencies for the package. The caller is
responsible for evaluating the requirements with the current build
environment settings to determine if they are actually needed.
.. py:currentmodule:: fromager.finders
.. autofromagerhook:: default_expected_source_archive_name
The ``expected_source_archive_name()`` function is used to re-discover a
source archive downloaded by a previous step, especially if the
filename does not match the standard naming scheme for an sdist.
The arguments are the ``Requirement`` being evaluated and the version to
look for.
The return value should be a string with the base filename (no paths)
for the archive.
.. autofromagerhook:: default_expected_source_directory_name
The ``expected_source_directory_name()`` function is used to re-discover
the location of a source tree prepared by a previous step, especially
if the name does not match the standard naming scheme for an sdist.
The arguments are the ``Requirement`` being evaluated and the version to
look for.
The return value should be a string with the name of the source root
directory relative to the ``ctx.work_dir`` where it was prepared.
For a configured Git checkout, this path uses the canonical distribution
name. Return the actual prepared path; an override bypasses the standard
source directory search.
.. currentmodule:: fromager.resolver
.. autofromagerhook:: default_resolver_provider
The ``get_resolver_provider()`` function allows an override to change
the way requirement specifications are converted to fixed
versions. The default implementation looks for published versions on a
Python package index. Most overrides do not need to implement this
hook unless they are building versions of packages not released to
https://pypi.org.
For examples, refer to ``fromager.resolver.PyPIProvider``,
``fromager.resolver.GitHubTagProvider``, and ``fromager.resolver.GitLabTagProvider``.
The arguments are the ``WorkContext``, the ``Requirement`` being
evaluated, a boolean indicating whether source distributions should be
included, a boolean indicating whether built wheels should be
included, and the URL for the package index. The hook also receives
``req_type`` and ``ignore_platform`` when those parameters are supported
by its signature.
The ``hook-sdist`` and ``hook-prebuilt`` source profiles call this hook
when it exists. They pass the following keyword arguments:
* ``ctx``
* ``req``
* ``include_sdists``
* ``include_wheels``
* ``sdist_server_url``
* ``req_type``
* ``ignore_platform``
Older hooks may omit newer arguments; unsupported keyword arguments are
filtered for compatibility. A return value that is not a
:class:`~fromager.resolver.BaseProvider` is an error.
Without this hook, a hook profile finds versions on its ``index_url`` and
the ``download_source`` hook gets the artifact. A package with neither
hook is an error; use ``pypi-sdist`` or ``pypi-prebuilt`` instead.
.. versionchanged:: 0.103.0
``hook-sdist`` and ``hook-prebuilt`` no longer require this hook when
the package has a ``download_source`` hook.
For a configured hook profile, ``sdist_server_url`` is the profile's
``index_url``. ``hook-sdist`` defaults that field to PyPI;
``hook-prebuilt`` requires it so a pass-through hook cannot silently
resolve public PyPI wheels. The hook may use, ignore, or replace the URL.
Command-line and caller URLs are ignored for configured ``source:``
profiles; Fromager logs a warning when it ignores a URL other than PyPI.
Other legacy calls to this hook, such as prebuilt-wheel fallback and
diagnostic resolution, continue to pass their existing wheel-server or
PyPI URLs. They are not configured source resolution.
The return value must be an instance of
:class:`~fromager.resolver.BaseProvider`.
The expectation is that it acts as an engine for any sort of package resolution
whether it is for wheels or sources. The provider can
therefore use any value as the "URL" that will help it decide what to
download. For example, the ``GitHubTagProvider`` returns the actual tag
name in case that is different from the version number encoded within
that tag name.
The ``GenericProvider`` is a convenient base class, or can be instantiated
directly if given a ``version_source`` callable that returns an iterator of
version values as ``str`` or ``Version`` objects.
.. code-block:: python
from fromager import resolver
VERSIONS = {
"https://pkg.example/pkg-1.0.tar.gz": "1.0",
"https://pkg.example/pkg-2.0.tar.gz": "2.0",
}
def _version_source(
identifier: str,
) -> typing.Iterable[tuple[str, Version]]:
return VERSIONS.items()
def get_resolver_provider(
ctx, req, include_sdists, include_wheels, sdist_server_url,
req_type=None, ignore_platform=False,
):
return resolver.GenericProvider(version_source=_version_source, constraints=ctx.constraints)
``GenericProvider``, ``GitHubTagProvider``, and ``GitLabTagProvider`` take
an optional ``matcher`` argument. The matcher can be a
``Callable[[str, str], Version | None]`` or a regular expression pattern
object from ``re.compile`` The default match function attempts to convert
a tag into a ``Version`` and ignores all errors.
.. code-block:: python
def custom_tag_match(identifier: str, item: str) -> Version | None:
# project-1_2_3 -> 1.2.3
if item.startswith("project-"):
return Version(item[8:].replace("_", "."))
# ignore other tags
return None
.. currentmodule:: fromager.sources
.. versionremoved:: 0.80.0 The ``resolve_source`` hook and ``default_resolve_source`` function were removed. Define a :ref:`resolver_provider <resolver_provider_hook>` hook to resolve sources.
.. autofromagerhook:: default_download_source
The ``download_source()`` function is responsible for downloading the
source from a URL.
The arguments are the ``WorkContext``, the ``Requirement`` being
evaluated, version of the package being downloaded, the download URL,
and the output directory in which the source should be downloaded.
``download_url`` is the value from the resolver provider. It can be a
ref, such as a commit SHA, instead of a URL.
For packages without ``source:`` configuration, return a ``pathlib.Path``
to the downloaded source. This legacy contract is unchanged.
.. versionchanged:: 0.103.0
The ``hook-sdist`` and ``hook-prebuilt`` source profiles accept
:class:`~fromager.packagesettings.DownloadedSource` with a local
``pathlib.Path`` and an explicit
:class:`~fromager.packagesettings.DownloadKind`:
* ``hook-sdist`` accepts ``sdist``, ``tarball``, or ``git_checkout``.
* ``hook-prebuilt`` accepts only ``prebuilt_wheel``.
A bare ``Path`` remains supported for legacy hooks: ``hook-sdist``
treats ``.tar.gz`` and ``.zip`` paths as ``sdist``; ``hook-prebuilt``
treats ``.whl`` paths as ``prebuilt_wheel``. Bare directories and other
path forms are rejected. The sdist default preserves legacy archive
semantics; it does not infer the artifact kind from archive contents.
Hooks needing tarball or git-checkout semantics must return a typed
result with the explicit kind. ``any_source`` and ``not_available``
are not valid concrete result kinds.
Invalid return types, malformed typed fields, and kinds incompatible
with the profile raise errors identifying the package and hook.
Hook exceptions are chained to a ``RuntimeError`` with that context;
neither failures nor invalid results trigger a fallback download.
If the download hook is absent, ``hook-sdist`` downloads the candidate
URL as an sdist into ``ctx.sdists_downloads``, saved as
``{canonical name}-{version}.tar.gz`` (or ``.zip``) whatever the URL's
filename. ``hook-prebuilt`` downloads it as a wheel into
``ctx.wheels_prebuilt``. These defaults use the standard archive and
wheel validation. Git downloads require a download hook.
Without a ``get_resolver_provider`` hook, ``hook-sdist`` finds
versions on its ``index_url`` in sdists and wheels for any platform,
and ``hook-prebuilt`` finds them in wheels for this platform. The
download hook then gets the artifact. ``download_url`` is the URL of
a matching file on the index, which can be a wheel, so the hook
should get the source from its own location, such as a git tag.
If the hook only clones a tag, use ``pypi-git`` instead and drop
the hook.
To migrate a raw-tarball hook to ``hook-sdist``, specify ``build_sdist``
in YAML and return its artifact kind explicitly:
.. code-block:: python
import pathlib
from packaging.requirements import Requirement
from packaging.version import Version
from fromager import downloads
from fromager.context import WorkContext
from fromager.packagesettings import DownloadedSource, DownloadKind
def download_source(
ctx: WorkContext,
req: Requirement,
version: Version,
download_url: str,
sdists_downloads_dir: pathlib.Path,
) -> DownloadedSource:
"""Download a raw source tarball and identify its kind."""
path = downloads.download_url(
destination_dir=sdists_downloads_dir,
url=download_url,
destination_filename=f"{req.name}-{version}.tar.gz",
)
return DownloadedSource(path=path, kind=DownloadKind.tarball)
A hook that clones sources returns the checkout directory with
``kind=DownloadKind.git_checkout``. Typed results are supported through
the configured hook profiles; packages using legacy settings still require
a bare ``Path``.
.. autofromagerhook:: default_prepare_source
The ``prepare_source()`` function sets up a source tree for building. It
selects ``default_prepare_source()`` for archives or
``default_prepare_git_checkout()`` for a Git checkout according to the
downloaded ``DownloadKind``. Both defaults apply patches and other project
preparation.
In-process callers pass the ``path`` and ``kind`` from ``DownloadedSource``
to ``prepare_source()``. The Python function accepts a source ``Path`` and
optional ``download_kind``; path-only callers default to ``sdist``.
A configured ``source:`` resolver does not disable the package
``prepare_source`` override. The override receives the ``WorkContext``,
``Requirement``, source ``Path``, and version, but not ``download_kind``.
An archive-specific override can call ``sources.default_prepare_source()``;
a Git-specific override can call
``sources.default_prepare_git_checkout()`` before its own changes.
The override returns the source root ``Path`` or a ``(Path, bool)`` pair.
Git checkouts must use the double-nested package/version layout under
``ctx.work_dir`` so the later bootstrap and step commands can find them.
Use :func:`~fromager.gitutils.git_checkout_dir` to construct this path;
it canonicalizes the package name before building the directory names.
.. versionchanged:: 0.103.0
Passing ``DownloadKind.git_checkout`` prepares a clean Git checkout.
.. autofromagerhook:: default_build_sdist
The ``build_sdist()`` function is responsible for creating a new source
distribution from the prepared source tree and placing it in
``ctx.sdists_build``. The dist name and version of the sdist file must
match the ``Requirement`` and ``Version``.
The arguments are the ``WorkContext``, the ``Requirement`` being evaluated, and the
`Path` to the root of the source tree.
The return value is the ``Path`` to the newly created source distribution.
The following helpers are available for use in custom source hooks:
.. autofunction:: fromager.gitutils.git_checkout_dir
.. autofunction:: fromager.sources.default_prepare_git_checkout
.. autofunction:: fromager.sources.ensure_pkg_info
.. autofunction:: fromager.sources.pep517_build_sdist
.. autofunction:: fromager.sources.unpack_source
.. currentmodule:: fromager.wheels
.. autofromagerhook:: default_build_wheel
The ``build_wheel()`` function is responsible for creating a wheel from
the prepared source tree and placing it in ``ctx.wheels_build``. The
default implementation uses :pep:`517` pyproject hook. The dist name
and version of the wheel must match the ``Requirement`` and ``Version``.
The arguments are the ``WorkContext``, the ``Path`` to a virtualenv
prepared with the build dependencies, a ``dict`` with extra environment
variables to pass to the build, the ``Requirement`` being evaluated, and
the ``Path`` to the root of the source tree.
The return value is ignored.
.. autofromagerhook:: default_add_extra_metadata_to_wheels
The ``add_extra_metadata_to_wheels()`` function is responsible to return any
data the user would like to include in the wheels that fromager builds. This
data will be added to the ``fromager-build-settings`` file under the
``.dist-info`` directory of the wheels. This file already contains the
settings used to build that package.
The arguments available are ``WorkContext``, ``Requirement`` being
evaluated, the resolved ``Version`` of that requirement, a ``dict`` with extra
environment variables, a ``Path`` to the root directory of the source
distribution and a ``Path`` to the ``.dist-info`` directory of the wheel.
The return value must be a ``dict``, otherwise it will be ignored.
.. autoclass:: fromager.build_environment.BuildEnvironment :members: run .. autoproperty:: python
.. autoclass:: fromager.context.WorkContext
.. autoclass:: fromager.resolver.PyPIProvider
.. autoclass:: fromager.resolver.GenericProvider
.. autoclass:: fromager.resolver.GitHubTagProvider
.. autoclass:: fromager.resolver.GitLabTagProvider
.. autofunction:: fromager.sources.prepare_new_source