Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs/concepts/hooks-and-overrides.rst
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,12 @@ arguments.
artifact kind. Package overrides remain active. See :ref:`prepare_source_hook`
for their arguments and the public Git checkout helper.

.. versionchanged:: 0.104.0

With ``source:``, set ``build_sdist: hook`` to use the package's
custom sdist builder. The hook must be installed, or the build fails
early. See :ref:`build_sdist_hook`.

Global Hooks
------------

Expand Down
5 changes: 5 additions & 0 deletions docs/concepts/package-settings.rst
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,11 @@ Settings Loading Flow
``PackageBuildInfo`` objects are created lazily the first time a
package is queried and cached for the rest of the run.

.. versionchanged:: 0.104.0
Loading settings does not check ``source.build_sdist: hook``. A build
checks it before downloading the source. Only the selected variant's
hook is required.

``PackageSettings`` validates source settings when it loads each package.
A package-level ``source`` cannot be combined with an explicitly set
variant ``pre_built`` or ``wheel_server_url``. A variant-level ``source``
Expand Down
36 changes: 28 additions & 8 deletions docs/concepts/resolver-architecture.rst
Original file line number Diff line number Diff line change
Expand Up @@ -212,17 +212,37 @@ Building the Sdist
Git checkouts can be prepared directly. Configured source resolvers
also select the sdist build method.

For source builds, the configured resolver selects one of two methods:
For source builds, ``source.build_sdist`` selects one of three methods:

- ``pep517`` calls the package's build backend. Git checkout resolvers
default to it.
- ``pep517`` builds the sdist with the package's build backend.
Git checkout resolvers default to it.
- ``tarball`` repacks the prepared source without VCS metadata such as
``.git``. PyPI sdist resolvers default to it.
``.git``. ``pypi-sdist`` and ``pypi-download`` default to it.
- ``hook`` runs the package's ``build_sdist`` override hook, which can
generate files or choose which directory to pack.

``hook-sdist`` requires an explicit ``build_sdist: pep517`` or
``build_sdist: tarball`` setting. A configured method takes precedence
over a ``build_sdist`` override and logs a warning when the override is
ignored; packages without ``source:`` keep using the override.
The choice follows these rules:

- ``hook`` requires an installed, callable hook. If it is missing, the
build fails before downloading the source. A cached wheel needs no
hook, so it is used without the check.
- A configured ``pep517`` or ``tarball`` skips an installed hook with a
warning.
- Without ``source:``, the hook runs when present; otherwise the default
method is used.
- ``hook-sdist`` has no default and always needs an explicit setting.

With ``source:`` configured, ``default_build_sdist()`` leaves VCS metadata
out of the archive. Pass ``exclude_vcs=False`` to keep it. Hooks that build
their own archives choose their own contents.

See :ref:`build_sdist_hook` for the hook's arguments and an example.

.. versionchanged:: 0.104.0

``source.build_sdist: hook`` enables package overrides for configured
source profiles, and ``default_build_sdist()`` excludes VCS metadata
by default for them.

Replaying a Build Order
-----------------------
Expand Down
5 changes: 5 additions & 0 deletions docs/proposals/new-resolver-config.md
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,11 @@ hook and `source` settings, then at least one `source` setting (top-level or
variant) must use `provider: hook-sdist` or `provider: hook-prebuilt`. The
rule ensures that the hooks are used.

Set `build_sdist: hook` under `source` to use the package's custom sdist
builder with any profile that builds from source. A missing hook is an
error. `pep517` and `tarball` skip the hook with a warning. Packages
without `source` keep their existing hook behavior.

### git clone

Git cloning uses the more efficient `git_clone_fast` function. Like pip's
Expand Down
7 changes: 7 additions & 0 deletions docs/reference/config-reference.rst
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,13 @@ Source Resolver

.. autoattribute:: pep517
.. autoattribute:: tarball
.. autoattribute:: hook

.. versionchanged:: 0.104.0

``pypi-sdist`` and ``pypi-download`` accept an explicit
``build_sdist: pep517``. Their default remains ``tarball`` because some
build backends require VCS metadata that source archives do not contain.

.. autoclass:: DownloadedSource
:members: path, kind
Expand Down
41 changes: 34 additions & 7 deletions docs/reference/hooks.rst
Original file line number Diff line number Diff line change
Expand Up @@ -361,17 +361,44 @@ Source hooks
.. versionchanged:: 0.103.0
Passing ``DownloadKind.git_checkout`` prepares a clean Git checkout.

.. _build_sdist_hook:

.. autofromagerhook:: default_build_sdist

The ``build_sdist()`` function is responsible for creating a new source
distribution from the prepared source tree and placing it in
``ctx.sdists_build``. The dist name and version of the sdist file must
match the ``Requirement`` and ``Version``.
Pack the prepared sources into an sdist file. Return its ``Path``.
The file must exist in ``ctx.sdists_builds`` and match the package
name and version.

The hook receives ``ctx``, ``extra_environ``, ``req``, ``version``,
``sdist_root_dir``, ``build_env``, and ``build_dir``. Use it to
generate files or to pack a directory other than the wheel build
directory.

The hook needs its build dependencies installed. ``step build-sdist``
does not install them. Run ``step prepare-build`` first.

.. versionchanged:: 0.104.0

To use this hook with ``source:``, set ``build_sdist: hook``:

.. code-block:: yaml

build_dir: python
source:
provider: pypi-download
download_url: https://sources.example/project-{version}.tar.gz
download_kind: tarball
build_sdist: hook

The hook must be installed. Without it, the build fails before
downloading the source. ``pep517`` and ``tarball`` ignore the hook and log a
warning. Packages without ``source:`` use the hook when it exists.

The arguments are the ``WorkContext``, the ``Requirement`` being evaluated, and the
`Path` to the root of the source tree.
Any profile that builds from source can select ``hook``.
``hook-sdist`` still needs a resolver or download hook.

The return value is the ``Path`` to the newly created source distribution.
A hook that builds its own archive picks what goes in it. It must
leave out VCS metadata such as ``.git`` itself.

Source helper functions
~~~~~~~~~~~~~~~~~~~~~~~
Expand Down
4 changes: 4 additions & 0 deletions e2e/source_settings_hook/stevedore.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
source:
provider: github-tag-git
project_url: https://github.com/python-wheel-build/stevedore-test-repo
build_sdist: hook
2 changes: 2 additions & 0 deletions e2e/stevedore_override/src/package_plugins/stevedore.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,8 @@ def build_sdist(
version: Version,
build_env: build_environment.BuildEnvironment,
) -> pathlib.Path:
"""Build the sdist through the PEP 517 backend, like a custom hook would."""
logger.info("build_sdist override hook invoked for %s", req.name)
return sources.pep517_build_sdist(
ctx=ctx,
extra_environ=extra_environ,
Expand Down
50 changes: 48 additions & 2 deletions e2e/test_build_source_config.sh
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,9 @@
# The git profile resolves stevedore tags on GitHub and builds from a
# git checkout. The archive profile resolves on PyPI and downloads a
# tarball from a URL template. The hook profiles use installed plugins
# to resolve a source archive and download a prebuilt wheel.
# to resolve a source archive and download a prebuilt wheel, and the
# build_sdist hook profile selects the stevedore `build_sdist` override
# through `source.build_sdist: hook`.

SCRIPTDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
source "$SCRIPTDIR/common.sh"
Expand All @@ -20,6 +22,7 @@ BUILT_SDIST="sdists-repo/builds/${DIST}-${VERSION}.tar.gz"

GIT_SETTINGS="$SCRIPTDIR/source_settings_git"
ARCHIVE_SETTINGS="$SCRIPTDIR/source_settings_archive"
HOOK_SETTINGS="$SCRIPTDIR/source_settings_hook"

pass=true

Expand All @@ -40,7 +43,7 @@ expect_file() {

expect_no_source_download() {
if [ -n "$(find "$OUTDIR/sdists-repo/downloads/" -name "${DIST}-*")" ]; then
echo "FAIL: git profile downloaded a source archive for ${DIST}" 1>&2
echo "FAIL: build downloaded a source archive for ${DIST}" 1>&2
pass=false
fi
}
Expand All @@ -52,6 +55,13 @@ expect_ignored_index_url() {
fi
}

expect_hook_sdist() {
if ! grep -q "build_sdist override hook invoked" "$1"; then
echo "FAIL: source.build_sdist: hook did not invoke the override in $1" 1>&2
pass=false
fi
}

# 1. Bootstrap with the git profile. This also builds the build
# dependencies that the later steps install from the local wheel server.
fromager \
Expand Down Expand Up @@ -220,5 +230,41 @@ then
echo "FAIL: build order does not record the hook-prebuilt wheel" 1>&2
pass=false
fi
# 7. The hook profile fails before downloading when no build_sdist
# hook is installed. The github override example registers the stevedore
# override module but provides no build_sdist method.
remove_dist
if fromager \
--log-file="$OUTDIR/build-hook-missing.log" \
--sdists-repo="$OUTDIR/sdists-repo" \
--wheels-repo="$OUTDIR/wheels-repo" \
--work-dir="$OUTDIR/work-dir" \
--settings-dir="$HOOK_SETTINGS" \
build "${DIST}" "${VERSION}" "unreachable_url"
then
echo "FAIL: build succeeded without a build_sdist hook installed" 1>&2
pass=false
fi
expect_no_source_download

# 8. Build with the hook profile. The installed stevedore override's
# build_sdist hook builds the sdist through source.build_sdist: hook.
# Uninstall the github override example first: both plugins register
# the stevedore override entry point.
remove_dist
pip uninstall -y github-example-with-package-plugin
pip install "$SCRIPTDIR/stevedore_override"
fromager \
--log-file="$OUTDIR/build-hook.log" \
--sdists-repo="$OUTDIR/sdists-repo" \
--wheels-repo="$OUTDIR/wheels-repo" \
--work-dir="$OUTDIR/work-dir" \
--settings-dir="$HOOK_SETTINGS" \
build "${DIST}" "${VERSION}" "unreachable_url"

expect_file "$WHEEL"
expect_file "$BUILT_SDIST"
expect_no_source_download
expect_hook_sdist "$OUTDIR/build-hook.log"

$pass
2 changes: 2 additions & 0 deletions src/fromager/bootstrapper/_prepare_source.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,8 @@ def _bg_prepare_source(
sdist_root_dir=unpacked / unpacked.stem,
cached_wheel_filename=cached_wheel,
)
# Fail before downloading a source that cannot produce an sdist.
sources.require_build_sdist_hook(ctx=ctx, req=req)
downloaded_source = sources.download_source(
ctx=ctx, req=req, version=resolved_version, download_url=source_url
)
Expand Down
Loading
Loading