diff --git a/docs/concepts/hooks-and-overrides.rst b/docs/concepts/hooks-and-overrides.rst index e030fbf9b..2022c85aa 100644 --- a/docs/concepts/hooks-and-overrides.rst +++ b/docs/concepts/hooks-and-overrides.rst @@ -66,6 +66,12 @@ arguments. artifact kind. Package overrides remain active. See :ref:`prepare_source_hook` for their arguments and the public Git checkout helper. +.. versionchanged:: 0.104.0 + + With ``source:``, set ``build_sdist: hook`` to use the package's + custom sdist builder. The hook must be installed, or the build fails + early. See :ref:`build_sdist_hook`. + Global Hooks ------------ diff --git a/docs/concepts/package-settings.rst b/docs/concepts/package-settings.rst index 6d0890e48..9c89b1a7e 100644 --- a/docs/concepts/package-settings.rst +++ b/docs/concepts/package-settings.rst @@ -35,6 +35,11 @@ Settings Loading Flow ``PackageBuildInfo`` objects are created lazily the first time a package is queried and cached for the rest of the run. +.. versionchanged:: 0.104.0 + Loading settings does not check ``source.build_sdist: hook``. A build + checks it before downloading the source. Only the selected variant's + hook is required. + ``PackageSettings`` validates source settings when it loads each package. A package-level ``source`` cannot be combined with an explicitly set variant ``pre_built`` or ``wheel_server_url``. A variant-level ``source`` diff --git a/docs/concepts/resolver-architecture.rst b/docs/concepts/resolver-architecture.rst index 1d5923320..72a1ab284 100644 --- a/docs/concepts/resolver-architecture.rst +++ b/docs/concepts/resolver-architecture.rst @@ -212,17 +212,37 @@ Building the Sdist Git checkouts can be prepared directly. Configured source resolvers also select the sdist build method. -For source builds, the configured resolver selects one of two methods: +For source builds, ``source.build_sdist`` selects one of three methods: -- ``pep517`` calls the package's build backend. Git checkout resolvers - default to it. +- ``pep517`` builds the sdist with the package's build backend. + Git checkout resolvers default to it. - ``tarball`` repacks the prepared source without VCS metadata such as - ``.git``. PyPI sdist resolvers default to it. + ``.git``. ``pypi-sdist`` and ``pypi-download`` default to it. +- ``hook`` runs the package's ``build_sdist`` override hook, which can + generate files or choose which directory to pack. -``hook-sdist`` requires an explicit ``build_sdist: pep517`` or -``build_sdist: tarball`` setting. A configured method takes precedence -over a ``build_sdist`` override and logs a warning when the override is -ignored; packages without ``source:`` keep using the override. +The choice follows these rules: + +- ``hook`` requires an installed, callable hook. If it is missing, the + build fails before downloading the source. A cached wheel needs no + hook, so it is used without the check. +- A configured ``pep517`` or ``tarball`` skips an installed hook with a + warning. +- Without ``source:``, the hook runs when present; otherwise the default + method is used. +- ``hook-sdist`` has no default and always needs an explicit setting. + +With ``source:`` configured, ``default_build_sdist()`` leaves VCS metadata +out of the archive. Pass ``exclude_vcs=False`` to keep it. Hooks that build +their own archives choose their own contents. + +See :ref:`build_sdist_hook` for the hook's arguments and an example. + +.. versionchanged:: 0.104.0 + + ``source.build_sdist: hook`` enables package overrides for configured + source profiles, and ``default_build_sdist()`` excludes VCS metadata + by default for them. Replaying a Build Order ----------------------- diff --git a/docs/proposals/new-resolver-config.md b/docs/proposals/new-resolver-config.md index 3895f8082..a7f3b87bf 100644 --- a/docs/proposals/new-resolver-config.md +++ b/docs/proposals/new-resolver-config.md @@ -174,6 +174,11 @@ hook and `source` settings, then at least one `source` setting (top-level or variant) must use `provider: hook-sdist` or `provider: hook-prebuilt`. The rule ensures that the hooks are used. +Set `build_sdist: hook` under `source` to use the package's custom sdist +builder with any profile that builds from source. A missing hook is an +error. `pep517` and `tarball` skip the hook with a warning. Packages +without `source` keep their existing hook behavior. + ### git clone Git cloning uses the more efficient `git_clone_fast` function. Like pip's diff --git a/docs/reference/config-reference.rst b/docs/reference/config-reference.rst index bd5c07f92..24ce5cb09 100644 --- a/docs/reference/config-reference.rst +++ b/docs/reference/config-reference.rst @@ -86,6 +86,13 @@ Source Resolver .. autoattribute:: pep517 .. autoattribute:: tarball + .. autoattribute:: hook + +.. versionchanged:: 0.104.0 + + ``pypi-sdist`` and ``pypi-download`` accept an explicit + ``build_sdist: pep517``. Their default remains ``tarball`` because some + build backends require VCS metadata that source archives do not contain. .. autoclass:: DownloadedSource :members: path, kind diff --git a/docs/reference/hooks.rst b/docs/reference/hooks.rst index f98830450..acee8ac4a 100644 --- a/docs/reference/hooks.rst +++ b/docs/reference/hooks.rst @@ -361,17 +361,44 @@ Source hooks .. versionchanged:: 0.103.0 Passing ``DownloadKind.git_checkout`` prepares a clean Git checkout. +.. _build_sdist_hook: + .. autofromagerhook:: default_build_sdist - The ``build_sdist()`` function is responsible for creating a new source - distribution from the prepared source tree and placing it in - ``ctx.sdists_build``. The dist name and version of the sdist file must - match the ``Requirement`` and ``Version``. + Pack the prepared sources into an sdist file. Return its ``Path``. + The file must exist in ``ctx.sdists_builds`` and match the package + name and version. + + The hook receives ``ctx``, ``extra_environ``, ``req``, ``version``, + ``sdist_root_dir``, ``build_env``, and ``build_dir``. Use it to + generate files or to pack a directory other than the wheel build + directory. + + The hook needs its build dependencies installed. ``step build-sdist`` + does not install them. Run ``step prepare-build`` first. + + .. versionchanged:: 0.104.0 + + To use this hook with ``source:``, set ``build_sdist: hook``: + + .. code-block:: yaml + + build_dir: python + source: + provider: pypi-download + download_url: https://sources.example/project-{version}.tar.gz + download_kind: tarball + build_sdist: hook + + The hook must be installed. Without it, the build fails before + downloading the source. ``pep517`` and ``tarball`` ignore the hook and log a + warning. Packages without ``source:`` use the hook when it exists. - The arguments are the ``WorkContext``, the ``Requirement`` being evaluated, and the - `Path` to the root of the source tree. + Any profile that builds from source can select ``hook``. + ``hook-sdist`` still needs a resolver or download hook. - The return value is the ``Path`` to the newly created source distribution. + A hook that builds its own archive picks what goes in it. It must + leave out VCS metadata such as ``.git`` itself. Source helper functions ~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/e2e/source_settings_hook/stevedore.yaml b/e2e/source_settings_hook/stevedore.yaml new file mode 100644 index 000000000..aff2697e0 --- /dev/null +++ b/e2e/source_settings_hook/stevedore.yaml @@ -0,0 +1,4 @@ +source: + provider: github-tag-git + project_url: https://github.com/python-wheel-build/stevedore-test-repo + build_sdist: hook diff --git a/e2e/stevedore_override/src/package_plugins/stevedore.py b/e2e/stevedore_override/src/package_plugins/stevedore.py index a7f3759be..f4b3a7d45 100644 --- a/e2e/stevedore_override/src/package_plugins/stevedore.py +++ b/e2e/stevedore_override/src/package_plugins/stevedore.py @@ -39,6 +39,8 @@ def build_sdist( version: Version, build_env: build_environment.BuildEnvironment, ) -> pathlib.Path: + """Build the sdist through the PEP 517 backend, like a custom hook would.""" + logger.info("build_sdist override hook invoked for %s", req.name) return sources.pep517_build_sdist( ctx=ctx, extra_environ=extra_environ, diff --git a/e2e/test_build_source_config.sh b/e2e/test_build_source_config.sh index 6534f4038..debcba8f6 100755 --- a/e2e/test_build_source_config.sh +++ b/e2e/test_build_source_config.sh @@ -7,7 +7,9 @@ # The git profile resolves stevedore tags on GitHub and builds from a # git checkout. The archive profile resolves on PyPI and downloads a # tarball from a URL template. The hook profiles use installed plugins -# to resolve a source archive and download a prebuilt wheel. +# to resolve a source archive and download a prebuilt wheel, and the +# build_sdist hook profile selects the stevedore `build_sdist` override +# through `source.build_sdist: hook`. SCRIPTDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" source "$SCRIPTDIR/common.sh" @@ -20,6 +22,7 @@ BUILT_SDIST="sdists-repo/builds/${DIST}-${VERSION}.tar.gz" GIT_SETTINGS="$SCRIPTDIR/source_settings_git" ARCHIVE_SETTINGS="$SCRIPTDIR/source_settings_archive" +HOOK_SETTINGS="$SCRIPTDIR/source_settings_hook" pass=true @@ -40,7 +43,7 @@ expect_file() { expect_no_source_download() { if [ -n "$(find "$OUTDIR/sdists-repo/downloads/" -name "${DIST}-*")" ]; then - echo "FAIL: git profile downloaded a source archive for ${DIST}" 1>&2 + echo "FAIL: build downloaded a source archive for ${DIST}" 1>&2 pass=false fi } @@ -52,6 +55,13 @@ expect_ignored_index_url() { fi } +expect_hook_sdist() { + if ! grep -q "build_sdist override hook invoked" "$1"; then + echo "FAIL: source.build_sdist: hook did not invoke the override in $1" 1>&2 + pass=false + fi +} + # 1. Bootstrap with the git profile. This also builds the build # dependencies that the later steps install from the local wheel server. fromager \ @@ -220,5 +230,41 @@ then echo "FAIL: build order does not record the hook-prebuilt wheel" 1>&2 pass=false fi +# 7. The hook profile fails before downloading when no build_sdist +# hook is installed. The github override example registers the stevedore +# override module but provides no build_sdist method. +remove_dist +if fromager \ + --log-file="$OUTDIR/build-hook-missing.log" \ + --sdists-repo="$OUTDIR/sdists-repo" \ + --wheels-repo="$OUTDIR/wheels-repo" \ + --work-dir="$OUTDIR/work-dir" \ + --settings-dir="$HOOK_SETTINGS" \ + build "${DIST}" "${VERSION}" "unreachable_url" +then + echo "FAIL: build succeeded without a build_sdist hook installed" 1>&2 + pass=false +fi +expect_no_source_download + +# 8. Build with the hook profile. The installed stevedore override's +# build_sdist hook builds the sdist through source.build_sdist: hook. +# Uninstall the github override example first: both plugins register +# the stevedore override entry point. +remove_dist +pip uninstall -y github-example-with-package-plugin +pip install "$SCRIPTDIR/stevedore_override" +fromager \ + --log-file="$OUTDIR/build-hook.log" \ + --sdists-repo="$OUTDIR/sdists-repo" \ + --wheels-repo="$OUTDIR/wheels-repo" \ + --work-dir="$OUTDIR/work-dir" \ + --settings-dir="$HOOK_SETTINGS" \ + build "${DIST}" "${VERSION}" "unreachable_url" + +expect_file "$WHEEL" +expect_file "$BUILT_SDIST" +expect_no_source_download +expect_hook_sdist "$OUTDIR/build-hook.log" $pass diff --git a/src/fromager/bootstrapper/_prepare_source.py b/src/fromager/bootstrapper/_prepare_source.py index 76f5212de..e9a844401 100644 --- a/src/fromager/bootstrapper/_prepare_source.py +++ b/src/fromager/bootstrapper/_prepare_source.py @@ -43,6 +43,8 @@ def _bg_prepare_source( sdist_root_dir=unpacked / unpacked.stem, cached_wheel_filename=cached_wheel, ) + # Fail before downloading a source that cannot produce an sdist. + sources.require_build_sdist_hook(ctx=ctx, req=req) downloaded_source = sources.download_source( ctx=ctx, req=req, version=resolved_version, download_url=source_url ) diff --git a/src/fromager/commands/build.py b/src/fromager/commands/build.py index a41b10d44..ca7fbf166 100644 --- a/src/fromager/commands/build.py +++ b/src/fromager/commands/build.py @@ -1,4 +1,5 @@ import concurrent.futures +import contextlib import dataclasses import datetime import json @@ -314,6 +315,36 @@ def _create_table( return table +def _new_wheel_log_handler(wheel_log: pathlib.Path) -> logging.FileHandler: + """Create a handler that logs the current thread to the per-wheel file.""" + # Keep only the calling thread's messages so parallel builds do + # not interleave in each other's log files. + thread_name = threading.current_thread().name + handler = logging.FileHandler(filename=str(wheel_log)) + handler.setFormatter(logging.Formatter(VERBOSE_LOG_FMT)) + handler.addFilter(ThreadLogFilter(thread_name)) + return handler + + +@contextlib.contextmanager +def _wheel_log_handler(wheel_log: pathlib.Path) -> typing.Iterator[None]: + """Capture this thread's log messages in the per-wheel log file. + + The handler is removed and closed even when the build fails. + """ + # Attach to the root logger so messages from every logger, + # including build dependencies, reach the wheel's log file. + root_logger = logging.getLogger() + + handler = _new_wheel_log_handler(wheel_log) + root_logger.addHandler(handler) + try: + yield + finally: + root_logger.removeHandler(handler) + handler.close() + + def _build( wkctx: context.WorkContext, resolved_version: Version, @@ -332,122 +363,113 @@ def _build( wheel_filename: pathlib.Path | None = None use_exiting_wheel: bool = False - # Set up a log file for all of the details of the build for this one wheel. - # We attach a handler to the root logger so that all messages are logged to - # the file, and we add a filter to the handler so that only messages from - # the current thread are logged for when we build in parallel. - root_logger = logging.getLogger(None) + # Log the details of the build for this one wheel to its own file. module_name = overrides.pkgname_to_override_module(req.name) wheel_log = wkctx.logs_dir / f"{module_name}-{resolved_version}.log" - file_handler = logging.FileHandler(filename=str(wheel_log)) - file_handler.setFormatter(logging.Formatter(VERBOSE_LOG_FMT)) - file_handler.addFilter(ThreadLogFilter(threading.current_thread().name)) - root_logger.addHandler(file_handler) + with _wheel_log_handler(wheel_log): + logger.info("starting processing") + pbi = wkctx.package_build_info(req) + prebuilt = pbi.pre_built + + # See if we can reuse an existing wheel. + if not force: + wheel_filename = _is_wheel_built( + wkctx, + req.name, + resolved_version, + cache_wheel_server_url=cache_wheel_server_url, + ) + if wheel_filename: + logger.info("using existing wheel from %s", wheel_filename) + use_exiting_wheel = True - logger.info("starting processing") - pbi = wkctx.package_build_info(req) - prebuilt = pbi.pre_built + # Handle prebuilt wheels. + if prebuilt: + if not wheel_filename: + logger.info("downloading prebuilt wheel") + wheel_filename = wheels.download_prebuilt_wheel( + ctx=wkctx, + req=req, + version=resolved_version, + download_url=source_download_url, + direct_download_directory=wkctx.wheels_build, + ) + else: + # already downloaded prebuilt wheel + use_exiting_wheel = True - # See if we can reuse an existing wheel. - if not force: - wheel_filename = _is_wheel_built( - wkctx, - req.name, - resolved_version, - cache_wheel_server_url=cache_wheel_server_url, - ) - if wheel_filename: - logger.info("using existing wheel from %s", wheel_filename) - use_exiting_wheel = True + # Run hooks for both cached and freshly downloaded prebuilt wheels. + hooks.run_prebuilt_wheel_hooks( + ctx=wkctx, + req=req, + dist_name=req.name, + dist_version=str(resolved_version), + wheel_filename=wheel_filename, + ) - # Handle prebuilt wheels. - if prebuilt: + # If we get here and still don't have a wheel filename, then we need to + # build the wheel. if not wheel_filename: - logger.info("downloading prebuilt wheel") - wheel_filename = wheels.download_prebuilt_wheel( + # Fail before downloading a source that cannot produce an sdist. + sources.require_build_sdist_hook(ctx=wkctx, req=req) + downloaded_source = sources.download_source( ctx=wkctx, req=req, version=resolved_version, download_url=source_download_url, - direct_download_directory=wkctx.wheels_build, ) - else: - # already downloaded prebuilt wheel - use_exiting_wheel = True - - # Run hooks for both cached and freshly downloaded prebuilt wheels. - hooks.run_prebuilt_wheel_hooks( - ctx=wkctx, - req=req, - dist_name=req.name, - dist_version=str(resolved_version), - wheel_filename=wheel_filename, - ) - - # If we get here and still don't have a wheel filename, then we need to - # build the wheel. - if not wheel_filename: - downloaded_source = sources.download_source( - ctx=wkctx, - req=req, - version=resolved_version, - download_url=source_download_url, - ) - logger.debug( - "saved source of version %s from %s to %s", - resolved_version, - source_download_url, - downloaded_source.path, - ) - - # Prepare source - source_root_dir = sources.prepare_source( - ctx=wkctx, - req=req, - source_filename=downloaded_source.path, - version=resolved_version, - download_kind=downloaded_source.kind, - ) + logger.debug( + "saved source of version %s from %s to %s", + resolved_version, + source_download_url, + downloaded_source.path, + ) - # Build environment - build_env = build_environment.prepare_build_environment( - ctx=wkctx, - req=req, - version=resolved_version, - sdist_root_dir=source_root_dir, - ) + # Prepare source + source_root_dir = sources.prepare_source( + ctx=wkctx, + req=req, + source_filename=downloaded_source.path, + version=resolved_version, + download_kind=downloaded_source.kind, + ) - # Make a new source distribution, in case we patched the code. - sdist_filename = sources.build_sdist( - ctx=wkctx, - req=req, - version=resolved_version, - sdist_root_dir=source_root_dir, - build_env=build_env, - ) + # Build environment + build_env = build_environment.prepare_build_environment( + ctx=wkctx, + req=req, + version=resolved_version, + sdist_root_dir=source_root_dir, + ) - # Build - wheel_filename = wheels.build_wheel( - ctx=wkctx, - req=req, - sdist_root_dir=source_root_dir, - version=resolved_version, - build_env=build_env, - ) + # Make a new source distribution, in case we patched the code. + sdist_filename = sources.build_sdist( + ctx=wkctx, + req=req, + version=resolved_version, + sdist_root_dir=source_root_dir, + build_env=build_env, + ) - hooks.run_post_build_hooks( - ctx=wkctx, - req=req, - dist_name=canonicalize_name(req.name), - dist_version=str(resolved_version), - sdist_filename=sdist_filename, - wheel_filename=wheel_filename, - ) + # Build + wheel_filename = wheels.build_wheel( + ctx=wkctx, + req=req, + sdist_root_dir=source_root_dir, + version=resolved_version, + build_env=build_env, + ) - wkctx.clean_build_dirs(source_root_dir, build_env) + hooks.run_post_build_hooks( + ctx=wkctx, + req=req, + dist_name=canonicalize_name(req.name), + dist_version=str(resolved_version), + sdist_filename=sdist_filename, + wheel_filename=wheel_filename, + ) - root_logger.removeHandler(file_handler) - file_handler.close() + wkctx.clean_build_dirs(source_root_dir, build_env) if ( prebuilt diff --git a/src/fromager/packagesettings/_resolver.py b/src/fromager/packagesettings/_resolver.py index 6a05a5d81..402fb7d47 100644 --- a/src/fromager/packagesettings/_resolver.py +++ b/src/fromager/packagesettings/_resolver.py @@ -28,8 +28,15 @@ class BuildSDist(enum.StrEnum): + """Source distribution build method.""" + pep517 = "pep517" tarball = "tarball" + hook = "hook" + """Call the package's ``build_sdist`` override hook. + + .. versionadded:: 0.104.0 + """ class DownloadKind(enum.StrEnum): @@ -71,10 +78,14 @@ class AbstractResolver(pydantic.BaseModel): provider: str build_sdist: BuildSDist | None - """Sdist build strategy; None for resolvers that do not build from source.""" + """Sdist build strategy; None for resolvers that do not build from source. + + ``hook`` requires a package ``build_sdist`` override independently of + the profile's resolver and download hooks. + """ supports_override_hooks: typing.ClassVar[bool] = False - """Does resolver support override hooks?""" + """Does resolver support resolution and download override hooks?""" download_kinds: typing.ClassVar[DownloadKindSet] = frozenset() """Set of download kinds this resolver can return.""" @@ -248,9 +259,9 @@ class PyPISDistResolver(AbstractPyPIResolver): provider: typing.Literal["pypi-sdist"] - # It is not safe to use PEP 517 to re-generate a source distribution. - # Some PEP 517 backends require VCS to generate correct sdist. - build_sdist: typing.Literal[BuildSDist.tarball] = BuildSDist.tarball + # Tarball repacking is the safe default because some PEP 517 backends + # require VCS metadata to generate a correct sdist. + build_sdist: BuildSDist = BuildSDist.tarball download_kinds: typing.ClassVar[DownloadKindSet] = frozenset({DownloadKind.sdist}) def resolver_provider( @@ -362,7 +373,7 @@ class PyPIDownloadResolver(AbstractPyPIResolver): URL must contain '{version}' template string. """ - build_sdist: typing.Literal[BuildSDist.tarball] = BuildSDist.tarball + build_sdist: BuildSDist = BuildSDist.tarball download_kinds: typing.ClassVar[DownloadKindSet] = frozenset( {DownloadKind.sdist, DownloadKind.tarball} ) @@ -808,7 +819,7 @@ class AbstractHookResolver(AbstractResolver, CooldownMixin): """Abstract base class for hook-based resolvers""" supports_override_hooks: typing.ClassVar[bool] = True - """Hook resolvers support override hooks.""" + """Hook resolvers support resolution and download override hooks.""" hookless_provider: typing.ClassVar[str] """Profile to suggest for packages without hooks.""" @@ -953,7 +964,8 @@ class HookSDistResolver(AbstractHookResolver): ``Path`` treated as an sdist. Without a download hook, the resolved URL is downloaded as an sdist. Specify ``build_sdist`` explicitly: ``tarball`` to repack an existing - sdist, or ``pep517`` to build from a git checkout. + sdist, ``pep517`` to build from a git checkout, or ``hook`` to call + the package's ``build_sdist`` override. Example:: diff --git a/src/fromager/sources.py b/src/fromager/sources.py index 226542ab6..7846720f3 100644 --- a/src/fromager/sources.py +++ b/src/fromager/sources.py @@ -615,6 +615,28 @@ def prepare_new_source( vendor_rust.vendor_rust(req, pbi.build_dir(source_root_dir)) +def require_build_sdist_hook(*, ctx: context.WorkContext, req: Requirement) -> None: + """Raise ``ValueError`` if ``source.build_sdist: hook`` has no callable hook. + + Packages that do not select ``hook`` are not checked, so their plugins + are not loaded. Build commands call this after the wheel cache check + and before downloading the source. + + .. versionadded:: 0.104.0 + """ + source_resolver = ctx.package_build_info(req).source_resolver + if ( + source_resolver is None + or source_resolver.build_sdist is not packagesettings.BuildSDist.hook + ): + return + if not callable(overrides.find_override_method(req.name, "build_sdist")): + raise ValueError( + f"{req.name}: source.build_sdist='hook' requires a callable " + "build_sdist override hook" + ) + + @metrics.timeit(description="build sdist") def build_sdist( *, @@ -624,12 +646,33 @@ def build_sdist( sdist_root_dir: pathlib.Path, build_env: build_environment.BuildEnvironment, ) -> pathlib.Path: - """Build a source distribution using the configured strategy or legacy hook.""" + """Build a source distribution using the configured strategy or legacy hook. + + ``source.build_sdist: hook`` requires a package override. Configured + ``pep517`` and ``tarball`` strategies take precedence over overrides. + + .. versionchanged:: 0.104.0 + Configured source profiles can select the ``build_sdist`` override. + """ pbi = ctx.package_build_info(req) source_resolver = pbi.source_resolver + strategy = source_resolver.build_sdist if source_resolver is not None else None build_dir = pbi.build_dir(sdist_root_dir) logger.info(f"building {ctx.variant} source distribution for {req} in {build_dir}") + # Build commands check before download; this covers `step build-sdist`. + require_build_sdist_hook(ctx=ctx, req=req) + if ( + source_resolver is not None + and strategy is not packagesettings.BuildSDist.hook + and overrides.find_override_method(req.name, "build_sdist") is not None + ): + logger.warning( + "%s: ignoring build_sdist override because source.build_sdist is %s; " + "set source.build_sdist: hook to use it", + req.name, + strategy, + ) extra_environ = packagesettings.get_extra_environ( ctx=ctx, req=req, @@ -637,45 +680,30 @@ def build_sdist( sdist_root_dir=sdist_root_dir, build_env=build_env, ) - if source_resolver is not None: - if overrides.find_override_method(req.name, "build_sdist") is not None: - logger.warning( - "%s: ignoring build_sdist override because source.build_sdist is %s", - req.name, - source_resolver.build_sdist, - ) - if ( - source_resolver is not None - and source_resolver.build_sdist is packagesettings.BuildSDist.pep517 - ): - sdist_file = pep517_build_sdist( + if source_resolver is None or strategy is packagesettings.BuildSDist.hook: + sdist_file: pathlib.Path = overrides.find_and_invoke( + req.name, + "build_sdist", + default_build_sdist, ctx=ctx, extra_environ=extra_environ, req=req, version=version, sdist_root_dir=sdist_root_dir, + build_dir=build_dir, build_env=build_env, ) - elif ( - source_resolver is not None - and source_resolver.build_sdist is packagesettings.BuildSDist.tarball - ): - sdist_file = default_build_sdist( + elif strategy is packagesettings.BuildSDist.pep517: + sdist_file = pep517_build_sdist( ctx=ctx, extra_environ=extra_environ, req=req, version=version, sdist_root_dir=sdist_root_dir, - build_dir=build_dir, build_env=build_env, - # Git checkouts keep their `.git`; keep it out of the sdist. - exclude_vcs=True, ) - elif source_resolver is None: - sdist_file = overrides.find_and_invoke( - req.name, - "build_sdist", - default_build_sdist, + elif strategy is packagesettings.BuildSDist.tarball: + sdist_file = default_build_sdist( ctx=ctx, extra_environ=extra_environ, req=req, @@ -686,8 +714,7 @@ def build_sdist( ) else: raise ValueError( - f"{req.name}: unsupported source.build_sdist strategy " - f"{source_resolver.build_sdist!r}" + f"{req.name}: unsupported source.build_sdist strategy {strategy!r}" ) logger.info(f"built source distribution {sdist_file}") @@ -713,12 +740,20 @@ def default_build_sdist( build_env: build_environment.BuildEnvironment, build_dir: pathlib.Path, *, - exclude_vcs: bool = False, + exclude_vcs: bool | None = None, ) -> pathlib.Path: """Repack *build_dir* as an sdist. - With *exclude_vcs*, VCS metadata such as ``.git`` is left out. + With *exclude_vcs*, VCS metadata such as ``.git`` is left out. By default, + exclude it when ``source:`` is configured, and retain it otherwise. + Explicit booleans override this policy. + + .. versionchanged:: 0.104.0 + With ``exclude_vcs=None``, exclude VCS metadata when the effective + package or variant has ``source:`` configured; retain it otherwise. """ + if exclude_vcs is None: + exclude_vcs = ctx.package_build_info(req).source_resolver is not None # It seems like the "correct" way to do this would be to run the # PEP 517 API in the source tree we have modified. However, quite # a few packages assume their source distribution is being built diff --git a/tests/test_bootstrap_requirement_resolver.py b/tests/test_bootstrap_requirement_resolver.py index c8b7fb9a6..050b94716 100644 --- a/tests/test_bootstrap_requirement_resolver.py +++ b/tests/test_bootstrap_requirement_resolver.py @@ -947,3 +947,28 @@ def test_resolve_warns_when_resolver_returns_no_results( assert results == [] assert f"resolver returned no results ({mode} version mode)" in caplog.text + + +def test_resolve_does_not_check_build_sdist_hook( + tmp_context: WorkContext, +) -> None: + """Resolve without the selected sdist hook; the build checks it later.""" + req = Requirement("pkg==1.0") + tmp_context.settings._package_settings[canonicalize_name("pkg")] = ( + packagesettings.PackageSettings.from_string( + "pkg", "source:\n provider: pypi-sdist\n build_sdist: hook\n" + ) + ) + bt_resolver = BootstrapRequirementResolver(tmp_context) + + with ( + patch("fromager.overrides.find_override_method", return_value=None), + patch.object(bt_resolver, "_resolve_from_graph", return_value=None), + patch( + "fromager.bootstrap_requirement_resolver.resolver.find_all_matching_from_provider", + return_value=[("https://pkg.test/pkg-1.0.tar.gz", Version("1.0"))], + ), + ): + results = bt_resolver.resolve(req, RequirementType.TOP_LEVEL) + + assert results == [("https://pkg.test/pkg-1.0.tar.gz", Version("1.0"))] diff --git a/tests/test_bootstrap_test_mode.py b/tests/test_bootstrap_test_mode.py index ee8e8740c..ed8edf49a 100644 --- a/tests/test_bootstrap_test_mode.py +++ b/tests/test_bootstrap_test_mode.py @@ -14,8 +14,9 @@ import pytest from packaging.requirements import Requirement +from packaging.version import Version -from fromager import bootstrapper, context +from fromager import bootstrapper, context, packagesettings class TestBootstrapperInitialization: @@ -295,3 +296,41 @@ def test_resolution_failure_raises_in_normal_mode( ): with pytest.raises(RuntimeError, match="Version resolution failed"): bt.bootstrap([req]) + + def test_missing_build_sdist_hook_uses_prebuilt_fallback( + self, tmp_context: context.WorkContext + ) -> None: + """A missing sdist hook is a build failure, so test mode tries a prebuilt wheel.""" + ps = packagesettings.PackageSettings.from_string( + "pkg", "source:\n provider: pypi-sdist\n build_sdist: hook\n" + ) + tmp_context.settings._package_settings[ps.name] = ps + bt = bootstrapper.Bootstrapper(ctx=tmp_context, test_mode=True) + + with ( + mock.patch("fromager.overrides.find_override_method", return_value=None), + mock.patch.object( + bt._resolver, + "resolve", + return_value=[("https://pkg.test/pkg-1.0.tar.gz", Version("1.0"))], + ), + mock.patch( + "fromager.bootstrapper._cache.find_cached_wheel", + return_value=(None, None), + ), + mock.patch("fromager.sources.download_source") as download_source, + mock.patch.object( + bt, "_handle_test_mode_failure", return_value=None + ) as fallback, + ): + bt.bootstrap([Requirement("pkg==1.0")]) + + download_source.assert_not_called() + fallback.assert_called_once() + assert isinstance(fallback.call_args.kwargs["build_error"], ValueError) + assert len(bt.failed_packages) == 1 + failure = bt.failed_packages[0] + assert failure["package"] == "pkg" + assert failure["version"] == "1.0" + assert failure["failure_type"] == "bootstrap" + assert "requires a callable" in failure["exception_message"] diff --git a/tests/test_bootstrapper.py b/tests/test_bootstrapper.py index d67cc9b49..e788dce82 100644 --- a/tests/test_bootstrapper.py +++ b/tests/test_bootstrapper.py @@ -1496,6 +1496,69 @@ def test_bg_prepare_source_accepts_git_checkout(tmp_context: WorkContext) -> Non assert (checkout.parent / "build-meta.json").is_file() +def _select_build_sdist_hook(ctx: WorkContext) -> None: + """Configure ``pkg`` to build its sdist with the ``build_sdist`` hook.""" + ps = PackageSettings.from_string( + "pkg", "source:\n provider: pypi-sdist\n build_sdist: hook\n" + ) + ctx.settings._package_settings[ps.name] = ps + + +def test_bg_prepare_source_rejects_missing_build_sdist_hook_before_download( + tmp_context: WorkContext, +) -> None: + """Fail before downloading a source that cannot produce an sdist.""" + _select_build_sdist_hook(tmp_context) + + with ( + patch("fromager.overrides.find_override_method", return_value=None), + patch( + "fromager.bootstrapper._cache.find_cached_wheel", return_value=(None, None) + ), + patch("fromager.sources.download_source") as download_source, + ): + with pytest.raises( + ValueError, match=r"pkg: source\.build_sdist='hook' requires a callable" + ): + _bg_prepare_source( + ctx=tmp_context, + cache_wheel_server_url=None, + req=Requirement("pkg==1.0"), + resolved_version=Version("1.0"), + source_url="https://pkg.test/pkg-1.0.tar.gz", + ) + + download_source.assert_not_called() + + +def test_bg_prepare_source_uses_cached_wheel_without_build_sdist_hook( + tmp_context: WorkContext, +) -> None: + """Use a cached wheel without the hook, because no sdist is built.""" + _select_build_sdist_hook(tmp_context) + cached_wheel = tmp_context.wheels_downloads / "pkg-1.0-py3-none-any.whl" + unpacked = tmp_context.work_dir / "pkg-1.0" + + with ( + patch("fromager.overrides.find_override_method", return_value=None), + patch( + "fromager.bootstrapper._cache.find_cached_wheel", + return_value=(cached_wheel, unpacked), + ), + patch("fromager.sources.download_source") as download_source, + ): + result = _bg_prepare_source( + ctx=tmp_context, + cache_wheel_server_url=None, + req=Requirement("pkg==1.0"), + resolved_version=Version("1.0"), + source_url="https://pkg.test/pkg-1.0.tar.gz", + ) + + assert result.cached_wheel_filename == cached_wheel + download_source.assert_not_called() + + def test_bootstrapper_download_source_preserves_download_kind( tmp_context: WorkContext, ) -> None: diff --git a/tests/test_commands.py b/tests/test_commands.py index bb1113bbd..d722c8d0e 100644 --- a/tests/test_commands.py +++ b/tests/test_commands.py @@ -1,6 +1,7 @@ import concurrent.futures import io import json +import logging import pathlib import subprocess import tarfile @@ -14,7 +15,7 @@ from packaging.requirements import Requirement from packaging.version import Version -from fromager import packagesettings, server, wheels +from fromager import build_environment, packagesettings, server, wheels from fromager.commands import bootstrap, build, download_sequence, step from fromager.context import WorkContext from fromager.log import requirement_ctxvar @@ -1251,6 +1252,54 @@ def test_default_source_kind_keeps_archive_fallback( assert result is packagesettings.DownloadKind.sdist +def test_step_build_sdist_uses_selected_hook( + cli_runner: CliRunner, tmp_context: WorkContext +) -> None: + """Use a configured sdist hook through the step command and source lookup.""" + _configure_package( + tmp_context, "source:\n provider: pypi-sdist\n build_sdist: hook\n" + ) + source_root = tmp_context.work_dir / "pkg-1.0" / "pkg-1.0" + source_root.mkdir(parents=True) + build_env = Mock(spec=build_environment.BuildEnvironment) + hook_environments: list[build_environment.BuildEnvironment] = [] + output = tmp_context.sdists_builds / "pkg-1.0.tar.gz" + + def hook( + ctx: WorkContext, + extra_environ: dict[str, str], + req: Requirement, + version: Version, + sdist_root_dir: pathlib.Path, + build_env: build_environment.BuildEnvironment, + build_dir: pathlib.Path, + ) -> pathlib.Path: + """Record the build environment and return the selected hook's output.""" + assert sdist_root_dir == source_root + assert build_dir == source_root + hook_environments.append(build_env) + output.touch() + return output + + with ( + patch( + "fromager.overrides.find_override_method", + side_effect=lambda name, method: hook if method == "build_sdist" else None, + ), + patch("fromager.packagesettings.get_extra_environ", return_value={}), + patch.object( + step.build_environment, "BuildEnvironment", return_value=build_env + ), + ): + result = cli_runner.invoke( + step.step, ["build-sdist", "pkg", "1.0"], obj=tmp_context + ) + + assert result.exit_code == 0, result.output + assert result.output.strip() == str(output) + assert hook_environments == [build_env] + + def test_step_prepare_source_accepts_explicit_hook_path( cli_runner: CliRunner, tmp_context: WorkContext, tmp_path: pathlib.Path ) -> None: @@ -1359,3 +1408,65 @@ def test_build_passes_downloaded_kind_to_prepare_source( download_kind=packagesettings.DownloadKind.sdist, ) assert result.wheel_filename == tmp_context.wheels_downloads / wheel_filename.name + + +def test_build_rejects_missing_build_sdist_hook_before_download( + tmp_context: WorkContext, +) -> None: + """Fail a replay build before downloading a source that cannot make an sdist.""" + _configure_package( + tmp_context, "source:\n provider: pypi-sdist\n build_sdist: hook\n" + ) + req = Requirement("pkg==1.0") + handlers_before = list(logging.getLogger().handlers) + + with ( + patch("fromager.overrides.find_override_method", return_value=None), + patch.object(build, "_is_wheel_built", return_value=None), + patch.object(build.sources, "download_source") as download_source, + ): + with pytest.raises( + ValueError, match=r"pkg: source\.build_sdist='hook' requires a callable" + ): + build._build( + tmp_context, + Version("1.0"), + req, + "https://pkg.test/pkg-1.0.tar.gz", + False, + None, + ) + + download_source.assert_not_called() + # The per-wheel log handler is removed even when the build fails. + assert logging.getLogger().handlers == handlers_before + + +def test_build_uses_existing_wheel_without_build_sdist_hook( + tmp_context: WorkContext, +) -> None: + """Reuse a built wheel without the hook, because no sdist is built.""" + _configure_package( + tmp_context, "source:\n provider: pypi-sdist\n build_sdist: hook\n" + ) + req = Requirement("pkg==1.0") + wheel = tmp_context.wheels_downloads / "pkg-1.0-py3-none-any.whl" + + with ( + patch("fromager.overrides.find_override_method", return_value=None), + patch.object(build, "_is_wheel_built", return_value=wheel), + patch.object(build.sources, "download_source") as download_source, + patch.object(build.server, "update_wheel_mirror"), + ): + result = build._build( + tmp_context, + Version("1.0"), + req, + "https://pkg.test/pkg-1.0.tar.gz", + False, + None, + ) + + assert result.skipped is True + assert result.wheel_filename == wheel + download_source.assert_not_called() diff --git a/tests/test_packagesettings.py b/tests/test_packagesettings.py index 6a37a8266..3baec86ba 100644 --- a/tests/test_packagesettings.py +++ b/tests/test_packagesettings.py @@ -13,6 +13,7 @@ from fromager.packagesettings import ( Annotations, BuildDirectory, + BuildSDist, EnvVars, ExternalCommands, GitOptions, @@ -1260,6 +1261,34 @@ def test_pbi_source_resolver_variant_overrides_package( assert isinstance(pbi.source_resolver, PyPIPrebuiltResolver) +def test_settings_lookups_do_not_check_build_sdist_hook( + tmp_path: pathlib.Path, +) -> None: + """Listing settings works when an unused package lacks its selected hook.""" + hook_pkg = PackageSettings.from_string( + "hook-pkg", "source:\n provider: pypi-sdist\n build_sdist: hook\n" + ) + other_pkg = PackageSettings.from_string("other-pkg", "env:\n VALUE: set\n") + settings = Settings( + settings=SettingsFile(), + package_settings=[hook_pkg, other_pkg], + variant="cpu", + patches_dir=tmp_path, + max_jobs=1, + ) + + with patch( + "fromager.overrides.find_override_method", return_value=None + ) as find_override: + pre_built = settings.list_pre_built() + pbi = settings.package_build_info("hook-pkg") + + assert pre_built == set() + assert pbi.source_resolver is not None + assert pbi.source_resolver.build_sdist is BuildSDist.hook + find_override.assert_not_called() + + @pytest.mark.parametrize( ("yaml_str", "expected"), [ diff --git a/tests/test_packagesettings_resolver.py b/tests/test_packagesettings_resolver.py index 5f05ec123..5773c5c01 100644 --- a/tests/test_packagesettings_resolver.py +++ b/tests/test_packagesettings_resolver.py @@ -104,7 +104,7 @@ def test_resolver_requires_build_sdist_strategy() -> None: @pytest.mark.parametrize( "provider", - ["pypi-sdist", "pypi-prebuilt", "hook-prebuilt", "not-available"], + ["pypi-prebuilt", "hook-prebuilt", "not-available"], ) def test_resolver_rejects_unsupported_build_sdist_strategy(provider: str) -> None: """Keep fixed resolver strategies from accepting a different build method.""" @@ -137,6 +137,18 @@ def test_default_index_url(self) -> None: r = PyPISDistResolver(provider="pypi-sdist") assert str(r.index_url) == "https://pypi.org/simple/" + def test_parse_explicit_pep517(self) -> None: + """Allow opting into PEP 517 for a downloaded sdist.""" + r = _parse( + """\ + source: + provider: pypi-sdist + index_url: https://pypi.test/simple + build_sdist: pep517 + """ + ) + assert r.build_sdist is BuildSDist.pep517 + def test_cooldown_override(self) -> None: r = PyPISDistResolver(provider="pypi-sdist", min_release_age=7) assert r.min_release_age == 7 @@ -240,6 +252,20 @@ def test_parse(self) -> None: assert r.download_kind == DownloadKind.tarball assert r.download_kinds == frozenset({DownloadKind.sdist, DownloadKind.tarball}) + def test_parse_explicit_pep517(self) -> None: + """Allow opting into PEP 517 for a custom source archive.""" + r = _parse( + """\ + source: + provider: pypi-download + index_url: https://pypi.test/simple + download_url: https://download.test/pkg-{version}.tar.gz + download_kind: tarball + build_sdist: pep517 + """ + ) + assert r.build_sdist is BuildSDist.pep517 + def test_download_kind_sdist(self) -> None: r = _parse("""\ source: @@ -1607,6 +1633,57 @@ def hook( # -- Discriminated union validation ------------------------------------------- +@pytest.mark.parametrize( + "profile_yaml", + [ + TestPyPISDistResolver.YAML, + TestPyPIDownloadResolver.YAML, + TestPyPIGitResolver.YAML, + TestGitHubTagDownloadResolver.YAML, + TestGitHubTagCloneResolver.YAML, + TestGitLabTagDownloadResolver.YAML, + TestGitLabTagCloneResolver.YAML, + TestHookSDistResolver.YAML, + ], + ids=[ + "pypi-sdist", + "pypi-download", + "pypi-git", + "github-tag-download", + "github-tag-git", + "gitlab-tag-download", + "gitlab-tag-git", + "hook-sdist", + ], +) +def test_source_profiles_accept_build_sdist_hook(profile_yaml: str) -> None: + """Allow source profiles to select a build hook independently of acquisition.""" + data = yaml.safe_load(textwrap.dedent(profile_yaml)) + data["source"]["build_sdist"] = "hook" + + source = _SourceWrapper.model_validate(data).source + + assert source.build_sdist is BuildSDist.hook + + +@pytest.mark.parametrize( + "profile_yaml", + [ + TestPyPIPrebuiltResolver.YAML, + TestHookPrebuiltResolver.YAML, + TestNotAvailableResolver.YAML, + ], + ids=["pypi-prebuilt", "hook-prebuilt", "not-available"], +) +def test_non_source_profiles_reject_build_sdist_hook(profile_yaml: str) -> None: + """Prebuilt and unavailable profiles cannot select an sdist build hook.""" + data = yaml.safe_load(textwrap.dedent(profile_yaml)) + data["source"]["build_sdist"] = "hook" + + with pytest.raises(pydantic.ValidationError, match="build_sdist"): + _SourceWrapper.model_validate(data) + + def test_invalid_provider() -> None: with pytest.raises(pydantic.ValidationError): _parse("""\ diff --git a/tests/test_sources.py b/tests/test_sources.py index 54732d583..5ad7542b8 100644 --- a/tests/test_sources.py +++ b/tests/test_sources.py @@ -11,7 +11,7 @@ from packaging.requirements import Requirement from packaging.version import Version -from fromager import context, packagesettings, resolver, sources +from fromager import build_environment, context, packagesettings, resolver, sources from fromager.candidate import Cooldown from fromager.requirements_file import RequirementType @@ -371,6 +371,29 @@ def test_resolve_source_uses_configured_source_resolver( ) +@patch("fromager.resolver.find_all_matching_from_provider") +def test_resolve_source_does_not_check_build_sdist_hook( + find_all_matching_from_provider: Mock, + tmp_context: context.WorkContext, +) -> None: + """Resolve without the selected sdist hook, so a download-only step works.""" + ps = packagesettings.PackageSettings.from_string( + "test-pkg", "source:\n provider: pypi-sdist\n build_sdist: hook\n" + ) + tmp_context.settings._package_settings[ps.name] = ps + req = Requirement("test-pkg>=1.0") + find_all_matching_from_provider.return_value = [("url", Version("1.0"))] + + with patch("fromager.overrides.find_override_method", return_value=None): + url, version = sources.resolve_source( + ctx=tmp_context, + req=req, + sdist_server_url=resolver.PYPI_SERVER_URL, + ) + + assert (url, version) == ("url", Version("1.0")) + + @patch("fromager.resolver.find_all_matching_from_provider") @patch("fromager.downloads.download_sdist") def test_resolve_source_from_settings( @@ -1804,13 +1827,25 @@ def test_prepare_source_plugin_returns_bad_tuple( packagesettings.HookSDistResolver( provider="hook-sdist", build_sdist=packagesettings.BuildSDist.pep517 ), + packagesettings.PyPISDistResolver( + provider="pypi-sdist", build_sdist=packagesettings.BuildSDist.pep517 + ), + packagesettings.PyPIDownloadResolver( + provider="pypi-download", + download_url=pydantic.HttpUrl("https://pkg.test/pkg-{version}.tar.gz"), + download_kind=packagesettings.DownloadKind.tarball, + build_sdist=packagesettings.BuildSDist.pep517, + ), ], - ids=["git-clone", "hook-source"], + ids=["git-clone", "hook-source", "pypi-sdist", "pypi-download"], ) @pytest.mark.parametrize("has_override", [False, True]) def test_build_sdist_uses_pep517_for_configured_source( source_resolver: ( - packagesettings.GitHubTagCloneResolver | packagesettings.HookSDistResolver + packagesettings.GitHubTagCloneResolver + | packagesettings.HookSDistResolver + | packagesettings.PyPISDistResolver + | packagesettings.PyPIDownloadResolver ), has_override: bool, tmp_context: context.WorkContext, @@ -1858,40 +1893,117 @@ def test_build_sdist_uses_pep517_for_configured_source( ) find_and_invoke.assert_not_called() find_override.assert_called_once_with(req.name, "build_sdist") - warning = "ignoring build_sdist override because source.build_sdist is pep517" + warning = ( + "ignoring build_sdist override because source.build_sdist is pep517; " + "set source.build_sdist: hook to use it" + ) assert (warning in caplog.text) is has_override if override is not None: override.assert_not_called() @pytest.mark.parametrize( - "source_resolver", + ("source_config", "has_override", "uses_hook"), [ - None, - packagesettings.HookSDistResolver( - provider="hook-sdist", build_sdist=packagesettings.BuildSDist.tarball + pytest.param("", False, True, id="legacy"), + pytest.param("", True, True, id="legacy-override"), + pytest.param( + "source:\n provider: pypi-sdist\n", False, False, id="pypi-sdist" ), - packagesettings.GitHubTagCloneResolver( - provider="github-tag-git", - project_url=pydantic.HttpUrl("https://github.test/org/pkg"), - build_sdist=packagesettings.BuildSDist.tarball, + pytest.param( + "source:\n provider: pypi-sdist\n", True, False, id="pypi-sdist-override" + ), + pytest.param( + "source:\n provider: hook-sdist\n build_sdist: tarball\n", + False, + False, + id="hook-sdist-tarball", + ), + pytest.param( + "source:\n provider: hook-sdist\n build_sdist: tarball\n", + True, + False, + id="hook-sdist-tarball-override", + ), + pytest.param( + "source:\n" + " provider: github-tag-git\n" + " project_url: https://github.test/org/pkg\n" + " build_sdist: tarball\n", + False, + False, + id="git-tarball", + ), + pytest.param( + "source:\n" + " provider: github-tag-git\n" + " project_url: https://github.test/org/pkg\n" + " build_sdist: tarball\n", + True, + False, + id="git-tarball-override", + ), + pytest.param( + "source:\n provider: pypi-sdist\n build_sdist: hook\n", + True, + True, + id="pypi-sdist-hook", + ), + pytest.param( + "source:\n" + " provider: pypi-download\n" + " download_url: https://pkg.test/pkg-{version}.tar.gz\n" + " download_kind: tarball\n" + " build_sdist: hook\n", + True, + True, + id="pypi-download-hook", + ), + pytest.param( + "source:\n provider: hook-sdist\n build_sdist: hook\n", + True, + True, + id="hook-sdist-hook", + ), + pytest.param( + "source:\n" + " provider: github-tag-git\n" + " project_url: https://github.test/org/pkg\n" + " build_sdist: hook\n", + True, + True, + id="git-hook", + ), + pytest.param( + "source:\n provider: pypi-sdist\n" + "variants:\n cpu:\n source:\n" + " provider: pypi-sdist\n build_sdist: hook\n", + True, + True, + id="variant-hook", + ), + pytest.param( + "source:\n provider: pypi-sdist\n build_sdist: hook\n" + "variants:\n cpu:\n source:\n provider: pypi-sdist\n", + True, + False, + id="variant-tarball", ), ], - ids=["legacy", "hook-source", "configured-tarball"], ) -@pytest.mark.parametrize("has_override", [False, True]) -def test_build_sdist_uses_configured_tarball_or_legacy_hook( - source_resolver: ( - packagesettings.HookSDistResolver - | packagesettings.GitHubTagCloneResolver - | None - ), +def test_build_sdist_uses_configured_strategy_or_legacy_hook( + source_config: str, has_override: bool, + uses_hook: bool, tmp_context: context.WorkContext, tmp_path: pathlib.Path, caplog: pytest.LogCaptureFixture, ) -> None: - """Use configured tarball builds directly and keep overrides for legacy settings.""" + """Use configured strategies directly and keep hooks for legacy and hook settings.""" + # Arrange + if source_config: + ps = packagesettings.PackageSettings.from_string("pkg", source_config) + tmp_context.settings._package_settings[ps.name] = ps req = Requirement("pkg==1.0") version = Version("1.0") build_env = Mock() @@ -1900,14 +2012,11 @@ def test_build_sdist_uses_configured_tarball_or_legacy_hook( sdist_file.write_text("fake sdist") override = Mock() if has_override else None + # Act with ( - patch.object( - packagesettings.PackageBuildInfo, - "source_resolver", - new_callable=PropertyMock, - return_value=source_resolver, - ), - patch("fromager.packagesettings.get_extra_environ", return_value={}), + patch( + "fromager.overrides.find_override_method", return_value=override + ) as find_override, patch( "fromager.sources.overrides.find_and_invoke", return_value=sdist_file ) as find_and_invoke, @@ -1915,9 +2024,7 @@ def test_build_sdist_uses_configured_tarball_or_legacy_hook( patch( "fromager.sources.default_build_sdist", return_value=sdist_file ) as tarball, - patch( - "fromager.sources.overrides.find_override_method", return_value=override - ) as find_override, + patch("fromager.packagesettings.get_extra_environ", return_value={}), ): result = sources.build_sdist( ctx=tmp_context, @@ -1927,6 +2034,7 @@ def test_build_sdist_uses_configured_tarball_or_legacy_hook( build_env=build_env, ) + # Assert assert result == sdist_file.resolve() build_kwargs = dict( ctx=tmp_context, @@ -1937,21 +2045,275 @@ def test_build_sdist_uses_configured_tarball_or_legacy_hook( build_dir=sdist_root_dir, build_env=build_env, ) - if source_resolver is None: + if uses_hook: find_and_invoke.assert_called_once_with( req.name, "build_sdist", tarball, **build_kwargs ) tarball.assert_not_called() - find_override.assert_not_called() else: - tarball.assert_called_once_with(**build_kwargs, exclude_vcs=True) find_and_invoke.assert_not_called() + tarball.assert_called_once_with(**build_kwargs) + pep517.assert_not_called() + if source_config: + # Either the hook requirement or the skipped-override warning looks + # up the hook, but a configured strategy never invokes it. find_override.assert_called_once_with(req.name, "build_sdist") - if override is not None: + if override is not None and not uses_hook: override.assert_not_called() + else: + find_override.assert_not_called() warning = "ignoring build_sdist override because source.build_sdist is tarball" - assert (warning in caplog.text) is (source_resolver is not None and has_override) + assert (warning in caplog.text) is (not uses_hook and has_override) + + +@pytest.mark.parametrize( + "source_config", + [ + pytest.param("", id="legacy"), + pytest.param( + "source:\n provider: pypi-sdist\n build_sdist: hook\n", + id="pypi-sdist-hook", + ), + ], +) +def test_build_sdist_hook_can_archive_repository( + source_config: str, + tmp_context: context.WorkContext, + tmp_path: pathlib.Path, +) -> None: + """Selected hooks can archive the whole prepared repository.""" + # Arrange + ps = packagesettings.PackageSettings.from_string( + "pkg", "build_dir: python\n" + source_config + ) + tmp_context.settings._package_settings[ps.name] = ps + req = Requirement("pkg==1.0") + version = Version("1.0") + build_env = Mock(spec=build_environment.BuildEnvironment) + sdist_root_dir = tmp_path / "pkg-1.0" + build_dir = sdist_root_dir / "python" + build_dir.mkdir(parents=True) + (build_dir / "setup.py").write_text("# Wheel build also needs ../native.c.\n") + (sdist_root_dir / "native.c").write_text("int value(void) { return 1; }\n") + hook_calls: list[pathlib.Path] = [] + + def archive_repository( + ctx: context.WorkContext, + extra_environ: dict[str, str], + req: Requirement, + version: Version, + sdist_root_dir: pathlib.Path, + build_env: build_environment.BuildEnvironment, + build_dir: pathlib.Path, + ) -> pathlib.Path: + """Keep the whole repository while the wheel builds from a subdirectory.""" + hook_calls.append(build_dir) + (sdist_root_dir / "generated.txt").write_text(extra_environ["BUILD_VALUE"]) + return sources.default_build_sdist( + ctx=ctx, + extra_environ=extra_environ, + req=req, + version=version, + sdist_root_dir=sdist_root_dir, + build_env=build_env, + build_dir=sdist_root_dir, + ) + + # Act + with ( + patch( + "fromager.overrides.find_override_method", return_value=archive_repository + ), + patch( + "fromager.packagesettings.get_extra_environ", + return_value={"BUILD_VALUE": "prepared"}, + ), + ): + sdist_file = sources.build_sdist( + ctx=tmp_context, + req=req, + version=version, + sdist_root_dir=sdist_root_dir, + build_env=build_env, + ) + + # Assert + assert sdist_file == tmp_context.sdists_builds / "pkg-1.0.tar.gz" + assert hook_calls == [build_dir] + with tarfile.open(sdist_file, "r:gz") as sdist: + names = sdist.getnames() + generated = sdist.extractfile("pkg-1.0/generated.txt") + assert generated is not None + assert generated.read() == b"prepared" + assert "pkg-1.0/PKG-INFO" in names + assert "pkg-1.0/native.c" in names + assert "pkg-1.0/generated.txt" in names + assert "pkg-1.0/python/setup.py" in names + + +@pytest.mark.parametrize("hook", [None, "not-callable"]) +def test_build_sdist_requires_selected_hook( + hook: str | None, + tmp_context: context.WorkContext, + tmp_path: pathlib.Path, +) -> None: + """Reject a missing selected hook before invoking environment or build hooks.""" + req = Requirement("pkg==1.0") + ps = packagesettings.PackageSettings.from_string( + "pkg", "source:\n provider: pypi-sdist\n build_sdist: hook\n" + ) + tmp_context.settings._package_settings[ps.name] = ps + with ( + patch("fromager.overrides.find_override_method", return_value=hook), + patch("fromager.packagesettings.get_extra_environ") as extra_environ, + patch("fromager.sources.pep517_build_sdist") as pep517, + patch("fromager.sources.default_build_sdist") as tarball, + ): + with pytest.raises( + ValueError, + match=r"pkg: source\.build_sdist='hook' requires a callable build_sdist override hook", + ): + sources.build_sdist( + ctx=tmp_context, + req=req, + version=Version("1.0"), + sdist_root_dir=tmp_path, + build_env=Mock(spec=build_environment.BuildEnvironment), + ) + + extra_environ.assert_not_called() pep517.assert_not_called() + tarball.assert_not_called() + + +@pytest.mark.parametrize( + "source_config", + [ + pytest.param( + "source:\n provider: pypi-sdist\n build_sdist: hook\n", + id="package-hook", + ), + pytest.param( + "variants:\n cpu:\n source:\n" + " provider: pypi-sdist\n build_sdist: hook\n", + id="variant-hook", + ), + pytest.param( + "source:\n provider: pypi-sdist\n build_sdist: hook\n" + "variants:\n cpu:\n env:\n BUILD_VALUE: prepared\n", + id="inherited-hook", + ), + ], +) +@pytest.mark.parametrize("hook", [None, "not-callable"]) +def test_require_build_sdist_hook_rejects_unusable_hook( + source_config: str, hook: str | None, tmp_context: context.WorkContext +) -> None: + """Reject a missing or non-callable hook for the selected variant.""" + ps = packagesettings.PackageSettings.from_string("test-pkg", source_config) + tmp_context.settings._package_settings[ps.name] = ps + + with patch("fromager.overrides.find_override_method", return_value=hook): + with pytest.raises( + ValueError, + match=r"test-pkg: source\.build_sdist='hook' requires a callable", + ): + sources.require_build_sdist_hook( + ctx=tmp_context, req=Requirement("test-pkg") + ) + + +@pytest.mark.parametrize( + "source_config", + [ + pytest.param("", id="legacy"), + pytest.param("source:\n provider: pypi-sdist\n", id="tarball"), + pytest.param( + "source:\n provider: pypi-sdist\n build_sdist: hook\n" + "variants:\n cpu:\n source:\n provider: pypi-sdist\n", + id="variant-replaces-package-hook", + ), + pytest.param( + "variants:\n cuda:\n source:\n" + " provider: pypi-sdist\n build_sdist: hook\n", + id="inactive-variant-hook", + ), + ], +) +def test_require_build_sdist_hook_skips_unselected_hook( + source_config: str, tmp_context: context.WorkContext +) -> None: + """Do not look up a hook unless the effective settings select it.""" + ps = packagesettings.PackageSettings.from_string("test-pkg", source_config) + tmp_context.settings._package_settings[ps.name] = ps + + with patch("fromager.overrides.find_override_method") as find_override: + sources.require_build_sdist_hook(ctx=tmp_context, req=Requirement("test-pkg")) + + find_override.assert_not_called() + + +@pytest.mark.parametrize( + ("output", "error_type", "message"), + [ + ("missing", FileNotFoundError, "pkg-1.0.tar.gz"), + ("outside", ValueError, "is not in sdists build directory"), + ("other-1.0.tar.gz", ValueError, "does not match requirement"), + ("pkg-2.0.tar.gz", ValueError, "does not match public version"), + ("exception", RuntimeError, "sdist generation failed"), + ], +) +def test_build_sdist_validates_selected_hook_result( + output: str, + error_type: type[Exception], + message: str, + tmp_context: context.WorkContext, + tmp_path: pathlib.Path, +) -> None: + """Reject invalid hook output and propagate hook failures without fallback.""" + ps = packagesettings.PackageSettings.from_string( + "pkg", "source:\n provider: pypi-sdist\n build_sdist: hook\n" + ) + tmp_context.settings._package_settings[ps.name] = ps + if output == "outside": + sdist_file = tmp_path / "pkg-1.0.tar.gz" + else: + filename = "pkg-1.0.tar.gz" if output == "missing" else output + sdist_file = tmp_context.sdists_builds / filename + if output not in {"missing", "exception"}: + sdist_file.touch() + + def hook( + ctx: context.WorkContext, + extra_environ: dict[str, str], + req: Requirement, + version: Version, + sdist_root_dir: pathlib.Path, + build_env: build_environment.BuildEnvironment, + build_dir: pathlib.Path, + ) -> pathlib.Path: + """Return invalid output or raise a generation error.""" + if output == "exception": + raise RuntimeError("sdist generation failed") + return sdist_file + + with ( + patch("fromager.overrides.find_override_method", return_value=hook), + patch("fromager.packagesettings.get_extra_environ", return_value={}), + patch("fromager.sources.pep517_build_sdist") as pep517, + patch("fromager.sources.default_build_sdist") as tarball, + ): + with pytest.raises(error_type, match=message): + sources.build_sdist( + ctx=tmp_context, + req=Requirement("pkg==1.0"), + version=Version("1.0"), + sdist_root_dir=tmp_path, + build_env=Mock(spec=build_environment.BuildEnvironment), + ) + + pep517.assert_not_called() + tarball.assert_not_called() @pytest.mark.parametrize("strategy", [None, "unsupported"]) @@ -2120,13 +2482,29 @@ def test_default_build_sdist_normalizes_name_and_root( assert "foo_bar-1.0/setup.py" in names -@pytest.mark.parametrize("exclude_vcs", [False, True]) +@pytest.mark.parametrize( + "source_config", + [ + pytest.param("", id="legacy"), + pytest.param("source:\n provider: pypi-sdist\n", id="package-source"), + pytest.param( + "variants:\n cpu:\n source:\n" + " provider: github-tag-git\n" + " project_url: https://github.test/org/pkg\n", + id="variant-source", + ), + ], +) +@pytest.mark.parametrize("exclude_vcs", [None, False, True]) def test_default_build_sdist_exclude_vcs( - exclude_vcs: bool, + source_config: str, + exclude_vcs: bool | None, tmp_context: context.WorkContext, tmp_path: pathlib.Path, ) -> None: - """Leave VCS metadata out of the sdist only when asked.""" + """Use the source profile's VCS policy unless explicitly overridden.""" + ps = packagesettings.PackageSettings.from_string("pkg", source_config) + tmp_context.settings._package_settings[ps.name] = ps build_dir = tmp_path / "pkg-1.0" (build_dir / ".git" / "objects").mkdir(parents=True) (build_dir / ".git" / "HEAD").write_text("ref: refs/heads/main\n") @@ -2149,6 +2527,7 @@ def test_default_build_sdist_exclude_vcs( names = tar.getnames() assert "pkg-1.0/setup.py" in names has_vcs = {"pkg-1.0/.git/HEAD", "pkg-1.0/sub/.git"} <= set(names) - assert has_vcs is not exclude_vcs - if exclude_vcs: - assert not any("/.git" in name for name in names) + should_exclude = bool(source_config) if exclude_vcs is None else exclude_vcs + assert has_vcs is not should_exclude + if should_exclude: + assert not any(".git" in pathlib.PurePosixPath(name).parts for name in names)