Publish nine hosts instead of four - #161
Merged
Merged
Conversation
build_musl_host() produced only the SDK tarball and refused outright if the matrix asked for a packaged musl host. musl is native inside its own Alpine container, unlike a canadian cross, so the same container can also fetch the vdpm bundle, build core-package/bootstrap-archive, and run the bootstrap smoke test before anything leaves the container. libarchive-tools joins the apk list: create-core-package.sh drives bsdtar for the .pkg.tar.xz format. The artifact-staging/provenance tail shared with build_and_stage() is now stage_and_write_provenance(), reused instead of duplicated a second time.
x86_64/aarch64-linux-musl (stage 2) and x86_64/aarch64-unknown-freebsd (stage 3) now get the packaged treatment, taking the published core matrix from 4 hosts to 8. FreeBSD is a canadian cross already handled by build_and_stage(); musl needed build_musl_host() to learn packaging first. x86_64-apple-darwin stays unpackaged: it still cross-builds from arm64-apple-darwin and moving it to a native runner is separate work.
Every stage and the grouping job required every upstream stage to have zero failed matrix legs, so one bad host anywhere -- including a host that nothing downstream of it depends on -- skipped every later stage and group, publishing nothing at all. Which hosts are required for a publishable release is a policy decision for the caller of this reusable workflow, not something the shell should encode; it now only still skips on an explicit cancellation, and lets whatever built reach grouping. create-core-repositories.sh already refuses to group an architecture that is missing either the core or the client package, so a half-built host still fails loudly there instead of publishing thin.
Loosening the gates so a host that fails to build only removes itself also let a failed Windows bootstrap smoke test through, which is a different thing: that host built something broken, and Windows is a required host, so the tree would have carried it to publication.
Every leg re-fetched each source tarball from its upstream host, so one of them answering 503 -- pyyaml.org did, in the first check of the autobuilds PR -- failed a whole leg. Eight publishing hosts multiply that exposure, so the downloads directory is cached the way the older workflow already cached it, keyed per OS since hosts of a kind fetch the same tarballs.
It was cross-built from Linux, so nothing ever executed what it produced -- no toolchain contract, no bootstrap smoke test -- which is why it stayed unpublished. A macos-15-intel runner builds it natively like every other host that can run its own output, and it publishes with the rest.
ld64 looks for a dylib in every search path before it considers a static library in any of them, so gmp built here -- static only -- lost to Homebrew's copy, which the runner's preinstalled GCC drags in. It only bites on Intel, where Homebrew lives in /usr/local and that is an implicit search path; on Apple Silicon it sits in /opt/homebrew and never enters the search. isl reported it as "gmp library too old", which is what its link probe says whatever the reason it failed.
FreeBSD's cross compiler answers to x86_64-unknown-freebsd14 while the host publishes as x86_64-unknown-freebsd, and one variable was serving both. So the SDK tarball, the bootstrap archive, the core package, the pacman architecture and the host handed to the vdpm bundle all carried a release number that vdpm's own release-info.txt does not, and the bundle check refused them -- silently, since that grep had no message. Everywhere the name is an identity now uses the published one, which the executor passes from the lock; the toolchain lookups keep the triplet. Every other host derives the same string either way, which is why nothing noticed until FreeBSD had to publish.
An autotools check says why it failed in config.log and nowhere else; what reaches the build log is whatever string that check carries, which sent this repository chasing a gmp that was never too old. The old autobuilds job dumped these on failure and the executor did not inherit it.
Its x86_64 assembly reaches lookup tables like __gmp_binvert_limb_table from the text segment, and Apple's linker refuses those relocations inside a position-independent executable -- so every link against the gmp built here died on Intel macOS, and isl reported it as "gmp library too old", the string its probe prints for any failed link. Reproduced and fixed in isolation before touching this: gmp and isl alone on an Intel runner, where a configure that took half an hour to reach here takes eight minutes. The search-paths flag from the previous attempt goes with it -- the linker was never looking in the wrong place.
gdb was the one component still building with NLS, so on Intel macOS it linked Homebrew's libintl -- a library no user's machine has, which is why the dependency audit refused the SDK. Its siblings all disable it already. The audit now reports every unexpected dependency before failing rather than the first: each one costs a full build to find, and a host that drags in one library from the machine tends to drag in several.
frangarcj
added a commit
that referenced
this pull request
Aug 25, 2026
x86_64-apple-darwin became native in #161 because nothing ever executed what the cross produced: no toolchain contract, no bootstrap smoke test, so it could not be published. That was the right call for the evidence and it cost the time. In the first real nine-host publication it took 62 minutes against 14 for its arm64 sibling, and it is the pole every other host waits behind. "Only a native build can run its own output" is true of a canadian cross to Windows or FreeBSD and false of Apple: an arm64 Mac runs x86_64 Mach-O through Rosetta, so the machine that cross-builds the Intel SDK is the one that can check it. The gate now asks how to run this host's binaries here rather than whether the build is native, and passes the launcher to both the contract and the smoke test. Windows and FreeBSD answer "cannot" and are skipped exactly as before. Measured end to end in the probe repository before touching this: cross 650s, contract 24s, bootstrap smoke 30s, 12m18s for the whole job against 62m22s native -- same checks, same published host.
frangarcj
added a commit
that referenced
this pull request
Aug 26, 2026
hosts.json says x86_64-apple-darwin is stage 3, cross-built on the arm64 runner; build-host.sh has called it native since #161, which made it so in both places at once. db4d1a5 turned the lock back into a cross and left the recipe alone, so stage 3 configured with the machine's own compiler: "-- Host: arm64-apple-darwin", every component at --host=aarch64-apple- darwin, and an arm64 SDK published under the Intel name. It stayed green through the SDK, the static validation and the toolchain contract -- the contract runs run.sh under arch -x86_64, and a Rosetta process execs an arm64 binary natively, so it never demanded the slice. What demanded it was the bootstrap smoke test, 18 minutes in: arch -x86_64 bootstrap-installed/bin/arm-vita-eabi-gcc, Bad CPU type in executable. The recipe restored here is the one #161 deleted, unchanged: the wrappers from setup-macos-cross.sh in PATH and the toolchain file that pins the triplet. The test is what was missing. The lock and the case block are two halves of one decision and nothing tied them together, so it runs the real case block for every host in hosts.json and fails when one that names a build_host gets no toolchain file, or one that names none gets a toolchain file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The client has shipped bundles for nine hosts since v0.1.2; the core has published for four. This closes the gap, so a musl, FreeBSD or Intel-Mac user installs the same way everyone else does.
musl needed real work. Its leg builds inside an Alpine container and only ever produced the SDK tarball — there was even a guard refusing to let it publish, precisely so nobody flipped the flag and shipped a host with no packages. Now the container also fetches the vdpm bundle, builds
core-packageandbootstrap-archive, and runs the bootstrap smoke test on itself, which it can do because musl is native there, unlike a canadian cross. The bundle is downloaded on the runner rather than inside the container, so the image never needs network.libarchive-toolsis the one dependency Alpine was missing —bsdtar, which the packaging drives.macOS x86_64 moves from cross to native. It was cross-built from Linux, so nothing ever executed what it produced: no toolchain contract, no bootstrap smoke test. That is why it stayed unpublished. It builds on
macos-15-intelnow, like every other host that can run its own output, and the cross toolchain path goes away with it.FreeBSD needed the flag and a name of its own. Its build path is Windows': packaged from a host that cannot execute it, which already works. But its cross compiler answers to
x86_64-unknown-freebsd14while the host publishes asx86_64-unknown-freebsd, and a single variable was serving both — so the SDK tarball, the bootstrap archive, the core package, the pacman architecture and the host handed to the vdpm bundle all carried a release number that vdpm's ownrelease-info.txtdoes not, and the bundle check refused them silently, since that grep had no message. Everywhere the name is an identity now uses the published one, which the executor passes down from the lock; the toolchain lookups keep the triplet. Every other host derives the same string either way, which is why nothing noticed until FreeBSD had to publish.Two components were reaching for the build machine's libraries. gmp now builds
--with-pic: its x86_64 assembly addresses lookup tables from the text segment, and Apple's linker refuses those relocations inside a position-independent executable. And gdb now builds--disable-nls, as every other component here already did — left on, it links the host's libintl, which on Intel macOS is Homebrew's: the dependency audit refuses it, and a user's machine would not have it.One bad host no longer blocks the rest. The grouping job used to require every stage to succeed, so a flaky runner on any host stopped the whole publication — including the four that have always worked. Who is indispensable is release policy, and that lives in
autobuilds' required-host list, so a host that fails to build now only removes itself and the caller rejects a tree missing something it needs. A failed smoke test still blocks the tree, though: that is a host that built something broken, which is a different thing from a host that did not build.Source downloads are cached. Every leg re-fetched each tarball from its upstream host, and one of them answering 503 — pyyaml.org did, in the first check of the autobuilds PR — failed a whole leg. Nine legs multiply that exposure, so the downloads directory is cached the way the older workflow already cached it.
Two changes so the next failure costs one build instead of three. A leg that dies now dumps the
config.logof whatever refused to configure: an autotools component says why there and nowhere else, and what reaches the build log is whatever generic string the failing check happens to carry. And the dependency audit reports every offender before exiting rather than the first — each one costs a full build to discover, and a host that drags in one library from the machine usually drags in several.A real lock built from this branch carries ten host entries, nine of them publishing; the tenth is the stage-1 bootstrap that has never published, and
i686-w64-mingw32is pruned out entirely, as it has been since it was dropped from the publication scope.All ten stage jobs are green on this head, and each of the nine publishing hosts produced its SDK tarball, core package and bootstrap archive — the first time musl ×2, FreeBSD ×2 and macOS x86_64 have. Merging does not change who is indispensable:
REQUIRED_HOSTSinautobuildsstays at the four it has always been, so the five new hosts publish best-effort and one bad runner cannot stop a nightly.AI tools were used in preparing this PR (Claude Sonnet 5, Claude Fable 5 and Claude Opus 5, Anthropic).