Skip to content

reproducible-builds.md

Latest commit

 

History

History
193 lines (149 loc) · 8.36 KB

reproducible-builds.md

File metadata and controls

193 lines (149 loc) · 8.36 KB
Raw

Reproducible Builds

See doc/docker.md for build-environment reproducibility.

These practices follow the reproducible-builds.org project's documentation. Every mechanism below contributes to producing bit-identical output across independent environments (CI and local).

Cross-compiler (musl-cross-make)

BUILD = x86_64-pc-linux-gnu pins config.guess so different CI runners produce the same build triplet instead of probing the Docker host kernel. -Wa,--no-pad-sections prevents gas from padding section ends (non-deterministic alignment). --with-debug-prefix-map=$(pwd)=. normalizes build paths in debug info. --enable-compressed-debug-sections=no disables zlib debug-section compression. SOURCE_DATE_EPOCH=0 (extracted tarballs lack .git; the build system cannot derive a commit timestamp from extracted tarballs, so modules/musl-cross-make falls back to echo 0 when git log fails) prevents __DATE__/__TIME__ embedding during the GCC build.

Userland compiler flags

heads_cc (Makefile) injects -fdebug-prefix-map=$(pwd)=heads and -gno-record-gcc-switches for every userland module, normalizing build paths and suppressing non-deterministic compiler flag recording in debug info. modules/libnitrokey additionally uses -ffile-prefix-map=$(pwd)=heads.

Kernel

EXTRA_FLAGS passes -fdebug-prefix-map=$(pwd)=heads -gno-record-gcc-switches to the kernel build. KBUILD_BUILD_USER (pinned to the Linux config filename), KBUILD_BUILD_HOST=linuxboot, KBUILD_BUILD_TIMESTAMP="1970-00-00", and KBUILD_BUILD_VERSION=0 pin all kernel build-identity variables. Modules are stripped with strip --strip-debug --preserve-dates.

Prefix normalization

Most autotools-based modules use --prefix "/" or --prefix "" combined with DESTDIR="$(INSTALL)" so generated Makefiles carry fixed paths; the install target redirects output to the actual build tree. (modules/bash is an exception, using --prefix="/usr".) modules/pciutils additionally sets IDSDIR="/" and PREFIX="/" so libpci.so.3 is path-independent.

rpath removal

modules/gpg2 and modules/cryptsetup2 use --disable-rpath to prevent build paths from being embedded in binaries. modules/tpm2-tss and modules/tpm2-tools use sed to rewrite libtool's hardcode_libdir_flag_spec or hardcode_into_libs in generated configure scripts. modules/cairo rewrites the same variables in the generated libtool file post-configure. modules/util-linux removes .la libtool files post-install.

gpg2

--disable-tpm2d in gpg2_configure eliminates the entire TPM probe chain, removing the non-deterministic BUILD_WITH_TPM2D/HAVE_INTEL_TSS/HAVE_LIBTSS defines from config.h.

coreboot

BUILD_TIMELESS=1 is passed to coreboot's build to produce reproducible ROMs.

Busybox

SOURCE_DATE_EPOCH=0 in busybox_target triggers patches/busybox-1.36.1/0004-trylink-reproducible.patch, which disables ld.bfd --gc-sections (ASLR-influenced hash tables in binutils 2.44). patches/busybox-1.36.1/0001-messages.patch replaces AUTOCONF_TIMESTAMP with a fixed "(heads)" string. The install rule copies the binary and runs applets/install.sh directly, avoiding make install's FORCE re-link.

Patches for reproducibility

Patch Mechanism
bash-5.1.16.patch Drops -b from mkversion.sh — removes build timestamp from version.h
busybox-1.36.1/0001-messages.patch Replaces AUTOCONF_TIMESTAMP with fixed "(heads)"
busybox-1.36.1/0004-trylink-reproducible.patch Disables --gc-sections when SOURCE_DATE_EPOCH set
coreboot-4.11/0073-build-race-condition-fixes.patch Fixes parallel-make race conditions (non-deterministic ordering)
openssl-3.0.8.patch SOURCE_DATE_EPOCH replaces time() in mkbuildinf.pl; compiler flags replaced with fixed literal
tpm2-tools-5.6.patch Disables git describe --tags --dirty > VERSION

Version and timestamp pins

Module Mechanism
modules/hotp-verification GITVERSION="" — removes git-derived version string
modules/tpm2-tools echo version > ./VERSION — pinned version file
modules/bash LDFLAGS="-s", CFLAGS="-g0 -Os" — strip symbols, no debug
modules/fbwhiptail LDFLAGS="-s", CFLAGS="-g0 -Os" — strip symbols, no debug
modules/zstd CFLAGS="-g0 -Os" — no debug info

Archive determinism

bin/cpio-clean.pl rewrites every newc cpio entry for determinism: files sorted by name, inodes derived from MD5(filename), timestamps/uid/gid zeroed, nlink=0, devmajor/devminor=0, check=0, and 512-byte trailing padding. blobs/dev.cpio is a pre-built, git-tracked archive providing a reproducible /dev/console.

modules/linux adds a FORCE dependency on modules.cpio so hashes.txt is always complete on every rebuild, not just cold builds. The do-cpio macro uses cmp --quiet to short-circuit identical output, avoiding unnecessary rewrites.

Tarball downloads

bin/fetch_source_archive.sh uses --timeout=30 --tries=3 -4 (IPv4 preference, fast wget timeout). All download attempts are logged to build/mirror_fallbacks.log. bin/fetch_musl_cross_make_archive.sh pre-seeds musl-cross-make component tarballs into packages/ via fetch_source_archive.sh.

Verifying ROM Reproducibility

Prerequisites

  • Same git commit on both CI and local (different commits produce different ROMs — see below)
  • Build with docker_repro.sh locally (same Docker image as CI)
  • Clean working tree (no uncommitted changes) — a dirty tree adds dirty to GIT_STATUS in build metadata, which changes ROM output
  • Download CI hashes.txt from CircleCI artifacts for the same commit (see Downloading Heads for details). Example URL: 
    wget -O /tmp/ci-hashes.txt \
  "https://output.circle-artifacts.com/output/job/circleci-job-id/artifacts/0/build/x86/EOL_t480-hotp-maximized/hashes.txt"

Output files

A build produces these hash-related files under build/<arch>/<board>/:

File Content
hashes.txt SHA-256 of every build artifact (cpio archives, bzImage, ROM) plus per-file hashes inside each cpio. Reset at each make invocation; appended by each build rule. The authoritative source for reproducibility verification.
sizes.txt Byte sizes of each artifact, matching the hashes.txt entries.
sha256sum.txt SHA-256 of the final ROM only. Packaged inside the update zip for integrity checks during flash updates.

Both hashes.txt and sha256sum.txt are included in the update zip for offline reproducibility verification.

Understanding hashes.txt

build/$ARCH/$BOARD/hashes.txt records the SHA256 of every file inside every cpio archive, not just the cpio archives themselves. Each cpio section is separated by ----- lines:

<hash>  /path/to/modules.cpio
-----
<hash>  ./lib/modules/usbhid.ko
<hash>  ./lib/modules/e1000e.ko
...
-----
<hash>  /path/to/tools.cpio
-----
<hash>  ./bin/busybox
<hash>  ./bin/kexec
...
-----

Same commit: everything should match

When CI and local build the same git commit, the ROM hash should match. If it does, the build is reproducible:

grep '\.rom' /tmp/ci-hashes.txt build/x86/EOL_t480-hotp-maximized/hashes.txt

If the ROM differs, step down: initrd.cpio.xz/bzImage → tools.cpio → individual files. The innermost differing file (e.g. ./bin/busybox) is the root cause — fix it and the cascade resolves. hashes.txt records every file at every level so no diffoscope is needed until you've identified what differs.

For a comprehensive same-commit check:

diff <(grep '^[0-9a-f]\{64\}' /tmp/ci-hashes.txt | sort) \
     <(grep '^[0-9a-f]\{64\}' build/x86/EOL_t480-hotp-maximized/hashes.txt | sort)

Zero output = all hash and path entries match.

Different commits: ROMs always differ

tools.cpio contains ./etc/config, which embeds GIT_HASH from git rev-parse HEAD. Every commit changes GIT_HASH, so ./etc/config differs between ANY two commits. This cascades: ./etc/config → tools.cpio → initrd.cpio.xz → ROM. The ROM hash WILL differ between different commits — this is expected.

When ./etc/config is the only differing file inside tools.cpio, the build is still reproducible — all binaries (busybox, kexec, gpg, etc.) are byte-identical. A binary mismatch is the actual reproducibility bug to investigate.