ci: build the compiler image on a prebuilt LLVM image (#5671)

* ci: build the compiler image on a prebuilt LLVM image

The Docker workflow built LLVM as the first stages of the compiler image. The
only thing that prevented a rebuild was the BuildKit registry layer cache. A
layer cache is a best-effort optimisation, so CI sometimes built LLVM again
although llvm-version.txt did not change.

Move the LLVM stages to Dockerfile.llvm and make the compiler image start from
that image through an LLVM_IMAGE build argument. The compiler build now holds
no LLVM build step, so it cannot build LLVM again.

Tag the LLVM image with the LLVM revision and a hash of the files that set the
content of the image: llvm-version.txt, Dockerfile.llvm, GNUmakefile, and the
files in make/. tools/llvm-image-tag.sh prints the tag, and CI and developers
use the same script. The workflow builds and pushes the LLVM image only when
the registry does not hold that tag.

Remove the object files and the git history from the LLVM image in the same
layer as the build. The CI caches already link against that subset, see
.github/actions/setup-llvm/action.yml.

Delete llvm.yml. It made an image that nothing used, and it needed a push to a
special branch. The Docker workflow now does the same work when it is
necessary, and the force-llvm input of the manual trigger makes the image
again.

The LLVM image goes to GHCR only, because only CI uses it. The compiler image
continues to go to Docker Hub and GHCR.

* ci: remove the LLVM git history in the layer that makes it

The removal was in the tinygo-llvm-build stage. This stage is below the
layer that makes the shallow clone. A later layer only hides files from a
parent layer. Thus the pack files stayed in the published image.

* ci: link the LLVM image to the repository

The label puts the package in the repository package list, so the package
settings are easy to find. BUILDING.md now tells you to build LLVM locally
if you are not able to pull the image.

Author: deadprogram

.github/workflows/docker.yml

@@ -2,18 +2,85 @@
 # If you are looking for the tinygo/tinygo "release" Docker image please see
 # https://github.com/tinygo-org/docker
 #
+# The compiler image is built on top of an LLVM image. The LLVM image is built
+# only when the registry holds no image for the current LLVM version and build
+# recipe, so a change to TinyGo alone does not build LLVM again. An LLVM build
+# takes at least 1-2 hours.
+#
+# Use the manual trigger with force-llvm to build the LLVM image again.
 name: Docker
 on:
   push:
-    branches: [ dev, fix-docker-llvm-build ]
+    branches: [ dev ]
+  workflow_dispatch:
+    inputs:
+      force-llvm:
+        description: Build the LLVM image again
+        type: boolean
+        default: false
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}
   cancel-in-progress: true
 
 jobs:
+  llvm:
+    name: build-push-llvm
+    runs-on: ubuntu-latest
+    permissions:
+      packages: write
+      contents: read
+    outputs:
+      image: ${{ steps.tag.outputs.image }}
+    steps:
+      - name: Free Disk space
+        shell: bash
+        run: |
+          df -h
+          sudo rm -rf /opt/hostedtoolcache
+          sudo rm -rf /usr/local/lib/android
+          sudo rm -rf /usr/share/dotnet
+          sudo rm -rf /opt/ghc
+          sudo rm -rf /usr/local/graalvm
+          sudo rm -rf /usr/local/share/boost
+          df -h
+      - name: Check out the repo
+        uses: actions/checkout@v6
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v4
+      - name: Log in to Github Container Registry
+        uses: docker/login-action@v4
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+      - name: Get the image tag
+        id: tag
+        shell: bash
+        run: |
+          tag=$(sh tools/llvm-image-tag.sh)
+          echo "image=ghcr.io/${{ github.repository_owner }}/llvm-22:$tag" >> "$GITHUB_OUTPUT"
+      - name: Look for the image
+        id: check
+        shell: bash
+        run: |
+          if docker buildx imagetools inspect "${{ steps.tag.outputs.image }}" > /dev/null 2>&1; then
+            echo "exists=true" >> "$GITHUB_OUTPUT"
+          else
+            echo "exists=false" >> "$GITHUB_OUTPUT"
+          fi
+      - name: Build and push
+        if: steps.check.outputs.exists != 'true' || inputs.force-llvm
+        uses: docker/build-push-action@v7
+        with:
+          context: .
+          file: Dockerfile.llvm
+          push: true
+          tags: ${{ steps.tag.outputs.image }}
+
   push_to_registry:
     name: build-push-dev
+    needs: llvm
     runs-on: ubuntu-latest
     permissions:
       packages: write
@@ -62,7 +129,9 @@ jobs:
         with:
           context: .
           push: true
+          build-args: |
+            LLVM_IMAGE=${{ needs.llvm.outputs.image }}
           tags: ${{ steps.meta.outputs.tags }}
           labels: ${{ steps.meta.outputs.labels }}
           cache-from: type=registry,ref=ghcr.io/${{ github.repository_owner }}/tinygo-dev:buildcache
-          cache-to: type=registry,ref=ghcr.io/${{ github.repository_owner }}/tinygo-dev:buildcache,mode=max
+          cache-to: type=registry,ref=ghcr.io/${{ github.repository_owner }}/tinygo-dev:buildcache,mode=max,ignore-error=true

.github/workflows/llvm.yml

@@ -45,6 +45,9 @@ It provides a help target for quick reference:
     tinygo                          Build the TinyGo compiler
     lint                            Lint source tree
     spell                           Spellcheck source tree
+    llvm-image-tag                  Print the tag of the prebuilt LLVM Docker image
+    docker-llvm                     Build the LLVM base image (slow)
+    docker-tinygo                   Build the TinyGo compiler image
 
 ## Download the source
 
@@ -54,8 +57,8 @@ the git repository). Then, inside the directory, download the LLVM source:
     make llvm-source
 
 The LLVM commit to use is pinned in `llvm-version.txt`. A change to that file
-makes CI build LLVM again, because the file is part of the LLVM cache key. All
-other changes reuse the cached LLVM build.
+makes CI build LLVM again. The file is part of the LLVM cache key and of the
+Docker image tag. All other changes reuse the cached LLVM build.
 
 You can also store LLVM outside of the TinyGo root directory by setting the
 `LLVM_BUILDDIR`, `CLANG_SRC` and `LLD_SRC` make variables, but that is not
@@ -105,6 +108,29 @@ On macOS, use otool -L:
 
 The result should not contain libclang or libLLVM.
 
+## Build with Docker
+
+The Docker build uses two images. The LLVM image holds the LLVM build, and the
+compiler image holds TinyGo. The LLVM image changes only when
+`llvm-version.txt`, `Dockerfile.llvm`, or the make files change, so the slow
+LLVM build does not run again for each change to TinyGo.
+
+To build both images:
+
+    make docker-llvm
+    make docker-tinygo
+
+The first command takes 1-2 hours. To use the LLVM image that CI published
+instead of a local build:
+
+    docker build -t tinygo-dev \
+        --build-arg LLVM_IMAGE=ghcr.io/tinygo-org/llvm-22:$(sh tools/llvm-image-tag.sh) .
+
+`tools/llvm-image-tag.sh` prints the tag of the LLVM image for the current
+source tree, and `make llvm-image-tag` does the same. CI uses that script, so
+the tag agrees. If the registry does not hold that tag, or you are not able to
+pull it, build the LLVM image with `make docker-llvm`.
+
 ## Make a release tarball
 
 Now that we have a working static build, it's time to make a release tarball:

BUILDING.md

@@ -1,29 +1,12 @@
-# tinygo-llvm stage obtains the llvm source for TinyGo
-FROM golang:1.27 AS tinygo-llvm
-
-RUN apt-get update && \
-    apt-get install -y apt-utils make cmake clang-17 ninja-build && \
-    rm -rf \
-        /var/lib/apt/lists/* \
-        /var/log/* \
-        /var/tmp/* \
-        /tmp/*
-
-COPY ./GNUmakefile /tinygo/GNUmakefile
-COPY ./make /tinygo/make
-COPY ./llvm-version.txt /tinygo/llvm-version.txt
-
-RUN cd /tinygo/ && \
-    make llvm-source
-
-# tinygo-llvm-build stage build the custom llvm with xtensa support
-FROM tinygo-llvm AS tinygo-llvm-build
-
-RUN cd /tinygo/ && \
-    make llvm-build
+# Build the TinyGo compiler on top of a prebuilt LLVM image.
+# Build the base image first with:
+#   docker build -t tinygo-llvm-build -f Dockerfile.llvm .
+# Or use the image that CI published:
+#   docker build --build-arg LLVM_IMAGE=ghcr.io/tinygo-org/llvm-22:<tag> .
+ARG LLVM_IMAGE=tinygo-llvm-build
 
 # tinygo-compiler-build stage builds the compiler itself
-FROM tinygo-llvm-build AS tinygo-compiler-build
+FROM ${LLVM_IMAGE} AS tinygo-compiler-build
 
 COPY . /tinygo
 

Dockerfile

@@ -0,0 +1,39 @@
+# Build the LLVM base image for the TinyGo compiler image.
+# It is built again only when llvm-version.txt or the build recipe changes.
+# tools/llvm-image-tag.sh prints the tag and lists the files that set it.
+
+# tinygo-llvm stage obtains the llvm source for TinyGo
+FROM golang:1.27 AS tinygo-llvm
+
+RUN apt-get update && \
+    apt-get install -y apt-utils make cmake clang-17 ninja-build && \
+    rm -rf \
+        /var/lib/apt/lists/* \
+        /var/log/* \
+        /var/tmp/* \
+        /tmp/*
+
+COPY ./GNUmakefile /tinygo/GNUmakefile
+COPY ./make /tinygo/make
+COPY ./llvm-version.txt /tinygo/llvm-version.txt
+
+# Remove the git history in the layer that makes it. A later layer only hides
+# files from a parent layer.
+RUN cd /tinygo/ && \
+    make llvm-source && \
+    rm -rf llvm-project/.git
+
+# tinygo-llvm-build stage build the custom llvm with xtensa support.
+# The object files are removed in the same layer as the build to keep the image
+# small. The static libraries and the headers stay, because TinyGo links to
+# them. The CI caches keep the same subset, see
+# .github/actions/setup-llvm/action.yml.
+FROM tinygo-llvm AS tinygo-llvm-build
+
+# Link the package to the repository. See
+# https://docs.github.com/en/packages/learn-github-packages/connecting-a-repository-to-a-package
+LABEL org.opencontainers.image.source=https://github.com/tinygo-org/tinygo
+
+RUN cd /tinygo/ && \
+    make llvm-build && \
+    find llvm-build -name CMakeFiles -prune -exec rm -r '{}' \;

Dockerfile.llvm

@@ -26,6 +26,18 @@ spell: tools ## Spellcheck source tree
 spellfix: tools ## Same as spell, but fixes what it finds
 	misspell -w --dict misspell.csv -i 'ackward,devided,extint,rela' $$( $(SPELLDIRSCMD) ) *.go *.md
 
+.PHONY: llvm-image-tag
+llvm-image-tag: ## Print the tag of the prebuilt LLVM Docker image
+	@sh tools/llvm-image-tag.sh
+
+.PHONY: docker-llvm
+docker-llvm: ## Build the LLVM base image (slow)
+	docker build -t tinygo-llvm-build -f Dockerfile.llvm .
+
+.PHONY: docker-tinygo
+docker-tinygo: ## Build the TinyGo compiler image
+	docker build -t tinygo-dev .
+
 # https://www.client9.com/self-documenting-makefiles/
 .PHONY: help
 help:

make/tools.mk

@@ -0,0 +1,10 @@
+#!/bin/sh
+# Print the tag of the prebuilt LLVM Docker image for this source tree.
+# The tag changes when the LLVM revision or the build recipe changes.
+# Keep the file list in agreement with the COPY lines in Dockerfile.llvm.
+set -e
+cd "$(dirname "$0")/.."
+rev=$(cut -c1-12 llvm-version.txt)
+hash=$(cat llvm-version.txt Dockerfile.llvm GNUmakefile make/*.mk \
+        | git hash-object --stdin | cut -c1-12)
+echo "$rev-$hash"