ci: self-hosted burst runners for the kvm label

The VM end-to-end test cannot run on GitHub's hosted runners - no nested
virtualisation, not enough disk - so it has been dispatch-only with nowhere
to dispatch to. This adds the runner side, modelled on the ADO burst agents:
spawn per job, one job per runner, nothing running when the queue is empty.

- Dockerfile.runner / runner-entrypoint.sh: an ephemeral runner that takes a
  single job, deregisters and exits. It drives the host's libvirt over the
  mounted socket rather than running qemu itself.
- autoscaler.py: stdlib-only loop that counts queued jobs asking for the
  label, tops up runner containers to a ceiling, and serves gha_ci_* metrics
  on :9826 for Prometheus.
- deploy-unraid.sh: builds both images and starts the autoscaler, refusing to
  run without a credential and saying how to create one.

Security, because this repo is public: vm-test.yml stays workflow_dispatch
only, so a fork PR can never reach the runner, and nothing else uses the kvm
label. Runners are ephemeral, and the credential is bind-mounted read-only and
read at point of use, so it is not in the image, the environment, or
`docker inspect`.

Also makes the VM script work from inside a container: OVMF paths are
overridable and the emulator is now asked of libvirt itself, since firmware
and qemu live on the host rather than on the machine running the script.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: louij2

.github/workflows/vm-test.yml

@@ -35,12 +35,19 @@ jobs:
       - uses: actions/checkout@v4
 
       - name: Run the end-to-end VM test
-        run: sudo tools/vm-test/run-vm-test.sh ${{ inputs.build && format('--build {0}', inputs.build) || '' }}
+        # VM_TEST_WORKDIR is set by the runner: it must be a path that exists
+        # identically on the libvirt host, since libvirt resolves the disk
+        # images there. It must not be RAM-backed - see the script's guard.
+        run: |
+          sudo --preserve-env=VM_TEST_WORKDIR,OVMF_CODE,OVMF_VARS \
+            tools/vm-test/run-vm-test.sh \
+            --workdir "${VM_TEST_WORKDIR:-/var/tmp/steamos-vm-test}" \
+            ${{ inputs.build && format('--build {0}', inputs.build) || '' }}
 
       - name: Publish the guest console log
         if: always()
         uses: actions/upload-artifact@v4
         with:
           name: vm-console-log
-          path: /var/tmp/steamos-vm-test/console.log
+          path: ${{ env.VM_TEST_WORKDIR || '/var/tmp/steamos-vm-test' }}/console.log
           if-no-files-found: warn

tools/ci-runner/Dockerfile.autoscaler

@@ -0,0 +1,13 @@
+# Watches the repo's job queue and spawns ephemeral runner containers.
+#
+# Needs the host's docker socket, because it starts sibling containers rather
+# than running jobs itself. It never runs job code.
+FROM python:3.12-alpine
+
+RUN apk add --no-cache docker-cli
+
+COPY autoscaler.py /usr/local/bin/autoscaler.py
+RUN chmod +x /usr/local/bin/autoscaler.py
+
+EXPOSE 9826
+ENTRYPOINT ["python3", "-u", "/usr/local/bin/autoscaler.py"]

tools/ci-runner/Dockerfile.runner

@@ -0,0 +1,32 @@
+# Ephemeral GitHub Actions runner able to drive a libvirt host.
+#
+# It does NOT run qemu itself: it talks to the host's libvirt over the mounted
+# socket, so the VM runs on the host while this container only orchestrates.
+# What it does need locally is the disk plumbing - losetup, mount, btrfs - which
+# is why it runs privileged with the host's /dev.
+FROM ubuntu:24.04
+
+ARG RUNNER_VERSION=2.337.0
+ARG RUNNER_ARCH=x64
+
+ENV DEBIAN_FRONTEND=noninteractive \
+    RUNNER_ALLOW_RUNASROOT=1 \
+    RUNNER_MANUALLY_TRAP_SIG=1
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+      ca-certificates curl jq git sudo unzip tar gzip bzip2 xz-utils \
+      libvirt-clients qemu-utils \
+      util-linux mount udev btrfs-progs e2fsprogs dosfstools \
+      coreutils gawk sed grep procps iproute2 \
+      libicu74 \
+    && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /actions-runner
+RUN curl -fsSL -o runner.tar.gz \
+      "https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-${RUNNER_ARCH}-${RUNNER_VERSION}.tar.gz" \
+    && tar xzf runner.tar.gz && rm runner.tar.gz
+
+COPY runner-entrypoint.sh /usr/local/bin/runner-entrypoint.sh
+RUN chmod +x /usr/local/bin/runner-entrypoint.sh
+
+ENTRYPOINT ["/usr/local/bin/runner-entrypoint.sh"]

tools/ci-runner/README.md

@@ -0,0 +1,129 @@
+# `tools/ci-runner/` — self-hosted burst runners
+
+Spawn-per-job GitHub Actions runners for the `kvm` label, so the
+[VM end-to-end test](../vm-test/README.md) can run in CI. GitHub's hosted
+runners cannot: they have no nested virtualisation and not enough disk.
+
+Nothing runs while the queue is empty. One small autoscaler container watches
+for queued jobs and starts an **ephemeral** runner per job; each runner takes
+exactly one job, deregisters itself and exits.
+
+```
+autoscaler (always on, ~20 MB)
+    │  polls the repo for queued jobs asking for 'kvm'
+    ▼
+gha-burst-<ts>-<n>   ephemeral runner container, one job then gone
+    │  drives libvirt over the host socket
+    ▼
+steamos-install-test  VM on the host, torn down by the test itself
+```
+
+## Security: read this before pointing it at a public repo
+
+A self-hosted runner executes whatever the workflow says. On a **public**
+repository that is a real risk, because anyone can open a pull request.
+
+The protections here:
+
+- **`vm-test.yml` is `workflow_dispatch` only.** It never triggers on
+  `pull_request`, so a fork cannot cause it to run. Nothing else uses the
+  `kvm` label — every other job stays on GitHub's hosted runners.
+- **Ephemeral runners.** One job per container, then it is discarded. Nothing
+  persists between jobs.
+- **The credential is never in the image or the environment.** It is
+  bind-mounted read-only as a file and read at the moment it is used, so
+  `docker inspect` does not reveal it.
+
+Worth also setting, once:
+
+```bash
+gh api -X PUT repos/OWNER/REPO/actions/permissions/workflow -f default_workflow_permissions=read
+```
+
+and in **Settings → Actions → General → Fork pull request workflows**, choose
+*Require approval for all outside collaborators*.
+
+Do not add the `kvm` label to a workflow that runs on `pull_request` unless you
+have thought hard about it. The runner is privileged and sits on your LAN.
+
+## Setting it up
+
+On the libvirt host (not your laptop):
+
+```bash
+git clone https://github.com/louij2/steamos_custom_install.git
+cd steamos_custom_install
+sudo ./tools/ci-runner/deploy-unraid.sh
+```
+
+It refuses to start without a credential and tells you how to create one. The
+token needs **`Administration: read and write`** on the one repository — that
+is what mints runner registration tokens. Create a fine-grained PAT scoped to
+that single repo at <https://github.com/settings/personal-access-tokens/new>,
+then put it in place yourself:
+
+```bash
+install -d -m 700 /mnt/user/appdata/gh-autoscaler
+printf '%s' 'YOUR_TOKEN' > /mnt/user/appdata/gh-autoscaler/gh_pat
+chmod 600 /mnt/user/appdata/gh-autoscaler/gh_pat
+```
+
+## Configuration
+
+Every knob is an environment variable on `deploy-unraid.sh`:
+
+| Variable | Default | Meaning |
+|---|---|---|
+| `GITHUB_REPO` | `louij2/steamos_custom_install` | Repository to serve |
+| `RUNNER_LABEL` | `kvm` | Label a job must request |
+| `MAX_RUNNERS` | `3` | Ceiling on concurrent burst runners |
+| `POLL_SECONDS` | `30` | Seconds between queue checks |
+| `METRICS_PORT` | `9826` | Prometheus endpoint |
+| `VM_WORKDIR` | `/mnt/user/isos/steamos-vm-test` | Image cache, must be real storage |
+| `MAX_RUNNERS=1` | | Effectively serialises, if the host is busy |
+
+Raise `MAX_RUNNERS` only as far as the host can take: each concurrent VM test
+wants ~8 GB RAM, 4 vCPU and ~21 GB of disk.
+
+Unlike the Azure DevOps equivalent there is **no billing ceiling** — self-hosted
+runners on a public repository have unlimited concurrency, so the only limit is
+the hardware.
+
+## Metrics
+
+Prometheus endpoint on `:9826/metrics`:
+
+| Metric | Meaning |
+|---|---|
+| `gha_ci_queued_jobs` | Queued jobs asking for the label |
+| `gha_ci_burst_running` | Burst containers alive now |
+| `gha_ci_max_runners` | Configured ceiling |
+| `gha_ci_spawned_total` | Runners started since the autoscaler booted |
+| `gha_ci_errors_total` | Poll or spawn failures |
+| `gha_ci_last_tick_timestamp_seconds` | Staleness check for the loop |
+
+Alert on `time() - gha_ci_last_tick_timestamp_seconds > 300` (the loop is dead)
+and on `gha_ci_queued_jobs > 0 and gha_ci_burst_running == 0` sustained (jobs
+are queuing but nothing is starting — usually an expired credential).
+
+## Why the runner container is privileged
+
+The VM test needs `losetup`, `mount` and `btrfs property` against real devices
+to inject the harness into the recovery image. It does **not** run qemu: it
+talks to the host's libvirt over the mounted socket, so the VM belongs to the
+host. That is also why the work directory is bind-mounted at the *same path*
+inside and out — libvirt resolves the disk image paths on the host, not in the
+container.
+
+## Operating it
+
+```bash
+docker logs -f gh-autoscaler          # what it is deciding
+docker ps --filter label=gha-burst=1  # runners alive right now
+curl -s localhost:9826/metrics        # current state
+docker rm -f gh-autoscaler            # stop scaling; running jobs finish
+```
+
+Runners that vanish mid-job leave a registration behind; they are ephemeral so
+GitHub clears them, but `gh api repos/OWNER/REPO/actions/runners` shows what is
+registered if you suspect otherwise.