ci: shard the C-suite legs, prove cross-shard coverage, warm compiler caches

PR iteration was bounded by single-job suite runs (~40-50 min critical
path) and by every first-of-ref build starting cold. Same suites, same
sanitizers, same gates — the schedule, the cache reuse, and a new
runtime coverage proof change.

- run-tests-parallel.sh accepts CBM_TEST_SHARD="i/N" and runs a
  deterministic slice of --list-suites; unset selects everything, so the
  path is inert outside CI. Known-heavy suites are dealt to shards in
  weight order first (naive modulo stacked the two slowest, store_arch
  and daemon_runtime, onto one job), the parallel wave and the serial
  tail are sliced separately so every shard keeps its own quiet tail,
  and the per-shard union guard proves each job ran exactly its slice.
- shard-completeness job: per-shard guards cannot see a mis-plumbed
  CBM_TEST_SHARD (two jobs running the same slice passes every local
  check while a slice runs nowhere). Every leg now uploads a manifest
  (leg, i/N, sha of the full suite list, its slice) and one aggregation
  job re-proves per leg that the shards agreed on the list, indices are
  exactly 1..N, and the union of slices IS the list. Runs on unsharded
  topologies too, where each leg's single manifest must cover the list.
- workflow topology (shard_suites input, off by default = byte-identical
  to the pre-shard matrix): ubuntu legs run 3 shards, Windows 2 (every
  extra Windows shard re-pays ~5 min of MSYS2 setup), macOS stays at 1
  (not the critical path; mac runner concurrency ceilings are the
  tightest). The Windows test job also pre-creates build\c with a
  protected DACL so build products inherit it — workspace drive roots
  grant Authenticated Users Modify by inheritance, which the activation
  transaction's source-directory policy correctly refuses.
- ccache: a final ref-less restore key lets a fresh PR start from the
  newest cache GitHub's scoping permits, and a nightly build-only
  cache-warm workflow keeps main-scoped caches at most a day stale —
  without it the fallback had no warm source, since nothing built in
  main's scope. The strictly-per-ref policy this replaces cost ~8-12
  minutes on every first-of-ref build for no safety gain:
  CCACHE_COMPILERCHECK=content makes a stale or foreign cache able to
  miss but never to return wrong output. Sharded legs restore-all/
  save-one (every shard builds the identical objects, so shard 1's
  cache carries the full set) — save volume stays flat and warm caches
  stop being evicted by per-shard duplicates.

- suite timeouts: daemon_runtime joins the slow tier. It measures ~610s
  solo on arm64 under ASan (10-round loop, no hang), so the 900s default
  under a loaded 4-job CI runner was a slowness kill masquerading as a
  hang detector.

Validated locally: 3-shard union over the real suite list is complete
with no duplicates, one top-heavy suite per shard, and the summed
3-shard totals reproduce the unsharded run exactly (6776 passed,
0 failed, 4 skipped).

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>

Author: DeusData

.github/workflows/_test.yml

@@ -12,6 +12,10 @@ on:
         description: 'Test the broad platform matrix (older glibc + extra OS versions) instead of the core set'
         type: boolean
         default: false
+      shard_suites:
+        description: 'Split each C-suite leg across parallel shard jobs (ubuntu x3, windows x2); off = one job per leg, byte-identical to the pre-shard topology'
+        type: boolean
+        default: false
 
 permissions:
   contents: read
@@ -31,6 +35,7 @@ jobs:
         id: set
         env:
           BROAD: ${{ inputs.broad_platforms }}
+          SHARDS: ${{ inputs.shard_suites }}
         run: |
           CORE_UNIX='[
             {"os":"ubuntu-latest","cc":"gcc","cxx":"g++"},
@@ -63,6 +68,25 @@ jobs:
             UNIX=$(jq -cn --argjson a "$CORE_UNIX" '$a')
             WIN=$(jq -cn --argjson a "$CORE_WIN" '$a')
           fi
+          # Suite sharding (opt-in via the shard_suites input): each platform's
+          # C-suite run is split across N runner jobs. CBM_TEST_SHARD="i/N"
+          # slices deterministically inside run-tests-parallel.sh; each
+          # shard's union guard proves it ran exactly its slice; shards are
+          # minted HERE from one expansion so a missing shard cannot be
+          # configured silently; and the shard-completeness job re-proves the
+          # cross-shard union at runtime. Ubuntu runners are plentiful (3
+          # shards); Windows gets 2 (every extra shard re-pays ~5 min of
+          # MSYS2 setup); macOS stays at 1 — mac legs are not the critical
+          # path and mac runner concurrency ceilings are the tightest.
+          if [ "$SHARDS" = "true" ]; then
+            UNIX=$(jq -c '[.[] | . as $e
+              | (if ($e.os | startswith("ubuntu")) then 3 else 1 end) as $n
+              | range(1; $n + 1) as $i
+              | $e + {shard: "\($i)/\($n)"}]' <<< "$UNIX")
+            WIN=$(jq -c '[.[] | . as $e
+              | range(1; 3) as $i
+              | $e + {shard: "\($i)/2"}]' <<< "$WIN")
+          fi
           echo "unix={\"include\":$UNIX}" >> "$GITHUB_OUTPUT"
           echo "windows={\"include\":$WIN}" >> "$GITHUB_OUTPUT"
 
@@ -91,24 +115,54 @@ jobs:
       # on the compiler-binary CONTENT plus the fully preprocessed input, so a
       # hit is provably the identical compilation — a stale or foreign cache
       # can only miss, never return wrong output (see scripts/env.sh).
-      # Keys embed github.ref on top of GitHub's own branch scoping: caches are
-      # STRICTLY per-ref by policy — no base-branch fallback, a new PR builds
-      # cold once and only its own pushes warm it.
-      - name: Compiler cache (content-verified)
-        uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+      # Keys embed github.ref; the final ref-less restore key falls back to
+      # the newest cache GitHub's scoping permits (a PR can only ever see its
+      # own ref and the base branch, and base-branch caches are written only
+      # by base-branch runs), so a fresh PR starts from main's warm cache
+      # instead of building cold. Content verification is what makes that
+      # safe; the fallback buys ~8-12 minutes on every first-of-ref build.
+      - name: Compiler cache (content-verified, restore)
+        uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: ${{ github.workspace }}/.ccache
           key: ccache-test-${{ matrix.os }}-${{ matrix.cc }}-${{ github.ref }}-${{ github.sha }}
+          # The cross-ref fallback is what keeps fresh PRs warm: without it a
+          # new PR always started from a cold cache (observed 0% hit legs).
+          # ccache verifies content hashes itself, so a stale cache can only
+          # miss, never mis-compile.
           restore-keys: |
             ccache-test-${{ matrix.os }}-${{ matrix.cc }}-${{ github.ref }}-
+            ccache-test-${{ matrix.os }}-${{ matrix.cc }}-
 
       - name: Test
         run: scripts/test.sh CC=${{ matrix.cc }} CXX=${{ matrix.cxx }}
         env:
+          CBM_TEST_SHARD: ${{ matrix.shard }}
+          CBM_TEST_LEG: ${{ matrix.os }}-${{ matrix.cc }}
           CBM_SKIP_PERF: ${{ inputs.skip_perf && '1' || '' }}
           CCACHE_DIR: ${{ github.workspace }}/.ccache
           CCACHE_MAXSIZE: 1500M
 
+      # Every shard of a leg builds the identical runner, so one shard's
+      # cache carries the complete object set: shard 1 (or the unsharded
+      # job) saves, the rest only restore — same save volume as before
+      # sharding, no quota thrash, no same-key save races. Saving on failure
+      # too keeps red iterations warm.
+      - name: Shard manifest (completeness proof)
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: shard-manifest-${{ matrix.os }}-${{ matrix.cc }}-${{ strategy.job-index }}
+          path: build/c/test-logs/shard-manifest.txt
+          if-no-files-found: error
+
+      - name: Compiler cache (save)
+        if: always() && (matrix.shard == null || startsWith(matrix.shard, '1/'))
+        uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+        with:
+          path: ${{ github.workspace }}/.ccache
+          key: ccache-test-${{ matrix.os }}-${{ matrix.cc }}-${{ github.ref }}-${{ github.sha }}
+
       - name: ccache stats
         if: always()
         run: ccache -s || true
@@ -152,6 +206,7 @@ jobs:
           key: ccache-tsan-${{ matrix.os }}-${{ github.ref }}-${{ github.sha }}
           restore-keys: |
             ccache-tsan-${{ matrix.os }}-${{ github.ref }}-
+            ccache-tsan-${{ matrix.os }}-
 
       - name: ThreadSanitizer tests
         # Calls make directly (no env.sh), so route the compilers through
@@ -225,14 +280,28 @@ jobs:
           $acl.AddAccessRule($rule) | Out-Null
           Set-Acl -LiteralPath $root -AclObject $acl
           "CBM_CI_TEMP_ROOT=$root" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
+          # Pre-create the build directory with the same protected DACL so
+          # every build product inherits it. The workspace drive root grants
+          # Authenticated Users Modify by inheritance — a shape real user
+          # checkouts (under the profile) do not have — and the activation
+          # transaction's source-directory policy correctly refuses it, which
+          # would fail the install-flow tests on the environment, not the code.
+          $build = Join-Path $env:GITHUB_WORKSPACE "build\c"
+          New-Item -ItemType Directory -Path $build -Force | Out-Null
+          $bacl = [System.Security.AccessControl.DirectorySecurity]::new()
+          $bacl.SetOwner($sid)
+          $bacl.SetAccessRuleProtection($true, $false)
+          $bacl.AddAccessRule($rule) | Out-Null
+          Set-Acl -LiteralPath $build -AclObject $bacl
 
       - name: Compiler cache (content-verified)
-        uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+        uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: ${{ github.workspace }}/.ccache
           key: ccache-test-${{ matrix.os }}-${{ matrix.msystem }}-${{ github.ref }}-${{ github.sha }}
           restore-keys: |
             ccache-test-${{ matrix.os }}-${{ matrix.msystem }}-${{ github.ref }}-
+            ccache-test-${{ matrix.os }}-${{ matrix.msystem }}-
 
       - name: Test
         shell: msys2 {0}
@@ -256,6 +325,8 @@ jobs:
           export TMPDIR="$(cygpath -u "$CBM_CI_TEMP_ROOT")"
           scripts/test.sh CC=clang CXX=clang++ ${{ matrix.os == 'windows-11-arm' && 'SANITIZE=-fsanitize=undefined -fsanitize-trap=undefined -fstack-protector-strong -fno-omit-frame-pointer' || '' }}
         env:
+          CBM_TEST_SHARD: ${{ matrix.shard }}
+          CBM_TEST_LEG: ${{ matrix.os }}-${{ matrix.msystem }}
           CBM_SKIP_PERF: ${{ inputs.skip_perf && '1' || '' }}
           CCACHE_DIR: ${{ github.workspace }}/.ccache
           CCACHE_MAXSIZE: 1500M
@@ -267,6 +338,82 @@ jobs:
         env:
           CCACHE_DIR: ${{ github.workspace }}/.ccache
 
+      - name: Shard manifest (completeness proof)
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: shard-manifest-${{ matrix.os }}-${{ matrix.msystem }}-${{ strategy.job-index }}
+          path: build/c/test-logs/shard-manifest.txt
+          if-no-files-found: error
+
+      # Shard 1 (or the unsharded job) saves the compiler cache; every shard
+      # builds the identical runner, so one cache carries the full object set.
+      - name: Compiler cache (save)
+        if: always() && (matrix.shard == null || startsWith(matrix.shard, '1/'))
+        uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+        with:
+          path: ${{ github.workspace }}/.ccache
+          key: ccache-test-${{ matrix.os }}-${{ matrix.msystem }}-${{ github.ref }}-${{ github.sha }}
+
+  # Cross-shard completeness: re-proves at runtime that each leg's shards
+  # agreed on the full suite list and that the union of their slices IS that
+  # list. This is the guard the per-shard union checks cannot provide: a
+  # mis-plumbed CBM_TEST_SHARD (two jobs running the same slice) passes every
+  # per-shard guard while a slice runs nowhere — only a cross-shard view
+  # catches it. Runs and gates on unsharded topologies too (every leg then
+  # contributes one 1/1 manifest whose slice must equal the full list).
+  shard-completeness:
+    needs: [test-unix, test-windows]
+    if: always()
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+        with:
+          pattern: shard-manifest-*
+          path: manifests
+
+      - name: Verify cross-shard union per leg
+        run: |
+          set -eu
+          files=$(find manifests -name shard-manifest.txt | sort)
+          if [ -z "$files" ]; then
+            echo "FAIL: no shard manifests were uploaded" >&2
+            exit 1
+          fi
+          rc=0
+          for leg in $(grep -h '^leg=' $files | sort -u | sed 's/^leg=//'); do
+            leg_files=$(grep -l "^leg=$leg\$" $files)
+            n=$(grep -h '^shard=' $leg_files | sed 's|.*/||' | sort -u)
+            if [ "$(printf '%s\n' "$n" | wc -l)" -ne 1 ]; then
+              echo "FAIL: $leg shards disagree on shard count: $n" >&2
+              rc=1
+              continue
+            fi
+            indices=$(grep -h '^shard=' $leg_files | sed 's/^shard=//;s|/.*||' | sort -n)
+            if [ "$indices" != "$(seq 1 "$n")" ]; then
+              echo "FAIL: $leg shard indices [$indices] != 1..$n" >&2
+              rc=1
+              continue
+            fi
+            list_sha=$(grep -h '^list_sha256=' $leg_files | sed 's/^list_sha256=//' | sort -u)
+            if [ "$(printf '%s\n' "$list_sha" | wc -l)" -ne 1 ]; then
+              echo "FAIL: $leg shards saw different suite lists" >&2
+              rc=1
+              continue
+            fi
+            union_sha=$(for f in $leg_files; do
+              sed -n '/^--- slice ---$/,$p' "$f" | tail -n +2
+            done | sort | sha256sum | awk '{print $1}')
+            if [ "$union_sha" != "$list_sha" ]; then
+              echo "FAIL: $leg union of shard slices != full suite list (GATE-QUALITY LOSS)" >&2
+              rc=1
+              continue
+            fi
+            echo "OK: $leg — $n shard(s), union of slices == full suite list"
+          done
+          exit "$rc"
+
   # Windows product-surface regression guards. Distinct from test-windows above
   # (the sanitizer C suite): these drive a real product binary + embedded HTTP UI
   # over stdio / CLI / HTTP and fail if a Windows bug already fixed on main comes
@@ -307,6 +454,7 @@ jobs:
           key: ccache-guards-${{ github.ref }}-${{ github.sha }}
           restore-keys: |
             ccache-guards-${{ github.ref }}-
+            ccache-guards-
 
       - name: Build product binary with embedded UI
         shell: msys2 {0}

.github/workflows/cache-warm.yml

@@ -0,0 +1,123 @@
+# Nightly compiler-cache warming for main's scope. GitHub cache isolation
+# lets a PR restore only caches written by its own ref or the base branch —
+# and nothing used to write test caches in main's scope (the test workflow
+# runs on pull_request and workflow_dispatch only), so the cross-ref
+# fallback restore keys in _test.yml had no warm source and every fresh PR
+# built cold. This build-only job (no tests) keeps main-scoped caches at
+# most a day stale; CCACHE_COMPILERCHECK=content turns staleness into
+# partial misses, never wrong output. The build commands and ccache
+# environment mirror scripts/test.sh's build phase exactly — a warm entry
+# only hits when flags and compiler content match, so fidelity here is what
+# makes the cache useful.
+name: Cache warm
+
+on:
+  schedule:
+    - cron: '17 2 * * *'
+  workflow_dispatch: {}
+
+permissions:
+  contents: read
+
+jobs:
+  warm-unix:
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - { os: ubuntu-latest, cc: gcc, cxx: g++ }
+          - { os: ubuntu-24.04-arm, cc: gcc, cxx: g++ }
+          - { os: macos-14, cc: cc, cxx: c++ }
+          - { os: macos-15-intel, cc: cc, cxx: c++ }
+    runs-on: ${{ matrix.os }}
+    timeout-minutes: 60
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+
+      - name: Install deps (Ubuntu)
+        if: startsWith(matrix.os, 'ubuntu')
+        run: sudo apt-get update && sudo apt-get install -y zlib1g-dev ccache
+
+      - name: Install ccache (macOS)
+        if: startsWith(matrix.os, 'macos')
+        run: command -v ccache >/dev/null 2>&1 || brew install ccache
+
+      - name: Compiler cache (restore)
+        uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+        with:
+          path: ${{ github.workspace }}/.ccache
+          key: ccache-test-${{ matrix.os }}-${{ matrix.cc }}-${{ github.ref }}-${{ github.sha }}
+          restore-keys: |
+            ccache-test-${{ matrix.os }}-${{ matrix.cc }}-${{ github.ref }}-
+            ccache-test-${{ matrix.os }}-${{ matrix.cc }}-
+
+      - name: Build (test runner + product, no tests)
+        run: |
+          source scripts/env.sh
+          make -j"$(getconf _NPROCESSORS_ONLN)" -f Makefile.cbm \
+            build/c/test-runner cbm CC=${{ matrix.cc }} CXX=${{ matrix.cxx }}
+        env:
+          CCACHE_DIR: ${{ github.workspace }}/.ccache
+          CCACHE_MAXSIZE: 1500M
+
+      - name: ccache stats
+        run: ccache -sv || ccache -s
+        env:
+          CCACHE_DIR: ${{ github.workspace }}/.ccache
+
+      - name: Compiler cache (save)
+        if: always()
+        uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+        with:
+          path: ${{ github.workspace }}/.ccache
+          key: ccache-test-${{ matrix.os }}-${{ matrix.cc }}-${{ github.ref }}-${{ github.sha }}
+
+  warm-windows:
+    runs-on: windows-latest
+    timeout-minutes: 90
+    defaults:
+      run:
+        shell: msys2 {0}
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+
+      - uses: msys2/setup-msys2@66cd2cce69caa17b53920067426061ca1de3a884 # v2
+        with:
+          msystem: CLANG64
+          path-type: inherit
+          install: >-
+            mingw-w64-clang-x86_64-clang
+            mingw-w64-clang-x86_64-compiler-rt
+            mingw-w64-clang-x86_64-zlib
+            mingw-w64-clang-x86_64-ccache
+            make
+            git
+
+      - name: Compiler cache (restore)
+        uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+        with:
+          path: ${{ github.workspace }}/.ccache
+          key: ccache-test-windows-latest-CLANG64-${{ github.ref }}-${{ github.sha }}
+          restore-keys: |
+            ccache-test-windows-latest-CLANG64-${{ github.ref }}-
+            ccache-test-windows-latest-CLANG64-
+
+      - name: Build (test runner + product, no tests)
+        run: |
+          source scripts/env.sh
+          make -j"$(nproc)" -f Makefile.cbm build/c/test-runner cbm CC=clang CXX=clang++
+        env:
+          CCACHE_DIR: ${{ github.workspace }}/.ccache
+          CCACHE_MAXSIZE: 1500M
+
+      - name: ccache stats
+        run: ccache -sv || ccache -s
+        env:
+          CCACHE_DIR: ${{ github.workspace }}/.ccache
+
+      - name: Compiler cache (save)
+        if: always()
+        uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+        with:
+          path: ${{ github.workspace }}/.ccache
+          key: ccache-test-windows-latest-CLANG64-${{ github.ref }}-${{ github.sha }}