Skip to content
CI

v0.4 Trust Closure: execution-correctness and contract-closure batch #201

Sign in to view logs

v0.4 Trust Closure: execution-correctness and contract-closure batch

v0.4 Trust Closure: execution-correctness and contract-closure batch #201

Workflow file for this run

.github/workflows/ci.yml at fd7abcb
name: CI
on:
push:
branches: [main]
pull_request:
permissions:
contents: read
pull-requests: write
issues: write
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
# Fastest feedback of all: static analysis needs no build. SwiftLint and
# SwiftFormat run against the whole tree using the repo's own configs
# (.swiftlint.yml / .swiftformat), which were tuned against this
# codebase's actual style rather than stock defaults.
#
# `swiftlint lint` runs `--strict --baseline .swiftlint-baseline.json`:
# 145 pre-existing violations by identity (rule + file + line), dominated
# by `line_length` (82, mostly long test lines) with the complexity/length
# remainder (function_body_length, type_body_length, cyclomatic_complexity,
# file_length, large_tuple) concentrated in, but not limited to, a few
# deliberately large orchestration files (see .swiftlint.yml's comments
# for the full breakdown, including which CLI command handlers are
# baselined too) — that aren't worth a risky structural refactor just to
# silence a linter. The baseline freezes exactly those pre-existing
# violations by identity (rule + file + line), so `--strict` still fails
# the build on any *new* violation anywhere, including a new one in an
# already-baselined file. Shrink the baseline over time by fixing an
# entry and regenerating with
# `swiftlint lint --write-baseline .swiftlint-baseline.json`; never grow
# it to paper over new debt.
#
# SwiftLint is pinned to an exact version (matching the version the
# baseline was generated with) rather than `brew install`'s floating
# latest: baseline identity and violation detection are both
# SwiftLint-version-sensitive, so an unpinned upgrade could silently
# change what the baseline recognizes, in either direction. Bump
# SWIFTLINT_VERSION and regenerate .swiftlint-baseline.json together, as
# one deliberate change, not independently.
lint:
name: Lint & format check
runs-on: macos-15
# Generous relative to observed reality (~20-25s on this exact job) —
# bounds a hang (a stuck `brew install`/download, say) rather than
# tuning for the common case. See the `unit` job's own comment for
# why every job here now has an explicit ceiling.
timeout-minutes: 10
env:
SWIFTLINT_VERSION: "0.63.2"
steps:
- uses: actions/checkout@v4
# Enforces the documented toolchain floor (README "Install",
# docs/apple-support-matrix.md "Toolchain floor"): Xcode 16+ and
# Swift 6.0+. The macos-15 runner image drifts over time, so printing
# the versions (as every job's `Toolchain` step does) is not enough —
# this fails the build if the floor regresses. Placed first so a
# drift fails fast, before the installs below spend any time.
- name: Toolchain floor (Xcode 16+, Swift 6+)
shell: bash
run: |
set -euo pipefail
# `xcodebuild -version` piped directly into `head -n 1` crashes:
# `head` closes the read end after its first line, xcodebuild's
# own Foundation-based stdout write then hits a broken pipe,
# which xcodebuild turns into an uncaught NSFileHandleOperationException
# (SIGABRT, exit 134) instead of the plain SIGPIPE a non-Foundation
# tool would just die from quietly — reproduced for real on this
# exact runner image. Capturing full output to a variable first
# (command substitution reads to EOF, no early pipe closure) sidesteps it.
xcode_version_output="$(xcodebuild -version)"
xcode_major="$(head -n 1 <<<"$xcode_version_output" | awk '{print $2}' | cut -d. -f1)"
swift_major="$(swift --version 2>&1 | grep -oE 'Apple Swift version [0-9]+' | grep -oE '[0-9]+')"
echo "Xcode major: $xcode_major, Swift major: $swift_major"
if [ -z "$xcode_major" ] || [ "$xcode_major" -lt 16 ]; then
echo "::error::Xcode 16+ required, found '${xcode_major:-unknown}'. See docs/apple-support-matrix.md."
exit 1
fi
if [ -z "$swift_major" ] || [ "$swift_major" -lt 6 ]; then
echo "::error::Swift 6+ required, found '${swift_major:-unknown}'. See docs/apple-support-matrix.md."
exit 1
fi
- name: Install SwiftFormat
run: brew install swiftformat
- name: Install pinned SwiftLint
run: |
curl -fsSL -o portable_swiftlint.zip \
"https://github.com/realm/SwiftLint/releases/download/${SWIFTLINT_VERSION}/portable_swiftlint.zip"
unzip -o portable_swiftlint.zip swiftlint
chmod +x swiftlint
sudo mv swiftlint /usr/local/bin/swiftlint
rm portable_swiftlint.zip
installed_version="$(swiftlint version)"
if [ "$installed_version" != "$SWIFTLINT_VERSION" ]; then
echo "::error::Installed SwiftLint $installed_version does not match pinned SWIFTLINT_VERSION $SWIFTLINT_VERSION"
exit 1
fi
- name: SwiftLint
run: swiftlint lint --strict --config .swiftlint.yml --baseline .swiftlint-baseline.json Sources Tests
- name: SwiftFormat (check only)
run: swiftformat --lint --config .swiftformat .
# Cognitive-complexity gate. SwiftLint caps `cyclomatic_complexity`
# (error: 25, see .swiftlint.yml) but ships no cognitive-complexity rule
# at all, so this job is the only thing watching the metric that tracks
# how hard a function is to hold in your head — nesting depth and
# interleaved control flow, which cyclomatic counting is blind to (a flat
# 20-case `switch` and a five-deep nest of loops and `if let`s score the
# same cyclomatic and wildly different cognitive).
#
# `--threshold 25` gates on `max(cyclomatic, cognitive)` — verified
# against this exact pinned version, where a cyclomatic-16/cognitive-15
# function reports at `--threshold 16` and not at 17 — so the cyclomatic
# half deliberately lands on SwiftLint's own error ceiling and the
# cognitive half is the coverage that is genuinely new.
#
# Scans Tests/ as well as Sources/, the same scope the `lint` job above
# uses. Tests/ passes at this threshold today with zero violations, so
# including it costs nothing now and keeps test-side complexity from
# growing unwatched later.
#
# `--report-suppressions` prints every `// swift-complexity:disable`
# comment together with the suppressed function's current metrics, to
# stderr, so a suppression stays visible in the job log instead of
# silently outliving the reason it was added.
#
# Syntax-only — no build, no toolchain beyond the analyzer itself — so
# this belongs in the same fast, always-required tier as `lint`.
complexity:
name: Complexity gate (swift-complexity)
runs-on: macos-15
# Generous relative to observed reality (a few seconds of analysis over
# the whole tree) — bounds a stuck `brew install`, not the analysis.
timeout-minutes: 10
env:
# Pinned for the same reason this workflow pins SWIFTLINT_VERSION
# above: this tool decides what the gate *means*, and an unannounced
# upstream change to how cognitive complexity is counted would move
# the bar in either direction without a single line of this repo
# changing. `brew install` always fetches the tap's latest, so the
# version is asserted after installing rather than pinned at install
# time — an upstream release then surfaces as an explicit, reviewable
# CI failure instead of a quietly different verdict.
SWIFT_COMPLEXITY_VERSION: "1.4.0"
steps:
- uses: actions/checkout@v4
- name: Install pinned swift-complexity
shell: bash
run: |
set -euo pipefail
brew install fummicc1/tap/swift-complexity
installed_version="$(swift-complexity --version)"
if [ "$installed_version" != "$SWIFT_COMPLEXITY_VERSION" ]; then
echo "::error::Installed swift-complexity $installed_version does not match pinned SWIFT_COMPLEXITY_VERSION $SWIFT_COMPLEXITY_VERSION. Re-run the gate locally against the new version, confirm the reported metrics did not move, then bump this pin."
exit 1
fi
- name: Complexity gate
shell: bash
run: swift-complexity Sources Tests --recursive --threshold 25 --report-suppressions
# Compiler-aware lint: `unused_import` only (see .swiftlint.yml's own
# `analyzer_rules` comment). Needs a clean xcodebuild log, so it pays for
# a full `MutantKit-Package` build where `lint` needs none — a separate
# job rather than a step there, so the fast static checks and this gate
# never block each other. The pre-existing set (74 true positives on this
# tree, not the 66 a syntax-only per-file scan over a *different* file set
# would suggest -- private targets excluded from the public snapshot
# change the count) was removed outright before this gate was switched on,
# so any finding here is new debt.
#
# NOT in merge-gate's `needs` (2026-09-05): `swiftlint analyze` segfaults
# deterministically on this runner image's Xcode 16.4 -- reproduced twice
# on the identical commit, same crash both times (SourceKit's
# IDEInspectionSecondPassRequest, type-checking
# Sources/MutationPlanner/SourceFileWalker.swift's `walk()`, exit 139).
# This is a SwiftLint/SourceKit toolchain-compatibility crash, not an
# unused-import finding -- `swiftlint analyze` never reached the point of
# reporting a violation, so there is nothing here for `--strict` to have
# caught correctly or incorrectly. Deliberately not papered over with
# `continue-on-error` (that would make this job always-green and useless
# as a signal) or silently dropped from CI (the unused-import gate itself,
# and the fixes it already found, stay real product value on any
# toolchain where the analyzer doesn't crash -- e.g. this exact command
# runs clean locally on Xcode 26.6). It stays informational -- expect it
# red until the runner image's Xcode moves past 16.4 or upstream fixes the
# crash -- and required-again once one of those happens.
analyze:
name: Unused imports (swiftlint analyze)
runs-on: macos-15
timeout-minutes: 20
env:
SWIFTLINT_VERSION: "0.63.2"
steps:
- uses: actions/checkout@v4
- name: Install pinned SwiftLint
run: |
curl -fsSL -o portable_swiftlint.zip \
"https://github.com/realm/SwiftLint/releases/download/${SWIFTLINT_VERSION}/portable_swiftlint.zip"
unzip -o portable_swiftlint.zip swiftlint
chmod +x swiftlint
sudo mv swiftlint /usr/local/bin/swiftlint
rm portable_swiftlint.zip
installed_version="$(swiftlint version)"
if [ "$installed_version" != "$SWIFTLINT_VERSION" ]; then
echo "::error::Installed SwiftLint $installed_version does not match pinned SWIFTLINT_VERSION $SWIFTLINT_VERSION"
exit 1
fi
- name: Clean build for analyzer
shell: bash
run: |
set -o pipefail
xcodebuild -scheme MutantKit-Package -destination 'generic/platform=macOS' clean build 2>&1 | tee xcodebuild.log
- name: swiftlint analyze
shell: bash
run: swiftlint analyze --strict --config .swiftlint.yml --compiler-log-path xcodebuild.log Sources
# Fast feedback: everything that does not build another project.
unit:
name: Unit tests
runs-on: macos-15
# A real, reproducible hang (this job's own `Test` step sitting for
# 68+ minutes with no completion, versus ~15-20s for the identical
# `swift test` locally) burned real Actions minutes on this account
# before three compounding process-handling bugs were found and
# fixed (Sources/BenchmarkRunner/ToolRunner.swift). This ceiling is
# this project's own "nothing runs unbounded" principle applied to
# its own CI: generous relative to known-good local runtime (cold
# build + full unit suite) so it never fires on a real, if slow, run,
# but bounded so a recurrence (this exact class of bug, or a new one)
# fails fast and visibly instead of silently burning an hour+ again.
# Raised from 15 to 25 when the coverage lane was added: a cold build
# with `--enable-code-coverage` plus the lcov export/Sonar/Codecov
# steps measured ~15m35s on this runner (every step individually
# succeeded, but the job as a whole was reported `cancelled` at the
# old 15-minute mark) — still an order of magnitude below the 68+
# minute hang this ceiling exists to catch, not a loosening of it.
timeout-minutes: 25
steps:
# fetch-depth: 0 (a full clone, not this action's normal shallow
# default) so the Sonar/Codecov steps below have the git history they
# need for blame/PR-diff coverage reporting.
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Toolchain
shell: bash
run: |
swift --version && xcodebuild -version
echo "CPU count: $(sysctl -n hw.ncpu)"
- name: Build
shell: bash
run: |
set -o pipefail
swift build --build-tests 2>&1 | tee build.log
# Diagnostic instrumentation for the same real CI hang this job's
# own `timeout-minutes` above exists for. `swift test` runs in the
# background, writing straight to a file (never through a pipe
# `tee`/GitHub's own log-streaming could stall on) — file-based
# logging already ruled out a log-pipe-buffering artifact directly:
# a prior run's own `test.log` genuinely froze (byte count static
# for the full ~12 minutes before timeout), proving a real stall
# inside the test process itself, not a reporting artifact.
#
# Detection: `test.log` not growing for 240s (sixteen 15s checks) is
# treated as a stall — don't wait for this job's own 15-minute
# `timeout-minutes` ceiling, which exists as an outer backstop only.
#
# 240s, not the original 75s: a real CI run on commit 8ed7d31 tripped
# this detector, and both a stack sample and the raw heartbeat log
# proved it was a false positive, not a real hang. The sample showed
# the sole busy thread inside `ProcessSupervisor.wait()`'s own
# ordinary poll loop (`usleep`/`nanosleep`), not stuck in any
# Foundation-internal call; the heartbeat log showed `test.log`
# frozen at a constant line count for exactly the-then-75s threshold
# before the kill fired. Root cause: this same investigation's own
# earlier widenings this session (`promptExitTimeoutSeconds` and
# `eventuallyNoSurvivors`'s timeout, both raised 10-30s -> 60s, plus
# `retryingKnownForkRaceWindow`'s up to 3 fresh attempts on the
# documented fork-race miss in `ProcessSupervisorResidueTests.swift`)
# legitimately raised the worst-case span a single test can run with
# zero `test.log` output well past the old 75s threshold — a single
# attempt alone can legitimately wait up to
# `promptExitTimeoutSeconds` + `eventuallyNoSurvivors`'s ceiling
# (60s + 60s = 120s) with no output at all before it either succeeds
# or retries. 240s covers two such attempts back to back (a real,
# if uncommon, case — not the fully compounded 3-attempt worst case,
# which would need multiple independent rare fork-race misses to
# land back to back and is treated as astronomically unlikely enough
# not to design the stall threshold around) while staying well under
# the 900s `timeout-minutes` outer backstop, so a genuine multi-
# minute hang is still caught and sampled long before that backstop
# would silently discard the evidence.
#
# On stall, this samples the *actual* stuck work, not just the
# parent: `swift-test`'s own process typically shows only "waiting
# on child" once sampled, since the real work — and therefore the
# real stuck stack trace — runs in `swiftpm-testing-helper`. Both
# are sampled, `swiftpm-testing-helper` first/foremost, plus
# `lsof` on the helper to see directly whether it is holding an
# unexpected pipe fd open (confirming or ruling out the
# file-descriptor-inheritance hypothesis directly, rather than
# inferring it).
#
# Then this step kills the stuck process tree itself and exits 124
# (the conventional "timed out" exit code) — deliberately *not*
# left to this job's own `timeout-minutes`, which marks a job
# `cancelled`, not `failed`; `if: failure()` (the existing
# "Upload unit failure logs" step's own condition before this
# comment was written) does not run on a `cancelled` job, so a
# GitHub-initiated timeout would silently discard every sample
# this step just captured. Exiting 124 from inside the step itself
# makes this an ordinary step failure, which does trigger the
# upload.
#
# SWT_EXPERIMENTAL_MAXIMUM_PARALLELIZATION_WIDTH: CI-only (never set
# for local development), added after a real, repeated CI failure
# pattern in ProcessSupervisorResidueTests and similar tests that
# depend on the OS actually finishing real, timing-sensitive work
# (reaping a killed descendant, a stall watchdog's own deadline) in
# a bounded window. This runner has confirmed only 3 vCPUs (see the
# "Toolchain" step above); Swift Testing schedules every test's own
# concurrency onto the cooperative thread pool by default, sized to
# the *full* core count with no user-facing throttle -- `swift
# test`'s own `--num-workers`/`--parallel` flags were tried first and
# confirmed, by direct local experiment, to have zero effect on
# Swift Testing's scheduling (they only govern legacy XCTest-style
# process-level parallelism). This environment variable is Swift
# Testing's own internal mechanism instead -- found via `strings` on
# the Testing framework binary (it is not documented public API, and
# its own name says "EXPERIMENTAL"), then verified directly, locally,
# to actually bound real concurrent test execution before trusting
# it here. Set to 2, not 1: leaves the supervisory/monitoring
# machinery real headroom rather than fully serializing the whole
# suite, which this project's own standing policy is not to do
# blindly for a speed-vs-reliability trade this narrow a problem
# does not need. If a future toolchain silently drops support for
# this variable, the worst case is reverting to today's already-
# documented behavior, not a regression beyond it.
- name: Test
shell: bash
env:
SWT_EXPERIMENTAL_MAXIMUM_PARALLELIZATION_WIDTH: "2"
run: |
set -o pipefail
mkdir -p samples
swift test --enable-code-coverage > test.log 2>&1 &
test_pid=$!
echo "swift test started, pid=$test_pid"
last_lines=0
stall_seconds=0
while kill -0 "$test_pid" 2>/dev/null; do
sleep 15
lines=$(wc -l < test.log | tr -d ' ')
if [ "$lines" -eq "$last_lines" ]; then
stall_seconds=$((stall_seconds + 15))
else
stall_seconds=0
fi
last_lines=$lines
echo "--- heartbeat $(date -u +%H:%M:%S) --- test.log lines: $lines (no growth for ${stall_seconds}s) ---"
# The real work happens in child processes (swiftpm-testing-helper,
# swift-package) the parent `swift` pid's own %cpu never reflects —
# snapshotting every swift-related process, not just $test_pid,
# is what actually shows whether real work is still happening.
ps -eo pid,ppid,%cpu,%mem,etime,comm | grep -iE "swift|xctest" | grep -v grep || true
tail -n 5 test.log
if [ "$stall_seconds" -ge 240 ]; then
echo ">>> STALL DETECTED (test.log has not grown in ${stall_seconds}s) — capturing diagnostics <<<"
ps -ef > samples/ps-tree.txt
parent_pid=$(pgrep -f "swift-test" | head -1)
if [ -n "$parent_pid" ]; then
echo "sampling parent swift-test pid=$parent_pid (for completeness — the real work is in the helper below)"
sample "$parent_pid" 5 -file "samples/sample-parent-${parent_pid}.txt" 2>&1 | tail -n 3 || true
fi
for helper_pid in $(pgrep -f "swiftpm-testing-helper"); do
echo "sampling swiftpm-testing-helper pid=$helper_pid — the actual test-worker process"
sample "$helper_pid" 5 -file "samples/sample-helper-${helper_pid}.txt" 2>&1 | tail -n 3 || true
lsof -p "$helper_pid" > "samples/lsof-helper-${helper_pid}.txt" 2>&1 || true
done
echo ">>> killing the stuck process tree and failing this step (exit 124) so the job registers as an ordinary failure, not a GitHub-initiated cancellation <<<"
kill -9 "$test_pid" 2>/dev/null
pkill -9 -f swiftpm-testing-helper 2>/dev/null
pkill -9 -f "swift-test" 2>/dev/null
cat test.log
exit 124
fi
done
wait "$test_pid"
exit_code=$?
echo "=== swift test exited with code $exit_code ==="
cat test.log
# A `swift test` that matched and ran nothing exits 0. Without this
# the whole public gate — every job below depends on this one
# passing — could report green having executed no test at all,
# which is the "zero work is never success" invariant this project
# applies to its own verdicts, unapplied to its own CI. The
# acceptance and schemata jobs below have had this since P8; the
# `unit` job never did.
#
# `|| exit_code=1`, never a bare call: the step sets `set -o
# pipefail` but NOT `set -e` (see the top of this run block), so a
# bare invocation's non-zero status is discarded and the trailing
# `exit $exit_code` would hand back the green it already had. Same
# idiom as the acceptance job's own call. A real test failure above
# must still win, which it does — this can only ever set the code
# to 1, never clear it.
Scripts/assert-tests-ran.sh test.log "unit (all suites)" || exit_code=1
exit $exit_code
# Coverage-lane scope, v0 (this round) vs. later:
# v0: Codecov coverage (project + patch, informational) and
# Sonar static analysis/duplication (no coverage import,
# informational). Test Analytics / JUnit output is explicitly
# NOT a goal of this round — SwiftPM's `--xunit-output`
# fidelity with Swift Testing was never independently
# verified, so it is left out rather than wired on trust.
# later: Codecov Test Analytics (once JUnit output is verified
# reliable), Sonar coverage import (generic coverage XML
# converted from this same lcov, once a converter is worth
# the added scope), and promoting either gate to blocking
# (once a few real runs show what the noise/denominator
# actually look like).
#
# One test execution (the `swift test --enable-code-coverage` above)
# feeds both Sonar and Codecov — deliberately not a second `swift
# test` invocation just to collect coverage. SwiftPM writes raw
# profile data to `<bin path>/codecov/default.profdata`, keyed to the
# test binary at `<bin path>/<Package>PackageTests.xctest/Contents/
# MacOS/<Package>PackageTests`; `llvm-cov export` turns that pair into
# a single portable lcov.info that both tools consume. The bin path
# itself is resolved via `swift build --show-bin-path` rather than
# hardcoded as `.build/debug` — it actually varies by toolchain/SDK
# (verified: a local Xcode 26.6/Swift 6.3.3 run and this job's own
# macos-15/Xcode 16.4 runner resolved to different paths, and a
# hardcoded guess silently no-op'd here on the very first live run).
# `always()`: coverage is informational, so still attempt it (and let
# the step no-op if the earlier build never got far enough to produce
# profile data) even when the Test step above reported a failure.
- name: Export coverage (lcov)
if: always()
shell: bash
run: |
set -o pipefail
bin_path="$(swift build --show-bin-path)"
profdata="$bin_path/codecov/default.profdata"
test_bundle="$(find "$bin_path" -maxdepth 1 -name '*.xctest' -type d | head -n1)"
if [ -z "$test_bundle" ] || [ ! -f "$profdata" ]; then
echo "No coverage profile data found under $bin_path (profdata=$profdata, test bundle=${test_bundle:-<none>}) — skipping lcov export."
exit 0
fi
test_bin="$test_bundle/Contents/MacOS/$(basename "$test_bundle" .xctest)"
if [ ! -f "$test_bin" ]; then
echo "Found test bundle $test_bundle but no binary at $test_bin — skipping lcov export."
exit 0
fi
xcrun llvm-cov export "$test_bin" -instr-profile "$profdata" -format=lcov > lcov.info
wc -l lcov.info
# Sonar/Codecov are informational-only at this stage (point 8 of the
# coverage-lane plan): `continue-on-error: true` on both so a Sonar
# outage, a missing SONAR_TOKEN (see below), or a Codecov upload
# hiccup never fails this job or blocks merge-gate.
#
# This workflow triggers on plain `pull_request`, not
# `pull_request_target` — GitHub already withholds repository secrets
# from fork-PR runs under `pull_request`. Codecov's action supports a
# tokenless upload for public repos, so it stays unconditional and
# just uploads without a token on a fork PR. Sonar has no such
# fallback — it hard-requires SONAR_TOKEN — so on a fork PR the step
# is skipped outright rather than left to run as an expected,
# continue-on-error'd failure every time. Do not switch this workflow
# to `pull_request_target` to give forks a token instead — that would
# hand fork-controlled code execution access to a real secret.
#
# `always()` is required here, not implied: unless a step's own `if`
# uses one of always()/failure()/cancelled(), GitHub Actions silently
# ANDs it with success() — so without this, the earlier Test step
# failing (which coverage/Sonar/Codecov are meant to survive) would
# skip this step even on a same-repo push/PR. Verified live: the
# first real run on this branch had exactly that happen.
- name: SonarCloud Scan
if: >
always() &&
(github.event_name != 'pull_request' ||
github.event.pull_request.head.repo.full_name == github.repository)
continue-on-error: true
uses: SonarSource/sonarqube-scan-action@22918119ff8e1ca75a623e15c8296b6ea4fbe28f # v8.2.1
env:
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
- name: Upload coverage to Codecov
if: always()
continue-on-error: true
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
with:
token: ${{ secrets.CODECOV_TOKEN }}
files: lcov.info
fail_ci_if_error: false
# `always()`, not `failure()`: a job that hits its own
# `timeout-minutes` is reported as *cancelled*, not failed, and
# `if: failure()` does not run for a cancelled job's steps — the
# exact scenario this whole diagnostic exists to capture evidence
# from. Uploading unconditionally is the only way the stack
# samples and logs from a timeout are ever actually retained.
- name: Upload unit failure logs
if: always()
uses: actions/upload-artifact@v4
with:
name: unit-failure-logs
if-no-files-found: ignore
path: |
build.log
test.log
samples/
- name: Publish unit failure excerpt
if: failure() && github.event_name == 'pull_request'
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
const marker = '<!-- mutantkit-ci-unit-failure -->';
const readTail = (path) => {
if (!fs.existsSync(path)) return '';
const lines = fs.readFileSync(path, 'utf8').split('\n');
return lines.slice(-180).join('\n');
};
const build = readTail('build.log');
const test = readTail('test.log');
const excerpt = (test || build || 'No captured build/test log was available.').slice(-30000);
const body = `${marker}\n### Latest unit CI failure\n\n\`\`\`text\n${excerpt}\n\`\`\``;
const { owner, repo } = context.repo;
const issue_number = context.issue.number;
const comments = await github.paginate(github.rest.issues.listComments, { owner, repo, issue_number });
const existing = comments.find(c => c.body && c.body.includes(marker));
if (existing) {
await github.rest.issues.updateComment({ owner, repo, comment_id: existing.id, body });
} else {
await github.rest.issues.createComment({ owner, repo, issue_number, body });
}
# The suites that build and mutate real projects. Slow, and the only thing that
# proves the tool works rather than merely compiles: every wiring bug this
# project has had — a sandbox handed the wrong excludes, xcodebuild pointed at
# unmutated sources, concurrent mutants fighting over one simulator — was
# invisible to the unit tests and produced a confident, wrong score.
acceptance:
name: Acceptance (${{ matrix.fixture }})
runs-on: macos-15
# Generous: real xcodebuild/simulator work, legitimately slower than
# `unit`'s own plain unit suite — but still bounded, per the same
# "nothing runs unbounded" reasoning as `unit`'s own ceiling above.
timeout-minutes: 30
strategy:
# Never cancel siblings: which fixtures fail together is a diagnosis.
fail-fast: false
matrix:
include:
- fixture: swift-package
filter: SwiftPackageMacOSAcceptanceTests
simulator: "0"
- fixture: swift-package-coverage
filter: SwiftPackageMacOSCoverageAcceptanceTests
simulator: "0"
- fixture: shard-merge
filter: ShardMergeAcceptanceTests
simulator: "0"
# Restored: `Scripts/ci-fixtures.json` defines 18 fixtures and this
# matrix listed 17. The missing one was the quick-start path — the
# first thing a new user runs, and the only fixture never exercised
# publicly since P13.
#
# Kept in `ci-fixtures.json`'s own order so the two lists can be
# compared by eye. That there are two lists at all is the real
# defect: `Scripts/ci-fixtures.json`'s own header states that both
# `Scripts/ci-route.sh` and this workflow's full-matrix fallback are
# supposed to read that one file "rather than two independently-
# maintained copies that can silently drift apart". This hardcoded
# copy exists only because the job that derived the matrix from the
# JSON is not currently wired up here. Re-deriving it is deliberately
# deferred — enabling a new CI control plane immediately before a
# release is the wrong trade — and until that lands, the two lists
# are held in step by a pre-publish check rather than by hope.
- fixture: golden-path-onboarding
filter: GoldenPathOnboardingAcceptanceTests
simulator: "0"
- fixture: swift-package-ios
filter: SwiftPackageIOSAcceptanceTests
simulator: "1"
- fixture: xcode-project
filter: XcodeProjectAcceptanceTests
simulator: "1"
- fixture: xcode-workspace
filter: XcodeWorkspaceAcceptanceTests
simulator: "1"
- fixture: xcode-app-debug-dylib
filter: XcodeAppDebugDylibAcceptanceTests
simulator: "1"
- fixture: xcode-unlinked-source
filter: XcodeUnlinkedSourceAcceptanceTests
simulator: "1"
# `simulator: "1"`, not "0": this suite is not purely a host-side
# SwiftPM fixture — `CLICommandsAcceptanceTests
# .initDetectsXcodeDestinationDespiteSchemeAmbiguity` runs a real
# `mutantkit init` against `Fixtures/XcodeProject`, which calls
# `XcodeConfigDetector.detectDestination()` -> a real
# `xcrun simctl list devices available --json`. Classified "0"
# originally, which skips the "Available simulators" preflight
# step below (gated on `matrix.simulator == '1'`) — a real CI
# failure traced this exactly: `Detected: ...`/`Multiple schemes
# found (...)` printed correctly, but `Detected destination: ...`
# never appeared, and the test itself reported "failed after
# 66.165 seconds" — just past `SimulatorPool`'s 60s default
# timeout on `simctl`, on a runner whose CoreSimulator subsystem
# this job's own matrix classification told it not to expect to
# need. A pre-existing test-topology misclassification P4's own
# longer runtime happened to expose, not a P4 product regression
# (confirmed: `InitCommand.swift`/`XcodeConfigDetector.swift`/
# `SimulatorPool.swift`/this test file are all untouched across
# every P4 commit).
- fixture: cli-commands
filter: CLICommandsAcceptanceTests
simulator: "1"
# Real-machine XcodeConfigDetector coverage (a real `simctl` call
# finding a real destination, including against a real
# ambiguous-scheme project), relocated out of the always-run unit
# suite: these are integration tests, not deterministic unit
# tests, and CI must explicitly declare a simulator dependency for
# them rather than run them unconditionally with no such
# declaration at all. See `XcodeConfigDetectorAcceptanceTests.swift`'s
# own header comment.
- fixture: xcode-config-detector
filter: XcodeConfigDetectorAcceptanceTests
simulator: "1"
- fixture: process-supervision
filter: ProcessSupervisionAcceptanceTests
simulator: "0"
# The four below all build the same real .xcodeproj through the
# batching/incremental/coverage-selection paths that a 100-mutant
# benchmark against a real external project found broken while
# every unit test (all fakes, no real xcodebuild invocation or
# .xctestrun) stayed green. They are the whole reason this job
# exists rather than trusting `unit` alone.
- fixture: xcode-batch-testing
filter: XcodeBatchTestingAcceptanceTests
simulator: "1"
- fixture: xcode-batch-testing-ui-target
filter: XcodeBatchTestingUITargetAcceptanceTests
simulator: "1"
- fixture: xcode-coverage-selection
filter: XcodeCoverageSelectionAcceptanceTests
simulator: "1"
- fixture: xcode-incremental-batch-testing
filter: XcodeIncrementalBatchTestingAcceptanceTests
simulator: "1"
# Real `swiftc -typecheck` proof that RelationalOperatorReplacement-
# SchemataLowerer's ternary lowering type-checks at every eligible
# operand shape — the same discipline as the batch/incremental
# fixtures above, applied to a lowerer still gated out of
# SchemataLowererRegistry.builtIn. No schemata runtime binary
# involved (see the ror-schemata-differential job below for that),
# so it fits this matrix's plain build+test shape.
- fixture: ror-schemata-compile
filter: RORSchemataCompileViabilityAcceptanceTests
simulator: "0"
# P8 (CI Safe-Skip Policy): `Acceptance.waveEnabled`
# (`Tests/MutantKitTests/Acceptance/AcceptanceWaveSupport.swift`)
# gates `XcodeWaveEarlyKillAcceptanceTests` behind
# `MUTANTKIT_WAVE_ACCEPTANCE`, which no workflow ever set —
# confirmed by `git log` that wave-based early kill has been real,
# shipped functionality on `main` for a while (this file's own doc
# comment describing "wave execution does not exist yet" predates
# that and was never updated), so this real, differential, Xcode
# acceptance gate had never executed in any CI run. Reuses the same
# `XcodeProject` fixture the `xcode-project` entry above already
# builds — no new fixture, no new infra, just the missing opt-in.
- fixture: xcode-wave-early-kill
filter: XcodeWaveEarlyKillAcceptanceTests
simulator: "1"
wave: "1"
steps:
- uses: actions/checkout@v4
- name: Toolchain
run: swift --version && xcodebuild -version
# The suites pick whichever iPhone this machine actually has rather than
# pinning a model, so this is context for a failure, not a gate.
- name: Available simulators
if: matrix.simulator == '1'
run: xcrun simctl list devices available | grep -i iphone || true
- name: Build
shell: bash
run: |
set -o pipefail
swift build --build-tests 2>&1 | tee build.log
# Fixture-scoped environment for the `xcode-project` entry only: its
# filter (`XcodeProjectAcceptanceTests`) also matches
# `SchemataSupportedMatrixXcodeProjectAcceptanceTests`, whose two
# provenance/evidence tests need more than a plain build+test —
# without both steps below they fail deterministically (not flakily):
#
# - `brew install xcodegen`: `startupHitAndReceiptUUIDMatchOnIOSSimulator`
# stages a throwaway .xcodeproj via `xcodegen generate`, which fails
# with `env: xcodegen: No such file or directory` (exit 127) when the
# tool is absent. Only this suite shells out to xcodegen, so only
# this entry installs it.
# - Schemata runtime archives + override: `runtimeProvenanceIsOverride`
# asserts `SchemataRuntimeLibraryLocator.locate(for: .iOSSimulator)`
# resolves with `.override` provenance, and the evidence test links
# the runtime into a real iOS-Simulator chunk — both fail closed with
# "no bundled schemata runtime manifest ..." when neither a release
# install nor `MUTANTKIT_SCHEMATA_RUNTIME_LIB_OVERRIDE` provides one.
# Same proven recipe as the `ios-simulator-schemata-runtime` job
# (release macOS archive + `build-schemata-runtime.sh` iPhoneSimulator
# slice), exported via $GITHUB_ENV so the `Acceptance` step below —
# and only the runs on this entry — see it.
- name: Install xcodegen (xcode-project only)
if: matrix.fixture == 'xcode-project'
run: brew install xcodegen
- name: Schemata runtime archives (xcode-project only)
if: matrix.fixture == 'xcode-project'
shell: bash
run: |
set -euo pipefail
swift build -c release --product MutantKitSchemataRuntime
scripts/build-schemata-runtime.sh "$GITHUB_WORKSPACE/.build/release"
echo "MUTANTKIT_SCHEMATA_RUNTIME_LIB_OVERRIDE=$GITHUB_WORKSPACE/.build/release" >> "$GITHUB_ENV"
- name: Acceptance
shell: bash
env:
MUTANTKIT_ACCEPTANCE: "1"
MUTANTKIT_ACCEPTANCE_SIMULATOR: ${{ matrix.simulator }}
MUTANTKIT_WAVE_ACCEPTANCE: ${{ matrix.wave }}
run: |
set -o pipefail
swift test --filter ${{ matrix.filter }} 2>&1 | tee acceptance.log
test_exit=$?
# P8: a filter matching zero tests must fail this job, not pass it
# silently -- see scripts/assert-tests-ran.sh. A real test failure
# above must still win even if the count check itself passes.
scripts/assert-tests-ran.sh acceptance.log "fixture ${{ matrix.fixture }} (filter ${{ matrix.filter }})" || test_exit=1
exit $test_exit
- name: Upload acceptance failure logs
if: failure()
uses: actions/upload-artifact@v4
with:
name: acceptance-${{ matrix.fixture }}-failure-logs
if-no-files-found: ignore
path: |
build.log
acceptance.log
# Separate from the `acceptance` matrix above because this suite links and
# runs the real schemata runtime binary (SchemataMutationRunner drives a
# genuine `swift build`/`swift test` through it) rather than only
# type-checking lowered source — it needs
# MUTANTKIT_SCHEMATA_RUNTIME_LIB_OVERRIDE pointing at a real built
# `libMutantKitSchemataRuntime.a`, which the matrix's plain build+test
# steps never produce. Runs the exact MutationIDs discovered from one
# fixture through both MutationRunner (isolated) and SchemataMutationRunner
# (schemata, invoked directly — RelationalOperatorReplacementSchemataLowerer
# is still not registered in SchemataLowererRegistry.builtIn) and asserts
# zero disagreement — the real gate before that registration can happen.
ror-schemata-differential:
name: ROR schemata isolated-vs-schemata differential
runs-on: macos-15
# Same reasoning as `acceptance`'s own ceiling: real build/link/test
# work, bounded rather than unbounded.
timeout-minutes: 30
steps:
- uses: actions/checkout@v4
- name: Toolchain
run: swift --version && xcodebuild -version
- name: Build the schemata runtime static library
run: swift build -c release --product MutantKitSchemataRuntime
- name: Build tests
shell: bash
run: |
set -o pipefail
swift build --build-tests 2>&1 | tee build.log
- name: Differential acceptance
shell: bash
env:
MUTANTKIT_ACCEPTANCE: "1"
MUTANTKIT_SCHEMATA_RUNTIME_LIB_OVERRIDE: ${{ github.workspace }}/.build/release
run: |
set -o pipefail
swift test --filter RORSchemataIsolatedDifferentialAcceptanceTests 2>&1 | tee acceptance.log
test_exit=$?
# P8: see scripts/assert-tests-ran.sh -- zero matched tests must fail this job.
scripts/assert-tests-ran.sh acceptance.log "RORSchemataIsolatedDifferentialAcceptanceTests" || test_exit=1
exit $test_exit
- name: Upload failure logs
if: failure()
uses: actions/upload-artifact@v4
with:
name: ror-schemata-differential-failure-logs
if-no-files-found: ignore
path: |
build.log
acceptance.log
# Separate from both `acceptance` and `ror-schemata-differential` for the same
# reason as the latter: it needs a real, non-default
# MUTANTKIT_SCHEMATA_RUNTIME_LIB_OVERRIDE layout that the plain matrix build+test
# steps never produce — here, BOTH the macOS archive (`swift build`'s normal
# product, needed as this suite's negative control) AND the iOS-Simulator archive
# (produced only by scripts/build-schemata-runtime.sh, which nothing else in CI
# runs) living side by side under one override directory, exactly the shape
# SchemataRuntimeLibraryLocator expects. An earlier draft of this feature had no
# job that actually exercised this combination in CI at all — the new acceptance
# suites existed but nothing wired MUTANTKIT_SCHEMATA_RUNTIME_LIB_OVERRIDE for
# them, so they would have failed closed with `missingOverride` the first time
# anyone ran them here, silently never proving anything past a developer's own
# laptop. This job is that wiring.
ios-simulator-schemata-runtime:
name: iOS-Simulator schemata runtime (build + link viability)
runs-on: macos-15
# Same reasoning as `acceptance`'s own ceiling: real build/link/test
# work, bounded rather than unbounded.
timeout-minutes: 30
steps:
- uses: actions/checkout@v4
- name: Toolchain
run: swift --version && xcodebuild -version
- name: Build the macOS schemata runtime static library (this suite's negative control)
run: swift build -c release --product MutantKitSchemataRuntime
- name: Build the iOS-Simulator schemata runtime static library
run: scripts/build-schemata-runtime.sh "$GITHUB_WORKSPACE/.build/release"
- name: Build tests
shell: bash
run: |
set -o pipefail
swift build --build-tests 2>&1 | tee build.log
- name: iOS-Simulator runtime acceptance
shell: bash
env:
MUTANTKIT_ACCEPTANCE: "1"
MUTANTKIT_SCHEMATA_RUNTIME_LIB_OVERRIDE: ${{ github.workspace }}/.build/release
run: |
set -o pipefail
swift test --filter "SchemataIOSSimulatorRuntimeArtifactAcceptanceTests|SchemataIOSSimulatorRuntimeLinkAcceptanceTests" 2>&1 | tee acceptance.log
test_exit=$?
# P8: see scripts/assert-tests-ran.sh -- zero matched tests must fail this job.
scripts/assert-tests-ran.sh acceptance.log "SchemataIOSSimulatorRuntime*AcceptanceTests" || test_exit=1
exit $test_exit
- name: Upload failure logs
if: failure()
uses: actions/upload-artifact@v4
with:
name: ios-simulator-schemata-runtime-failure-logs
if-no-files-found: ignore
path: |
build.log
acceptance.log
# The one required status. Every other job in this workflow is
# unconditional — this workflow has no `route`, so nothing here is ever
# deliberately skipped — which makes the expected result table static:
# all five must genuinely report `success`.
#
# `if: always()`, not the default `success()`: this job's entire purpose is
# to report on the others, including when one of them fails or GitHub
# cancels it. Under the default condition it would be skipped in exactly
# that case, leaving the one check meant to be authoritative silently
# absent rather than red — the CI-layer form of "zero work is never
# success", and the shape this repository's own trust invariants exist to
# rule out.
#
# Deliberately simpler than the private repo's own merge-gate, which
# derives each job's expected result from a `route` job's declared
# execution plan. With no `route` here there is no plan to compare
# against, so mirroring that machinery would add a permissive failure mode
# (a mis-derived expectation reading as an intentional skip) in exchange
# for nothing. If `route` is ever restored, this must be replaced by the
# route-aware form rather than left alongside it.
#
# A skipped or cancelled dependency fails this gate. That is intended: on
# this workflow a skip is never intentional, so it is a signal that
# something did not run, which is precisely what must not pass unnoticed.
merge-gate:
name: Merge gate
runs-on: ubuntu-latest
timeout-minutes: 5
needs:
- lint
- complexity
- unit
- acceptance
- ror-schemata-differential
- ios-simulator-schemata-runtime
if: always()
steps:
- name: Confirm every required job succeeded
shell: bash
run: |
set -euo pipefail
failed="false"
check() {
local name="$1" actual="$2"
echo "$name -> $actual (expected: success)"
if [ "$actual" != "success" ]; then
echo "::error::$name reported '$actual'; every job in this workflow is unconditional, so anything other than success — including a skip or a cancellation — means it did not run to completion"
failed="true"
fi
}
check "lint" "${{ needs.lint.result }}"
check "complexity" "${{ needs.complexity.result }}"
check "unit" "${{ needs.unit.result }}"
check "acceptance" "${{ needs.acceptance.result }}"
check "ror-schemata-differential" "${{ needs.ror-schemata-differential.result }}"
check "ios-simulator-schemata-runtime" "${{ needs.ios-simulator-schemata-runtime.result }}"
if [ "$failed" = "true" ]; then
echo "::error::merge-gate: at least one required job did not succeed"
exit 1
fi
echo "merge-gate: every required job succeeded"