Action smoke test #1
Workflow file for this run
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: Action smoke test | |
| # Exercises the repo-root `action.yml` composite action the same way an | |
| # external consumer would (`uses: juntaki/mutantkit@<ref>`), except pointed | |
| # at this checkout (`uses: ./`) so a change to action.yml is validated | |
| # before it ever reaches a tag. Manual/scheduled rather than on every push: | |
| # unlike `ci.yml` (which builds from source) this depends on a real | |
| # published GitHub Release existing to download, so it is not a source-of- | |
| # truth gate for a PR — `release.yml`'s own `clean-machine-e2e` job already | |
| # proves the underlying tarball/SHA256SUMS/PATH recipe works at release | |
| # time; this workflow's job is to prove action.yml wraps that recipe | |
| # correctly. | |
| # | |
| # Two jobs, two different slices of action.yml: | |
| # - `smoke-test` installs with no project present at all — the one thing | |
| # every consumer does before anything else, and the only path a project | |
| # with no `mutantkit.yml` can safely exercise. | |
| # - `run-mode` is the one this phase adds: install, then doctor -> plan -> | |
| # run -> gate -> job summary -> baseline cache -> artifact upload, | |
| # against a real (if small) Swift package, because composite-action YAML | |
| # wiring a `set -euo pipefail` shell step at a time cannot be proven | |
| # correct any other way than actually running it. | |
| on: | |
| workflow_dispatch: | |
| schedule: | |
| # Weekly, so a change on the GitHub Actions platform side (runner image, | |
| # curl/shasum/tar behavior) that silently breaks the action is caught | |
| # without anyone having to remember to run it by hand. | |
| - cron: "0 6 * * 1" | |
| jobs: | |
| smoke-test: | |
| name: Install via action.yml (${{ matrix.version }}) | |
| runs-on: macos-15 | |
| strategy: | |
| fail-fast: false | |
| matrix: | |
| # Both the floating default and an explicit pin — the two paths | |
| # action.yml's `inputs.version` branches on. | |
| version: ["latest"] | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - name: Install mutantkit via this repo's own action.yml | |
| id: mutantkit | |
| uses: ./ | |
| with: | |
| version: ${{ matrix.version }} | |
| - name: Confirm mutantkit is on PATH and reports a version | |
| shell: bash | |
| run: | | |
| set -euo pipefail | |
| command -v mutantkit | |
| reported="$(mutantkit --version)" | |
| echo "mutantkit --version: $reported" | |
| if [ -z "$reported" ]; then | |
| echo "::error::mutantkit --version produced no output" | |
| exit 1 | |
| fi | |
| if [ "$reported" != "${{ steps.mutantkit.outputs.version }}" ]; then | |
| echo "::error::action output version ('${{ steps.mutantkit.outputs.version }}') does not match a fresh 'mutantkit --version' ('$reported')" | |
| exit 1 | |
| fi | |
| - name: mutantkit doctor (no project — must report that honestly, not crash) | |
| shell: bash | |
| run: mutantkit doctor || true | |
| run-mode: | |
| name: Full run mode (doctor -> plan -> run -> gate -> summary -> artifacts) | |
| runs-on: macos-15 | |
| steps: | |
| # Checked out to a subdirectory, not the workspace root: action.yml's | |
| # own internal steps have no `working-directory` of their own, so they | |
| # always run against `$GITHUB_WORKSPACE` — that has to be free for the | |
| # fixture project below, not occupied by this repo's own Package.swift | |
| # and Sources/. `./mutantkit-src` (below) is how `uses:` finds | |
| # action.yml once it is no longer at the workspace root. | |
| - name: Checkout mutantkit (action.yml + its own SwiftPackageMacOS fixture) | |
| uses: actions/checkout@v4 | |
| with: | |
| path: mutantkit-src | |
| # `Fixtures/SwiftPackageMacOS` is this repo's own acceptance fixture — | |
| # see `Tests/MutantKitTests/Acceptance/SwiftPackageMacOSAcceptanceTests | |
| # .swift`, which plans/runs it with the exact `mutantkit.yml` written | |
| # below and asserts a specific, named set of mutants live and die. That | |
| # makes it a real, already-proven-correct, host-testable (no simulator, | |
| # no signing) Swift package — reusing it here means this workflow is | |
| # exercising a project mutantkit's own test suite already trusts, | |
| # rather than a fixture invented just for this YAML. It ships with no | |
| # `mutantkit.yml` of its own (other fixtures in this repo that need one | |
| # carry one; this one is driven with an inline configuration per-test | |
| # instead), so one is written here, copied verbatim from that | |
| # acceptance test's own pinned `configuration` string — not | |
| # regenerated by `mutantkit init` — so a future change to the `init` | |
| # template cannot silently change what this smoke test exercises. | |
| # Deliberately has no `qualityGate:` thresholds: this suite's fixed, | |
| # named mutants always fully kill under `execution.strategy: isolated` | |
| # (proven by the acceptance test above), so the only way `gate` can | |
| # fail is a real integrity violation somewhere in the | |
| # install/doctor/plan/run chain, which is exactly the failure mode | |
| # this smoke test exists to catch — a passing gate here is a | |
| # meaningful assertion, not a tautology. | |
| - name: Stage a real, minimal Swift package as the project under test | |
| shell: bash | |
| run: | | |
| set -euo pipefail | |
| cp -R mutantkit-src/Fixtures/SwiftPackageMacOS/. . | |
| cat > mutantkit.yml <<'YAML' | |
| version: 1 | |
| project: | |
| kind: swiftPackageMacOS | |
| sources: | |
| include: [Sources/**] | |
| operators: | |
| profile: default | |
| execution: | |
| strategy: isolated | |
| workers: 2 | |
| reports: [console, json] | |
| YAML | |
| # This single step is the entire orchestration this phase added: | |
| # install (with attestation verification) -> doctor -> plan -> run | |
| # (json/github-actions/html/ci-summary reports) -> restore baseline | |
| # cache -> gate -> write $GITHUB_STEP_SUMMARY -> save baseline cache -> | |
| # upload-artifact. No `with:` beyond the default `version`/`diff` | |
| # inputs — this exercises the plain, no-input path every consumer | |
| # starts from. | |
| - name: Run mutantkit via action.yml | |
| id: mutantkit | |
| uses: ./mutantkit-src | |
| # Everything below only reads what the step above produced — no | |
| # verdict is recomputed, matching action.yml's own rule for itself. | |
| # `if: always()` because a real assertion of "the pipeline produced | |
| # real output" is more useful on a failing gate than on a passing one, | |
| # and because a broken `run`/`gate` step must not hide behind this | |
| # job's own steps being skipped. | |
| - name: Assert the run actually produced real output | |
| if: always() | |
| shell: bash | |
| run: | | |
| set -euo pipefail | |
| echo "::group::report.json" | |
| test -s report.json || { echo "::error::report.json is missing or empty"; exit 1; } | |
| jq -e '.baseline.passed == true' report.json >/dev/null \ | |
| || { echo "::error::report.json's baseline test run did not pass"; cat report.json; exit 1; } | |
| # `score` (like `RunReport` itself) only ever serializes its stored | |
| # fields — `killed`/`survived`/`noCoverage`/`excluded` — never the | |
| # `tested`/`effective` percentages, which are computed properties | |
| # `Codable` does not encode (see `MutationScore.tested`/`.effective` | |
| # in Sources/MutationModel/MutationOutcome.swift). Likewise | |
| # `IntegrityReport.passed` is computed from `violations`, so an | |
| # empty `violations` array, not a `passed` key, is what "integrity | |
| # held" looks like on disk. | |
| jq -e '.score.killed != null and .score.survived != null' report.json >/dev/null \ | |
| || { echo "::error::report.json has no mutation score"; cat report.json; exit 1; } | |
| jq -e '.integrity.violations | length == 0' report.json >/dev/null \ | |
| || { echo "::error::report.json recorded an integrity violation"; jq '.integrity' report.json; exit 1; } | |
| jq '{score, integrityViolations: .integrity.violations}' report.json | |
| echo "::endgroup::" | |
| echo "::group::gate-result.json" | |
| test -s gate-result.json || { echo "::error::gate-result.json is missing or empty"; exit 1; } | |
| jq -e 'has("passed") and has("violations")' gate-result.json >/dev/null \ | |
| || { echo "::error::gate-result.json is not a QualityGateResult"; cat gate-result.json; exit 1; } | |
| cat gate-result.json | |
| # This fixture carries no qualityGate thresholds and its mutants | |
| # are proven (by the Swift acceptance suite this fixture is | |
| # shared with) to fully kill under `execution.strategy: isolated` | |
| # — so the *only* way this can be anything but a clean pass is a | |
| # real regression somewhere in install/doctor/plan/run. Asserting | |
| # `true` here, not just "gate produced valid JSON", is what makes | |
| # this a pipeline-correctness check rather than a shape check. | |
| if [ "$(jq -r '.passed' gate-result.json)" != "true" ]; then | |
| echo "::error::expected gate to pass against this fixture's known-good baseline; it did not" | |
| exit 1 | |
| fi | |
| echo "::endgroup::" | |
| echo "::group::\$GITHUB_STEP_SUMMARY" | |
| test -s "$GITHUB_STEP_SUMMARY" || { echo "::error::no job summary was written"; exit 1; } | |
| cat "$GITHUB_STEP_SUMMARY" | |
| grep -q ':white_check_mark: Mutation quality gate passed' "$GITHUB_STEP_SUMMARY" \ | |
| || { echo "::error::job summary is missing the expected pass banner"; exit 1; } | |
| grep -q '## Mutation testing' "$GITHUB_STEP_SUMMARY" \ | |
| || { echo "::error::job summary is missing CISummaryReporter's own section (summary.md was not folded in)"; exit 1; } | |
| grep -q 'Tested Mutation Score' "$GITHUB_STEP_SUMMARY" \ | |
| || { echo "::error::job summary is missing the score table"; exit 1; } | |
| echo "::endgroup::" | |
| echo "::group::other reports run wrote" | |
| test -s .mutantkit/report.html || { echo "::error::.mutantkit/report.html was not generated"; exit 1; } | |
| test -s .mutantkit/summary.md || { echo "::error::.mutantkit/summary.md was not generated"; exit 1; } | |
| echo "::endgroup::" | |
| # `actions/upload-artifact` writes to the Actions API rather than the | |
| # filesystem, so "an artifact was uploaded" cannot be asserted from | |
| # inside this same job with a plain file check — the assertion is | |
| # `actions/download-artifact` succeeding in a later step, against the | |
| # exact name/files action.yml's own upload step declares. | |
| - name: Download the artifact action.yml uploaded, to prove it exists | |
| if: always() | |
| uses: actions/download-artifact@v4 | |
| with: | |
| name: mutantkit-report | |
| path: downloaded-artifact | |
| - name: Assert the downloaded artifact has real content | |
| if: always() | |
| shell: bash | |
| run: | | |
| set -euo pipefail | |
| for f in report.json gate-result.json .mutantkit/report.html .mutantkit/summary.md; do | |
| path="downloaded-artifact/$f" | |
| test -s "$path" || { echo "::error::uploaded artifact is missing or empty: $f"; exit 1; } | |
| done | |
| echo "mutantkit-report artifact verified: $(find downloaded-artifact -type f | sort | tr '\n' ' ')" |