Mutation testing #23

Workflow file for this run

.github/workflows/mutation.yml at 2033ef0

	name: Mutation testing

	# Nightly rather than per pull request, for two measured reasons
	# (docs/spec/quality-policy.md):
	#
	# Cost. Timeouts dominate the run: a mutation that breaks a loop condition
	# burns the full timeout, which the plugin derives from the suite duration
	# and cannot be configured down. Locally that is ~2600 process-seconds; on a
	# four-core runner, over ten minutes for a job every other check beats by an
	# order of magnitude.
	#
	# Reproducibility. Two consecutive runs over the same unchanged code scored
	# 63.24% and 66.18%, because which mutations time out depends on machine
	# load. A blocking gate that moves three points on its own eventually fails a
	# pull request that changed nothing, and a gate contributors learn to re-run
	# is not a gate.
	#
	# It still fails loudly when the score drops, and workflow_dispatch runs it on
	# demand before a release.
	#
	# The schedule is when it may run, not what it measures: the `changed` job below
	# compares the default branch against the last run that reached a verdict and
	# skips when nothing has moved. Re-scoring identical code answers a question
	# already answered, and answers it differently, since the score is not
	# reproducible, so a quiet week would produce a week of contradictory numbers.
	on:
	schedule:
	- cron: '0 4 * * *'
	workflow_dispatch:

	permissions:
	contents: read
	# To read this workflow's own history and find the commit last scored.
	actions: read
	# Only to file the failure. A red job notifies whoever last touched the cron
	# and nobody else, and a scheduled run nobody watches is discovered whenever
	# someone next opens the Actions tab.
	issues: write

	concurrency:
	group: ${{ github.workflow }}
	cancel-in-progress: true

	jobs:
	changed:
	name: Has main moved?
	runs-on: ubuntu-latest

	outputs:
	run: ${{ steps.check.outputs.run }}

	steps:
	# Compared against the last run that reached a verdict, not against a
	# 24-hour window: a cancelled or delayed run would make a window lie, and
	# GitHub delays scheduled runs under load. Cancelled runs are excluded
	# deliberately: concurrency cancels them mid-flight, so their commit was
	# never actually scored.
	- name: Compare against the commit last scored
	id: check
	env:
	GH_TOKEN: ${{ github.token }}
	run: \|
	scored=$(gh api \
	"repos/${{ github.repository }}/actions/workflows/mutation.yml/runs?per_page=30" \
	--jq '[.workflow_runs[]
	\| select(.id != ${{ github.run_id }})
	\| select(.conclusion == "success" or .conclusion == "failure")][0].head_sha // ""')

	echo "last scored: ${scored:-none}"
	echo "current: ${{ github.sha }}"

	# A manual dispatch always runs, since it is the escape hatch, used before
	# a release and to re-check after a toolchain fix.
	if [ "${{ github.event_name }}" != 'schedule' ]; then
	echo 'run=true' >> "$GITHUB_OUTPUT"
	exit 0
	fi

	if [ "$scored" = "${{ github.sha }}" ]; then
	echo 'run=false' >> "$GITHUB_OUTPUT"
	echo "Skipped: \`${GITHUB_SHA:0:7}\` was already scored, and nothing has landed since." >> "$GITHUB_STEP_SUMMARY"
	else
	echo 'run=true' >> "$GITHUB_OUTPUT"
	fi

	mutate:
	name: ${{ matrix.namespace }}
	runs-on: ubuntu-latest

	needs: changed
	if: needs.changed.outputs.run == 'true'

	strategy:
	fail-fast: false
	matrix:
	# One runner per namespace: wall-clock becomes the slowest namespace
	# instead of the sum. src/Signing is by far the slowest, since every test
	# covering it signs a real PDF.
	#
	# Splitting by mutated path, not by --shard. --shard divides the test
	# suite, and mutation testing needs the whole suite available for every
	# mutation: a mutation killed by a test that landed in another shard is
	# reported as uncovered. Measured on src/Certificates, the full run
	# scores 64.71% with 8 uncovered, while shard 1/2 reports 61.76% with
	# 26 uncovered and shard 2/2 reports 69.12%. Faster, and wrong.
	#
	# Each floor sits a few points below the lowest measurement of that
	# namespace, because the score is not reproducible to the point: it
	# tracks how many mutations time out, which tracks machine load.
	#
	# Measurements below are the nightly runs of 2026-08-09 through
	# 2026-08-12, read from the logs rather than remembered. Every one of
	# them is higher than the pair each floor was originally set from, so
	# the floors were sitting six to twelve points low and had stopped
	# being able to fail.
	#
	# **Every measurement below predates two fixes and none of them has
	# been reproduced since.** The runs they came from installed none of
	# the verification tools, so the tests needing them skipped and every
	# mutation those would have killed counted as surviving; and they
	# carried --parallel, which generates no mutations at all. Treat them
	# as provenance, not as numbers, until a serial run with the full
	# toolchain has replaced them.
	include:
	- namespace: Certificates # 68.13 / 68.13 / 68.13 / 73.48
	min: 64
	- namespace: Signing # 70.05 / 70.02 / 72.75 / 73.98
	min: 66
	# Left where it is. The lowest observation, 75.19, clears this floor
	# by 0.19, which is the tightest margin here by an order of
	# magnitude. Raising it would be indefensible and lowering it is what
	# the rule in docs/spec/quality-policy.md forbids, so it stays and
	# the margin is written down instead of discovered on a red night.
	- namespace: Validation # 75.19 / 76.74 / 78.95 / 79.50
	min: 75
	# Added when two helpers moved here out of Signing and Validation,
	# which quietly took them out of the gate they had been under. The
	# floor was provisional at 65, from a single measurement of 83.44%.
	# Two consecutive runs have since scored lower than that one, which
	# is exactly why the rule asks for two.
	- namespace: Support # 78.26 / 79.26
	min: 74

	steps:
	- name: Checkout
	uses: actions/checkout@v7

	- name: Setup PHP
	uses: shivammathur/setup-php@v2
	with:
	php-version: '8.5'
	extensions: mbstring, dom, fileinfo, openssl, json, gd, imagick, bcmath, sqlite3
	coverage: pcov
	tools: composer:v2

	# The four installs below are main_action.yml's, verbatim, and they are
	# here for a sharper reason than going green.
	#
	# A test that cannot run cannot kill a mutation. An absent tool does
	# not merely skip: every mutation that test would have caught is reported
	# as surviving, so the score is understated and the floors above are
	# measured against a suite quietly smaller than the one a pull request
	# runs. It is the argument this file already makes against --shard, one
	# step further out.
	#
	# Kept verbatim so a diff against main_action.yml shows drift. The pins
	# have to move together: a validator or a grammar that differs between the
	# two workflows makes their verdicts incomparable.

	- name: Install qpdf
	run: sudo apt-get update -qq && sudo apt-get install -y -qq qpdf

	# veraPDF is the reference PDF/A validator and the only thing that can
	# establish a conformance verdict. Pinned rather than tracking "latest":
	# a validator that changes its verdicts between builds cannot be the
	# thing a gate is measured against.
	# See docs/decisions/0025-what-signing-does-to-pdf-a.md.
	- name: Install veraPDF
	env:
	VERAPDF_VERSION: '1.30.2'
	run: \|
	set -eux
	curl -sSL -o /tmp/verapdf.zip \
	"https://software.verapdf.org/releases/${VERAPDF_VERSION%.*}/verapdf-greenfield-${VERAPDF_VERSION}-installer.zip"
	unzip -q /tmp/verapdf.zip -d /tmp/verapdf-src
	cat > /tmp/verapdf-auto.xml <<'XML'
	<?xml version="1.0" encoding="UTF-8" standalone="no"?>
	<AutomatedInstallation langpack="eng">
	<com.izforge.izpack.panels.htmlhello.HTMLHelloPanel id="welcome"/>
	<com.izforge.izpack.panels.target.TargetPanel id="install_dir">
	<installpath>/opt/verapdf</installpath>
	</com.izforge.izpack.panels.target.TargetPanel>
	<com.izforge.izpack.panels.packs.PacksPanel id="sdk_pack_select">
	<pack index="0" name="veraPDF GUI" selected="true"/>
	<pack index="1" name="veraPDF Mac and *nix Scripts" selected="true"/>
	<pack index="3" name="veraPDF Corpus and Tests" selected="false"/>
	<pack index="4" name="veraPDF Documentation" selected="false"/>
	<pack index="5" name="veraPDF Sample Plugins" selected="false"/>
	</com.izforge.izpack.panels.packs.PacksPanel>
	<com.izforge.izpack.panels.install.InstallPanel id="install"/>
	<com.izforge.izpack.panels.finish.FinishPanel id="finish"/>
	</AutomatedInstallation>
	XML
	java -jar /tmp/verapdf-src/verapdf-greenfield-/verapdf-izpack-installer-.jar /tmp/verapdf-auto.xml
	sudo ln -s /opt/verapdf/verapdf /usr/local/bin/verapdf
	verapdf --version

	# pyHanko is the reader that enforces /DocMDP, which pdfsig cannot
	# surface at all. Pinned for the same reason veraPDF is.
	#
	# Two distributions: installing pyHanko alone gives no command, because
	# the CLI is packaged and versioned separately.
	# See docs/decisions/0031-certification-verified-by-a-reader.md.
	- name: Install pyHanko
	env:
	PYHANKO_VERSION: '0.36.2'
	PYHANKO_CLI_VERSION: '0.4.2'
	run: \|
	set -eux
	python3 -m venv /opt/pyhanko
	/opt/pyhanko/bin/pip install --quiet \
	"pyHanko==${PYHANKO_VERSION}" "pyhanko-cli==${PYHANKO_CLI_VERSION}"
	sudo ln -s /opt/pyhanko/bin/pyhanko /usr/local/bin/pyhanko
	pyhanko --version

	# The Arlington PDF Model, checked against the specification's own
	# grammar. Pinned by commit: the releases carry no binaries, and the TSV
	# model lives in the same tree, so one SHA pins tool and grammar together.
	# See docs/decisions/0037-what-we-write-against-the-grammar.md.
	- name: Install the Arlington PDF Model
	env:
	ARLINGTON_COMMIT: '9d75f6de8cdc3883d5519f83e804238329b9eb10'
	run: \|
	set -eux
	git clone --quiet https://github.com/pdf-association/arlington-pdf-model.git /tmp/arlington
	cd /tmp/arlington
	git checkout --quiet "${ARLINGTON_COMMIT}"
	cd TestGrammar
	cmake -B build -DPDFSDK_PDFIUM=ON -DCMAKE_BUILD_TYPE=Release .
	cmake --build build --config Release -j"$(nproc)"
	sudo cp bin/linux/TestGrammar /usr/local/bin/testgrammar
	sudo mkdir -p /opt/arlington
	sudo cp -r /tmp/arlington/tsv /opt/arlington/tsv
	testgrammar --help > /dev/null
	echo "ARLINGTON_TSV=/opt/arlington/tsv/latest" >> "$GITHUB_ENV"


	- name: Install dependencies
	run: composer update --prefer-dist --prefer-stable --no-interaction --no-progress

	# Raise a floor as its score improves; never lower one to make a run
	# pass.
	#
	# shell: bash for pipefail: the default shell would let tee's exit code
	# mask a failing run.
	- name: Run mutation testing
	id: mutate
	shell: bash
	run: >
	vendor/bin/pest --mutate
	--path=src/${{ matrix.namespace }}
	--exclude-group=network
	--min=${{ matrix.min }}
	\| tee mutation.log

	# The score lives nowhere but the log, so a red job cannot tell a
	# three-point timeout wobble from a real regression without someone
	# opening it, and calibrating a floor means reading two runs by hand.
	# Both become a glance at the run summary.
	- name: Record the score
	id: score
	if: always()
	shell: bash
	run: \|
	plain=$(sed -e 's/\x1b\[[0-9;]*m//g' mutation.log 2>/dev/null \|\| true)
	score=$(printf '%s\n' "$plain" \| grep -oE 'Score:[[:space:]]+[0-9.]+%' \| tail -1 \| grep -oE '[0-9.]+' \|\| true)
	mutations=$(printf '%s\n' "$plain" \| grep -oE 'Mutations:.*' \| tail -1 \| sed -e 's/[[:space:]]\+/ /g' \|\| true)

	echo "score=${score:-unknown}" >> "$GITHUB_OUTPUT"

	display="unavailable"
	[ -n "$score" ] && display="${score}%"

	{
	echo "### ${{ matrix.namespace }}"
	echo
	echo "\| \| \|"
	echo "\|---\|---\|"
	echo "\| Score \| ${display} \|"
	echo "\| Floor \| ${{ matrix.min }} \|"
	echo "\| Breakdown \| ${mutations:-unavailable} \|"
	echo
	echo "The timeout count is what moves the score between identical runs."
	} >> "$GITHUB_STEP_SUMMARY"

	# Scoped to the mutation step: a failure in checkout or composer is not a
	# score regression, and an issue titled as one would be worse than none.
	# Those still fall back to the e-mail.
	- name: Open or update the tracking issue
	if: failure() && steps.mutate.outcome == 'failure'
	uses: actions/github-script@v9
	env:
	NAMESPACE: ${{ matrix.namespace }}
	FLOOR: ${{ matrix.min }}
	SCORE: ${{ steps.score.outputs.score }}
	with:
	script: \|
	const { NAMESPACE, FLOOR, SCORE } = process.env
	const { owner, repo } = context.repo

	const run = `${context.serverUrl}/${owner}/${repo}/actions/runs/${context.runId}`

	// A run that produced no score did not score badly: it crashed, and
	// the two need different titles. Reporting a crash as a regression
	// sends whoever opens it hunting for a test that never weakened;
	// that happened on 2026-08-08, when php-code-coverage 14.2.4 broke
	// the plugin and three issues claimed a score of "unknown%".
	const crashed = !/^[0-9.]+$/.test(SCORE)

	const title = crashed
	? `Mutation run failed: ${NAMESPACE}`
	: `Mutation score below floor: ${NAMESPACE}`

	const body = crashed
	? [
	`The mutation run for \`src/${NAMESPACE}\` failed without producing a score.`,
	``,
	`Run: ${run}`,
	``,
	`This is not a score regression. No number was reported, so nothing can be`,
	`said about coverage quality from this run. Read the log before touching any`,
	`test. The usual cause is the toolchain rather than this repository, since`,
	`the job resolves its dependencies unpinned on every run.`,
	].join('\n')
	: [
	`\`src/${NAMESPACE}\` scored ${SCORE}% against a floor of ${FLOOR}.`,
	``,
	`Run: ${run}`,
	``,
	`Before acting on it: the score is not reproducible, and the variance tracks how`,
	`many mutations time out, which tracks machine load. \`Certificates\` swings about`,
	`three points between identical runs. One or two points is noise; a sustained`,
	`drop, or one that survives a re-run, is not.`,
	``,
	`Do not lower the floor to make this pass, see docs/spec/quality-policy.md.`,
	].join('\n')

	// One open issue per namespace, commented on rather than reopened
	// nightly: without this the same regression files a fresh issue
	// every night until someone fixes it.
	const open = await github.paginate(github.rest.issues.listForRepo, {
	owner,
	repo,
	state: 'open',
	labels: 'mutation',
	})

	const existing = open.find(issue => issue.title === title)

	if (existing) {
	await github.rest.issues.createComment({ owner, repo, issue_number: existing.number, body })
	return
	}

	await github.rest.issues.create({ owner, repo, title, body, labels: ['mutation'] })

Navigation Menu

Mutation testing #23

Workflow file

Mutation testing #23

Uh oh!

Workflow file for this run