Skip to content

Commit 2c5c4e3

Browse files
drowaudiodrowaudio
authored
Add --plugin to pluginval test and GitHub Actions for validate and acceptance tests (#193)
1 parent f257fb4 commit 2c5c4e3

22 files changed

Lines changed: 646 additions & 11 deletions

‎.github/actions/README.md‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
# pluginval GitHub Actions
2+
3+
Two composite actions for using pluginval from another repository's workflows.
4+
Both run on Linux, macOS and Windows runners, download the matching pluginval
5+
release (or use a binary you give them), run everything they are given, write a
6+
results table to the job summary and fail the step if anything failed.
7+
8+
On Linux the plugin is run under `xvfb-run` when there is no `DISPLAY` (install
9+
`xvfb` on runners that don't have it).
10+
11+
## validate
12+
13+
Validates one or more plugins.
14+
15+
```yaml
16+
- uses: Tracktion/pluginval/.github/actions/validate@v2
17+
with:
18+
plugins: |
19+
build/MyPlugin_artefacts/Release/VST3/MyPlugin.vst3
20+
build/MyPlugin_artefacts/Release/AU/MyPlugin.component
21+
strictness-level: 10
22+
args: --skip-gui-tests
23+
```
24+
25+
| Input | Default | |
26+
| --- | --- | --- |
27+
| `plugins` | (required) | The plugins to validate, one per line. Each is a path, a glob or an AU id. A glob that matches nothing fails. |
28+
| `strictness-level` | `5` | The strictness level (1-10). |
29+
| `args` | | Extra options for `pluginval validate`, split on whitespace. |
30+
| `pluginval-version` | `v2.0.0-beta1` | The pluginval release to download: a tag, or `latest`. |
31+
| `pluginval-path` | | An existing pluginval binary to use instead of downloading one. |
32+
33+
Every `pluginval validate` option can also be set as an environment variable on
34+
the step (e.g. `env: { TIMEOUT_MS: 60000 }`), which is the way to pass values
35+
containing spaces.
36+
37+
## acceptance
38+
39+
Runs the [acceptance tests](../../docs/Acceptance%20testing.md) for every config
40+
(`*.json`) found recursively under a directory, optionally against one built
41+
plugin.
42+
43+
```yaml
44+
- uses: Tracktion/pluginval/.github/actions/acceptance@v2
45+
with:
46+
config-dir: tests/acceptance
47+
plugin: build/MyPlugin_artefacts/Release/VST3/MyPlugin.vst3
48+
```
49+
50+
| Input | Default | |
51+
| --- | --- | --- |
52+
| `config-dir` | (required) | Directory searched recursively for configs. |
53+
| `plugin` | | Plugin path or AU id to test every config against, instead of the `plugin` in each config. |
54+
| `record-missing` | `false` | Record a missing reference and pass, instead of failing. |
55+
| `timeout-ms` | | Fail a test if it takes longer than this (pluginval's default if empty). |
56+
| `pluginval-version` | `v2.0.0-beta1` | The pluginval release to download: a tag, or `latest`. |
57+
| `pluginval-path` | | An existing pluginval binary to use instead of downloading one. |
58+
59+
Notes:
60+
61+
- Every `*.json` under `config-dir` is treated as a config, apart from the
62+
`<reference>.wav.json` manifests. Keep other JSON files (and plugin bundles,
63+
which contain their own) out of it.
64+
- A config holding an array of tests is reported as a failure: use one config
65+
per file.
66+
- Relative `plugin` paths resolve against the working directory; the input,
67+
state and reference paths in a config resolve against the config's directory.
68+
69+
## Outputs
70+
71+
Both actions set:
72+
73+
| Output | |
74+
| --- | --- |
75+
| `passed` / `failed` | How many plugins / configs passed and failed. |
76+
| `report-dir` | A directory holding a JUnit XML report for each one. For `acceptance` it also has a `diffs` directory with the `*-diff.wav` files found under `config-dir`. |
77+
78+
To keep the reports and diffs from a failed run:
79+
80+
```yaml
81+
- uses: Tracktion/pluginval/.github/actions/acceptance@v2
82+
id: acceptance
83+
with:
84+
config-dir: tests/acceptance
85+
plugin: build/MyPlugin_artefacts/Release/VST3/MyPlugin.vst3
86+
87+
- uses: actions/upload-artifact@v4
88+
if: always()
89+
with:
90+
name: pluginval-acceptance-${{ runner.os }}
91+
path: ${{ steps.acceptance.outputs.report-dir }}
92+
```

‎.github/actions/acceptance/acceptance.sh‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
#!/usr/bin/env bash
2+
# Runs "pluginval test" on every config found (recursively) under
3+
# INPUT_CONFIG_DIR, runs them all, and fails if any failed. See action.yml for
4+
# the inputs.
5+
6+
set -u
7+
source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/../common/common.sh"
8+
9+
resolve_pluginval
10+
11+
config_dir="${INPUT_CONFIG_DIR:-}"
12+
13+
if [ ! -d "$config_dir" ]; then
14+
echo "::error::config-dir is not a directory: $config_dir"
15+
exit 1
16+
fi
17+
18+
report_dir="${INPUT_REPORT_DIR:-$RUNNER_TEMP/pluginval-reports/acceptance}"
19+
mkdir -p "$report_dir"
20+
21+
options=()
22+
[ -n "${INPUT_PLUGIN:-}" ] && options+=(--plugin "$INPUT_PLUGIN")
23+
[ "${INPUT_RECORD_MISSING:-false}" = "true" ] && options+=(--record-missing)
24+
[ -n "${INPUT_TIMEOUT_MS:-}" ] && options+=(--timeout-ms "$INPUT_TIMEOUT_MS")
25+
26+
passed=0
27+
failed=0
28+
index=0
29+
rows=""
30+
31+
# Every *.json is a config, except the <reference>.wav.json manifests that
32+
# pluginval writes next to each reference.
33+
IFS=$'\n'
34+
configs=( $(find "$config_dir" -type f -name '*.json' ! -name '*.wav.json' | sort) )
35+
unset IFS
36+
37+
for config in ${configs[@]+"${configs[@]}"}; do
38+
index=$((index + 1))
39+
name="${config#"$config_dir"/}"
40+
report="$report_dir/$index-$(sanitise "$(basename "$config" .json)").xml"
41+
42+
echo "::group::pluginval test: $name"
43+
44+
# pluginval only runs the first entry of an array config, so reject them
45+
# rather than silently skipping tests.
46+
if [ "$(tr -d '[:space:]' < "$config" | head -c 1)" = "[" ]; then
47+
echo "Array configs aren't supported: use one config per file."
48+
code=1
49+
else
50+
run_pluginval test ${options[@]+"${options[@]}"} --junit-xml "$report" "$config"
51+
code=$?
52+
fi
53+
54+
echo "::endgroup::"
55+
56+
if [ $code -eq 0 ]; then
57+
passed=$((passed + 1))
58+
rows="$rows| \`$name\` | :white_check_mark: passed |"$'\n'
59+
else
60+
failed=$((failed + 1))
61+
rows="$rows| \`$name\` | :x: failed |"$'\n'
62+
echo "::error::pluginval acceptance test failed: $name"
63+
fi
64+
done
65+
66+
if [ $index -eq 0 ]; then
67+
echo "::error::No configs (*.json) found under $config_dir"
68+
exit 1
69+
fi
70+
71+
# Gather the diff WAVs written next to failing references, so they can be
72+
# uploaded along with the reports.
73+
find "$config_dir" -type f -name '*-diff.wav' | while IFS= read -r diff; do
74+
mkdir -p "$report_dir/diffs"
75+
cp "$diff" "$report_dir/diffs/$(sanitise "${diff#"$config_dir"/}")"
76+
done
77+
78+
{
79+
echo "### pluginval acceptance tests"
80+
echo
81+
echo "| Config | Result |"
82+
echo "| --- | --- |"
83+
printf '%s' "$rows"
84+
} >> "$GITHUB_STEP_SUMMARY"
85+
86+
{
87+
echo "passed=$passed"
88+
echo "failed=$failed"
89+
echo "report-dir=$report_dir"
90+
} >> "$GITHUB_OUTPUT"
91+
92+
echo "pluginval acceptance tests: $passed passed, $failed failed"
93+
[ $failed -eq 0 ]

‎.github/actions/acceptance/action.yml‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
name: pluginval acceptance tests
2+
description: Runs pluginval's golden-file acceptance tests for every config in a directory.
3+
4+
inputs:
5+
config-dir:
6+
description: >
7+
Directory searched recursively for test configs (*.json). The
8+
<reference>.wav.json manifests are skipped.
9+
required: true
10+
plugin:
11+
description: >
12+
Plugin path or AU id to test every config against, instead of the
13+
"plugin" in each config.
14+
default: ''
15+
record-missing:
16+
description: Record a missing reference and pass, instead of failing.
17+
default: 'false'
18+
timeout-ms:
19+
description: Fail a test if it takes longer than this (pluginval's default if empty).
20+
default: ''
21+
pluginval-version:
22+
description: The pluginval release to download - a tag (e.g. v2.0.0-beta1) or "latest".
23+
default: v2.0.0-beta1
24+
pluginval-path:
25+
description: Path to an existing pluginval binary to use instead of downloading one.
26+
default: ''
27+
28+
outputs:
29+
passed:
30+
description: The number of configs that passed.
31+
value: ${{ steps.run.outputs.passed }}
32+
failed:
33+
description: The number of configs that failed.
34+
value: ${{ steps.run.outputs.failed }}
35+
report-dir:
36+
description: Directory holding a JUnit XML report per config, plus diffs/ with any diff WAVs.
37+
value: ${{ steps.run.outputs.report-dir }}
38+
39+
runs:
40+
using: composite
41+
steps:
42+
- id: run
43+
shell: bash
44+
env:
45+
INPUT_CONFIG_DIR: ${{ inputs.config-dir }}
46+
INPUT_PLUGIN: ${{ inputs.plugin }}
47+
INPUT_RECORD_MISSING: ${{ inputs.record-missing }}
48+
INPUT_TIMEOUT_MS: ${{ inputs.timeout-ms }}
49+
INPUT_PLUGINVAL_VERSION: ${{ inputs.pluginval-version }}
50+
INPUT_PLUGINVAL_PATH: ${{ inputs.pluginval-path }}
51+
run: bash "$GITHUB_ACTION_PATH/acceptance.sh"

‎.github/actions/common/common.sh‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
#!/usr/bin/env bash
2+
# Shared helpers for the pluginval composite actions (sourced, not run).
3+
# Kept compatible with bash 3.2, which is what macOS runners ship.
4+
5+
# Defaults so the scripts can also be run outside of GitHub Actions.
6+
if [ -z "${RUNNER_OS:-}" ]; then
7+
if [ "$(uname -s)" = "Darwin" ]; then RUNNER_OS=macOS
8+
elif [ "$(uname -s)" = "Linux" ]; then RUNNER_OS=Linux
9+
else RUNNER_OS=Windows
10+
fi
11+
fi
12+
13+
: "${RUNNER_TEMP:=$(mktemp -d)}"
14+
: "${GITHUB_STEP_SUMMARY:=/dev/null}"
15+
: "${GITHUB_OUTPUT:=/dev/null}"
16+
17+
# Sets PLUGINVAL_BIN to the pluginval binary: INPUT_PLUGINVAL_PATH if given,
18+
# otherwise the INPUT_PLUGINVAL_VERSION release ("latest" or a tag like v2.0.0-beta1)
19+
# downloaded for this runner's OS.
20+
resolve_pluginval()
21+
{
22+
if [ -n "${INPUT_PLUGINVAL_PATH:-}" ]; then
23+
PLUGINVAL_BIN="$INPUT_PLUGINVAL_PATH"
24+
else
25+
local version="${INPUT_PLUGINVAL_VERSION:-latest}"
26+
local dir="$RUNNER_TEMP/pluginval-$version"
27+
local url="https://github.com/Tracktion/pluginval/releases/download/$version/pluginval_$RUNNER_OS.zip"
28+
29+
if [ "$version" = "latest" ]; then
30+
url="https://github.com/Tracktion/pluginval/releases/latest/download/pluginval_$RUNNER_OS.zip"
31+
fi
32+
33+
case "$RUNNER_OS" in
34+
macOS) PLUGINVAL_BIN="$dir/pluginval.app/Contents/MacOS/pluginval" ;;
35+
Windows) PLUGINVAL_BIN="$dir/pluginval.exe" ;;
36+
*) PLUGINVAL_BIN="$dir/pluginval" ;;
37+
esac
38+
39+
if [ ! -f "$PLUGINVAL_BIN" ]; then
40+
echo "Downloading $url"
41+
mkdir -p "$dir"
42+
43+
if ! curl -fsSL --retry 3 "$url" -o "$dir/pluginval.zip"; then
44+
echo "::error::Couldn't download pluginval $version for $RUNNER_OS ($url)"
45+
exit 1
46+
fi
47+
48+
unzip -q -o "$dir/pluginval.zip" -d "$dir"
49+
chmod +x "$PLUGINVAL_BIN"
50+
fi
51+
fi
52+
53+
if [ ! -f "$PLUGINVAL_BIN" ]; then
54+
echo "::error::pluginval binary not found: $PLUGINVAL_BIN"
55+
exit 1
56+
fi
57+
}
58+
59+
# Runs pluginval with the given arguments. On Linux without a display it is
60+
# wrapped in xvfb-run, as plugins generally need an X server to load.
61+
run_pluginval()
62+
{
63+
if [ "$RUNNER_OS" = "Linux" ] && [ -z "${DISPLAY:-}" ] && command -v xvfb-run > /dev/null; then
64+
xvfb-run --auto-servernum "$PLUGINVAL_BIN" "$@"
65+
else
66+
"$PLUGINVAL_BIN" "$@"
67+
fi
68+
}
69+
70+
# Prints a file-name-safe version of the argument.
71+
sanitise()
72+
{
73+
printf '%s' "$1" | tr -c 'A-Za-z0-9._-' '_'
74+
}

‎.github/actions/validate/action.yml‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
name: pluginval validate
2+
description: Validates one or more audio plugins with pluginval.
3+
4+
inputs:
5+
plugins:
6+
description: >
7+
The plugins to validate, one per line. Each is a path, a glob (e.g.
8+
build/*_artefacts/Release/VST3/*.vst3) or an AU id.
9+
required: true
10+
strictness-level:
11+
description: The strictness level (1-10).
12+
default: '5'
13+
args:
14+
description: >
15+
Extra options for "pluginval validate", split on whitespace (e.g.
16+
"--skip-gui-tests --timeout-ms 60000"). Any option can also be set as an
17+
environment variable on the step.
18+
default: ''
19+
pluginval-version:
20+
description: The pluginval release to download - a tag (e.g. v2.0.0-beta1) or "latest".
21+
default: v2.0.0-beta1
22+
pluginval-path:
23+
description: Path to an existing pluginval binary to use instead of downloading one.
24+
default: ''
25+
26+
outputs:
27+
passed:
28+
description: The number of plugins that passed.
29+
value: ${{ steps.run.outputs.passed }}
30+
failed:
31+
description: The number of plugins that failed.
32+
value: ${{ steps.run.outputs.failed }}
33+
report-dir:
34+
description: Directory holding a JUnit XML report per plugin.
35+
value: ${{ steps.run.outputs.report-dir }}
36+
37+
runs:
38+
using: composite
39+
steps:
40+
- id: run
41+
shell: bash
42+
env:
43+
INPUT_PLUGINS: ${{ inputs.plugins }}
44+
INPUT_STRICTNESS_LEVEL: ${{ inputs.strictness-level }}
45+
INPUT_ARGS: ${{ inputs.args }}
46+
INPUT_PLUGINVAL_VERSION: ${{ inputs.pluginval-version }}
47+
INPUT_PLUGINVAL_PATH: ${{ inputs.pluginval-path }}
48+
run: bash "$GITHUB_ACTION_PATH/validate.sh"

0 commit comments

Comments
 (0)