Skip to content

Commit 7557b04

Browse files
drowaudiodrowaudio
committed
Merge remote-tracking branch 'origin/v2' into dev/discrete-params-validation-b488be
2 parents ca4ea12 + 0175cbf commit 7557b04

35 files changed

Lines changed: 1804 additions & 109 deletions

‎.github/actions/README.md‎

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
# pluginval GitHub Actions
2+
3+
Two composite actions for using pluginval from another repository's workflows.
4+
Their `action.yml` files are in `validate/` and `acceptance/` at the repository
5+
root, which keeps the references short; the scripts they run are in this directory.
6+
Both run on Linux, macOS and Windows runners, download the matching pluginval
7+
release (or use a binary you give them), run everything they are given and fail
8+
the step if anything failed.
9+
10+
Results are shown in two places: the job summary gets a table with a row per
11+
plugin / config and its test count, followed by the name and message of every
12+
failed test; and each failed plugin / config gets an error annotation naming its
13+
failed tests, which also appears in the pull request's checks.
14+
15+
On Linux the plugin is run under `xvfb-run` when there is no `DISPLAY` (install
16+
`xvfb` on runners that don't have it).
17+
18+
## validate
19+
20+
Validates one or more plugins.
21+
22+
```yaml
23+
- uses: Tracktion/pluginval/validate@v2
24+
with:
25+
plugins: |
26+
build/MyPlugin_artefacts/Release/VST3/MyPlugin.vst3
27+
build/MyPlugin_artefacts/Release/AU/MyPlugin.component
28+
strictness-level: 10
29+
args: --skip-gui-tests
30+
```
31+
32+
| Input | Default | |
33+
| --- | --- | --- |
34+
| `plugins` | (required) | The plugins to validate, one per line. Each is a path, a glob or an AU id. A glob that matches nothing fails. |
35+
| `strictness-level` | `5` | The strictness level (1-10). |
36+
| `args` | | Extra options for `pluginval validate`, split on whitespace. |
37+
| `report-dir` | | Directory to write the JUnit XML reports to (a temporary directory if empty). |
38+
| `pluginval-version` | `v2.0.0-beta1` | The pluginval release to download: a tag, or `latest`. |
39+
| `pluginval-path` | | An existing pluginval binary to use instead of downloading one. |
40+
41+
Every `pluginval validate` option can also be set as an environment variable on
42+
the step (e.g. `env: { TIMEOUT_MS: 60000 }`), which is the way to pass values
43+
containing spaces.
44+
45+
## acceptance
46+
47+
Runs the [acceptance tests](../../docs/Acceptance%20testing.md) for every config
48+
(`*.json`) found recursively under a directory, optionally against one built
49+
plugin.
50+
51+
```yaml
52+
- uses: Tracktion/pluginval/acceptance@v2
53+
with:
54+
config-dir: tests/acceptance
55+
plugin: build/MyPlugin_artefacts/Release/VST3/MyPlugin.vst3
56+
```
57+
58+
| Input | Default | |
59+
| --- | --- | --- |
60+
| `config-dir` | (required) | Directory searched recursively for configs. |
61+
| `plugin` | | Plugin path or AU id to test every config against, instead of the `plugin` in each config. |
62+
| `record-missing` | `false` | Record a missing reference and pass, instead of failing. |
63+
| `timeout-ms` | | Fail a test if it takes longer than this (pluginval's default if empty). |
64+
| `report-dir` | | Directory to write the JUnit XML reports to (a temporary directory if empty). |
65+
| `pluginval-version` | `v2.0.0-beta1` | The pluginval release to download: a tag, or `latest`. |
66+
| `pluginval-path` | | An existing pluginval binary to use instead of downloading one. |
67+
68+
Notes:
69+
70+
- Every `*.json` under `config-dir` is treated as a config, apart from the
71+
`<reference>.wav.json` manifests. Keep other JSON files (and plugin bundles,
72+
which contain their own) out of it.
73+
- A config holding an array of tests is reported as a failure: use one config
74+
per file.
75+
- Relative `plugin` paths resolve against the working directory; the input,
76+
state and reference paths in a config resolve against the config's directory.
77+
78+
## Outputs
79+
80+
Both actions set:
81+
82+
| Output | |
83+
| --- | --- |
84+
| `passed` / `failed` | How many plugins / configs passed and failed. |
85+
| `report-dir` | A directory holding a JUnit XML report for each one. For `acceptance` it also has a `diffs` directory with the `*-diff.wav` files found under `config-dir`. |
86+
87+
To keep the reports and diffs from a failed run:
88+
89+
```yaml
90+
- uses: Tracktion/pluginval/acceptance@v2
91+
id: acceptance
92+
with:
93+
config-dir: tests/acceptance
94+
plugin: build/MyPlugin_artefacts/Release/VST3/MyPlugin.vst3
95+
96+
- uses: actions/upload-artifact@v4
97+
if: always()
98+
with:
99+
name: pluginval-acceptance-${{ runner.os }}
100+
path: ${{ steps.acceptance.outputs.report-dir }}
101+
```
102+
103+
### Per-test results as a pull request check
104+
105+
The actions report failures through the job summary and annotations, which need
106+
no extra permissions. If you would rather have every pluginval test listed as its
107+
own check on the pull request, feed the JUnit reports to a reporter action such
108+
as [mikepenz/action-junit-report](https://github.com/mikepenz/action-junit-report)
109+
or [dorny/test-reporter](https://github.com/dorny/test-reporter):
110+
111+
```yaml
112+
jobs:
113+
validate:
114+
runs-on: macos-latest
115+
permissions:
116+
contents: read
117+
checks: write
118+
steps:
119+
# ... build the plugin ...
120+
121+
- uses: Tracktion/pluginval/validate@v2
122+
with:
123+
plugins: build/MyPlugin_artefacts/Release/VST3/MyPlugin.vst3
124+
report-dir: pluginval-reports
125+
126+
- uses: mikepenz/action-junit-report@v5
127+
if: always()
128+
with:
129+
report_paths: pluginval-reports/*.xml
130+
check_name: pluginval
131+
```
132+
133+
This needs the `checks: write` permission, which workflows triggered by pull
134+
requests from forks don't get, so on those the reporter step can't publish.

‎.github/actions/acceptance/acceptance.sh‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
#!/usr/bin/env bash
2+
# Runs "pluginval test" on every config found (recursively) under
3+
# INPUT_CONFIG_DIR, runs them all, and fails if any failed. See action.yml for
4+
# the inputs.
5+
6+
set -u
7+
source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/../common/common.sh"
8+
9+
resolve_pluginval
10+
11+
config_dir="${INPUT_CONFIG_DIR:-}"
12+
13+
if [ ! -d "$config_dir" ]; then
14+
echo "::error::config-dir is not a directory: $config_dir"
15+
exit 1
16+
fi
17+
18+
report_dir="${INPUT_REPORT_DIR:-$RUNNER_TEMP/pluginval-reports/acceptance}"
19+
mkdir -p "$report_dir"
20+
21+
options=()
22+
[ -n "${INPUT_PLUGIN:-}" ] && options+=(--plugin "$INPUT_PLUGIN")
23+
[ "${INPUT_RECORD_MISSING:-false}" = "true" ] && options+=(--record-missing)
24+
[ -n "${INPUT_TIMEOUT_MS:-}" ] && options+=(--timeout-ms "$INPUT_TIMEOUT_MS")
25+
26+
passed=0
27+
failed=0
28+
index=0
29+
rows=""
30+
details=""
31+
32+
# Every *.json is a config, except the <reference>.wav.json manifests that
33+
# pluginval writes next to each reference.
34+
IFS=$'\n'
35+
configs=( $(find "$config_dir" -type f -name '*.json' ! -name '*.wav.json' | sort) )
36+
unset IFS
37+
38+
for config in ${configs[@]+"${configs[@]}"}; do
39+
index=$((index + 1))
40+
name="${config#"$config_dir"/}"
41+
report="$report_dir/$index-$(sanitise "$(basename "$config" .json)").xml"
42+
43+
echo "::group::pluginval test: $name"
44+
45+
# pluginval only runs the first entry of an array config, so reject them
46+
# rather than silently skipping tests.
47+
if [ "$(tr -d '[:space:]' < "$config" | head -c 1)" = "[" ]; then
48+
failure="Array configs aren't supported: use one config per file."
49+
echo "$failure"
50+
code=1
51+
else
52+
run_pluginval test ${options[@]+"${options[@]}"} --junit-xml "$report" "$config"
53+
code=$?
54+
failure="pluginval exited with code $code (see the log)"
55+
fi
56+
57+
echo "::endgroup::"
58+
59+
# The per-test results come from the JUnit report; pass / fail from the exit code.
60+
read_report "$report"
61+
62+
if [ $code -eq 0 ]; then
63+
passed=$((passed + 1))
64+
rows="$rows| \`$name\` | :white_check_mark: passed | $REPORT_TESTS |"$'\n'
65+
else
66+
failed=$((failed + 1))
67+
rows="$rows| \`$name\` | :x: failed | $REPORT_TESTS |"$'\n'
68+
[ -z "$REPORT_DETAILS" ] && REPORT_DETAILS=" - $failure"$'\n'
69+
details="$details- \`$name\`"$'\n'"$REPORT_DETAILS"
70+
annotate_error "pluginval acceptance test failed: $name" "${REPORT_NAMES:-$failure}"
71+
fi
72+
done
73+
74+
if [ $index -eq 0 ]; then
75+
echo "::error::No configs (*.json) found under $config_dir"
76+
exit 1
77+
fi
78+
79+
# Gather the diff WAVs written next to failing references, so they can be
80+
# uploaded along with the reports.
81+
find "$config_dir" -type f -name '*-diff.wav' | while IFS= read -r diff; do
82+
mkdir -p "$report_dir/diffs"
83+
cp "$diff" "$report_dir/diffs/$(sanitise "${diff#"$config_dir"/}")"
84+
done
85+
86+
{
87+
echo "### pluginval acceptance tests"
88+
echo
89+
echo "| Config | Result | Tests |"
90+
echo "| --- | --- | --- |"
91+
printf '%s' "$rows"
92+
93+
if [ -n "$details" ]; then
94+
echo
95+
echo "#### Failed tests"
96+
echo
97+
printf '%s' "$details"
98+
fi
99+
} >> "$GITHUB_STEP_SUMMARY"
100+
101+
{
102+
echo "passed=$passed"
103+
echo "failed=$failed"
104+
echo "report-dir=$report_dir"
105+
} >> "$GITHUB_OUTPUT"
106+
107+
echo "pluginval acceptance tests: $passed passed, $failed failed"
108+
[ $failed -eq 0 ]

‎.github/actions/common/common.sh‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
#!/usr/bin/env bash
2+
# Shared helpers for the pluginval composite actions (sourced, not run).
3+
# Kept compatible with bash 3.2, which is what macOS runners ship.
4+
5+
COMMON_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
6+
7+
# Defaults so the scripts can also be run outside of GitHub Actions.
8+
if [ -z "${RUNNER_OS:-}" ]; then
9+
if [ "$(uname -s)" = "Darwin" ]; then RUNNER_OS=macOS
10+
elif [ "$(uname -s)" = "Linux" ]; then RUNNER_OS=Linux
11+
else RUNNER_OS=Windows
12+
fi
13+
fi
14+
15+
: "${RUNNER_TEMP:=$(mktemp -d)}"
16+
: "${GITHUB_STEP_SUMMARY:=/dev/null}"
17+
: "${GITHUB_OUTPUT:=/dev/null}"
18+
19+
# Sets PLUGINVAL_BIN to the pluginval binary: INPUT_PLUGINVAL_PATH if given,
20+
# otherwise the INPUT_PLUGINVAL_VERSION release ("latest" or a tag like v2.0.0-beta1)
21+
# downloaded for this runner's OS.
22+
resolve_pluginval()
23+
{
24+
if [ -n "${INPUT_PLUGINVAL_PATH:-}" ]; then
25+
PLUGINVAL_BIN="$INPUT_PLUGINVAL_PATH"
26+
else
27+
local version="${INPUT_PLUGINVAL_VERSION:-latest}"
28+
local dir="$RUNNER_TEMP/pluginval-$version"
29+
local url="https://github.com/Tracktion/pluginval/releases/download/$version/pluginval_$RUNNER_OS.zip"
30+
31+
if [ "$version" = "latest" ]; then
32+
url="https://github.com/Tracktion/pluginval/releases/latest/download/pluginval_$RUNNER_OS.zip"
33+
fi
34+
35+
case "$RUNNER_OS" in
36+
macOS) PLUGINVAL_BIN="$dir/pluginval.app/Contents/MacOS/pluginval" ;;
37+
Windows) PLUGINVAL_BIN="$dir/pluginval.exe" ;;
38+
*) PLUGINVAL_BIN="$dir/pluginval" ;;
39+
esac
40+
41+
if [ ! -f "$PLUGINVAL_BIN" ]; then
42+
echo "Downloading $url"
43+
mkdir -p "$dir"
44+
45+
if ! curl -fsSL --retry 3 "$url" -o "$dir/pluginval.zip"; then
46+
echo "::error::Couldn't download pluginval $version for $RUNNER_OS ($url)"
47+
exit 1
48+
fi
49+
50+
unzip -q -o "$dir/pluginval.zip" -d "$dir"
51+
chmod +x "$PLUGINVAL_BIN"
52+
fi
53+
fi
54+
55+
if [ ! -f "$PLUGINVAL_BIN" ]; then
56+
echo "::error::pluginval binary not found: $PLUGINVAL_BIN"
57+
exit 1
58+
fi
59+
}
60+
61+
# Runs pluginval with the given arguments. On Linux without a display it is
62+
# wrapped in xvfb-run, as plugins generally need an X server to load.
63+
run_pluginval()
64+
{
65+
if [ "$RUNNER_OS" = "Linux" ] && [ -z "${DISPLAY:-}" ] && command -v xvfb-run > /dev/null; then
66+
xvfb-run --auto-servernum "$PLUGINVAL_BIN" "$@"
67+
else
68+
"$PLUGINVAL_BIN" "$@"
69+
fi
70+
}
71+
72+
# Prints a file-name-safe version of the argument.
73+
sanitise()
74+
{
75+
printf '%s' "$1" | tr -c 'A-Za-z0-9._-' '_'
76+
}
77+
78+
# Reads a JUnit XML report written by pluginval and sets:
79+
# REPORT_TESTS a summary-table cell, e.g. "17" or "2 of 17 failed"
80+
# REPORT_NAMES "name: message" per failed test, "; " separated (for the annotation)
81+
# REPORT_DETAILS a markdown list item per failed test (for the job summary)
82+
# All three are empty if there is no report.
83+
read_report()
84+
{
85+
REPORT_TESTS=""
86+
REPORT_NAMES=""
87+
REPORT_DETAILS=""
88+
89+
[ -f "$1" ] || return 0
90+
91+
local kind first second
92+
while IFS=$'\t' read -r kind first second; do
93+
if [ "$kind" = "T" ]; then
94+
if [ "$second" -gt 0 ]; then REPORT_TESTS="$second of $first failed"; else REPORT_TESTS="$first"; fi
95+
elif [ "$kind" = "F" ]; then
96+
REPORT_NAMES="${REPORT_NAMES:+$REPORT_NAMES; }$first: $second"
97+
REPORT_DETAILS="$REPORT_DETAILS - **${first//|/\\|}**: ${second//|/\\|}"$'\n'
98+
fi
99+
done < <(awk -f "$COMMON_DIR/junit.awk" "$1")
100+
}
101+
102+
# Prints an ::error annotation, escaping the message as workflow commands need.
103+
annotate_error()
104+
{
105+
local message="${2//%/%25}"
106+
echo "::error title=$1::$message"
107+
}

‎.github/actions/common/junit.awk‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# Summarises a pluginval JUnit XML report (--junit-xml) as tab-separated lines:
2+
# T <tests> <failed> the totals (failures + errors)
3+
# F <test name> <message> one per failed test
4+
# Written for POSIX awk (macOS ships BSD awk), so no gawk extensions.
5+
6+
function attr(s, key, start, rest)
7+
{
8+
start = index(s, " " key "=\"")
9+
if (start == 0) return ""
10+
rest = substr(s, start + length(key) + 3)
11+
return substr(rest, 1, index(rest, "\"") - 1)
12+
}
13+
14+
function unescape(s)
15+
{
16+
gsub(/&#1[03];/, " ", s)
17+
gsub(/&lt;/, "<", s)
18+
gsub(/&gt;/, ">", s)
19+
gsub(/&quot;/, "\"", s)
20+
gsub(/&apos;/, "\047", s)
21+
gsub(/&amp;/, "\\&", s)
22+
gsub(/[\t ]+/, " ", s)
23+
return s
24+
}
25+
26+
{ sub(/\r$/, ""); xml = xml " " $0 }
27+
28+
END {
29+
count = split(xml, cases, "<testcase ")
30+
printf "T\t%d\t%d\n", attr(cases[1], "tests"), attr(cases[1], "failures") + attr(cases[1], "errors")
31+
32+
for (i = 2; i <= count; ++i)
33+
{
34+
problem = index(cases[i], "<failure")
35+
if (problem == 0) problem = index(cases[i], "<error")
36+
if (problem == 0) continue
37+
38+
message = unescape(attr(substr(cases[i], problem), "message"))
39+
if (length(message) > 300) message = substr(message, 1, 300) "..."
40+
printf "F\t%s\t%s\n", unescape(attr(" " cases[i], "name")), message
41+
}
42+
}

0 commit comments

Comments
 (0)