Add --junit-xml and --json-results report output to validate and test (#192)

* Added --junit-xml and --json-results report output to validate and test

Both commands can now write machine-readable reports through a shared report_writer. A placeholder error report is written before a run and replaced on completion, and a timeout writes a timed-out report, so a report always agrees with the exit code. A report that can't be written fails the run.

* Report output: Renamed TestCase::Status to avoid the X11 Status macro on Linux

Author: drowaudio

CHANGELIST.md

@@ -3,6 +3,7 @@
 ### 2.0.0
 - Updated JUCE to 9.0.3
 - Fixed the crash handler so a backtrace is shown again when a plugin crashes, and made it async-signal-safe [#144]
+- Added `--junit-xml <file>` and `--json-results <file>` to both `pluginval validate` and `pluginval test` to write machine-readable reports for CI (also settable as `JUNIT_XML` / `JSON_RESULTS`). A report that can't be written fails the run. See `docs/Adding pluginval to CI.md`
 - Added acceptance testing: `pluginval test <config.json>` renders a plugin with a known state, input (audio/MIDI) and optional playhead, then compares the output against a checked-in reference WAV, exiting `0` on a match and `1` on a mismatch (with a diff WAV). Use `--record-missing` to record new references. See `docs/Acceptance testing.md`
 - Restructured the command line into subcommands: `pluginval validate [options] <plugin>` (the default), `pluginval run-tests` and `pluginval strictness-help [level]`. The plugin path is now a positional argument of `validate`
 - **Deprecated:** the flat flags `--validate <plugin>`, `--run-tests` and `--strictness-help [level]` still work as aliases (with a one-line notice) but will be removed in a future version. `pluginval <plugin>` remains a silent shorthand for `validate`

CLAUDE.md

@@ -84,6 +84,7 @@ pluginval/
 │   ├── PluginvalSettings.h   # Unified settings struct + JSON mapping + toPluginTestOptions()
 │   ├── SettingsParser.cpp/h  # CLI/env/config -> merged JSON -> settings; child-process handoff
 │   ├── SettingsSerializer.cpp/h # JSON load/save + value coercions (comma lists, hex seed)
+│   ├── ReportWriter.cpp/h    # JUnit XML + JSON report files (shared by validate and test)
 │   ├── CrashHandler.cpp/h    # Crash reporting utilities
 │   ├── TestUtilities.cpp/h   # Helper functions for tests
 │   ├── RTCheck.h             # Real-time safety checking macros
@@ -98,7 +99,7 @@ pluginval/
 │   │   ├── TestConfig.cpp/h        # snake_case JSON config struct (independent of PluginvalSettings)
 │   │   ├── AcceptanceTest.cpp/h    # plugin load + state + input + render; record-or-compare orchestrator
 │   │   ├── ReferenceComparator.cpp/h # Comparator interface + registry + SampleComparator
-│   │   └── TestReporter.cpp/h      # text + JSON result, exit code
+│   │   └── TestReporter.cpp/h      # text + JSON result, exit code, conversion to a report_writer::Report
 │   └── tests/                # Individual test implementations
 │       ├── BasicTests.cpp    # Core plugin tests (info, state, audio)
 │       ├── BusTests.cpp      # Audio bus configuration tests
@@ -316,6 +317,36 @@ spec: `tests/acceptance/Acceptance testing design.md`; end-user guide:
   multiplexing, automation, extra comparators, child-process isolation) are notes
   only.
 
+### Report Output (`--junit-xml` / `--json-results`)
+
+Both `validate` and `test` can write machine-readable report files through one
+shared writer, `report_writer` (`source/ReportWriter.cpp/h`). Stdout and the text
+log are unaffected.
+
+- **Model**: a neutral `report_writer::Report` (command, target, properties and a
+  list of `TestCase`s with a `passed` / `failed` / `error` status, messages,
+  duration and optional JSON `details`). `toJUnitXml()` builds a single
+  `<testsuite>` with `juce::XmlElement`; `toJson()` emits snake_case keys with a
+  `summary` block. `write()` skips unset files and creates parent directories; if it fails, the
+  run fails (exit `1`).
+- **Two adapters**: `toReport()` in `Validator.cpp` converts the
+  `juce::UnitTestRunner::TestResult` array collected in `runTests()` (one case per
+  `beginTest`); `acceptance::reporter::toReport()` converts an acceptance
+  `TestResult` (its existing JSON goes in `details`).
+- **Options**: for `validate` they are ordinary settings (`junitXml` /
+  `jsonResults` in `PluginvalSettings`, resolved to `juce::File`s on
+  `PluginTests::Options`), so env vars (`JUNIT_XML` / `JSON_RESULTS`), `--config`
+  and the child-process handoff all work. For `test` they are parsed in
+  `settings_parser::dispatch()` into `DispatchResult` and carried on
+  `acceptance::RunOptions`.
+- **Incomplete runs**: before a run starts, `makeIncomplete()` writes a
+  placeholder report with a single `error` case; the real report overwrites it on
+  completion, so a crash leaves an error report rather than a stale or missing
+  one. On a timeout the thread that ends the process writes a "timed out" error
+  report as its last act (`PluginsUnitTestRunner`'s `onTimeout` for validate, the
+  acceptance `Watchdog` for test), so the report always agrees with the exit code
+  even if the tests finish in the moment before the process is terminated.
+
 ### Test Framework
 
 Tests are self-registering. To find all tests, look for static instances:
@@ -467,6 +498,8 @@ Key options (for `validate`):
 - `--timeout-ms [ms]` - Test timeout (default: 30000, -1 for none)
 - `--verbose` - Enable verbose logging
 - `--output-dir [dir]` - Directory for log files
+- `--junit-xml [file]` - Write a JUnit XML report (also accepted by `test`)
+- `--json-results [file]` - Write a JSON report (also accepted by `test`)
 - `--sample-rates [list]` - Comma-separated sample rates
 - `--block-sizes [list]` - Comma-separated block sizes
 - `--rtcheck [disabled|enabled|relaxed]` - Real-time safety checking

CMakeLists.txt

@@ -134,6 +134,7 @@ set(SourceFiles
     source/MainComponent.h
     source/PluginTests.h
     source/PluginvalSettings.h
+    source/ReportWriter.h
     source/SettingsParser.h
     source/SettingsSerializer.h
     source/TestUtilities.h
@@ -151,6 +152,7 @@ set(SourceFiles
     source/Main.cpp
     source/MainComponent.cpp
     source/PluginTests.cpp
+    source/ReportWriter.cpp
     source/SettingsParser.cpp
     source/SettingsSerializer.cpp
     source/tests/BasicTests.cpp

docs/Acceptance testing.md

@@ -89,6 +89,13 @@ Any other option is an error.
   `RECORD_MISSING=1`.
 - `--timeout-ms <ms>` - fail (exit `1`) if the run takes longer than this
   (default `30000`, `-1` to never time out). Also settable as `TIMEOUT_MS`.
+- `--junit-xml <file>` - write a JUnit XML report, for CI systems that display
+  test results. Also settable as `JUNIT_XML`.
+- `--json-results <file>` - write a JSON report (the same result that is printed
+  to stdout, wrapped in a summary). Also settable as `JSON_RESULTS`.
+
+See [Adding pluginval to CI](Adding%20pluginval%20to%20CI.md#machine-readable-reports)
+for the report formats.
 
 The render runs on a background thread, as a host's audio thread would, leaving
 the message thread free for the plugin.

docs/Adding pluginval to CI.md

@@ -41,3 +41,62 @@ $ ./pluginval validate --output-dir "./bin" "<path_to_plugin>" || exit 1
 ```sh
 > choco install pluginval
 ```
+
+### Machine-readable reports
+Both `pluginval validate` and `pluginval test` can write their results to a file, as well as printing them:
+
+ - `--junit-xml <file>` writes a JUnit XML report, which most CI systems (GitHub Actions, GitLab, Jenkins, Azure Pipelines) can display as per-test results
+ - `--json-results <file>` writes the same results as JSON, for scripts and dashboards
+
+Relative paths are resolved against the working directory and any missing directories are created. Both options can be used together, and can also be set as the `JUNIT_XML` and `JSON_RESULTS` environment variables. The text output and log files are unchanged. If a report can't be written (e.g. the path isn't writable) the run fails with exit code `1`.
+
+```sh
+$ ./pluginval validate --strictness-level 5 --junit-xml "./reports/MyPlugin.xml" --json-results "./reports/MyPlugin.json" "<path_to_plugin>"
+```
+
+Each test pluginval runs becomes one `<testcase>`, with a `<failure>` holding the failure messages if it failed:
+
+```xml
+<testsuites name="pluginval" tests="20" failures="1" errors="0" time="12.345">
+  <testsuite name="pluginval validate: MyPlugin.vst3" tests="20" failures="1" errors="0" time="12.345" skipped="0" timestamp="...">
+    <testcase classname="pluginval.validate" name="Open plugin (cold)" time="0.150"/>
+    <testcase classname="pluginval.validate" name="Plugin state" time="0.020">
+      <failure message="...">...</failure>
+    </testcase>
+  </testsuite>
+</testsuites>
+```
+
+The JSON report has the same content:
+
+```json
+{
+  "pluginval_version": "2.0.0",
+  "command": "validate",
+  "target": "/path/to/MyPlugin.vst3",
+  "start_time": "...",
+  "duration_ms": 12345,
+  "properties": { "plugin_name": "MyPlugin", "strictness_level": "5" },
+  "summary": { "tests": 20, "passed": 19, "failed": 1, "errors": 0 },
+  "tests": [
+    { "name": "Open plugin (cold)", "status": "passed", "duration_ms": 150, "messages": [] }
+  ]
+}
+```
+
+`status` is `passed`, `failed` or `error`. For `pluginval test`, each test also has a `details` object holding the comparison results.
+
+If pluginval doesn't finish (the plugin crashes it, or the run times out), the report contains a single `error` entry saying so, so a report left over from an earlier run is never mistaken for a pass.
+
+##### GitHub Actions
+```yaml
+- name: Validate plugin
+  run: ./pluginval validate --strictness-level 5 --skip-gui-tests --junit-xml reports/pluginval.xml ./build/MyPlugin.vst3
+
+- name: Upload pluginval report
+  if: always()
+  uses: actions/upload-artifact@v4
+  with:
+    name: pluginval-report
+    path: reports/
+```

docs/Command line options.md

@@ -18,6 +18,8 @@ COMMANDS:
                                                     pass instead.
                                   --timeout-ms <ms> Fail if the run takes longer
                                                     (default 30000, -1 = never).
+                                  --junit-xml <file>    Write a JUnit XML report.
+                                  --json-results <file> Write a JSON report.
                                 Any other option is an error.
 
 The flat flags --validate <plugin>, --run-tests and --strictness-help [level]
@@ -46,6 +48,8 @@ OPTIONS (for the validate command):
           --output-dir TEXT   Directory in which to write the log files.
           --output-filename TEXT 
                               Filename to write logs into.
+          --junit-xml TEXT    File to write a JUnit XML report to.
+          --json-results TEXT File to write a JSON report to.
           --disabled-tests TEXT 
                               Comma-separated test names, or a path to a file listing them.
           --random-seed TEXT  Random seed (hex 0x.. or int) for replicable test runs.

source/CommandLine.cpp

@@ -232,6 +232,14 @@ void performCommandLine (CommandLineValidator& validator, const juce::String& co
         options.recordMissing = routed.recordMissing;
         options.timeoutMs = routed.timeoutMs;
 
+        const auto toReportFile = [] (const juce::String& path)
+        {
+            return path.isEmpty() ? juce::File() : juce::File::getCurrentWorkingDirectory().getChildFile (path);
+        };
+
+        options.junitXmlFile = toReportFile (routed.junitXmlPath);
+        options.jsonResultsFile = toReportFile (routed.jsonResultsPath);
+
         // Runs async so will quit itself when done
         validator.runAcceptanceTest (juce::File::getCurrentWorkingDirectory().getChildFile (routed.testConfigPath), options);
         return;