Skip to content

CHANGELIST.md

Latest commit

 

History

History
133 lines (109 loc) · 9.63 KB

CHANGELIST.md

File metadata and controls

133 lines (109 loc) · 9.63 KB
Raw

pluginval Change List

2.0.0

Upgrading from 1.x: the command line now uses subcommands (pluginval validate [options] <plugin>; pluginval <plugin> still works) and is stricter: unrecognised options and invalid values are errors instead of being ignored, and --vst3validator has been removed because the VST3 validator is now built in. CI scripts that download releases/latest will get 2.0.0 automatically, so check the Breaking items below.

New

  • Added acceptance testing: pluginval test <config.json> renders a plugin with a known state, input (audio/MIDI) and optional playhead, then compares the output against a checked-in reference WAV, exiting 0 on a match and 1 on a mismatch (with a diff WAV). Use --record-missing to record new references. See docs/Acceptance testing.md
  • Added --plugin <plugin> to pluginval test to run a config against a given plugin instead of the one it names (the config's plugin can then be omitted)
  • Added --junit-xml <file> and --json-results <file> to both pluginval validate and pluginval test to write machine-readable reports for CI (also settable as JUNIT_XML / JSON_RESULTS). A report that can't be written fails the run. See docs/Adding pluginval to CI.md
  • Added two GitHub Actions for use from other repositories: Tracktion/pluginval/validate (validate a list of plugins) and Tracktion/pluginval/acceptance (run every acceptance config in a directory against a built plugin). See .github/actions/README.md
  • Added real-time safety checking on macOS, using rtcheck: --rtcheck enabled checks the plugin's process calls during the tests for real-time-unsafe behaviour, and --rtcheck relaxed skips the first process call to allow for initialisation. Off by default [#156]
  • The Steinberg VST3 validator is now built into pluginval and runs automatically for VST3 plugins at strictness level 5 and above, so it no longer needs downloading and passing in separately [#162]
  • Added --config <file.json> to load settings from JSON (repeatable; later files win per key)
  • Added a "Discrete parameter state restoration" test (strictness 8) which checks a discrete parameter has the same value after restoring a state, whatever value it held beforehand. This fails for parameters that keep the raw normalised value from the host instead of mapping it to a step (e.g. juce::AudioParameterBool)
  • Added drag-and-drop of plug-in files onto the main window, with two drop zones to either validate the plug-in or add it to the plugin list [#170]
  • Enabled the editor stress and extreme (real-time allocation / oversized block) tests that were present in the source tree but had not been compiled into the build

Changes

  • Updated JUCE to 9.0.4
  • Restructured the command line into subcommands: pluginval validate [options] <plugin> (the default), pluginval run-tests and pluginval strictness-help [level]. The plugin path is now a positional argument of validate
  • Deprecated: the flat flags --validate <plugin>, --run-tests and --strictness-help [level] still work as aliases (with a one-line notice) but will be removed in a future version. pluginval <plugin> remains a silent shorthand for validate
  • Replaced the hand-rolled command-line parser with CLI11 and a single JSON-based settings pipeline
  • Settings precedence is now (lowest to highest): defaults, environment variables, --config, command-line options
  • Environment variable support is now automatic for every option (including DISABLED_TESTS)
  • --help output is now generated by CLI11
  • Below strictness level 8, two-step (on/off) parameters are now only sent 0 or 1 when tests randomise parameter values. Arbitrary values are still sent to them at level 8 and above, and to all other parameters at every level
  • Added static linking to the Windows runtime so it should run on more Windows systems (particularly non-dev machines)
  • Made PluginInfoTest run on the message thread as the functions it calls aren't thread-safe

Breaking

  • Removed --vst3validator <path>: the VST3 validator is now built in (see above), and passing the option is an error [#162]
  • Unrecognised options now produce an error instead of being ignored
  • Invalid option values now error instead of silently falling back (e.g. --rtcheck banana, --strictness-level abc)
  • Falsey flag environment variables now mean off (e.g. SKIP_GUI_TESTS=0 no longer enables skipping)

Fixes

  • Fixed the crash handler so a backtrace is shown again when a plugin crashes, and made it async-signal-safe [#144]
  • Fixed false failures in the "Plugin state restoration" test where the first parameter was compared against the last value sent by a previous test rather than the plugin's own value
  • Fixed the "Plugin state restoration" test's exact binary state comparison never running. It was documented as running at strictness level 8 and above but checked the test's own level instead of the run's
  • Fixed races between a timeout and the writing of the log and report files, which could leave a report that disagreed with the exit code [#198]
  • Fixed garbled non-ASCII characters in output from the validation child process when a character was split across reads [#198]
  • Fixed --output-filename without --output-dir not being resolved against the working directory [#198]
  • pluginval's operator delete replacement no longer throws (which ended the process), and its operator new throws std::bad_alloc instead of returning null [#198]
  • Fixed the macOS app failing to start on any machine other than the one that built it ("Library not loaded: @rpath/librtcheck.dylib"); this affected the v2.0.0-beta1 and v2.0.0-beta2 releases

1.0.4

  • Limit auval's stress test to 20 seconds (vs 600) [#135]
  • Fixed incorrect "ALL TESTS PASSED" message when validating out of process [#125]
  • Updated juce to 8.0.3 [#133]
  • Added LV2 support [#25]
  • Provide CMake module for registering pluginval tests with CTest [#123]
  • Changed minimum Linux version to Ubuntu 22.04

1.0.3

  • Fix VST3 host bus issue

1.0.2

  • Added com.apple.security.get-task-allow to allow debugging on pluginval [macOS]

1.0.1

  • Fixed support for Ubuntu 18.04

1.0.0

  • Added running auval as a test for Audio Units on macOS (levels 5 and up)
  • Added running VST3's validator as a test for VST3s (levels 5 and up)
  • Added better conformance to the VST3 standard when running at levels 5 and below
  • Refactored internals for better stability

0.2.9

  • Fixed support for macOS from 10.11 and Windows without the runtime DLLs

0.2.8

  • Added a test to process audio with different sample rate and block sizes but not call releaseResources in between

0.2.6

  • Avoided a deadlock when opening plugin windows for the second time on Linux

0.2.5

  • Fixed Linux architecture for pre-built binaries

0.2.4

  • Added tests for plugin programs
  • Added a new CLI option to pass a file with a list of tests to disable
  • Added notarisation to macOS builds for Catalina compatibility

0.2.3

  • Fixed some failing tests on startup

0.2.2

  • Added a test for disabled HiDPI awareness on Windows
  • Updated JUCE to v5.4.3

0.2.1

  • Added an EditorAutomationTest to adjust parameter values whilst showing the plugin editor

0.2.0

  • Removed built-in support for VST2 testing, this can be built with by setting the VST2_SDK_DIR environment variable before running the tests/ scripts
  • Added the option to repeat tests a specified number of times
  • Added the option to randomise the test order each repeat
  • Added some message loop running to editor tests to ensure the UI is actually displayed
  • Added a test for processing audio whilst showing the editor
  • Added an option to automatically save log files with a sensible name (see the "--output-dir" option)
  • Disabled the LargerThanPreparedBlockSizeTest for AU, VST & VST3 formats
  • Added a "Test File" button to the UI for validating plugins that won't scan (this improves the stack trace information in the log file)

0.1.5

  • Fixed a problem catching allocations in clang release builds
  • Avoided accidently setting the bypass parameter for VST3 plugins
  • Simplified some parameter iteration methods to avoid returning bypass or non-automatable parameters
  • Added MIDI note on and off messages for synth processing tests
  • Added a Parameter thread safety test which attempts to call setValue on parameters from multiple threads concurrently mimicking automation and GUI interaction
  • Added a PluginStateTestRestoration test which calculates some checksums before randomising and restoring plugin state to check state restoration is correct
  • Added the ability to set command line arguments as environment variables e.g. "--skip-gui-tests" > "SKIP_GUI_TESTS=1", "--timeout-ms 30000" > "TIMEOUT_MS=30000"
  • Added the ability to set a random seed to use for the tests

0.1.4

  • Added stack backtraces to crashed validation output
  • Added tests for detecting memory allocations and deallocations in the audio thread
  • Added a test for creating an editor with an uninitialised plugin and a 0 sample rate and block size prepared plugin
  • Added a test calling processBlock with more samples than initialised with

0.1.3

  • Reduced the amount of default logging
  • Added the "--verbose" command line flag
  • Added some basic bus tests

0.1.2

  • Added optional logging from the slave process
  • Added an option to validate in the same process in GUI mode
  • Added the "--validate-in-process" command line flag
  • Replaced the "strictnessLevel" command line option with the more canonical "strictness-level" form
  • Added a timeout option which can be set from the GUI "Options" menu or the command line "--timeoutMs [numMs]" flag