You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Background
Currently, Symphonia relies heavily on developers bringing their own private media collections and testing them locally using symphonia-check. While this is an excellent approach for broad coverage, it presents a few challenges as the project matures:
Barrier to Entry: New contributors may not have a diverse corpus of files containing obscure codec features, making it hard for them to confidently verify their changes.
Proposal: A FATE-style Test Vector Corpus
To ensure long-term stability and make regression testing robust, I propose establishing a hosted test vector corpus, similar to FFmpeg's FATE suite or libvpx's test vectors.
The architecture could look like this:
Storage: A curated set of minimal, targeted test vectors (typically only a few seconds long to save space) hosted externally (e.g., GitHub Releases, an S3 bucket, or a separate Git LFS repo).
Integration: A script (e.g., a cargo xtask or a simple python script similar to benchtest.py) that downloads the corpus into the git-ignored testvec/ directory and verifies SHA-256 checksums.
CI Pipeline: The GitHub Actions workflow is updated to optionally fetch this minimal corpus and run integration tests (perhaps a headless version of symphonia-check) over the files.
Starting Point
We already have the infrastructure for downloading files in benchtest.py (which fetches from archive.org). We could start small. For example, I have a 3-second repro.mp3 file generated specifically to test the CRC-protection edge case from #516 that I would be happy to contribute as the first official regression test vector.
I've placed it in a public repository located here:
Would the maintainers (@pdeljanov , any other interested parties) be interested to moving towards a reproducible, CI-integrated test corpus? I'd be happy to help draft the fetch scripts or CI changes if there's interest in a specific hosting approach.
Background
Currently, Symphonia relies heavily on developers bringing their own private media collections and testing them locally using
symphonia-check. While this is an excellent approach for broad coverage, it presents a few challenges as the project matures:Proposal: A FATE-style Test Vector Corpus
To ensure long-term stability and make regression testing robust, I propose establishing a hosted test vector corpus, similar to FFmpeg's FATE suite or
libvpx's test vectors.The architecture could look like this:
cargo xtaskor a simple python script similar tobenchtest.py) that downloads the corpus into the git-ignoredtestvec/directory and verifies SHA-256 checksums.symphonia-check) over the files.Starting Point
We already have the infrastructure for downloading files in
benchtest.py(which fetches from archive.org). We could start small. For example, I have a 3-secondrepro.mp3file generated specifically to test the CRC-protection edge case from #516 that I would be happy to contribute as the first official regression test vector.I've placed it in a public repository located here:
https://github.com/baylesj/symphonia-test-vectors
Would the maintainers (@pdeljanov , any other interested parties) be interested to moving towards a reproducible, CI-integrated test corpus? I'd be happy to help draft the fetch scripts or CI changes if there's interest in a specific hosting approach.