Skip to content

Hosted test vector corpus for CI? #518

Description

@baylesj

Background
Currently, Symphonia relies heavily on developers bringing their own private media collections and testing them locally using symphonia-check. While this is an excellent approach for broad coverage, it presents a few challenges as the project matures:

  1. Regressions: Edge cases (like the CRC-protected Xing/Info tags in mp3: Xing/Info tag rejected on CRC-protected frames, VBR duration estimate ends up way off #516) are hard to test end-to-end in CI. While unit tests with mocked byte arrays work, they don't provide the confidence of parsing a real file.
  2. Barrier to Entry: New contributors may not have a diverse corpus of files containing obscure codec features, making it hard for them to confidently verify their changes.

Proposal: A FATE-style Test Vector Corpus
To ensure long-term stability and make regression testing robust, I propose establishing a hosted test vector corpus, similar to FFmpeg's FATE suite or libvpx's test vectors.

The architecture could look like this:

  1. Storage: A curated set of minimal, targeted test vectors (typically only a few seconds long to save space) hosted externally (e.g., GitHub Releases, an S3 bucket, or a separate Git LFS repo).
  2. Integration: A script (e.g., a cargo xtask or a simple python script similar to benchtest.py) that downloads the corpus into the git-ignored testvec/ directory and verifies SHA-256 checksums.
  3. CI Pipeline: The GitHub Actions workflow is updated to optionally fetch this minimal corpus and run integration tests (perhaps a headless version of symphonia-check) over the files.

Starting Point
We already have the infrastructure for downloading files in benchtest.py (which fetches from archive.org). We could start small. For example, I have a 3-second repro.mp3 file generated specifically to test the CRC-protection edge case from #516 that I would be happy to contribute as the first official regression test vector.

I've placed it in a public repository located here:

https://github.com/baylesj/symphonia-test-vectors

Would the maintainers (@pdeljanov , any other interested parties) be interested to moving towards a reproducible, CI-integrated test corpus? I'd be happy to help draft the fetch scripts or CI changes if there's interest in a specific hosting approach.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions