Skip to content

Add Codecov Test Analytics (JUnit XML test-results reporting) #57

Description

@andremun

Summary

Codecov's coverage upload (.github/workflows/tests.yml's codecov/codecov-action step, from #51/#54) only reports line coverage. Codecov's separate Test Analytics product — test run times, failure rates, flaky-test detection, and a failed-tests report on PRs — needs a JUnit XML file of the actual test results (pass/fail/duration per test case), uploaded via a second action, codecov/test-results-action@v1.

Codecov's own setup instructions only list JUnit-generation snippets for pytest/Vitest/Jest/PHPUnit — no MATLAB option, since matlab.unittest isn't one of their first-party integrations. It's straightforward regardless: matlab.unittest.plugins.XMLPlugin.producingJUnitFormat(file) is MATLAB's own built-in JUnit-format test-report plugin, and Codecov's ingestion is format-based, not language-based — a JUnit XML file is a JUnit XML file to their action.

Proposed change

  1. test_integration.m: add a second plugin to the existing TestRunner, alongside the current CodeCoveragePlugin:
    import matlab.unittest.plugins.XMLPlugin
    ...
    junitReportFile = fullfile(repoRoot, 'junit.xml');
    runner.addPlugin(XMLPlugin.producingJUnitFormat(junitReportFile));
    (Same repoRoot-relative pattern already used for coverage.xml — see the matlab-ci-testing skill's path-resolution guidance if this file is touched again.)
  2. .github/workflows/tests.yml: add the upload step Codecov's wizard specifies, after the existing test-run step:
    - name: Upload test results to Codecov
      if: ${{ !cancelled() }}
      uses: codecov/test-results-action@v1
      with:
        token: ${{ secrets.CODECOV_TOKEN }}
        files: junit.xml
    Uses the same CODECOV_TOKEN secret as the existing coverage-upload step. Note the condition is !cancelled() (Codecov's own recommendation for this specific action), not always() like the existing coverage-upload/artifact steps — worth a moment's thought on whether to align them or whether the distinction is deliberate on Codecov's part.
  3. Consider also uploading junit.xml as a build artifact (actions/upload-artifact), matching the existing coverage-report artifact, for local inspection without needing the Codecov dashboard — not required by Codecov itself, just consistent with the existing pattern.

Motivation

40 test cases across 4 matlab.unittest.TestCase classes is enough surface area that flaky-test detection and failure-rate tracking are genuinely useful, not just a formality. This is pure test-infrastructure tooling — no pipeline algorithm or .m production-code behaviour changes anywhere.

Scope note

Sized for v0.9.2, not milestone-assigned yet (no v0.9.2 milestone object exists in this repo yet — see #30/#50/#52/#53 for the same caveat). Not algorithmic, but a new deliverable rather than a fix to existing behaviour.

Acceptance criteria

  • test_integration.m produces a junit.xml alongside the existing coverage.xml
  • CI uploads it via codecov/test-results-action@v1
  • CODECOV_TOKEN secret already exists from the coverage-upload work — confirm the test-results action can reuse it (should be able to; same repo, same token scope)
  • A run with at least one intentionally-failing test confirms the failed-test report actually appears (PR comment and/or the Codecov dashboard), not just that the upload step exits 0

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions