Skip to content

test: add coverage reporting and cover save_data - #10196

Open
clavedeluna wants to merge 2 commits into
stanfordnlp:mainfrom
clavedeluna:coverage
Open

clavedeluna wants to merge 2 commits into
stanfordnlp:mainfrom
clavedeluna:coverage

Conversation

@clavedeluna

@clavedeluna clavedeluna commented Aug 11, 2026 •

Copy link
Copy Markdown

📝 Changes Description

This PR contains the following changes:

  • Adds pytest-cov to the dev extra and wires --cov=dspy --cov-report=term-missing --cov-report=xml into the CI test job (confirmed compatible with the existing pytest-xdist / --dist worksteal setup), uploading coverage.xml as a workflow artifact. Documents the --cov flag for local runs.
  • Adds a regression test for save_data in dspy/clients/utils_finetune.py (tests/clients/test_utils_finetune.py), which had no coverage at all.

Why coverage: we currently have no visibility into which parts of dspy/ are actually exercised by the test suite. Making coverage visible in CI (and easy to reproduce locally with one flag) is a low-risk first step toward finding untested, high-risk code paths like the one below — it's observability for the test suite itself, not a gate (no minimum threshold is enforced by this PR).

Why the save_data test isn't just about the coverage number: save_data is the function that serializes fine-tuning examples to a JSONL file before BootstrapFinetune (and friends) hand that file off to an external, paid, slow fine-tuning API. It resolves its write location via get_finetune_directory() (which honors a DSPY_FINETUNEDIR override, falling back to DSPy's shared cache dir) and names the file by a content hash. A regression here — wrong directory, an ignored DSPY_FINETUNEDIR override, truncated/malformed JSONL — would not fail loudly at the point of the bug. It would surface expensively downstream: a fine-tuning job silently trains on the wrong or corrupted data, or writes into a shared cache dir a user expected to be sandboxed, and nobody notices until the resulting model behaves oddly, well after the (non-refundable) training run finished. That's exactly the kind of quiet, unverified, expensive-to-fail path coverage is good at surfacing. The new test locks in the contract every fine-tuning caller relies on: the returned path is absolute, lives under the configured (or overridden) fine-tune directory, ends in .jsonl, and round-trips the exact data it was given.

✅ Contributor Checklist

  • Pre-Commit checks are passing (locally and remotely)
  • Title of your PR / MR corresponds to the required format
  • Commit message follows required format {label}(dspy): {message}

⚠️ Warnings

  • This only makes coverage visible (terminal output + coverage.xml artifact) — it does not enforce a minimum coverage threshold or gate merges on it.

@clavedeluna clavedeluna changed the title test(dspy): add coverage reporting and cover save_data test: add coverage reporting and cover save_data Aug 11, 2026
run: uv run -p .venv pytest -vv -m 'not extra and not deno' -n auto --dist worksteal tests/
run: |
uv run -p .venv pytest -vv -m 'not extra and not deno' -n auto --dist worksteal \
--cov=dspy --cov-report=term-missing --cov-report=xml tests/

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think it's reasonable to ignore a dir like datasets?

@greptile-apps

greptile-apps Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Adds coverage reporting to the standard test workflow, uploads the generated XML report from the Python 3.14 run, and adds a regression test for fine-tuning JSONL serialization. The coverage-enabled focused test passed with xdist enabled, generated a valid root-level coverage.xml at the workflow upload path, and the locked development environment successfully exposed pytest-cov. No defects were found; this is safe to merge.

Confidence Score: 5/5

Safe to merge: the coverage plugin, test invocation, XML output path, and focused serialization regression test behaved as intended.

The exercised behavior confirms that locked development dependency installation provides pytest-cov, the coverage options run successfully with the workflow's parallel test settings, and the report artifact is created at the configured upload location.

Files Needing Attention: No files need follow-up attention.

T-Rex T-Rex Logs

What T-Rex did

  • Compared the previous workflow and lockfile state with this PR, confirming there was no pytest-cov entry and no coverage artifact upload before the change.
  • Synchronized locked development dependencies and verified that pytest-cov is available for coverage reporting.
  • Ran the focused fine-tuning serialization test with the workflow's coverage options; pytest loaded pytest-cov 7.1.0, the test passed, and a parseable coverage.xml was produced at the configured artifact upload path.
  • Post-run checks confirmed the Coverage XML file was created at the intended path and the workflow upload target exists and is valid.
  • The dev-extra installation remained consistent with no environment changes required, ensuring the new dependency/configuration remains compatible.

View all artifacts

T-Rex Ran code and verified through T-Rex

Reviews (1): Last reviewed commit: "test(dspy): add test for save_data" | Re-trigger Greptile

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant