Repository navigation
test: add coverage reporting and cover save_data - #10196
clavedeluna wants to merge 2 commits into
Conversation
| run: uv run -p .venv pytest -vv -m 'not extra and not deno' -n auto --dist worksteal tests/ | ||
| run: | | ||
| uv run -p .venv pytest -vv -m 'not extra and not deno' -n auto --dist worksteal \ | ||
| --cov=dspy --cov-report=term-missing --cov-report=xml tests/ |
There was a problem hiding this comment.
Do you think it's reasonable to ignore a dir like datasets?
Greptile SummaryAdds coverage reporting to the standard test workflow, uploads the generated XML report from the Python 3.14 run, and adds a regression test for fine-tuning JSONL serialization. The coverage-enabled focused test passed with xdist enabled, generated a valid root-level Confidence Score: 5/5Safe to merge: the coverage plugin, test invocation, XML output path, and focused serialization regression test behaved as intended. The exercised behavior confirms that locked development dependency installation provides pytest-cov, the coverage options run successfully with the workflow's parallel test settings, and the report artifact is created at the configured upload location. Files Needing Attention: No files need follow-up attention.
What T-Rex did
Reviews (1): Last reviewed commit: "test(dspy): add test for save_data" | Re-trigger Greptile |
📝 Changes Description
This PR contains the following changes:
pytest-covto thedevextra and wires--cov=dspy --cov-report=term-missing --cov-report=xmlinto the CItestjob (confirmed compatible with the existingpytest-xdist/--dist workstealsetup), uploadingcoverage.xmlas a workflow artifact. Documents the--covflag for local runs.save_dataindspy/clients/utils_finetune.py(tests/clients/test_utils_finetune.py), which had no coverage at all.Why coverage: we currently have no visibility into which parts of
dspy/are actually exercised by the test suite. Making coverage visible in CI (and easy to reproduce locally with one flag) is a low-risk first step toward finding untested, high-risk code paths like the one below — it's observability for the test suite itself, not a gate (no minimum threshold is enforced by this PR).Why the
save_datatest isn't just about the coverage number:save_datais the function that serializes fine-tuning examples to a JSONL file beforeBootstrapFinetune(and friends) hand that file off to an external, paid, slow fine-tuning API. It resolves its write location viaget_finetune_directory()(which honors aDSPY_FINETUNEDIRoverride, falling back to DSPy's shared cache dir) and names the file by a content hash. A regression here — wrong directory, an ignoredDSPY_FINETUNEDIRoverride, truncated/malformed JSONL — would not fail loudly at the point of the bug. It would surface expensively downstream: a fine-tuning job silently trains on the wrong or corrupted data, or writes into a shared cache dir a user expected to be sandboxed, and nobody notices until the resulting model behaves oddly, well after the (non-refundable) training run finished. That's exactly the kind of quiet, unverified, expensive-to-fail path coverage is good at surfacing. The new test locks in the contract every fine-tuning caller relies on: the returned path is absolute, lives under the configured (or overridden) fine-tune directory, ends in.jsonl, and round-trips the exact data it was given.✅ Contributor Checklist
coverage.xmlartifact) — it does not enforce a minimum coverage threshold or gate merges on it.