Keep eval fixtures out of the published wheel - #176
Open
alexkuzmik wants to merge 1 commit into
Open
Conversation
The `opik-diagnose` and `opik-explain` skills carry `evals/` fixture directories. `skills_catalog.EXCLUDED_DIRS` keeps them off both MCP surfaces, and `build_skills_pack.py` keeps them out of the published pack — but nothing kept them out of the WHEEL. 0.2.24 on PyPI is clean only because it predates those two skills. The next release would have been the first to put 41 KB across 18 files into every user's site-packages, including runnable fixture scripts (`agent.py`, `seed.py`, `grader.py`, `run_evals.py`). Verified by building the 0.2.27 tag: 39 skill files in the wheel, 21 of which the server will ever serve. One `exclude` in `[tool.hatch.build]` fixes it: 21 files, 286 KB instead of 307. Nothing in the suite looked at a built artifact before. `test_skills_packaged.py` resolves skills through `importlib.resources`, which in a dev checkout reads the EDITABLE install pointing at `src/` — it passes whatever the backend would or would not put in a wheel. So this shipped past every green test. `tests/e2e/test_wheel_contents.py` builds a real wheel with the real backend and reads it back, both directions: every served document is present, and nothing that will not be served is. Plus the end of the chain — install the wheel into a throwaway venv and have THAT interpreter enumerate the skills, which is the only thing a pip-installing user actually has. Verified by removing the exclusion: two tests fail, exactly the state that would have shipped. e2e-marked because it shells out to `uv build` (~3s) and because, like the stdio suite, the thing under test only exists once something is really built. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while checking why
read_skillwas missing from PyPI. This would have shipped for the first time in the next release.The defect
opik-diagnoseandopik-explaincarryevals/fixture directories. Three places decide what ships, and only two of them exclude those:evals/?read_skillskills_catalog.EXCLUDED_DIRSopik-skillspackbuild_skills_pack.pyimports the same ruleBuilding the
0.2.27tag gives 39 skill files, of which the server will ever serve 21. The other 18 are 41 KB of fixtures, including runnableagent.py,seed.py,grader.py,run_evals.py, landing in every user'ssite-packages.0.2.24on PyPI is clean only because it predates both skills — so nobody has seen this yet, and nobody will if it goes in before the next release.The fix
One
excludein[tool.hatch.build]. Wheel goes 307 KB → 286 KB, 39 skill files → 21.Why no test caught it
Nothing in the suite looked at a built artifact.
test_skills_packaged.pyresolves skills throughimportlib.resources, which in a dev checkout reads the editable install pointing atsrc/— it passes whatever the backend would or would not put in a wheel.tests/e2e/test_wheel_contents.pybuilds a real wheel with the real backend and reads it back, in both directions: every served document is present, and nothing that will not be served is. Plus the end of the chain — install the wheel into a throwaway venv and have that interpreter enumerate the skills, which is the only thing a pip-installing user actually has.Verified by removing the exclusion: two tests fail, exactly the state that would have shipped.
e2e-marked (shells out touv build, ~3s), so it runs in the existinge2eCI job.Release note
0.2.27is the tag carryingread_skill(merge commit2b8ac10). If this merges first, release the tag that lands after it instead, and PyPI gets both the new tool and a clean wheel.🤖 Generated with Claude Code