diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index e8b87a7..7ac9ce1 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -30,8 +30,8 @@ jobs: - name: Check formatting run: uv run ruff format --check . - - name: Run tests - run: uv run pytest + - name: Run tests with coverage + run: uv run pytest --cov=text_toolkit --cov-report=term-missing - name: Run type checking run: uv run mypy diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7410abc..a57ded4 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -189,7 +189,7 @@ Before opening a pull request with code changes, run the local quality checks: ```bash uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy uv build --no-sources ``` @@ -205,7 +205,7 @@ Then repeat the checks: ```bash uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy uv build --no-sources ``` diff --git a/README.md b/README.md index 29df75c..ab224ee 100644 --- a/README.md +++ b/README.md @@ -121,7 +121,7 @@ Run the same quality checks used by CI: ```bash uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy uv build --no-sources ``` diff --git a/docs/06_github_actions.md b/docs/06_github_actions.md index e472a70..49570f5 100644 --- a/docs/06_github_actions.md +++ b/docs/06_github_actions.md @@ -47,7 +47,7 @@ In this project, the local checks are: ```bash uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy uv build --no-sources ``` @@ -169,6 +169,7 @@ This is important in CI because the workflow should check the project using the The `--group dev` option installs development dependencies such as: - `pytest`, +- `pytest-cov`, - `mypy`, - `ruff`. @@ -213,16 +214,18 @@ The developer can fix formatting locally with: uv run ruff format . ``` -## Tests +## Tests and coverage -The workflow runs tests: +The workflow runs tests and prints a coverage report: ```yaml -- name: Run tests - run: uv run pytest +- name: Run tests with coverage + run: uv run pytest --cov=text_toolkit --cov-report=term-missing ``` -This verifies the behavior of the example package. +This verifies the behavior of the example package while measuring statement and +branch coverage. The terminal report shows missing source lines, but the project +does not fail the workflow based on a coverage percentage. At this stage, the tests check: diff --git a/docs/07_git_commits_branches_prs.md b/docs/07_git_commits_branches_prs.md index 0628e5b..dc191a4 100644 --- a/docs/07_git_commits_branches_prs.md +++ b/docs/07_git_commits_branches_prs.md @@ -334,7 +334,7 @@ For code changes, run: ```bash uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy ``` diff --git a/docs/08_common_mistakes.md b/docs/08_common_mistakes.md index fb22e0d..b984a9e 100644 --- a/docs/08_common_mistakes.md +++ b/docs/08_common_mistakes.md @@ -145,7 +145,7 @@ For code changes, run: ```bash uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy ``` diff --git a/docs/09_checklist.md b/docs/09_checklist.md index 0b86639..2fb9ef3 100644 --- a/docs/09_checklist.md +++ b/docs/09_checklist.md @@ -140,6 +140,24 @@ uv run mypy Type checking complements tests and Ruff; it does not replace them. +## Test coverage + +The project should use coverage as a diagnostic tool: + +- [ ] pytest-cov is installed as a development dependency +- [ ] coverage is limited to production source code +- [ ] statement and branch coverage are measured +- [ ] missing lines are visible in the terminal report +- [ ] a percentage is not treated as proof of test quality + +In this guide, tests and coverage run together: + +```bash +uv run pytest --cov=text_toolkit --cov-report=term-missing +``` + +There is no minimum percentage gate. + ## Continuous integration The project should run checks automatically in CI: @@ -150,7 +168,7 @@ The project should run checks automatically in CI: - [ ] CI installs dependencies from the lockfile - [ ] CI runs linting - [ ] CI checks formatting -- [ ] CI runs tests +- [ ] CI runs tests with coverage reporting - [ ] CI runs type checking - [ ] CI builds a wheel and source distribution - [ ] CI status is visible in the README @@ -198,7 +216,7 @@ For code changes, run: ```bash uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy uv build --no-sources ``` @@ -214,7 +232,7 @@ Then repeat: ```bash uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy uv build --no-sources ``` @@ -250,6 +268,7 @@ How do I install it? Where is the code? Where are the tests? How do I run the tests? +How do I inspect test coverage? How do I check formatting and linting? How do I check type annotations? How do I build distribution files? diff --git a/docs/12_test_coverage.md b/docs/12_test_coverage.md index a336a04..434ab1b 100644 --- a/docs/12_test_coverage.md +++ b/docs/12_test_coverage.md @@ -97,6 +97,17 @@ The important columns are: The exact counts will change when the package changes. +## Coverage in CI + +The `Quality checks` job runs the same command: + +```bash +uv run pytest --cov=text_toolkit --cov-report=term-missing +``` + +The report is visible in the GitHub Actions log. CI fails when tests or coverage +collection fail, but it does not fail because the percentage decreases. + ## Why there is no percentage gate This project intentionally does not configure `fail_under`. diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 341fcac..2421da4 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -96,6 +96,7 @@ These parts are already included in the repository. - [x] Run linting in CI - [x] Run formatting checks in CI - [x] Run tests in CI +- [x] Run coverage reporting in CI - [x] Run type checking in CI - [x] Build distributions in CI - [x] Add a CI badge to the README @@ -138,10 +139,6 @@ These parts are already included in the repository. These are good candidates for upcoming small pull requests. -### Test coverage - -- [ ] Run coverage reporting in CI and align workflow documentation - ### Repository polish - [ ] Add issue templates only if the repository starts receiving external feedback diff --git a/docs/cheatsheet.md b/docs/cheatsheet.md index 2a22d8d..e0a081e 100644 --- a/docs/cheatsheet.md +++ b/docs/cheatsheet.md @@ -62,6 +62,7 @@ Examples: uv add --group dev pytest uv add --group dev ruff uv add --group dev mypy +uv add --group dev pytest-cov ``` Sync the local environment: @@ -84,6 +85,12 @@ Run tests: uv run pytest ``` +Run tests with coverage: + +```bash +uv run pytest --cov=text_toolkit --cov-report=term-missing +``` + Run type checking: ```bash @@ -113,7 +120,7 @@ Run all local checks: ```bash uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy uv build --no-sources ``` @@ -136,7 +143,7 @@ If formatting fails: uv run ruff format . uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy uv build --no-sources ``` @@ -154,7 +161,7 @@ Then run: ```bash uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy ``` @@ -250,7 +257,7 @@ uv python install uv sync --locked --group dev uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy uv build --no-sources ``` @@ -305,7 +312,7 @@ git status uv sync uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy ``` diff --git a/docs/common_questions.md b/docs/common_questions.md index 3a2a02b..36e4995 100644 --- a/docs/common_questions.md +++ b/docs/common_questions.md @@ -224,7 +224,7 @@ This guide first teaches the commands directly: ```bash uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy ``` @@ -305,7 +305,7 @@ Then run the local checks: ```bash uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy ``` diff --git a/docs/example_pull_request.md b/docs/example_pull_request.md index 7938ccc..e3c8aa8 100644 --- a/docs/example_pull_request.md +++ b/docs/example_pull_request.md @@ -289,7 +289,7 @@ For code changes, run: ```bash uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy ``` diff --git a/docs/from_script_to_project.md b/docs/from_script_to_project.md index 74fae35..01c0acf 100644 --- a/docs/from_script_to_project.md +++ b/docs/from_script_to_project.md @@ -249,6 +249,7 @@ build-backend = "uv_build" dev = [ "mypy>=2.3.1", "pytest>=9.0.3", + "pytest-cov>=7.1.0", "ruff>=0.15.12", ] @@ -260,6 +261,13 @@ python_version = "3.12" files = ["src", "tests"] strict = true +[tool.coverage.run] +branch = true +source = ["text_toolkit"] + +[tool.coverage.report] +show_missing = true + [tool.ruff] line-length = 88 target-version = "py312" @@ -308,7 +316,7 @@ In this guide, the main local checks are: ```bash uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy uv build --no-sources ``` @@ -317,7 +325,7 @@ These commands check: - linting, - formatting, -- behavior, +- behavior and test coverage, - type annotations, - package distribution builds. @@ -348,7 +356,7 @@ uv python install uv sync --locked --group dev uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy uv build --no-sources ``` @@ -429,7 +437,7 @@ The project now has: - documentation in `docs/`, - dependencies described in `pyproject.toml`, - exact dependency versions in `uv.lock`, -- local checks with Ruff, pytest, and mypy, +- local checks with Ruff, pytest-cov, and mypy, - CI with GitHub Actions, - a branch and pull request workflow. diff --git a/docs/github_actions_logs.md b/docs/github_actions_logs.md index cedd4d3..f0eb565 100644 --- a/docs/github_actions_logs.md +++ b/docs/github_actions_logs.md @@ -71,7 +71,7 @@ jobs: - name: Install dependencies - name: Run Ruff linting - name: Check formatting - - name: Run tests + - name: Run tests with coverage - name: Run type checking - name: Build distributions ``` @@ -288,7 +288,7 @@ After fixing, run: ```bash uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy uv build --no-sources ``` @@ -326,7 +326,7 @@ Then run all checks again: ```bash uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy ``` @@ -339,17 +339,18 @@ git commit -m "style: format files" If the formatting change is part of another small documentation or code PR, it may be better to amend the existing commit instead of adding a separate formatting-only commit. -## Step: Run tests +## Step: Run tests with coverage ### What it does This step runs: ```bash -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing ``` -It checks whether the code behaves as expected. +It checks whether the code behaves as expected and reports statement and branch +coverage for `text_toolkit`. There is no minimum percentage gate. ### If it fails @@ -364,7 +365,7 @@ Pytest usually shows: Example: ```text -FAILED tests/test_text_stats.py::test_count_words_handles_repeated_whitespace +FAILED tests/test_text_stats.py::test_count_characters_includes_whitespace_by_default ``` This tells you which test failed. @@ -382,7 +383,7 @@ assertion error Then run tests locally: ```bash -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing ``` For more detail: @@ -404,7 +405,7 @@ Then run: ```bash uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy ``` @@ -490,7 +491,7 @@ git status uv sync uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy uv build --no-sources ``` @@ -620,7 +621,7 @@ Run quality checks: ```bash uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy ``` @@ -723,7 +724,7 @@ A green CI run means: - dependencies installed, - Ruff linting passed, - formatting check passed, -- tests passed, +- tests and coverage reporting passed, - type checking passed, - wheel and source distribution builds passed. diff --git a/docs/glossary.md b/docs/glossary.md index edec052..25cf4aa 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -40,7 +40,7 @@ The CI workflow runs: ```bash uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy uv build --no-sources ``` diff --git a/docs/markdown_review.md b/docs/markdown_review.md index 621ed05..9108564 100644 --- a/docs/markdown_review.md +++ b/docs/markdown_review.md @@ -169,7 +169,7 @@ Good: ```bash uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy ``` @@ -382,7 +382,7 @@ However, if the repository has existing quality checks, it is still fine to run: ```bash uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy ``` diff --git a/docs/quickstart_clone_and_ci.md b/docs/quickstart_clone_and_ci.md index af5ded3..19588cf 100644 --- a/docs/quickstart_clone_and_ci.md +++ b/docs/quickstart_clone_and_ci.md @@ -187,10 +187,10 @@ uv sync ## 6. Run local checks -Run tests: +Run tests with coverage: ```bash -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing ``` Run Ruff linting: @@ -216,7 +216,7 @@ A useful local check sequence is: ```bash uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy ``` @@ -231,7 +231,7 @@ Then repeat: ```bash uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy ``` @@ -241,7 +241,7 @@ If the project is set up correctly: - Ruff linting should pass, - Ruff formatting check should pass, -- pytest should pass, +- pytest and coverage reporting should pass, - mypy should report no issues. If one command fails, read the error message before changing anything. @@ -375,8 +375,8 @@ jobs: - name: Check formatting run: uv run ruff format --check . - - name: Run tests - run: uv run pytest + - name: Run tests with coverage + run: uv run pytest --cov=text_toolkit --cov-report=term-missing - name: Run type checking run: uv run mypy @@ -416,7 +416,7 @@ The CI workflow should run the same checks that developers run locally: ```bash uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy uv build --no-sources ``` @@ -505,7 +505,7 @@ git status uv sync uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy uv build --no-sources ``` @@ -522,7 +522,7 @@ uv sync uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy uv build --no-sources ``` @@ -534,7 +534,7 @@ uv python install uv sync --locked --group dev uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy uv build --no-sources ``` @@ -552,7 +552,7 @@ Before opening a pull request, run: ```bash uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy ``` diff --git a/docs/quickstart_uv.md b/docs/quickstart_uv.md index 806ea0b..1541c62 100644 --- a/docs/quickstart_uv.md +++ b/docs/quickstart_uv.md @@ -107,7 +107,7 @@ def add(left: int, right: int) -> int: return left + right ``` -## 4. Add pytest, Ruff, and mypy as development dependencies +## 4. Add pytest, pytest-cov, Ruff, and mypy as development dependencies Add `pytest`: @@ -127,6 +127,12 @@ Add `mypy`: uv add --group dev mypy ``` +Add `pytest-cov`: + +```bash +uv add --group dev pytest-cov +``` + ### Expected result The project should now contain development dependencies in `pyproject.toml`. @@ -161,8 +167,8 @@ This adds `rich` to the project dependencies. The runtime dependency should appear in the `[project]` dependencies section of `pyproject.toml`. -Development tools such as `pytest`, Ruff, and mypy should stay in the -development dependency group. +Development tools such as `pytest`, pytest-cov, Ruff, and mypy should stay in +the development dependency group. This separation keeps runtime dependencies and development tools easy to understand. @@ -170,7 +176,7 @@ Use runtime dependencies only when the actual package needs them. Do not add test or linting tools as runtime dependencies. -## 6. Configure pytest, Ruff, and mypy +## 6. Configure pytest, coverage, Ruff, and mypy Open `pyproject.toml`. @@ -193,6 +199,7 @@ build-backend = "uv_build" dev = [ "mypy>=2.3.1", "pytest>=9.0.3", + "pytest-cov>=7.1.0", "ruff>=0.15.12", ] @@ -204,6 +211,13 @@ python_version = "3.12" files = ["src", "tests"] strict = true +[tool.coverage.run] +branch = true +source = ["example_project"] + +[tool.coverage.report] +show_missing = true + [tool.ruff] line-length = 88 target-version = "py312" @@ -291,10 +305,10 @@ Do not commit `.venv/` to Git. ## 9. Run commands inside the environment -Run tests: +Run tests with coverage: ```bash -uv run pytest +uv run pytest --cov=example_project --cov-report=term-missing ``` Run Ruff linting: @@ -325,7 +339,7 @@ uv run ruff format . If everything is configured correctly: -- `uv run pytest` should pass, +- `uv run pytest --cov=example_project --cov-report=term-missing` should pass, - `uv run ruff check .` should pass, - `uv run ruff format --check .` should pass, - `uv run mypy` should report no issues. @@ -367,7 +381,7 @@ For everyday development, use: uv sync uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=example_project --cov-report=term-missing uv run mypy ``` @@ -377,7 +391,7 @@ If formatting fails: uv run ruff format . uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=example_project --cov-report=term-missing uv run mypy ``` @@ -420,13 +434,14 @@ touch tests/test_calculator.py uv add --group dev pytest uv add --group dev ruff uv add --group dev mypy +uv add --group dev pytest-cov uv add rich uv sync uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=example_project --cov-report=term-missing uv run mypy ``` diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 38fe071..9f9d7e5 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -14,7 +14,7 @@ uv --version uv sync uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy ``` @@ -261,7 +261,7 @@ Then check again: ```bash uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy ``` @@ -302,7 +302,7 @@ Then run all checks again: ```bash uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy ``` @@ -388,10 +388,40 @@ Then run: ```bash uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy ``` +## Coverage reporting fails + +### Problem + +Tests may pass without coverage, but this command fails: + +```bash +uv run pytest --cov=text_toolkit --cov-report=term-missing +``` + +### Possible causes + +- pytest-cov is missing from the development dependency group, +- the local environment is out of sync, +- the package name passed to `--cov` is incorrect, +- coverage configuration is invalid. + +### Fix + +Sync dependencies and retry: + +```bash +uv sync +uv run pytest --cov=text_toolkit --cov-report=term-missing +``` + +Check that `pyproject.toml` contains pytest-cov and the `[tool.coverage.run]` +section. A lower percentage by itself does not fail this project because no +`fail_under` threshold is configured. + ## `uv.lock` is out of sync ### Problem @@ -436,7 +466,7 @@ Run checks again: ```bash uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy ``` @@ -627,7 +657,7 @@ git status uv sync uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy ``` @@ -665,7 +695,7 @@ Run: uv sync uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy ``` @@ -765,7 +795,7 @@ git status uv sync uv run ruff check . uv run ruff format --check . -uv run pytest +uv run pytest --cov=text_toolkit --cov-report=term-missing uv run mypy ```