From d0f166fedce7d6c5fac046ddd126e4aa9cec6c7c Mon Sep 17 00:00:00 2001 From: Daniel Ochoa Date: Wed, 8 Apr 2026 15:05:10 -0500 Subject: [PATCH 1/3] FEA-107: Add PR validation gates, lockfile, and flaky test quarantine - Create requirements-dev.txt with pinned direct dev/CI dependencies (anthropic, mcp, pyright, pytest, PyYAML, ruff). Every CI job now installs via pip install -r requirements-dev.txt followed by pip check, which validates the resolved dependency graph is internally consistent and satisfies FEA-107 AC-004.1 (install must succeed without warnings or missing packages). File header documents the regeneration procedure for manual bumps. - Add [tool.pytest.ini_options] to pyproject.toml registering two markers: contract (repo-level contract tests) and quarantine (flaky, non-blocking). Registering the markers also silences pytest's PytestUnknownMarkWarning so future runs are warning-free. Also sets testpaths = ["plugins"]. - Rewrite .github/workflows/ci.yml with six jobs: - Lint: ruff check . - Type Check: pyright - Tests: pytest plugins/ -m 'not quarantine' (quarantined tests are now excluded from the blocking run) - Contract tests: pytest plugins/ -m contract, wrapped to treat exit code 5 (no tests collected) as success -- passes vacuously until tests are marked - Quarantined tests (non-blocking): pytest plugins/ -m quarantine with continue-on-error: true, also handling exit 5 gracefully - PR gates: aggregator with needs: [lint, typecheck, test, contract-tests] and if: always(); lets branch protection require one check long-term instead of four. - Add a concurrency group so stacked PR updates cancel in-flight runs. - Add docs/flaky-test-quarantine.md documenting the @pytest.mark.contract and @pytest.mark.quarantine conventions, how to quarantine/un-quarantine, and the policy for rolling review. - Add docs/ci-gates.md documenting the gate matrix and the gh api commands to update ruleset 13555155 once the new check contexts have been observed on a real PR run. The ruleset currently has zero required status checks. Testing: Ran the full suite locally against requirements-dev.txt in a Python 3.14 venv: pip check clean, ruff check . clean, pyright clean (0 errors, 0 warnings), pytest plugins/ -m 'not quarantine' = 577 passed, pytest plugins/ -m contract = exit 5 (wrapper treats as success), same for -m quarantine. Risks: The new check contexts (Lint, Type Check, Tests, Contract tests, Quarantined tests, PR gates) will not appear on existing PRs until the branch rebuilds. Branch protection ruleset 13555155 currently has zero required status checks -- none of the CI jobs are gating merges today. The ruleset must be updated to require the new contexts only after they have been observed on a real PR run; see docs/ci-gates.md. Pinned requirements-dev.txt is the minimum viable lockfile; it is bumped manually rather than via Dependabot. --- .github/workflows/ci.yml | 76 +++++++++++++++++++++++++++++++-- docs/ci-gates.md | 71 +++++++++++++++++++++++++++++++ docs/flaky-test-quarantine.md | 79 +++++++++++++++++++++++++++++++++++ pyproject.toml | 7 ++++ requirements-dev.txt | 30 +++++++++++++ 5 files changed, 259 insertions(+), 4 deletions(-) create mode 100644 docs/ci-gates.md create mode 100644 docs/flaky-test-quarantine.md create mode 100644 requirements-dev.txt diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index fca421eb..9a67683e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -5,6 +5,10 @@ on: branches: [main] pull_request: +concurrency: + group: ci-${{ github.event.pull_request.number || github.ref }} + cancel-in-progress: true + jobs: lint: name: Lint @@ -14,7 +18,8 @@ jobs: - uses: actions/setup-python@v5 with: python-version: "3.13" - - run: pip install ruff + - run: pip install -r requirements-dev.txt + - run: pip check - run: ruff check . typecheck: @@ -25,7 +30,8 @@ jobs: - uses: actions/setup-python@v5 with: python-version: "3.13" - - run: pip install pyright pyyaml anthropic mcp pytest + - run: pip install -r requirements-dev.txt + - run: pip check - run: pyright test: @@ -36,5 +42,67 @@ jobs: - uses: actions/setup-python@v5 with: python-version: "3.13" - - run: pip install pytest pyyaml anthropic - - run: pytest plugins/ + - run: pip install -r requirements-dev.txt + - run: pip check + - run: pytest plugins/ -m 'not quarantine' + + contract-tests: + name: Contract tests + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.13" + - run: pip install -r requirements-dev.txt + - run: pip check + - name: Run contract tests + shell: bash + run: | + set +e + pytest plugins/ -m contract + rc=$? + if [ $rc -eq 5 ]; then + echo "No contract tests collected -- gate passes vacuously." | tee -a "$GITHUB_STEP_SUMMARY" + exit 0 + fi + exit $rc + + quarantine: + name: Quarantined tests (non-blocking) + runs-on: ubuntu-latest + continue-on-error: true + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.13" + - run: pip install -r requirements-dev.txt + - run: pip check + - name: Run quarantined tests + shell: bash + run: | + set +e + echo "### Quarantined tests" >> "$GITHUB_STEP_SUMMARY" + pytest plugins/ -m quarantine 2>&1 | tee -a "$GITHUB_STEP_SUMMARY" + rc=${PIPESTATUS[0]} + if [ $rc -eq 5 ]; then + echo "No quarantined tests." | tee -a "$GITHUB_STEP_SUMMARY" + fi + exit 0 + + pr-gates: + name: PR gates + needs: [lint, typecheck, test, contract-tests] + if: always() + runs-on: ubuntu-latest + steps: + - name: Aggregate required gate results + run: | + for result in "${{ needs.lint.result }}" "${{ needs.typecheck.result }}" "${{ needs.test.result }}" "${{ needs.contract-tests.result }}"; do + if [ "$result" != "success" ]; then + echo "A required gate failed: $result" + exit 1 + fi + done + echo "All required gates passed." diff --git a/docs/ci-gates.md b/docs/ci-gates.md new file mode 100644 index 00000000..7c29d677 --- /dev/null +++ b/docs/ci-gates.md @@ -0,0 +1,71 @@ +# PR Validation Gates (FEA-107) + +This repo runs standardised PR validation gates on every pull request. The gates +live in `.github/workflows/ci.yml`. + +## Gate matrix + +| Job | Command | Blocking? | +| ---------------------------------- | ------------------------------------- | --------- | +| `Lint` | `ruff check .` | yes | +| `Type Check` | `pyright` | yes | +| `Tests` | `pytest plugins/ -m 'not quarantine'` | yes | +| `Contract tests` | `pytest plugins/ -m contract` | yes | +| `Quarantined tests (non-blocking)` | `pytest plugins/ -m quarantine` | **no** | +| `PR gates` | aggregates the four blocking jobs | yes | + +Every job installs dev dependencies via `pip install -r requirements-dev.txt` +followed by `pip check`, which validates the resolved dependency graph is +internally consistent. This satisfies FEA-107 AC-004.1 ("lockfile consistency +check; install must succeed without warnings or missing packages"). + +`requirements-dev.txt` is bumped manually. See the header comment in that +file for the regeneration procedure. + +The `Contract tests` and `Quarantined tests` jobs treat pytest exit code 5 +("no tests collected") as success so they pass vacuously until tests are +marked. + +See [`flaky-test-quarantine.md`](./flaky-test-quarantine.md) for the +`@pytest.mark.quarantine` and `@pytest.mark.contract` conventions. + +## Updating branch protection + +Ruleset `13555155` currently has **zero** required status checks. After a test +PR confirms the new gates are reporting, add this rule: + +```bash +# 1. Fetch the current ruleset JSON +gh api repos/closedloop-ai/claude-plugins/rulesets/13555155 > /tmp/plugins-ruleset.json + +# 2. Edit /tmp/plugins-ruleset.json -- add a new entry to the "rules" array: +# { +# "type": "required_status_checks", +# "parameters": { +# "strict_required_status_checks_policy": false, +# "do_not_enforce_on_create": false, +# "required_status_checks": [ +# { "context": "Lint", "integration_id": 15368 }, +# { "context": "Type Check", "integration_id": 15368 }, +# { "context": "Tests", "integration_id": 15368 }, +# { "context": "Contract tests", "integration_id": 15368 }, +# { "context": "PR gates", "integration_id": 15368 } +# ] +# } +# } + +# 3. Apply +gh api -X PUT repos/closedloop-ai/claude-plugins/rulesets/13555155 \ + --input /tmp/plugins-ruleset.json +``` + +Verify with: + +```bash +gh api repos/closedloop-ai/claude-plugins/rules/branches/main \ + | jq '.[] | select(.type=="required_status_checks").parameters.required_status_checks' +``` + +**Do not update the ruleset until the new check contexts have been observed on +at least one PR run.** Otherwise PRs will hang waiting for a context that does +not yet exist in the workflow history. diff --git a/docs/flaky-test-quarantine.md b/docs/flaky-test-quarantine.md new file mode 100644 index 00000000..86079514 --- /dev/null +++ b/docs/flaky-test-quarantine.md @@ -0,0 +1,79 @@ +# Flaky Test Quarantine & Contract Tests + +This repo uses pytest markers to categorize tests for CI gating. Two markers are +currently defined in `pyproject.toml` under `[tool.pytest.ini_options]`: + +- `@pytest.mark.contract` -- repo-level contract tests that assert invariants + across multiple plugins or modules. Run by the `contract-tests` CI job. +- `@pytest.mark.quarantine` -- known-flaky tests that should not block merges. + Run by the `quarantine` CI job with `continue-on-error: true`. + +## How CI uses the markers + +`.github/workflows/ci.yml` runs several gates, each installing from +`requirements-dev.txt` followed by `pip check`: + +| Job | Command | Blocking? | +| ---------------- | --------------------------------------- | --------- | +| `Lint` | `ruff check .` | yes | +| `Type Check` | `pyright` | yes | +| `Tests` | `pytest plugins/ -m 'not quarantine'` | yes | +| `Contract tests` | `pytest plugins/ -m contract` | yes | +| `Quarantined tests (non-blocking)` | `pytest plugins/ -m quarantine` | **no** | +| `PR gates` | aggregates the four blocking jobs | yes | + +The `Contract tests` and `Quarantined tests` jobs both treat pytest exit +code 5 ("no tests collected") as success so they pass vacuously until +tests are marked. + +## Quarantining a flaky test + +1. Add the marker to the top of the test function or class: + ```python + import pytest + + @pytest.mark.quarantine + def test_something_flaky(): + ... + ``` +2. Add a comment referencing the tracking issue: + ```python + # FLAKY: https://github.com/closedloop-ai/claude-plugins/issues/NNN + @pytest.mark.quarantine + def test_something_flaky(): + ... + ``` +3. Open or link a follow-up issue to fix or delete the test. + +Because the main `Tests` job runs with `-m 'not quarantine'`, quarantined tests +are excluded from the blocking run. The non-blocking `Quarantined tests` job +still executes them so signal is retained in the PR check summary. + +## Un-quarantining + +Remove the `@pytest.mark.quarantine` decorator (and any `FLAKY:` comment) once +the test has been stabilised and passes reliably in the quarantine job for +several PRs. + +## Adding a contract test + +Mark any test intended as a repo-level contract with `@pytest.mark.contract`: + +```python +import pytest + +@pytest.mark.contract +def test_all_plugins_declare_a_manifest(): + ... +``` + +The `contract-tests` CI job will pick it up automatically on the next PR. + +## Policy + +- Quarantined tests are reviewed on a rolling basis -- long-standing quarantines + should be fixed or deleted, not left indefinitely. +- Deleting a flaky test without a quarantine period loses signal; prefer + quarantine first. +- Both markers are registered in `pyproject.toml`, so pytest will not emit + `PytestUnknownMarkWarning` for them. diff --git a/pyproject.toml b/pyproject.toml index 0553f287..54d836f2 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -7,3 +7,10 @@ extraPaths = [ "plugins/code/tools/python", "plugins/code/skills/plan-validate/scripts", ] + +[tool.pytest.ini_options] +testpaths = ["plugins"] +markers = [ + "contract: repo-level contract tests (FEA-107)", + "quarantine: quarantined flaky tests, non-blocking (FEA-107)", +] diff --git a/requirements-dev.txt b/requirements-dev.txt new file mode 100644 index 00000000..2e8da162 --- /dev/null +++ b/requirements-dev.txt @@ -0,0 +1,30 @@ +# Pinned dev/CI dependencies for claude-plugins. +# +# Installed in every CI job via `pip install -r requirements-dev.txt`, then +# validated with `pip check` to guard against broken dependency graphs. +# +# Bumped manually. To regenerate: +# +# python3.13 -m venv /tmp/claude-plugins-lockfile +# /tmp/claude-plugins-lockfile/bin/pip install --upgrade pip +# /tmp/claude-plugins-lockfile/bin/pip install ruff pyright pytest pyyaml anthropic mcp +# /tmp/claude-plugins-lockfile/bin/pip freeze +# +# Then update the pinned direct dependencies below. Transitive dependencies +# are resolved by pip at install time; `pip check` verifies the resolved graph +# is internally consistent. +# +# Direct dependencies used by CI jobs: +# - ruff: linting (lint job) +# - pyright: type checking (typecheck job) +# - pytest: test runner (test / contract-tests / quarantine jobs) +# - pyyaml: YAML parsing in plugin tools + tests +# - anthropic: Anthropic SDK (imported by plugin code under test) +# - mcp: MCP SDK (imported by plugin code under test) + +anthropic==0.92.0 +mcp==1.27.0 +pyright==1.1.408 +pytest==9.0.3 +PyYAML==6.0.3 +ruff==0.15.9 From c33b7e0f03b3d0b71e4ce22129ebbadb899e0633 Mon Sep 17 00:00:00 2001 From: Daniel Ochoa Date: Wed, 8 Apr 2026 17:00:04 -0500 Subject: [PATCH 2/3] FEA-107: Surface real failures from quarantine gate The quarantine job's run script ended with an unconditional exit 0, forcing the step to succeed even when pytest reported a real failure (rc != 0 and rc != 5). Combined with continue-on-error: true at the job level, real flaky-test breakages would have rendered as silent green checks instead of yellow non-blocking warnings, defeating the gate. The exit 5 (no tests collected) special case is preserved -- it still exits 0 vacuously until quarantined tests exist. All other exit codes now propagate so pytest failures are visible. - Move the unconditional exit 0 inside the rc==5 branch. - Add explicit `exit $rc` for the general case so pytest's exit code flows through. - continue-on-error: true at the job level keeps the gate non-blocking. Note: PIPESTATUS[0] is already used to capture pytest's exit code across the tee pipeline, so this fix only needed to stop masking that captured rc. Testing: - The rc==5 branch is unchanged and still vacuously passes (no quarantined tests yet). - Same fix applied in parallel to closedloop-electron and symphony-alpha. Risks: - None. continue-on-error keeps merges unblocked; the change only restores the failure signal that was being silently dropped. --- .github/workflows/ci.yml | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 9a67683e..227b4236 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -86,10 +86,17 @@ jobs: echo "### Quarantined tests" >> "$GITHUB_STEP_SUMMARY" pytest plugins/ -m quarantine 2>&1 | tee -a "$GITHUB_STEP_SUMMARY" rc=${PIPESTATUS[0]} + # pytest exit code 5 = no tests collected. Treat as a vacuous pass + # until quarantined tests exist. if [ $rc -eq 5 ]; then echo "No quarantined tests." | tee -a "$GITHUB_STEP_SUMMARY" + exit 0 fi - exit 0 + # Job has continue-on-error: true, so a real failure here surfaces + # as a yellow non-blocking warning rather than blocking the merge. + # Do NOT mask the exit code with `exit 0` -- that would render real + # quarantine breakages as silent green checks. + exit $rc pr-gates: name: PR gates From 3030535e3eb446b18eb037fb34d7b3c3c476f76d Mon Sep 17 00:00:00 2001 From: Daniel Ochoa Date: Thu, 9 Apr 2026 10:51:21 -0500 Subject: [PATCH 3/3] FEA-107: Remove contract-tests and quarantine gates Aligning with the decision in closedloop-electron PR #94 where reviewers rejected the contract-tests and quarantine mechanisms. Cross-repo standardization was the strongest remaining argument for keeping this scaffolding in claude-plugins; with the reference repo (closedloop- electron) reverting, that argument no longer holds. Reverts: - Remove contract-tests, quarantine, and pr-gates jobs from ci.yml. - Revert the Tests job's -m 'not quarantine' filter back to pytest plugins/. - Remove the contract and quarantine markers from pyproject.toml (the entire [tool.pytest.ini_options] block is removed since testpaths is redundant with the explicit pytest plugins/ in CI). - Remove docs/ci-gates.md and docs/flaky-test-quarantine.md. Kept from the original FEA-107 commit: - requirements-dev.txt pinned lockfile (real improvement, independent of the gate shape). - pip install -r requirements-dev.txt && pip check in each job (install consistency check, satisfies the original AC-004.1 motivation). - The concurrency block that cancels superseded PR runs. Risks: - None. All removed pieces had no current users -- no tests were marked @pytest.mark.contract or @pytest.mark.quarantine, and neither job context was in the branch-protection ruleset. --- .github/workflows/ci.yml | 70 +------------------------------ docs/ci-gates.md | 71 ------------------------------- docs/flaky-test-quarantine.md | 79 ----------------------------------- pyproject.toml | 7 ---- 4 files changed, 1 insertion(+), 226 deletions(-) delete mode 100644 docs/ci-gates.md delete mode 100644 docs/flaky-test-quarantine.md diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 227b4236..11dcd16a 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -44,72 +44,4 @@ jobs: python-version: "3.13" - run: pip install -r requirements-dev.txt - run: pip check - - run: pytest plugins/ -m 'not quarantine' - - contract-tests: - name: Contract tests - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - uses: actions/setup-python@v5 - with: - python-version: "3.13" - - run: pip install -r requirements-dev.txt - - run: pip check - - name: Run contract tests - shell: bash - run: | - set +e - pytest plugins/ -m contract - rc=$? - if [ $rc -eq 5 ]; then - echo "No contract tests collected -- gate passes vacuously." | tee -a "$GITHUB_STEP_SUMMARY" - exit 0 - fi - exit $rc - - quarantine: - name: Quarantined tests (non-blocking) - runs-on: ubuntu-latest - continue-on-error: true - steps: - - uses: actions/checkout@v4 - - uses: actions/setup-python@v5 - with: - python-version: "3.13" - - run: pip install -r requirements-dev.txt - - run: pip check - - name: Run quarantined tests - shell: bash - run: | - set +e - echo "### Quarantined tests" >> "$GITHUB_STEP_SUMMARY" - pytest plugins/ -m quarantine 2>&1 | tee -a "$GITHUB_STEP_SUMMARY" - rc=${PIPESTATUS[0]} - # pytest exit code 5 = no tests collected. Treat as a vacuous pass - # until quarantined tests exist. - if [ $rc -eq 5 ]; then - echo "No quarantined tests." | tee -a "$GITHUB_STEP_SUMMARY" - exit 0 - fi - # Job has continue-on-error: true, so a real failure here surfaces - # as a yellow non-blocking warning rather than blocking the merge. - # Do NOT mask the exit code with `exit 0` -- that would render real - # quarantine breakages as silent green checks. - exit $rc - - pr-gates: - name: PR gates - needs: [lint, typecheck, test, contract-tests] - if: always() - runs-on: ubuntu-latest - steps: - - name: Aggregate required gate results - run: | - for result in "${{ needs.lint.result }}" "${{ needs.typecheck.result }}" "${{ needs.test.result }}" "${{ needs.contract-tests.result }}"; do - if [ "$result" != "success" ]; then - echo "A required gate failed: $result" - exit 1 - fi - done - echo "All required gates passed." + - run: pytest plugins/ diff --git a/docs/ci-gates.md b/docs/ci-gates.md deleted file mode 100644 index 7c29d677..00000000 --- a/docs/ci-gates.md +++ /dev/null @@ -1,71 +0,0 @@ -# PR Validation Gates (FEA-107) - -This repo runs standardised PR validation gates on every pull request. The gates -live in `.github/workflows/ci.yml`. - -## Gate matrix - -| Job | Command | Blocking? | -| ---------------------------------- | ------------------------------------- | --------- | -| `Lint` | `ruff check .` | yes | -| `Type Check` | `pyright` | yes | -| `Tests` | `pytest plugins/ -m 'not quarantine'` | yes | -| `Contract tests` | `pytest plugins/ -m contract` | yes | -| `Quarantined tests (non-blocking)` | `pytest plugins/ -m quarantine` | **no** | -| `PR gates` | aggregates the four blocking jobs | yes | - -Every job installs dev dependencies via `pip install -r requirements-dev.txt` -followed by `pip check`, which validates the resolved dependency graph is -internally consistent. This satisfies FEA-107 AC-004.1 ("lockfile consistency -check; install must succeed without warnings or missing packages"). - -`requirements-dev.txt` is bumped manually. See the header comment in that -file for the regeneration procedure. - -The `Contract tests` and `Quarantined tests` jobs treat pytest exit code 5 -("no tests collected") as success so they pass vacuously until tests are -marked. - -See [`flaky-test-quarantine.md`](./flaky-test-quarantine.md) for the -`@pytest.mark.quarantine` and `@pytest.mark.contract` conventions. - -## Updating branch protection - -Ruleset `13555155` currently has **zero** required status checks. After a test -PR confirms the new gates are reporting, add this rule: - -```bash -# 1. Fetch the current ruleset JSON -gh api repos/closedloop-ai/claude-plugins/rulesets/13555155 > /tmp/plugins-ruleset.json - -# 2. Edit /tmp/plugins-ruleset.json -- add a new entry to the "rules" array: -# { -# "type": "required_status_checks", -# "parameters": { -# "strict_required_status_checks_policy": false, -# "do_not_enforce_on_create": false, -# "required_status_checks": [ -# { "context": "Lint", "integration_id": 15368 }, -# { "context": "Type Check", "integration_id": 15368 }, -# { "context": "Tests", "integration_id": 15368 }, -# { "context": "Contract tests", "integration_id": 15368 }, -# { "context": "PR gates", "integration_id": 15368 } -# ] -# } -# } - -# 3. Apply -gh api -X PUT repos/closedloop-ai/claude-plugins/rulesets/13555155 \ - --input /tmp/plugins-ruleset.json -``` - -Verify with: - -```bash -gh api repos/closedloop-ai/claude-plugins/rules/branches/main \ - | jq '.[] | select(.type=="required_status_checks").parameters.required_status_checks' -``` - -**Do not update the ruleset until the new check contexts have been observed on -at least one PR run.** Otherwise PRs will hang waiting for a context that does -not yet exist in the workflow history. diff --git a/docs/flaky-test-quarantine.md b/docs/flaky-test-quarantine.md deleted file mode 100644 index 86079514..00000000 --- a/docs/flaky-test-quarantine.md +++ /dev/null @@ -1,79 +0,0 @@ -# Flaky Test Quarantine & Contract Tests - -This repo uses pytest markers to categorize tests for CI gating. Two markers are -currently defined in `pyproject.toml` under `[tool.pytest.ini_options]`: - -- `@pytest.mark.contract` -- repo-level contract tests that assert invariants - across multiple plugins or modules. Run by the `contract-tests` CI job. -- `@pytest.mark.quarantine` -- known-flaky tests that should not block merges. - Run by the `quarantine` CI job with `continue-on-error: true`. - -## How CI uses the markers - -`.github/workflows/ci.yml` runs several gates, each installing from -`requirements-dev.txt` followed by `pip check`: - -| Job | Command | Blocking? | -| ---------------- | --------------------------------------- | --------- | -| `Lint` | `ruff check .` | yes | -| `Type Check` | `pyright` | yes | -| `Tests` | `pytest plugins/ -m 'not quarantine'` | yes | -| `Contract tests` | `pytest plugins/ -m contract` | yes | -| `Quarantined tests (non-blocking)` | `pytest plugins/ -m quarantine` | **no** | -| `PR gates` | aggregates the four blocking jobs | yes | - -The `Contract tests` and `Quarantined tests` jobs both treat pytest exit -code 5 ("no tests collected") as success so they pass vacuously until -tests are marked. - -## Quarantining a flaky test - -1. Add the marker to the top of the test function or class: - ```python - import pytest - - @pytest.mark.quarantine - def test_something_flaky(): - ... - ``` -2. Add a comment referencing the tracking issue: - ```python - # FLAKY: https://github.com/closedloop-ai/claude-plugins/issues/NNN - @pytest.mark.quarantine - def test_something_flaky(): - ... - ``` -3. Open or link a follow-up issue to fix or delete the test. - -Because the main `Tests` job runs with `-m 'not quarantine'`, quarantined tests -are excluded from the blocking run. The non-blocking `Quarantined tests` job -still executes them so signal is retained in the PR check summary. - -## Un-quarantining - -Remove the `@pytest.mark.quarantine` decorator (and any `FLAKY:` comment) once -the test has been stabilised and passes reliably in the quarantine job for -several PRs. - -## Adding a contract test - -Mark any test intended as a repo-level contract with `@pytest.mark.contract`: - -```python -import pytest - -@pytest.mark.contract -def test_all_plugins_declare_a_manifest(): - ... -``` - -The `contract-tests` CI job will pick it up automatically on the next PR. - -## Policy - -- Quarantined tests are reviewed on a rolling basis -- long-standing quarantines - should be fixed or deleted, not left indefinitely. -- Deleting a flaky test without a quarantine period loses signal; prefer - quarantine first. -- Both markers are registered in `pyproject.toml`, so pytest will not emit - `PytestUnknownMarkWarning` for them. diff --git a/pyproject.toml b/pyproject.toml index 54d836f2..0553f287 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -7,10 +7,3 @@ extraPaths = [ "plugins/code/tools/python", "plugins/code/skills/plan-validate/scripts", ] - -[tool.pytest.ini_options] -testpaths = ["plugins"] -markers = [ - "contract: repo-level contract tests (FEA-107)", - "quarantine: quarantined flaky tests, non-blocking (FEA-107)", -]