FEA-107: Add PR validation gates, lockfile, and flaky test quarantine

.github/workflows/ci.yml

@@ -5,6 +5,10 @@ on:
     branches: [main]
   pull_request:
 
+concurrency:
+  group: ci-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
+
 jobs:
   lint:
     name: Lint
@@ -14,7 +18,8 @@ jobs:
       - uses: actions/setup-python@v5
         with:
           python-version: "3.13"
-      - run: pip install ruff
+      - run: pip install -r requirements-dev.txt
+      - run: pip check
       - run: ruff check .
 
   typecheck:
@@ -25,7 +30,8 @@ jobs:
       - uses: actions/setup-python@v5
         with:
           python-version: "3.13"
-      - run: pip install pyright pyyaml anthropic mcp pytest
+      - run: pip install -r requirements-dev.txt
+      - run: pip check
       - run: pyright
 
   test:
@@ -36,5 +42,74 @@ jobs:
       - uses: actions/setup-python@v5
         with:
           python-version: "3.13"
-      - run: pip install pytest pyyaml anthropic
-      - run: pytest plugins/
+      - run: pip install -r requirements-dev.txt
+      - run: pip check
+      - run: pytest plugins/ -m 'not quarantine'
+
+  contract-tests:
+    name: Contract tests
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+      - run: pip install -r requirements-dev.txt
+      - run: pip check
+      - name: Run contract tests
+        shell: bash
+        run: |
+          set +e
+          pytest plugins/ -m contract
+          rc=$?
+          if [ $rc -eq 5 ]; then
+            echo "No contract tests collected -- gate passes vacuously." | tee -a "$GITHUB_STEP_SUMMARY"
+            exit 0
+          fi
+          exit $rc
+
+  quarantine:
+    name: Quarantined tests (non-blocking)
+    runs-on: ubuntu-latest
+    continue-on-error: true
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+      - run: pip install -r requirements-dev.txt
+      - run: pip check
+      - name: Run quarantined tests
+        shell: bash
+        run: |
+          set +e
+          echo "### Quarantined tests" >> "$GITHUB_STEP_SUMMARY"
+          pytest plugins/ -m quarantine 2>&1 | tee -a "$GITHUB_STEP_SUMMARY"
+          rc=${PIPESTATUS[0]}
+          # pytest exit code 5 = no tests collected. Treat as a vacuous pass
+          # until quarantined tests exist.
+          if [ $rc -eq 5 ]; then
+            echo "No quarantined tests." | tee -a "$GITHUB_STEP_SUMMARY"
+            exit 0
+          fi
+          # Job has continue-on-error: true, so a real failure here surfaces
+          # as a yellow non-blocking warning rather than blocking the merge.
+          # Do NOT mask the exit code with `exit 0` -- that would render real
+          # quarantine breakages as silent green checks.
+          exit $rc
+
+  pr-gates:
+    name: PR gates
+    needs: [lint, typecheck, test, contract-tests]
+    if: always()
+    runs-on: ubuntu-latest
+    steps:
+      - name: Aggregate required gate results
+        run: |
+          for result in "${{ needs.lint.result }}" "${{ needs.typecheck.result }}" "${{ needs.test.result }}" "${{ needs.contract-tests.result }}"; do
+            if [ "$result" != "success" ]; then
+              echo "A required gate failed: $result"
+              exit 1
+            fi
+          done
+          echo "All required gates passed."

docs/ci-gates.md

@@ -0,0 +1,71 @@
+# PR Validation Gates (FEA-107)
+
+This repo runs standardised PR validation gates on every pull request. The gates
+live in `.github/workflows/ci.yml`.
+
+## Gate matrix
+
+| Job                                | Command                               | Blocking? |
+| ---------------------------------- | ------------------------------------- | --------- |
+| `Lint`                             | `ruff check .`                        | yes       |
+| `Type Check`                       | `pyright`                             | yes       |
+| `Tests`                            | `pytest plugins/ -m 'not quarantine'` | yes       |
+| `Contract tests`                   | `pytest plugins/ -m contract`         | yes       |
+| `Quarantined tests (non-blocking)` | `pytest plugins/ -m quarantine`       | **no**    |
+| `PR gates`                         | aggregates the four blocking jobs     | yes       |
+
+Every job installs dev dependencies via `pip install -r requirements-dev.txt`
+followed by `pip check`, which validates the resolved dependency graph is
+internally consistent. This satisfies FEA-107 AC-004.1 ("lockfile consistency
+check; install must succeed without warnings or missing packages").
+
+`requirements-dev.txt` is bumped manually. See the header comment in that
+file for the regeneration procedure.
+
+The `Contract tests` and `Quarantined tests` jobs treat pytest exit code 5
+("no tests collected") as success so they pass vacuously until tests are
+marked.
+
+See [`flaky-test-quarantine.md`](./flaky-test-quarantine.md) for the
+`@pytest.mark.quarantine` and `@pytest.mark.contract` conventions.
+
+## Updating branch protection
+
+Ruleset `13555155` currently has **zero** required status checks. After a test
+PR confirms the new gates are reporting, add this rule:
+
+```bash
+# 1. Fetch the current ruleset JSON
+gh api repos/closedloop-ai/claude-plugins/rulesets/13555155 > /tmp/plugins-ruleset.json
+
+# 2. Edit /tmp/plugins-ruleset.json -- add a new entry to the "rules" array:
+#    {
+#      "type": "required_status_checks",
+#      "parameters": {
+#        "strict_required_status_checks_policy": false,
+#        "do_not_enforce_on_create": false,
+#        "required_status_checks": [
+#          { "context": "Lint",           "integration_id": 15368 },
+#          { "context": "Type Check",     "integration_id": 15368 },
+#          { "context": "Tests",          "integration_id": 15368 },
+#          { "context": "Contract tests", "integration_id": 15368 },
+#          { "context": "PR gates",       "integration_id": 15368 }
+#        ]
+#      }
+#    }
+
+# 3. Apply
+gh api -X PUT repos/closedloop-ai/claude-plugins/rulesets/13555155 \
+  --input /tmp/plugins-ruleset.json
+```
+
+Verify with:
+
+```bash
+gh api repos/closedloop-ai/claude-plugins/rules/branches/main \
+  | jq '.[] | select(.type=="required_status_checks").parameters.required_status_checks'
+```
+
+**Do not update the ruleset until the new check contexts have been observed on
+at least one PR run.** Otherwise PRs will hang waiting for a context that does
+not yet exist in the workflow history.

docs/flaky-test-quarantine.md

@@ -0,0 +1,79 @@
+# Flaky Test Quarantine & Contract Tests
+
+This repo uses pytest markers to categorize tests for CI gating. Two markers are
+currently defined in `pyproject.toml` under `[tool.pytest.ini_options]`:
+
+- `@pytest.mark.contract` -- repo-level contract tests that assert invariants
+  across multiple plugins or modules. Run by the `contract-tests` CI job.
+- `@pytest.mark.quarantine` -- known-flaky tests that should not block merges.
+  Run by the `quarantine` CI job with `continue-on-error: true`.
+
+## How CI uses the markers
+
+`.github/workflows/ci.yml` runs several gates, each installing from
+`requirements-dev.txt` followed by `pip check`:
+
+| Job              | Command                                 | Blocking? |
+| ---------------- | --------------------------------------- | --------- |
+| `Lint`           | `ruff check .`                          | yes       |
+| `Type Check`     | `pyright`                               | yes       |
+| `Tests`          | `pytest plugins/ -m 'not quarantine'`   | yes       |
+| `Contract tests` | `pytest plugins/ -m contract`           | yes       |
+| `Quarantined tests (non-blocking)` | `pytest plugins/ -m quarantine` | **no**  |
+| `PR gates`       | aggregates the four blocking jobs       | yes       |
+
+The `Contract tests` and `Quarantined tests` jobs both treat pytest exit
+code 5 ("no tests collected") as success so they pass vacuously until
+tests are marked.
+
+## Quarantining a flaky test
+
+1. Add the marker to the top of the test function or class:
+   ```python
+   import pytest
+
+   @pytest.mark.quarantine
+   def test_something_flaky():
+       ...
+   ```
+2. Add a comment referencing the tracking issue:
+   ```python
+   # FLAKY: https://github.com/closedloop-ai/claude-plugins/issues/NNN
+   @pytest.mark.quarantine
+   def test_something_flaky():
+       ...
+   ```
+3. Open or link a follow-up issue to fix or delete the test.
+
+Because the main `Tests` job runs with `-m 'not quarantine'`, quarantined tests
+are excluded from the blocking run. The non-blocking `Quarantined tests` job
+still executes them so signal is retained in the PR check summary.
+
+## Un-quarantining
+
+Remove the `@pytest.mark.quarantine` decorator (and any `FLAKY:` comment) once
+the test has been stabilised and passes reliably in the quarantine job for
+several PRs.
+
+## Adding a contract test
+
+Mark any test intended as a repo-level contract with `@pytest.mark.contract`:
+
+```python
+import pytest
+
+@pytest.mark.contract
+def test_all_plugins_declare_a_manifest():
+    ...
+```
+
+The `contract-tests` CI job will pick it up automatically on the next PR.
+
+## Policy
+
+- Quarantined tests are reviewed on a rolling basis -- long-standing quarantines
+  should be fixed or deleted, not left indefinitely.
+- Deleting a flaky test without a quarantine period loses signal; prefer
+  quarantine first.
+- Both markers are registered in `pyproject.toml`, so pytest will not emit
+  `PytestUnknownMarkWarning` for them.

pyproject.toml

@@ -7,3 +7,10 @@ extraPaths = [
   "plugins/code/tools/python",
   "plugins/code/skills/plan-validate/scripts",
 ]
+
+[tool.pytest.ini_options]
+testpaths = ["plugins"]
+markers = [
+    "contract: repo-level contract tests (FEA-107)",
+    "quarantine: quarantined flaky tests, non-blocking (FEA-107)",
+]

requirements-dev.txt

@@ -0,0 +1,30 @@
+# Pinned dev/CI dependencies for claude-plugins.
+#
+# Installed in every CI job via `pip install -r requirements-dev.txt`, then
+# validated with `pip check` to guard against broken dependency graphs.
+#
+# Bumped manually. To regenerate:
+#
+#   python3.13 -m venv /tmp/claude-plugins-lockfile
+#   /tmp/claude-plugins-lockfile/bin/pip install --upgrade pip
+#   /tmp/claude-plugins-lockfile/bin/pip install ruff pyright pytest pyyaml anthropic mcp
+#   /tmp/claude-plugins-lockfile/bin/pip freeze
+#
+# Then update the pinned direct dependencies below. Transitive dependencies
+# are resolved by pip at install time; `pip check` verifies the resolved graph
+# is internally consistent.
+#
+# Direct dependencies used by CI jobs:
+#   - ruff:     linting (lint job)
+#   - pyright:  type checking (typecheck job)
+#   - pytest:   test runner (test / contract-tests / quarantine jobs)
+#   - pyyaml:   YAML parsing in plugin tools + tests
+#   - anthropic: Anthropic SDK (imported by plugin code under test)
+#   - mcp:      MCP SDK (imported by plugin code under test)
+
+anthropic==0.92.0
+mcp==1.27.0
+pyright==1.1.408
+pytest==9.0.3
+PyYAML==6.0.3
+ruff==0.15.9