Run the gates that were being run by hand - #65
Merged
Conversation
The skills in this repository are markdown. Nothing compiles, no test suite runs, and a defect therefore stays invisible until someone reads the exact line. Every check was run manually, which means it was run when someone remembered: a skill shipped with frontmatter that did not parse and had been loading with empty metadata, unable to auto-trigger; documented counts drifted nine behind the tree across four files; a credential-shaped file sat untracked and unignored while commits were being staged with `git add -A`. Three jobs now run on every pull request. scripts/validate-framework.py holds seven gates. `claude plugin validate` runs without credentials, reading the plugin off disk. install.sh gets a dry run across every editor, since it stays supported through the plugin transition. Each gate covers something that has actually broken here: frontmatter that does not parse; malformed plugin JSON, or a `version` key reappearing where it is omitted so the commit SHA is the version; a documented count that no longer matches the tree; a relative link pointing at nothing; a credential-shaped file being tracked; a reference to a removed mechanism; and a backend product named inside a skill rather than in its adapter reference. Every gate was verified twice: it passes on the clean tree, and it fails when its specific defect is injected. A gate that cannot fail is decoration, and this repository has spent enough time on checks that only appeared to run. Two link failures the gate found on the first run were in fenced template blocks — a README the technical writer emits into a user's project, where a link is illustrative rather than a reference. The gate skips fenced content rather than the files being changed to satisfy it.
There was a problem hiding this comment.
Pull request overview
Adds automated CI “gates” to catch recurring repository defects early (frontmatter parsing, manifest schema, doc count drift, broken links, tracked credential-shaped files, references to retired mechanisms, and vendor-neutrality rules), replacing prior manual validation.
Changes:
- Added
scripts/validate-framework.pyimplementing seven framework validation gates. - Added GitHub Actions workflow to run framework gates,
claude plugin validate, and aninstall.sh --dry-runon PRs andmain. - Updated
CONTRIBUTING.mdandCHANGELOG.mdto document the new validation process and rationale.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| scripts/validate-framework.py | New Python validator implementing 7 repository “gates” (frontmatter, manifests, counts, links, credentials, retired refs, vendor-neutrality). |
| .github/workflows/validate.yml | New CI workflow running the three validation jobs on PRs and pushes to main. |
| CONTRIBUTING.md | Documents how to run the gates locally and what each gate checks. |
| CHANGELOG.md | Adds Unreleased notes describing the new CI automation and the seven-gate validator. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…o, pin the validator Review findings on this branch, all three confirmed and all three in the gates themselves. G2 assumed a parsed manifest was an object. Valid JSON that is not — a list, a string — reached `.get()` and raised, which aborted the run before the remaining gates gets to execute. A malformed manifest would therefore have taken out every later check instead of reporting itself. It now reports the type it found and carries on. G4 resolved a link target but never confirmed the result was still inside the repository. Enough `../` segments, or a leading `/`, escape ROOT and the check then passes against whatever unrelated file happens to exist on the runner. That is a false negative in a gate, which is worse than having no gate: it reports safety it did not establish. Escaping links are now a failure in their own right, verified against both the relative and absolute forms. The workflow installed the validator unpinned, so a green pull request could turn red tomorrow with no change here, and a behaviour change in the checker would arrive silently. Pinned to the version this branch was verified against; bumping it is now a deliberate edit. Both script fixes were verified by injection: a list-valued manifest produces a G2 failure rather than a traceback, and both escaping link forms produce G4 failures. The clean tree still passes all seven.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The skills in this repository are markdown. Nothing compiles, no test suite runs, and a defect therefore stays invisible until someone reads the exact line. Every check was run manually, which means it was run when someone remembered.
Three things that reached
mainas a result:SKILL.mdwhose frontmatter did not parse. It had been loading with empty metadata — no description, so it could never auto-trigger. It looked correct in the file and passed review.git add -A.None of those are hard to detect. They survived because detecting them depended on somebody choosing to look.
What runs now
Three jobs, on every pull request and on pushes to
main:python3 scripts/validate-framework.pyclaude plugin validate ./claude— needs no credentials; it reads the plugin off disk./install.sh --dry-run --editors=all --scope=project --yesinstall.shis still the path for Cursor, Kiro and VS Code, so it has to keep working through the plugin transition.The seven gates
Each one covers something that has actually broken in this repository.
versionkey reappearing inplugin.json, which is omitted on purpose so the commit SHA is the version.mdlink pointing at nothingTests
Every gate was verified twice: it passes on the clean tree, and it fails when its specific defect is injected — broken YAML, an added
versionkey, a wrong count, a dangling link, a tracked fake password file, a reference to the removed learnings store, and a product name inside a skill. All seven fired, and the tree was restored after each.A gate that cannot fail is decoration, and this repository has spent enough time on checks that only appeared to run.
All three CI commands were also run locally exactly as the workflow invokes them; each exits 0.
One gate was wrong on its first run
G4 flagged two links in
technical-writer/SKILL.md. Both were inside fenced blocks — a template README the writer emits into a user's project, where a link is illustrative rather than a reference to anything here.The gate skips fenced content now. The files were not changed to satisfy it, and
CONTRIBUTING.mdsays the same thing for future cases: if a gate is wrong, change the gate deliberately and say why — do not work around it.