Skip to content

Commit 474a00a

Browse files
brabojclaude
andauthored
feat(tests): move stack, example and process checks into fork-owned modules (#1968)
* feat(tests): move stack, example and process checks into fork-owned modules Part of #1944: run_smoke.py and conformance.py take in fork_checks.py and fork_dispositions.py when present, so the split deletes two files instead of editing code both trees share. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(tests): SYS-18 accepts a tree with no exempt category Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: name the fork-owned test modules in CLAUDE.md and PLAYBOOK Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * style(tests): separate the SYS-18 floor comment from the line above Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 813cca0 commit 474a00a

7 files changed

Lines changed: 573 additions & 469 deletions

File tree

‎CLAUDE.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -356,6 +356,8 @@ The suite lives in `tests/`:
356356
| `tests/run_e2e.py` | E2E test runner (agent-based tests) |
357357
| `tests/providers.py` | Model backends the e2e runner selects between |
358358
| `tests/conformance.py` | Disposition per embedded check — run here, or skip with a reason |
359+
| `tests/fork_dispositions.py` | Dispositions for the blocks in fork-owned content: stacks, platforms, workflow, process sections |
360+
| `tests/fork_checks.py` | Smoke checks that read fork-owned content (examples, interview, e2e cases), loaded by `run_smoke.py` when present |
359361
| `tests/run_conformance.py` | Runs the templates' own embedded checks against this repository |
360362
| `tests/run_fork_sync.py` | Merges the change under test into a simulated fork and runs smoke on the result |
361363

@@ -459,7 +461,8 @@ Run `py tests/run_smoke.py` before every PR. It checks:
459461
still be red in CI: one such edit put a 127-character line into every
460462
generated chain, 36 findings, with smoke green over it
461463
- Adding a fenced `bash`/`python` block to a template MUST add its
462-
disposition in `tests/conformance.py` — run it here, or skip it with
464+
disposition in `tests/conformance.py`, or in `tests/fork_dispositions.py`
465+
for a block in fork-owned content — run it here, or skip it with
463466
the reason it does not apply. `py tests/run_conformance.py` fails on a
464467
block with no entry, and on a skip stating no reason. A block unrunnable
465468
only because it names a placeholder is not a skip: resolve the

‎docs/PLAYBOOK.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -434,7 +434,11 @@ checks that this repository OBEYS them — it extracts every fenced check in
434434
`templates/` and either runs it here or records why it does not apply.
435435
Adding a fenced check to a template without a disposition in
436436
`tests/conformance.py` fails the run, which is what stops a new check from
437-
arriving unexamined. Checks reaching GitHub need `gh` authenticated.
437+
arriving unexamined. A block in fork-owned content (stacks, platforms,
438+
workflow, the process sections) takes its entry in
439+
`tests/fork_dispositions.py` instead, and a smoke check reading such
440+
content goes in `tests/fork_checks.py`, so both runners stay the same file
441+
upstream and in every fork. Checks reaching GitHub need `gh` authenticated.
438442

439443
A check whose verdict is a judgement reports `REVIEW` rather than `PASS`,
440444
and the summary counts it as awaiting a reading. It does not fail the run

‎tests/conformance.py‎

Lines changed: 12 additions & 259 deletions
Original file line numberDiff line numberDiff line change
@@ -59,32 +59,6 @@
5959

6060
# -- core/docs.md ------------------------------------------------
6161

62-
{"file": "core/docs.md", "find": "community health files checked",
63-
"title": "Community health files resolve to one location each",
64-
"do": RUN, "expect": ["nonzero"]},
65-
66-
{"file": "core/docs.md",
67-
"find": "the naming convention drifted",
68-
"title": "ADR frontmatter schema", "do": RUN, "expect": ["nonzero"]},
69-
70-
{"file": "core/docs.md", "find": "bound by the layout, from",
71-
"title": "Decision records carry the layout's sections",
72-
"do": RUN, "expect": ["nonzero", "nonzero"]},
73-
74-
{"file": "core/docs.md", "find": "over a bound and not frozen",
75-
"title": "Decision-record prose stays within the declared bounds",
76-
"do": RUN,
77-
"expect": ["line", "line", "nonzero", "nonzero", "zero", "zero",
78-
"zero", "zero"]},
79-
80-
{"file": "core/docs.md", "find": "changelog entries measured",
81-
"title": "Changelog entries stay within the word bound",
82-
"do": RUN, "expect": ["nonzero", "nonzero", "any", "zero"]},
83-
84-
{"file": "core/docs.md", "find": "entries, fenced = [], False",
85-
"title": "Development journal entry form", "do": RUN,
86-
"expect": ["nonzero"]},
87-
8862
# The efficacy benchmark's arm files are a generator's verbatim output,
8963
# which its design forbids editing; `short`'s is held to its own budget
9064
# and the others to none. They are source material the benchmark
@@ -103,24 +77,6 @@
10377
'SOURCES = ("templates/", "generated/", "examples/", '
10478
'"tests/efficacy/arms/")'}},
10579

106-
{"file": "core/docs.md", "find": "(?:R|TD)[0-9]{2}",
107-
"title": "Risk and technical-debt identifiers stay in their owning doc",
108-
"do": RUN, "expect": ["nonzero", "nonzero", "nonzero"]},
109-
110-
{"file": "core/docs.md", "find": "git diff --word-diff=porcelain",
111-
"title": "A format-only ADR edit moved no claim", "do": SKIP,
112-
"reason": "Takes the format-only commit under review as its argument; "
113-
"`HEAD~1` is a placeholder. There is no standing corpus of "
114-
"format-only ADR edits to run it over."},
115-
116-
# -- core/examples.md ---------------------------------------------
117-
118-
{"file": "core/examples.md", "find": "example failed: $f",
119-
"title": "Every example runs against a consumer install", "do": SKIP,
120-
"reason": "Placeholder runner. This repository's `examples/` holds generated "
121-
"context files rather than executable programs, so there is nothing "
122-
"to execute."},
123-
12480
# -- core/git.md -------------------------------------------------
12581

12682
{"file": "core/git.md", "find": "<regenerate-command>",
@@ -141,81 +97,6 @@
14197
"reason": "PATHS is a placeholder naming the documents two open "
14298
"branches both edit; it is set per release, not per repo."},
14399

144-
{"file": "core/git.md", "find": "git diff --stat :3:<path>",
145-
"title": "A conflict resolution keeps both sides' content", "do": SKIP,
146-
"reason": "Runs only during an unresolved merge conflict, against the "
147-
"conflicting path."},
148-
149-
{"file": "core/git.md",
150-
"find": "settings leaving no safe deletion path",
151-
"title": "A merged head branch has a safe deletion path", "do": RUN,
152-
"expect": ["nonzero", "zero"]},
153-
154-
{"file": "core/git.md", "find": "MILESTONE",
155-
"title": "Every issue closed since the previous tag carries the "
156-
"milestone", "do": RUN, "expect": MANUAL,
157-
"reason": "Parameterised by MILESTONE, which is unset here, so the "
158-
"check reports that the release is not scoped to a "
159-
"milestone and does not apply."},
160-
161-
{"file": "core/git.md", "find": "<release-workflow>.yml",
162-
"title": "The release pipeline has executed before", "do": RUN,
163-
"expect": ["nonzero"],
164-
"substitute": {"<release-workflow>.yml": "release-gates.yml"}},
165-
166-
{"file": "core/git.md",
167-
"find": "closing issues the subject does not name",
168-
"title": "A single-commit branch names its issue in the subject",
169-
"do": RUN, "expect": ["nonzero", "nonzero", "nonzero", "zero"]},
170-
171-
{"file": "core/git.md",
172-
"find": "named by a merged commit and still open",
173-
"title": "No issue whose work already merged is still open",
174-
"do": RUN,
175-
"expect": ["line", "nonzero", "nonzero", "nonzero", "zero"]},
176-
177-
{"file": "core/git.md", "find": "ready but unmerged",
178-
"title": "Release ordering against other ready pull requests",
179-
"do": RUN, "expect": MANUAL,
180-
"reason": "Each pull request it lists as ready is a decision about "
181-
"which side of the tag the work lands on. The check states "
182-
"that is not a warning it can score."},
183-
184-
# DOCS is the project judgement the check names: the paths a consumer of
185-
# the release reads. Here that is the templates themselves, so an
186-
# identifier a template documents is documented in what ships. The
187-
# v2.90.0 cut named `uv.lock`, which only `stack/python-lib.md` carries,
188-
# and the default set read it as undocumented. `generated/` stays out as
189-
# a derivation of `templates/`; `tools/` and `tests/` stay out because a
190-
# set covering the whole tree passes whatever the code mentions, which
191-
# the rule calls the defect itself.
192-
{"file": "core/git.md", "find": "documentation files in the tree",
193-
"title": "The tree being tagged documents what the release claims",
194-
"do": RUN,
195-
"expect": ["nonzero", "nonzero", "line", "line", "any", "any", "zero"],
196-
"substitute": {'DOCS = ("README.md", "docs/")':
197-
'DOCS = ("README.md", "docs/", "templates/")'}},
198-
199-
{"file": "core/git.md",
200-
"find": "bullets ever under Unreleased in that range",
201-
"title": "A bullet deleted from Unreleased is reported",
202-
"do": RUN, "expect": ["nonzero", "nonzero", "zero"]},
203-
{"file": "core/git.md", "find": "entries in Unreleased",
204-
"title": "The Unreleased section accounts for what the release carries",
205-
"do": RUN, "expect": MANUAL,
206-
"reason": "Parameterised by RELEASE, which is unset here, so the check "
207-
"reports that no release is in preparation and does not "
208-
"apply. With it set, the two counts are deliberately not "
209-
"required to match and the operator reconciles them."},
210-
211-
{"file": "core/git.md", "find": "audit records found",
212-
"title": "The newest periodic-review record is inside the interval",
213-
"do": RUN, "expect": ["nonzero", "nonzero"]},
214-
215-
{"file": "core/git.md", "find": "<source-owner>/<source-repo>",
216-
"title": "A repository migration preserves both remotes", "do": SKIP,
217-
"reason": "Placeholder owner/repo pairs, run once during a migration."},
218-
219100
{"file": "core/git.md", "find": "BASE = \"origin/main\"",
220101
"title": "A diff touching an off-limits path says so", "do": RUN,
221102
"expect": MANUAL,
@@ -336,34 +217,6 @@
336217
"reason": "Compares two revisions of a module named as arguments; run "
337218
"during a module split, not against a tree."},
338219

339-
# -- language/python.md ------------------------------------------
340-
341-
{"file": "language/python.md", "find": "pkg/_accel.py",
342-
"title": "An optional accelerator is isolated to one module",
343-
"do": SKIP,
344-
"reason": "An illustration of the module layout, not a check."},
345-
346-
{"file": "language/python.md", "find": "_LOG.addHandler",
347-
"title": "A library core attaches a null handler",
348-
"do": SKIP,
349-
"reason": "An illustration of the module layout, not a check."},
350-
351-
{"file": "language/python.md", "find": "the library attaches no handler",
352-
"title": "A plain import leaves the library's logger silent",
353-
"do": SKIP,
354-
"reason": "Imports a package named `pkg`. This repository ships no "
355-
"importable package."},
356-
357-
{"file": "language/python.md",
358-
"find": "raises_nothing_outside_the_contract",
359-
"title": "The package raises nothing outside its error contract",
360-
"do": SKIP,
361-
"reason": "Imports a package named `pkg` and walks its raises. This "
362-
"repository ships no importable package; its own Python "
363-
"(tools/, tests/) raises SystemExit to carry a verdict and "
364-
"declares no error hierarchy, because nothing calls it as "
365-
"a library."},
366-
367220
# -- security/devsecops.md ---------------------------------------
368221

369222
{"file": "security/devsecops.md", "find": "sbom.json",
@@ -418,118 +271,18 @@
418271
"title": "Every stated check is in a form a tool can extract",
419272
"do": RUN, "expect": ["nonzero", "zero"]},
420273

421-
# -- workflow/scope.md -------------------------------------------
422-
423-
{"file": "workflow/scope.md", "find": "platform/[a-z-]+[.]md",
424-
"title": "The context file names exactly one platform", "do": SKIP,
425-
"reason": "Reads a generated project's `CLAUDE.md` for its platform "
426-
"axis. This repository's `CLAUDE.md` is authored, not "
427-
"generated, and declares no platform template."},
428-
429-
# -- workflow/two-pass-review.md ---------------------------------
430-
431-
{"file": "workflow/two-pass-review.md", "find": "usage: scan <artifact>",
432-
"title": "First-pass mechanical scan of an artifact under review",
433-
"do": SKIP,
434-
"reason": "Takes the artifact to scan as an argument. It is a review "
435-
"aid run per artifact, not a property of the tree."},
436-
437-
# -- platform/github.md -----------------------------------------------
438-
439-
{"file": "platform/github.md", "find": "gh run list --commit",
440-
"title": "A commit's checks have started, not merely not failed",
441-
"do": RUN, "expect": ["line", "nonzero", READ],
442-
"reading": "The rows under the count, one per run that did not "
443-
"succeed. Whether a run still going is acceptable depends "
444-
"on what is being decided, so the rows go to whoever the "
445-
"count summons. The count itself is decided: its pass "
446-
"condition says a zero must be resolved either way, since "
447-
"the check cannot tell a malformed query from a workflow "
448-
"that never fired."},
449-
450-
{"file": "platform/github.md", "find": "code-scanning/analyses",
451-
"title": "Code scanning is entitled before the workflow is committed",
452-
"do": SKIP,
453-
"reason": "Placeholder owner/repo, and it answers a question asked "
454-
"once when adding the workflow."},
455-
456-
{"file": "platform/github.md", "find": "state,headRefOid",
457-
"title": "A branch is deleted only against the merged pull request",
458-
"do": SKIP, "reason": "Takes a pull request number as a placeholder."},
459-
460-
{"file": "platform/github.md", "find": "refs/pull/<N>/head",
461-
"title": "A rewritten head is inspected rather than assumed",
462-
"do": SKIP, "reason": "Takes a pull request number as a placeholder."},
463-
464-
{"file": "platform/github.md", "find": "TYPES = re.compile",
465-
"title": "Every open issue carries one type and one priority label",
466-
"do": RUN, "expect": ["nonzero"]},
467-
468-
# -- stack/c-embedded.md ----------------------------------------------
469-
470-
{"file": "stack/c-embedded.md", "find": "Configure (cross-compile",
471-
"title": "Build commands for the embedded stack", "do": SKIP,
472-
"reason": "A stack's Commands section, not a check."},
473-
474-
# -- stack/go-lib.md ---------------------------------------------------
475-
476-
{"file": "stack/go-lib.md", "find": "for program in examples",
477-
"title": "Every example program builds and runs", "do": SKIP,
478-
"reason": "Runs `go vet`, `go test` and each program under "
479-
"`examples/`. This repository ships no Go module."},
480-
481-
# -- stack/nodejs-lib.md -----------------------------------------------
482-
483-
{"file": "stack/nodejs-lib.md", "find": "npm --prefix",
484-
"title": "Every example runs against the packed tarball", "do": SKIP,
485-
"reason": "Packs an npm distribution and installs it into a clean "
486-
"directory. This repository publishes no npm package."},
487-
488-
# -- stack/python-lib.md ----------------------------------------------
489-
490-
{"file": "stack/python-lib.md",
491-
"find": "git grep -n \"sys.path\" -- 'tests/*'",
492-
"title": "The suite imports the installation, not the working tree",
493-
"do": SKIP,
494-
"reason": "Scoped to a packaged library adopting `src/`. This "
495-
"repository ships no importable package, so its `tests/` "
496-
"has no installation to import instead."},
497-
498-
{"file": "stack/python-lib.md", "find": "assert 'pkg.heavy' not in",
499-
"title": "A plain import loads neither the submodule nor its dependency",
500-
"do": SKIP,
501-
"reason": "`pkg` is a placeholder for the library under test. This "
502-
"repository ships no importable package."},
503-
504-
{"file": "stack/python-lib.md", "find": "tar -tzf dist/",
505-
"title": "The sdist carries no vendored or submodule path", "do": SKIP,
506-
"reason": "Builds a source distribution and lists it. This repository "
507-
"builds no distribution."},
508-
509-
510-
{"file": "stack/python-lib.md", "find": "pathlib.Path(\"dist\").glob",
511-
"title": "The built wheel contains what it should", "do": SKIP,
512-
"reason": "Inspects a wheel under `dist/`. This repository builds no "
513-
"distribution."},
274+
]
514275

515-
{"file": "stack/python-lib.md", "find": "from importlib import import_module",
516-
"title": "A deferred import binds its name on first access", "do": SKIP,
517-
"reason": "This repository ships no importable package. The block is "
518-
"the module `__getattr__` pattern itself; the check that "
519-
"every advertised name resolves is `core/testing.md`'s "
520-
"and carries its own entry."},
521276

522-
{"file": "stack/python-lib.md", "find": "_RENAMED = {\"OldName\"",
523-
"title": "A renamed public symbol resolves through the deferred map",
524-
"do": SKIP,
525-
"reason": "This repository ships no importable package."},
277+
# A fork's dispositions cover the blocks only a fork's templates carry --
278+
# its stacks, platforms, workflow and process sections. They live in
279+
# fork_dispositions.py, which this registry takes in when the file exists,
280+
# so a tree without those templates carries no entry for a block it lacks.
281+
try:
282+
from fork_dispositions import CHECKS as FORK_CHECKS
283+
except ModuleNotFoundError as exc:
284+
if exc.name != "fork_dispositions":
285+
raise
286+
FORK_CHECKS = []
526287

527-
{"file": "stack/python-lib.md", "find": "READER = {",
528-
"title": "Every committed lock has a Dependabot ecosystem that reads it",
529-
"do": SKIP,
530-
"reason": "Pairs a committed Python lock with its Dependabot reader. "
531-
"This repository ships Markdown, commits no Python lock, and "
532-
"enrols no Python ecosystem, so the check would fail on the "
533-
"zero-locks branch that is a finding for a library and a "
534-
"fact here."},
535-
]
288+
CHECKS = CHECKS + FORK_CHECKS

0 commit comments

Comments
 (0)