Skip to content

Commit 07168f3

Browse files
brabojclaude
andauthored
feat(tests): merge each change into a simulated fork in CI (#1924) (#1962)
* feat(tests): add the fork-sync runner (#1924) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(tests): spec, index and CI step for the fork-sync runner (#1924) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 9fea599 commit 07168f3

7 files changed

Lines changed: 472 additions & 3 deletions

File tree

‎.github/workflows/smoke.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,10 @@ jobs:
3434
# mutation that does nothing reads as the judge holding.
3535
- name: Check the rubric control still builds its trees
3636
run: python tests/efficacy/control/control.py --self-test
37+
# A fork syncs by merging upstream. This merges the change under test
38+
# into a simulated fork and runs smoke on the result.
39+
- name: Check a fork that only appends merges this change
40+
run: python tests/run_fork_sync.py
3741
- name: Check generated docs are up to date
3842
run: python tools/sync.py --check
3943
- name: Check generated/ files are up to date

‎CLAUDE.md‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -357,6 +357,7 @@ The suite lives in `tests/`:
357357
| `tests/providers.py` | Model backends the e2e runner selects between |
358358
| `tests/conformance.py` | Disposition per embedded check — run here, or skip with a reason |
359359
| `tests/run_conformance.py` | Runs the templates' own embedded checks against this repository |
360+
| `tests/run_fork_sync.py` | Merges the change under test into a simulated fork and runs smoke on the result |
360361

361362
```bash
362363
py tests/run_smoke.py # structural checks
@@ -365,6 +366,8 @@ py tests/run_smoke.py SYS-01 # run one check by ID
365366
py tests/run_conformance.py # the templates' checks, against this repo
366367
py tests/run_conformance.py --list # dispositions only, run nothing
367368

369+
py tests/run_fork_sync.py # a fork that only appends merges HEAD
370+
368371
py tests/run_e2e.py # canary only (fixture stack, needs API key)
369372
py tests/run_e2e.py --all # all tests (live, needs API key)
370373
py tests/run_e2e.py --area=STK # run all stack tests
@@ -378,8 +381,9 @@ py tests/run_e2e.py --dry-run # build prompts, call no model, write no repo
378381
model and writes none
379382
- Spec files live in `tests/specs/` — see `tests/CODIFICATION.md`
380383
for the ID scheme and `tests/INDEX.md` for the full list
381-
- CI runs smoke, conformance, `sync.py --check`, `resolve.py --check`,
382-
the redundancy audit and gitleaks on PRs and on push to main
384+
- CI runs smoke, conformance, fork sync, `sync.py --check`,
385+
`resolve.py --check`, the redundancy audit and gitleaks on PRs and on
386+
push to main
383387
- Live e2e calls a model and costs money, so it runs on a cadence
384388
rather than per change: the STK-24 canary at every release cut,
385389
recorded in the cut PR, and the full suite at each periodic review

‎docs/PLAYBOOK.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -362,13 +362,14 @@ any entry that no longer applies.
362362
py tests/run_smoke.py # structural checks — seconds
363363
py tests/run_conformance.py # the templates' own checks, run here
364364
py tests/run_conformance.py --list # dispositions only, run nothing
365+
py tests/run_fork_sync.py # a fork that only appends merges HEAD
365366
py tests/run_e2e.py # canary test (fixture stack)
366367
py tests/run_e2e.py --all # all agent tests
367368
py tests/run_e2e.py STK-01 FMT-01 # specific tests only
368369
py tests/run_e2e.py --dry-run # build prompts, call no model, write no report
369370
```
370371

371-
The efficacy benchmark is a fourth runner and deliberately not in that
372+
The efficacy benchmark is another runner and deliberately not in that
372373
list: it costs quota, takes hours, and answers a different question from
373374
whether this repository is sound. `tests/efficacy/README.md` owns its
374375
procedure, and `docs/design/efficacy-benchmark.md` owns the method.

‎tests/CODIFICATION.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,7 @@ reused for a different component within the same area.
9797
| `03` | Core tier — resolved chains include core tier files |
9898
| `04` | Prompt assembly — prompt builds for all stacks |
9999
| `05` | Parser parity — resolve.py resolution matches the PyYAML resolution |
100+
| `06` | Fork sync — a simulated fork merges an upstream change cleanly |
100101

101102
### STK - Stack
102103

‎tests/INDEX.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,7 @@ restated in each.
5959
| `SAIT-INT-MNF-03-001A` | INT | P0 | All resolved chains include core tier files |
6060
| `SAIT-INT-MNF-04-001A` | INT | P1 | Prompt builds for all stacks |
6161
| `SAIT-INT-MNF-05-001A` | INT | P0 | resolve.py resolution matches the PyYAML resolution |
62+
| `SAIT-INT-MNF-06-001A` | INT | P1 | A fork that only appends merges an upstream change cleanly |
6263
| `SAIT-INT-ITV-01-001A` | INT | P0 | All REQUIRED interview questions are asked before output is generated |
6364
| `SAIT-INT-ITV-02-001A` | INT | P1 | DEFAULTED sections are pre-filled from the selected stack template |
6465
| `SAIT-INT-ITV-03-001A` | INT | P0 | Interview answers override stack and base template rules in the output |

0 commit comments

Comments
 (0)