Skip to content

Commit 52174d0

Browse files
brabojclaude
andauthored
feat(tests): fail smoke when a fork edits an upstream manifest key (#1923) (#1960)
SYS-20 compares core, base, backend and frontend with the revision a fork last merged, the merge base of HEAD and SAIT_UPSTREAM_REF, and names each entry added, removed or changed. Upstream sets no ref and compares the manifest with itself; planted pairs exercise the failing path every run. Per-directory manifest fragments are not added: the check enforces the rule, and fragments would change the schema 3.x freezes. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent fcb8585 commit 52174d0

6 files changed

Lines changed: 252 additions & 0 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,9 @@ alters no template carries no entry.
3030
- `tools/resolve.py` exits 1 naming a `depends_on` ID with no manifest
3131
entry and the entry naming it, or a dependency cycle as a path, where it
3232
used to resolve a shorter chain or crash.
33+
- Smoke check SYS-20 fails when a fork edits an upstream manifest key
34+
(`core`, `base`, `backend`, `frontend`), naming the key and entry,
35+
against the upstream revision it last merged, set by `SAIT_UPSTREAM_REF`.
3336

3437
### Breaking changes
3538

‎CLAUDE.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -427,6 +427,9 @@ Run `py tests/run_smoke.py` before every PR. It checks:
427427
exemption's recorded reason still holds (ADR-044)
428428
- SYS-19: `resolve.py` names a missing dependency and a dependency cycle,
429429
each with its own message and exit 1
430+
- SYS-20: a fork leaves the manifest's upstream keys (`core`, `base`,
431+
`backend`, `frontend`) as it last merged them, against the ref in
432+
`SAIT_UPSTREAM_REF`; upstream sets none and passes against itself
430433
- Prompt builds for all stacks
431434

432435
## 6. Session protocol

‎docs/PLAYBOOK.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -392,6 +392,18 @@ See `tests/CODIFICATION.md` for the ID scheme and `tests/INDEX.md` for the
392392
full list of specs. Requires `py -m pip install pyyaml` for the manifest
393393
check.
394394

395+
In a fork, set `SAIT_UPSTREAM_REF` to a ref holding upstream's history
396+
before running smoke, so SYS-20 compares the manifest's upstream keys with
397+
the revision the fork last merged:
398+
399+
```bash
400+
git fetch upstream
401+
SAIT_UPSTREAM_REF=upstream/main py tests/run_smoke.py SYS-20
402+
```
403+
404+
Unset, SYS-20 treats the checkout as upstream and compares the manifest
405+
with itself.
406+
395407
`run_e2e.py` is the one runner that calls a model and costs money, so it
396408
runs on a cadence rather than on every change: the STK-24 canary runs live
397409
at every release cut, recorded in the cut's pull request, and the full

‎tests/INDEX.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@ restated in each.
4141
| `SAIT-SMK-SYS-17-001A` | SMK | P2 | Every registered check names a spec document that exists |
4242
| `SAIT-SMK-SYS-18-001A` | SMK | P1 | A stack outside the constrained categories resolves the gate tier |
4343
| `SAIT-SMK-SYS-19-001A` | SMK | P1 | resolve.py names a missing dependency and a dependency cycle |
44+
| `SAIT-SMK-SYS-20-001A` | SMK | P1 | A fork leaves upstream's manifest keys as it last merged them |
4445
| `SAIT-SMK-TPL-04-001A` | SMK | P1 | All EXTEND and OVERRIDE directives reference existing IDs |
4546
| `SAIT-INT-TPL-01-001A` | INT | P0 | DEPENDS ON chain assembles a complete rule set |
4647
| `SAIT-INT-TPL-02-001A` | INT | P1 | EXTEND adds rules without removing base rules |

‎tests/run_smoke.py‎

Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2462,6 +2462,151 @@ def check_sys_19():
24622462
return seen.failures() + failures, seen.notes()
24632463

24642464

2465+
# ---------------------------------------------------------------------------
2466+
# SYS-20 -- a fork leaves upstream's manifest keys as it last merged them
2467+
# ---------------------------------------------------------------------------
2468+
2469+
# A fork appends under `stacks:`, `platform:` and keys of its own, and never
2470+
# edits the keys upstream owns. An edit there is what turns the fork's next
2471+
# upstream merge into a conflict, and nothing else notices it until then.
2472+
#
2473+
# In a fork, SAIT_UPSTREAM_REF names a ref holding upstream's history, and
2474+
# the baseline is the revision the fork last merged: the merge base of HEAD
2475+
# and that ref. Upstream sets no ref, so the manifest is compared with
2476+
# itself and the check passes trivially. Because that run reads no edit,
2477+
# planted manifest pairs exercise the failing path on every run.
2478+
2479+
UPSTREAM_KEYS = ("core", "base", "backend", "frontend")
2480+
UPSTREAM_REF_ENV = "SAIT_UPSTREAM_REF"
2481+
2482+
2483+
def upstream_key_edits(was, now):
2484+
"""Each edit to an upstream-owned manifest key, as one finding."""
2485+
found = []
2486+
for key in UPSTREAM_KEYS:
2487+
before, after = was.get(key) or [], now.get(key) or []
2488+
if key == "core":
2489+
if before != after:
2490+
found.append(f"core: {before} became {after}")
2491+
continue
2492+
old = {e["id"]: e for e in before}
2493+
new = {e["id"]: e for e in after}
2494+
for eid in sorted(set(old) - set(new)):
2495+
found.append(f"{key}: {eid} removed")
2496+
for eid in sorted(set(new) - set(old)):
2497+
found.append(f"{key}: {eid} added")
2498+
for eid in sorted(set(old) & set(new)):
2499+
fields = sorted(k for k in set(old[eid]) | set(new[eid])
2500+
if old[eid].get(k) != new[eid].get(k))
2501+
if fields:
2502+
found.append(f"{key}: {eid} changed ({', '.join(fields)})")
2503+
return found
2504+
2505+
2506+
OWNED_BASE = """core: [base-a]
2507+
base:
2508+
- id: base-a
2509+
file: templates/core/quality.md
2510+
backend:
2511+
- id: backend-a
2512+
file: templates/backend/http.md
2513+
depends_on: [base-a]
2514+
frontend: []
2515+
stacks:
2516+
- id: stack-a
2517+
file: templates/stack/a.md
2518+
depends_on: [backend-a]
2519+
"""
2520+
2521+
# Each case is a fork's manifest against OWNED_BASE, and the findings the
2522+
# check owes it. The appends are the control: a check that flagged them
2523+
# would fail every fork that did what the contract allows.
2524+
OWNERSHIP_CASES = [
2525+
("a stack and a platform appended, and a key of the fork's own",
2526+
OWNED_BASE + """ - id: stack-b
2527+
file: templates/stack/b.md
2528+
platform:
2529+
- id: platform-b
2530+
file: templates/platform/b.md
2531+
fork:
2532+
- id: fork-note
2533+
""", []),
2534+
("the core tier edited",
2535+
OWNED_BASE.replace("core: [base-a]", "core: [base-a, backend-a]"),
2536+
["core: ['base-a'] became ['base-a', 'backend-a']"]),
2537+
("an upstream entry changed",
2538+
OWNED_BASE.replace("depends_on: [base-a]", "depends_on: []"),
2539+
["backend: backend-a changed (depends_on)"]),
2540+
("an entry added under an upstream key",
2541+
OWNED_BASE.replace("frontend: []", """frontend:
2542+
- id: frontend-b
2543+
file: templates/frontend/b.md"""),
2544+
["frontend: frontend-b added"]),
2545+
("an upstream entry removed",
2546+
OWNED_BASE.replace(""" - id: backend-a
2547+
file: templates/backend/http.md
2548+
depends_on: [base-a]
2549+
""", ""),
2550+
["backend: backend-a removed"]),
2551+
]
2552+
2553+
2554+
def check_sys_20():
2555+
if not HAS_YAML:
2556+
return [MISSING_YAML]
2557+
2558+
seen = Inspected()
2559+
failures = []
2560+
2561+
seen.count("planted fork manifests", OWNERSHIP_CASES)
2562+
base = yaml.safe_load(OWNED_BASE)
2563+
for label, text, want in OWNERSHIP_CASES:
2564+
got = upstream_key_edits(base, yaml.safe_load(text))
2565+
if got != want:
2566+
failures.append(f" planted, {label}: found {got}, owed {want}")
2567+
2568+
manifest = os.path.join(ROOT, "templates", "manifest.yaml")
2569+
with io.open(manifest, encoding="utf-8") as handle:
2570+
now_text = handle.read()
2571+
2572+
ref = os.environ.get(UPSTREAM_REF_ENV)
2573+
if ref:
2574+
merged = subprocess.run(["git", "merge-base", "HEAD", ref],
2575+
cwd=ROOT, capture_output=True, text=True)
2576+
if merged.returncode != 0:
2577+
failures.append(
2578+
f" {UPSTREAM_REF_ENV}={ref} shares no history with HEAD, so "
2579+
f"the revision the fork last merged is unknown: "
2580+
f"{merged.stderr.strip()}"
2581+
)
2582+
return seen.failures() + failures, seen.notes()
2583+
revision = merged.stdout.strip()
2584+
shown = subprocess.run(
2585+
["git", "show", f"{revision}:templates/manifest.yaml"],
2586+
cwd=ROOT, capture_output=True, text=True, encoding="utf-8")
2587+
if shown.returncode != 0:
2588+
failures.append(f" the manifest at {revision[:12]} could not be "
2589+
f"read: {shown.stderr.strip()}")
2590+
return seen.failures() + failures, seen.notes()
2591+
was_text = shown.stdout
2592+
basis = (f" baseline: the merge base {revision[:12]} of HEAD and "
2593+
f"{ref}")
2594+
else:
2595+
was_text = now_text
2596+
basis = (f" baseline: {UPSTREAM_REF_ENV} is unset, so this is "
2597+
f"upstream and the manifest is compared with itself")
2598+
2599+
was, now = yaml.safe_load(was_text), yaml.safe_load(now_text)
2600+
seen.count("upstream keys compared",
2601+
[k for k in UPSTREAM_KEYS if k in was])
2602+
for finding in upstream_key_edits(was, now):
2603+
failures.append(
2604+
f" {finding}: a fork appends under stacks:, platform: and its "
2605+
f"own keys, and leaves upstream's as it last merged them"
2606+
)
2607+
2608+
return seen.failures() + failures, seen.notes() + [basis]
2609+
24652610
CHECKS = [
24662611
{"id": "SYS-01", "spec": "SAIT-SMK-SYS-01-001A",
24672612
"title": "DEPENDS ON paths resolve to existing files", "fn": check_sys_01},
@@ -2545,6 +2690,9 @@ def check_sys_19():
25452690
{"id": "SYS-19", "spec": "SAIT-SMK-SYS-19-001A",
25462691
"title": "resolve.py names a missing dependency and a dependency cycle",
25472692
"fn": check_sys_19},
2693+
{"id": "SYS-20", "spec": "SAIT-SMK-SYS-20-001A",
2694+
"title": "A fork leaves upstream's manifest keys as it last merged them",
2695+
"fn": check_sys_20},
25482696
]
25492697

25502698

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
---
2+
id: SAIT-SMK-SYS-20-001A
3+
title: A fork leaves upstream's manifest keys as it last merged them
4+
product: sait
5+
type: smoke
6+
area: SYS
7+
priority: p1
8+
status: ready
9+
environment: [local, ci]
10+
automatable: yes
11+
created: 2026-10-17
12+
author: Branimir Georgiev
13+
product-version: "2.x"
14+
tags: [manifest, fork, composition, merge]
15+
---
16+
17+
## Short description
18+
19+
> **Given** a fork's manifest and the upstream revision the fork last merged
20+
> **When** the keys upstream owns — `core`, `base`, `backend`, `frontend` —
21+
> are compared between the two
22+
> **Then** any entry added, removed or changed under them fails, naming the
23+
> key and the entry, because that edit is what turns the fork's next
24+
> upstream merge into a conflict
25+
26+
## Results
27+
28+
| Result | Condition |
29+
|--------|-----------|
30+
| PASSED | Every planted case yields exactly its owed findings, and the real manifest's upstream keys match the baseline |
31+
| FAILED | A planted case yields findings other than those it owes |
32+
| FAILED | An upstream key's entry is added, removed or changed against the baseline |
33+
| FAILED | `SAIT_UPSTREAM_REF` is set and shares no history with HEAD, or the baseline manifest cannot be read |
34+
| FAILED | No planted case or no upstream key was compared |
35+
| SKIPPED | — |
36+
| BLOCKED | — |
37+
| ERROR | PyYAML is not installed |
38+
39+
## Steps
40+
41+
### Prerequisites
42+
43+
- Repository cloned locally, with git
44+
- Python 3 and PyYAML
45+
- In a fork: a ref holding upstream's history, named in `SAIT_UPSTREAM_REF`
46+
47+
### Setup
48+
49+
— (operates on the committed tree and the working manifest)
50+
51+
### Execution
52+
53+
1. Compare each planted fork manifest in `OWNERSHIP_CASES` with the planted
54+
upstream manifest, and compare the findings with those the case owes
55+
2. Take the baseline: with `SAIT_UPSTREAM_REF` set, the manifest at the
56+
merge base of HEAD and that ref; unset, the working manifest itself
57+
3. Compare the working manifest's upstream keys with the baseline's
58+
4. Report the planted-case count, the upstream keys compared, and the
59+
baseline used
60+
61+
### Assertions
62+
63+
1. Assert a planted fork that appends a stack and a platform, and adds a
64+
key of its own, yields no finding — the control that legal appends pass
65+
2. Assert planted edits yield one finding each: the core tier changed, an
66+
upstream entry changed, an entry added under an upstream key, and an
67+
upstream entry removed
68+
3. Assert the real manifest's upstream keys match the baseline
69+
4. Assert both counts are non-zero, per ADR-034
70+
71+
### Negative controls
72+
73+
1. **The defect that motivated the check.** A fork edits an upstream key
74+
and nothing notices until its next upstream merge conflicts. Run with
75+
`SAIT_UPSTREAM_REF` at a revision before an upstream-key change and the
76+
check names that entry
77+
2. **A comparison that finds nothing.** A comparison that returns no
78+
findings passes upstream's run against itself; the planted edits fail
79+
against it, each by its owed finding
80+
81+
## Related
82+
83+
- ADR-049 — a fork appends under `stacks:`, `platform:` and its own keys,
84+
and never edits upstream's
85+
- ADR-034 — why the counts are reported and floored

0 commit comments

Comments
 (0)