Skip to content

Commit e925bfb

Browse files
LeadGoEngineerPaperclip-Paperclip
andcommitted
ci(compat): verify CONTRACT §X citations against CONTRACT.md
Adds TestCONTRACTCitationsAreReal in compat/citations_test.go: walks every .go file in the compat module, finds comments of the form `CONTRACT §N: "..."`, and asserts the quoted fragment is a substring of section N of CONTRACT.md. Picks up via the existing compat CI job's `go test ./...` step — no workflow changes needed. Motivation: two consecutive PRs (#6, #7) landed `CONTRACT §5: "..."` quotes that referred to text §5 (Auth) does not contain. Both passed Go-side review. The structural fix is to make the next fabricated quote die in CI, not in a post-merge incident comment. CONTRIBUTING.md gains a "Citing the contract from compat code" subsection that documents the rule for contributors. This guard fails CI on `main` today because of the residual §5 hallucination in compat/dates/dates.go. Cleaning that up is the out-of-scope half tracked in QUA-15; this PR cannot squash-merge until that lands. Refs QUA-20. Co-Authored-By: Paperclip <noreply@paperclip.ing>
1 parent 000912b commit e925bfb

2 files changed

Lines changed: 162 additions & 0 deletions

File tree

CONTRIBUTING.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,17 @@ Rules:
8181
- Compat tests run in CI on every PR and on `main`. A failing compat test on `main` means at least one shipped CLI no longer matches the contract, and that's a release-blocker incident, not a flake.
8282
- The Status table in `CONTRACT.md` distinguishes **machine-attested** rows (covered by `compat/`) from **human-attested** rows (still verified by reviewer judgment). Promoting a row from human to machine attestation is itself a worthwhile PR.
8383

84+
### Citing the contract from compat code
85+
86+
If you write a comment in `compat/` that quotes the contract, use the form `CONTRACT §N: "exact text"`. Every such quote is auto-verified at PR time by `TestCONTRACTCitationsAreReal` in `compat/citations_test.go` — the test walks the compat tree and fails CI if the quoted fragment is not a substring of section N of `CONTRACT.md`. The check uses double-quoted fragments only; backtick-wrapped code references (e.g. `` `--format` ``) are ignored.
87+
88+
Two ways to make a failing citation green:
89+
90+
- Drop the quote, or restate the property without claiming it comes from the contract.
91+
- Update `CONTRACT.md` so §N actually contains the text, then re-run `go test ./...` from `compat/`.
92+
93+
The check exists because two consecutive PRs landed `CONTRACT §5:` quotes that referenced text §5 didn't contain. That class of error should die in CI now, not in a post-merge incident comment.
94+
8495
**Bar for a new exporter:** the exporter's CI must build its binary and run `dates.RunContract` against it green. The `formats.RunContract` bundle ships, but its CSV subtest assumes an exporter that implements all three §4 codecs (markdown, json, csv) — until the framework gains a `Runner.SupportedFormats` affordance, exporters whose CSV writer is incomplete cannot adopt the bundle as a hard gate. Wire it in as exporter parity catches up. See [`compat/README.md`](compat/README.md) for the one-file integration pattern.
8596

8697
## License and sign-off

compat/citations_test.go

Lines changed: 151 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
package compat_test
2+
3+
import (
4+
"fmt"
5+
"go/parser"
6+
"go/token"
7+
"io/fs"
8+
"os"
9+
"path/filepath"
10+
"regexp"
11+
"sort"
12+
"strings"
13+
"testing"
14+
)
15+
16+
// citationRE matches `CONTRACT §N: "..."` in a comment, where N is one
17+
// or more digits and "..." is a double-quoted prose fragment. The
18+
// fragment may have been wrapped across two or more comment lines in
19+
// source — go/ast.CommentGroup.Text() joins consecutive comment lines
20+
// with single spaces before we run the regex, so the match is always
21+
// against a single logical line.
22+
//
23+
// Backtick-quoted fragments (e.g. `[]` for json) are deliberately
24+
// excluded: those are usually inline code references, not contract
25+
// quotations, and treating them as the latter would create noise. If a
26+
// contributor wants the harness to verify a quote, they write it with
27+
// double quotes.
28+
var citationRE = regexp.MustCompile(`CONTRACT §(\d+):\s*"([^"]+)"`)
29+
30+
// sectionHeaderRE matches `## N. <title>` headers in CONTRACT.md and
31+
// captures the section number.
32+
var sectionHeaderRE = regexp.MustCompile(`(?m)^##\s+(\d+)\.\s+`)
33+
34+
// TestCONTRACTCitationsAreReal walks every .go file in the compat
35+
// module and, for every comment that quotes a section of the contract
36+
// in the form `CONTRACT §N`-colon-double-quoted-fragment, verifies the
37+
// fragment is a substring of section N of CONTRACT.md. A false quote
38+
// fails the test loudly so it cannot silently survive review.
39+
//
40+
// The test exists because two consecutive PRs landed comments
41+
// attributing prose to §5 that §5 does not contain (§5 is the Auth
42+
// section, with no language about hermeticity or `--help`). Go-side
43+
// review missed both. This guard is the structural fix: the next
44+
// fabricated quote dies in CI instead of in a post-merge incident
45+
// comment.
46+
func TestCONTRACTCitationsAreReal(t *testing.T) {
47+
contractPath := filepath.Join("..", "CONTRACT.md")
48+
body, err := os.ReadFile(contractPath)
49+
if err != nil {
50+
t.Fatalf("read CONTRACT.md (looked at %s): %v", contractPath, err)
51+
}
52+
sections := splitContractSections(string(body))
53+
if len(sections) == 0 {
54+
t.Fatalf("no `## N. <title>` headers found in %s; the regex assumes the contract uses that header style", contractPath)
55+
}
56+
57+
type problem struct {
58+
path string
59+
section string
60+
quote string
61+
reason string
62+
}
63+
var problems []problem
64+
65+
walkErr := filepath.WalkDir(".", func(path string, d fs.DirEntry, err error) error {
66+
if err != nil {
67+
return err
68+
}
69+
if d.IsDir() {
70+
return nil
71+
}
72+
if !strings.HasSuffix(path, ".go") {
73+
return nil
74+
}
75+
fset := token.NewFileSet()
76+
f, err := parser.ParseFile(fset, path, nil, parser.ParseComments)
77+
if err != nil {
78+
return fmt.Errorf("parse %s: %w", path, err)
79+
}
80+
for _, cg := range f.Comments {
81+
text := cg.Text()
82+
for _, m := range citationRE.FindAllStringSubmatch(text, -1) {
83+
section := m[1]
84+
quote := m[2]
85+
secBody, ok := sections[section]
86+
if !ok {
87+
problems = append(problems, problem{path, section, quote, "no `## " + section + ". ` section exists in CONTRACT.md"})
88+
continue
89+
}
90+
if !strings.Contains(normalize(secBody), normalize(quote)) {
91+
problems = append(problems, problem{path, section, quote, "quoted text is not a substring of CONTRACT.md §" + section})
92+
}
93+
}
94+
}
95+
return nil
96+
})
97+
if walkErr != nil {
98+
t.Fatalf("walk compat tree: %v", walkErr)
99+
}
100+
101+
if len(problems) == 0 {
102+
return
103+
}
104+
105+
sort.Slice(problems, func(i, j int) bool {
106+
if problems[i].path != problems[j].path {
107+
return problems[i].path < problems[j].path
108+
}
109+
return problems[i].section < problems[j].section
110+
})
111+
112+
var b strings.Builder
113+
fmt.Fprintf(&b, "%d CONTRACT §X citation(s) do not match CONTRACT.md:\n\n", len(problems))
114+
for _, p := range problems {
115+
fmt.Fprintf(&b, " %s\n", p.path)
116+
fmt.Fprintf(&b, " cite : CONTRACT §%s\n", p.section)
117+
fmt.Fprintf(&b, " quote: %q\n", p.quote)
118+
fmt.Fprintf(&b, " why : %s\n\n", p.reason)
119+
}
120+
b.WriteString("Fix by either dropping the quote or updating CONTRACT.md so §N actually contains the text.\n")
121+
b.WriteString("See compat/citations_test.go for the matching rule.")
122+
t.Fatal(b.String())
123+
}
124+
125+
// splitContractSections returns a map from section number ("3") to the
126+
// raw markdown body of `## 3. <title>` up to (but not including) the
127+
// next `## N. <title>` header. The section header itself is included
128+
// in the body so that the title text is also citable.
129+
func splitContractSections(md string) map[string]string {
130+
idxs := sectionHeaderRE.FindAllStringSubmatchIndex(md, -1)
131+
out := map[string]string{}
132+
for i, m := range idxs {
133+
section := md[m[2]:m[3]]
134+
var end int
135+
if i+1 < len(idxs) {
136+
end = idxs[i+1][0]
137+
} else {
138+
end = len(md)
139+
}
140+
out[section] = md[m[0]:end]
141+
}
142+
return out
143+
}
144+
145+
// normalize collapses runs of whitespace to single spaces so a quote
146+
// that was wrapped across two source-comment lines (joined with one
147+
// space by CommentGroup.Text) still matches a sentence in CONTRACT.md
148+
// that lives on a single line.
149+
func normalize(s string) string {
150+
return strings.Join(strings.Fields(s), " ")
151+
}

0 commit comments

Comments
 (0)