Skip to content

Commit e9b80c5

Browse files
authored
ci: validate every member with mcpp 2026.9.28.2, and a failing member names its issue (#483)
N1: validate.yml runs every member with mcpp 2026.9.28.2 (mcpp-community/mcpp#730) and latest_mcpp moves to it; min_mcpp does not move. The full sweep on the branch (run 36376460690) passed all 22 shards. N2: tests/known-red.tsv lists the members known to fail with the issue that owns each; the job 'red members name their issues' (tests/red_members.py, self-test in lint) lists every failing member with its issue and fails when one has none. The sweep alert sets GH_REPO, which it lacked when the sweep of 2026-09-26 failed, and also runs when mirror-cn-reachable fails (#482).
1 parent 1041a95 commit e9b80c5

4 files changed

Lines changed: 228 additions & 5 deletions

File tree

‎.github/workflows/validate.yml‎

Lines changed: 56 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -357,7 +357,21 @@ env:
357357
# every member and platform.
358358
#
359359
# `index.toml` min_mcpp does not move: no descriptor grammar changed.
360-
MCPP_VERSION: "2026.9.28.1"
360+
# ── 2026-09-28: raised 2026.9.28.1 -> 2026.9.28.2 ─────────────────────────
361+
#
362+
# 2026.9.28.2 (mcpp-community/mcpp#730) changes four things a member's build
363+
# can see: on Windows the files beside a program are decided by one runtime
364+
# placement resolver, which places the toolset's C++ runtime over an older
365+
# copy a dependency ships and states that copy once as a packaging fault; on
366+
# Windows every action runs with the toolset's runtime directory first on
367+
# PATH, so each action re-runs once after the raise; a GNU-dialect compile
368+
# on Windows writes a depfile, so those members rebuild once; and matching
369+
# conditional tables apply in order of selector specificity. The one
370+
# descriptor with two `target_cfg` predicates, compat.libarchive, names two
371+
# that never match one target (linux, macos), so no descriptor's flags move.
372+
#
373+
# `index.toml` min_mcpp does not move: no descriptor grammar changed.
374+
MCPP_VERSION: "2026.9.28.2"
361375

362376
jobs:
363377
lint:
@@ -463,6 +477,8 @@ jobs:
463477
# cases are about the rule and not about any member.
464478
- name: The compatibility classifier agrees with its own rules
465479
run: python3 tests/openkal/compat.py selftest
480+
- name: The red-member rule agrees with its own cases
481+
run: python3 tests/red_members.py selftest
466482
# ── Single-source-of-truth grammar check ─────────────────────────
467483
# `mcpp xpkg parse` uses EXACTLY the resolver's parser, so what
468484
# passes here is what builds for users of the pinned MCPP_VERSION.
@@ -1584,6 +1600,32 @@ jobs:
15841600
path: member-timings.tsv
15851601
retention-days: 90
15861602

1603+
# ── Every red member names its issue ────────────────────────────────────
1604+
# The full sweeps of 2026-09-26 failed on a member and on the CN-mirror job,
1605+
# and no issue recorded either. A failing member is either listed in
1606+
# tests/known-red.tsv with the open issue that owns it, or it fails this job;
1607+
# the run summary lists every failing member with its issue (task N2 of the
1608+
# 2026-09-28 ecosystem design). `always()`: it reads the rows the members'
1609+
# shards uploaded, whatever the members' job concluded.
1610+
red-members:
1611+
name: red members name their issues
1612+
needs: [select, workspace]
1613+
if: always() && needs.select.outputs.members != ''
1614+
runs-on: ubuntu-latest
1615+
steps:
1616+
- uses: actions/checkout@v4
1617+
- uses: actions/download-artifact@v4
1618+
with:
1619+
pattern: timings-*
1620+
path: timings
1621+
continue-on-error: true
1622+
- name: Every failing member names its issue
1623+
run: |
1624+
set -o pipefail
1625+
mkdir -p timings
1626+
python3 tests/red_members.py report timings tests/known-red.tsv \
1627+
--members-result "${{ needs.workspace.result }}" | tee -a "$GITHUB_STEP_SUMMARY"
1628+
15871629
# ─────────────────────────────────────────────────────────────────────────
15881630
# A full sweep that fails must be SEEN.
15891631
#
@@ -1602,10 +1644,10 @@ jobs:
16021644
# already visible on the PR.
16031645
sweep-alert:
16041646
name: full sweep failed — open an issue
1605-
needs: [workspace]
1647+
needs: [workspace, mirror-cn-reachable, red-members]
16061648
if: >-
16071649
always()
1608-
&& needs.workspace.result == 'failure'
1650+
&& (needs.workspace.result == 'failure' || needs.mirror-cn-reachable.result == 'failure')
16091651
&& (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch')
16101652
runs-on: ubuntu-latest
16111653
permissions:
@@ -1614,7 +1656,14 @@ jobs:
16141656
- name: Open or update the sweep-failure issue
16151657
env:
16161658
GH_TOKEN: ${{ github.token }}
1659+
# This job checks nothing out, so `gh` cannot infer the repository
1660+
# from a work tree: on 2026-09-26 every `gh issue` call here failed
1661+
# with "not a git repository", and the failing sweep opened no issue.
1662+
GH_REPO: ${{ github.repository }}
16171663
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
1664+
FAILED_JOBS: >-
1665+
${{ needs.workspace.result == 'failure' && 'workspace' || '' }}
1666+
${{ needs.mirror-cn-reachable.result == 'failure' && 'mirror-cn-reachable' || '' }}
16181667
run: |
16191668
set -euo pipefail
16201669
TITLE="full workspace sweep is failing"
@@ -1629,7 +1678,10 @@ jobs:
16291678
"This is the only signal that catches a published package broken by an" \
16301679
"upstream change — selective CI cannot select a member no diff names." \
16311680
"" \
1632-
"Check the \`workspace\` job's failed shards for which members regressed.")
1681+
"Failed jobs: $FAILED_JOBS." \
1682+
"" \
1683+
"The \`red members name their issues\` job lists each failing member with" \
1684+
"the issue that owns it (tests/known-red.tsv).")
16331685
if [ -n "$NUM" ]; then
16341686
gh issue reopen "$NUM" 2>/dev/null || true
16351687
gh issue comment "$NUM" --body "$BODY"

‎index.toml‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -115,7 +115,12 @@
115115
# descriptor grammar. An engine from this release on that a later floor
116116
# leaves below min_mcpp reports the floor as a closing tip of the command that
117117
# refreshed the index, not as an error at the start of every command.
118+
# ── 2026-09-28: latest_mcpp -> 2026.9.28.2, min_mcpp unchanged ─────────────
119+
#
120+
# Moved in the pull request that moves validate.yml's pin to 2026.9.28.2,
121+
# which runs every member on every platform with it. The release changes no
122+
# descriptor grammar.
118123
[index]
119124
spec = "1"
120125
min_mcpp = "2026.9.18.3"
121-
latest_mcpp = "2026.9.28.1"
126+
latest_mcpp = "2026.9.28.2"

‎tests/known-red.tsv‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Members known to fail in the full sweep, one line each: <leg>\t<member>\t<issue>.
2+
# A leg is <platform>-<toolchain> (linux-default, linux-llvm, macos-default,
3+
# windows-default), or * for every leg. tests/red_members.py fails the sweep's
4+
# summary when a member fails and has no line here; a line is removed when its
5+
# issue is closed. See .github/workflows/validate.yml, job `red-members`.

‎tests/red_members.py‎

Lines changed: 161 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
#!/usr/bin/env python3
2+
"""A failing member names its issue (task N2 of the 2026-09-28 ecosystem design,
3+
mcpp-community/mcpp `.agents/docs/2026-09-28-ecosystem-design-and-optimisation-plan.md`).
4+
5+
WHY. The two full sweeps of 2026-09-26 on `main` failed -- one on the
6+
`mirror-cn-reachable` job, one on the member `pangocairo` (linux) -- and nothing
7+
recorded either: the job that opens the sweep issue failed as well, and a red
8+
weekly sweep that no issue names is read by nobody (#482).
9+
10+
`tests/known-red.tsv` lists the members known to fail, per leg, each with the
11+
open issue that owns it. After the members have run, this script reads every
12+
shard's timing rows (`<seconds>\t<member>\t<ok|FAIL>`, in the artifacts
13+
`timings-<platform>-<toolchain>-<shard>`), and:
14+
15+
- lists every failing member with its issue, in the run summary;
16+
- fails when a failing member has no issue on its leg;
17+
- fails when the members' job failed and no member row says FAIL (a shard that
18+
failed before testing: nothing can be attributed);
19+
- lists a known-red member that passed, so its line can be removed.
20+
21+
red_members.py report <timings dir> <known-red.tsv> [--members-result R]
22+
red_members.py selftest
23+
24+
A leg is `<platform>-<toolchain>` (`linux-default`, `linux-llvm`,
25+
`macos-default`, `windows-default`); `*` in known-red.tsv matches every leg.
26+
"""
27+
from __future__ import annotations
28+
29+
import pathlib
30+
import sys
31+
import tempfile
32+
33+
34+
def read_known(path: pathlib.Path) -> dict[tuple[str, str], str]:
35+
known: dict[tuple[str, str], str] = {}
36+
for number, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
37+
if not line.strip() or line.lstrip().startswith("#"):
38+
continue
39+
parts = line.split("\t")
40+
if len(parts) != 3 or not all(p.strip() for p in parts):
41+
raise SystemExit(f"{path}:{number}: expected <leg>\\t<member>\\t<issue>, got {line!r}")
42+
leg, member, issue = (p.strip() for p in parts)
43+
known[(leg, member)] = issue
44+
return known
45+
46+
47+
def read_rows(timings: pathlib.Path) -> list[tuple[str, str, str]]:
48+
"""(leg, member, status) for every row of every shard."""
49+
rows = []
50+
for tsv in sorted(timings.glob("timings-*/timings.tsv")):
51+
# timings-<platform>-<toolchain>-<shard>
52+
name = tsv.parent.name.split("-")
53+
if len(name) < 4:
54+
continue
55+
leg = f"{name[1]}-{name[2]}"
56+
for line in tsv.read_text(encoding="utf-8").splitlines():
57+
parts = line.split("\t")
58+
if len(parts) == 3:
59+
rows.append((leg, parts[1], parts[2].strip()))
60+
return rows
61+
62+
63+
def report(timings: pathlib.Path, known_path: pathlib.Path, members_result: str,
64+
out=sys.stdout) -> int:
65+
known = read_known(known_path)
66+
rows = read_rows(timings)
67+
failing = sorted({(leg, m) for leg, m, s in rows if s == "FAIL"})
68+
passing = {(leg, m) for leg, m, s in rows if s == "ok"}
69+
70+
def issue_of(leg: str, member: str) -> str | None:
71+
return known.get((leg, member)) or known.get(("*", member))
72+
73+
unattributed = [(leg, m) for leg, m in failing if not issue_of(leg, m)]
74+
print("### Red members", file=out)
75+
print("", file=out)
76+
if failing:
77+
print("| leg | member | issue |", file=out)
78+
print("|---|---|---|", file=out)
79+
for leg, m in failing:
80+
print(f"| {leg} | `{m}` | {issue_of(leg, m) or '**none**'} |", file=out)
81+
else:
82+
print("No member failed.", file=out)
83+
healed = sorted((leg, m) for (leg, m) in known
84+
if leg != "*" and (leg, m) in passing)
85+
if healed:
86+
print("", file=out)
87+
print("Listed in tests/known-red.tsv and passing now (remove the line once "
88+
"its issue is closed):", file=out)
89+
for leg, m in healed:
90+
print(f"- {leg} `{m}` ({known[(leg, m)]})", file=out)
91+
status = 0
92+
if unattributed:
93+
print("", file=out)
94+
print("A failing member without an issue fails this job: open one and add "
95+
"`<leg>\\t<member>\\t<issue>` to tests/known-red.tsv, or fix the member.",
96+
file=out)
97+
status = 1
98+
if members_result == "failure" and not failing:
99+
print("", file=out)
100+
print("The members' job failed, and no member row says FAIL: a shard failed "
101+
"before testing, and nothing can be attributed to a member.", file=out)
102+
status = 1
103+
return status
104+
105+
106+
def selftest() -> int:
107+
def run(rows_by_artifact: dict[str, str], known: str, result: str) -> tuple[int, str]:
108+
with tempfile.TemporaryDirectory() as d:
109+
root = pathlib.Path(d)
110+
for artifact, text in rows_by_artifact.items():
111+
(root / artifact).mkdir()
112+
(root / artifact / "timings.tsv").write_text(text, encoding="utf-8")
113+
known_path = root / "known-red.tsv"
114+
known_path.write_text(known, encoding="utf-8")
115+
import io
116+
buf = io.StringIO()
117+
code = report(root, known_path, result, out=buf)
118+
return code, buf.getvalue()
119+
120+
header = "# <leg>\t<member>\t<issue>\n"
121+
cases = [
122+
# (name, rows, known, members result, expected status, text that must appear)
123+
("all green", {"timings-linux-default-0": "10\ta\tok\n"}, header, "success", 0, "No member failed"),
124+
("a red member without an issue", {"timings-linux-default-0": "10\ta\tFAIL\n"}, header,
125+
"failure", 1, "**none**"),
126+
("a red member with its issue", {"timings-linux-default-0": "10\ta\tFAIL\n"},
127+
header + "linux-default\ta\t#1\n", "failure", 0, "| linux-default | `a` | #1 |"),
128+
("an issue on another leg does not cover", {"timings-linux-llvm-1": "10\ta\tFAIL\n"},
129+
header + "linux-default\ta\t#1\n", "failure", 1, "**none**"),
130+
("a wildcard leg covers", {"timings-windows-default-0": "10\ta\tFAIL\n"},
131+
header + "*\ta\t#2\n", "failure", 0, "#2"),
132+
("a shard that failed before testing", {"timings-linux-default-0": "10\ta\tok\n"}, header,
133+
"failure", 1, "failed before testing"),
134+
("a known-red member that passes is named", {"timings-macos-default-0": "10\ta\tok\n"},
135+
header + "macos-default\ta\t#3\n", "success", 0, "passing now"),
136+
]
137+
failed = 0
138+
for name, rows, known, result, want_status, want_text in cases:
139+
status, text = run(rows, known, result)
140+
ok = status == want_status and want_text in text
141+
print(f"{'ok' if ok else 'FAIL'}: {name}")
142+
if not ok:
143+
failed += 1
144+
print(text)
145+
return 1 if failed else 0
146+
147+
148+
def main(argv: list[str]) -> int:
149+
if argv[:1] == ["selftest"]:
150+
return selftest()
151+
if argv[:1] == ["report"] and len(argv) >= 3:
152+
result = ""
153+
if "--members-result" in argv:
154+
result = argv[argv.index("--members-result") + 1]
155+
return report(pathlib.Path(argv[1]), pathlib.Path(argv[2]), result)
156+
print(__doc__)
157+
return 2
158+
159+
160+
if __name__ == "__main__":
161+
sys.exit(main(sys.argv[1:]))

0 commit comments

Comments
 (0)