Skip to content

Commit 9db5217

Browse files
committed
fix(vm): require the completion marker of the full leg only
The verdict guard added a few commits ago demanded `=== All tests passed ===` from every run. Only the full leg prints it. `scripts/test.sh --suites` stops after the suites it was named and says nothing more, because it has no end to reach — so every `win.sh test <suites>` run came back red while passing: === test.sh: ITERATION mode — suites: artifact (incremental build) === === artifact === 23 passed, 1 skipped GUARD: the log has a summary but no completion marker (runner rc=0) That is the same defect as the one the guard exists to prevent, pointing the other way. A guard against false greens is not allowed to invent false reds, and this one broke the documented iteration workflow on its first real use. vm_verdict now takes a mode. The full leg is unchanged: failures, then the marker, then the note about a lost exit status. A named subset is judged on its summaries and, since it has no marker to lean on, on its exit status — it is a developer tool rather than a gate, so there it obeys rc instead of overriding it. The contract test grew four cases, and they are self-proving rather than inert: the SAME markerless log must be green in iteration mode and red for the full leg, so a build that ignored the mode would fail them. That gap is why the original eight passed while the shipped path was broken — they only ever fed it full-leg logs, which is the case I was thinking about rather than the case I shipped. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1 parent dae1e7d commit 9db5217

2 files changed

Lines changed: 42 additions & 11 deletions

File tree

‎test-infrastructure/vm/vm-run-tests.sh‎

Lines changed: 25 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -49,9 +49,17 @@ esac
4949
# logic with synthetic logs instead of a copy of it that can drift
5050
# (tests/test_vm_verdict_contract.sh). See the two-channel note at the call
5151
# site for why the log outranks the exit status.
52+
# `mode` is "full" for the whole venue leg and "iteration" for a named subset of
53+
# suites. Only the full leg prints the completion marker, because only the full
54+
# leg HAS an end to reach: scripts/test.sh --suites finishes after the suites it
55+
# was given and says nothing more. Requiring the marker in both modes made every
56+
# `win.sh test <suites>` run report failure while passing -- 23 passed, 1
57+
# skipped, rc=0, called red. A guard against false greens is not allowed to
58+
# invent false reds.
5259
vm_verdict() {
5360
local log="$1"
5461
local rc="$2"
62+
local mode="${3:-full}"
5563
if ! grep -Eq '[0-9]+ passed' "$log"; then
5664
echo "GUARD: test runner produced no completion summary — the suites did" \
5765
"not validly run; treating as failure (runner rc=$rc)" >&2
@@ -66,11 +74,19 @@ vm_verdict() {
6674
echo "GUARD: the log reports $failed_total failed test(s) (runner rc=$rc)" >&2
6775
return 1
6876
fi
69-
if [ "$complete" -eq 0 ]; then
77+
if [ "$mode" = "full" ] && [ "$complete" -eq 0 ]; then
7078
echo "GUARD: the log has a summary but no completion marker — the leg" \
7179
"stopped before the end (runner rc=$rc)" >&2
7280
return 1
7381
fi
82+
if [ "$mode" != "full" ] && [ "${rc:-1}" -ne 0 ]; then
83+
# No marker to lean on here, so a non-zero status is the only evidence
84+
# that the run ended badly after its last summary line. Iteration mode
85+
# is a developer tool, not a gate, so it obeys rc rather than overriding
86+
# it the way the full leg does.
87+
echo "GUARD: suites reported no failures but the run exited $rc" >&2
88+
return 1
89+
fi
7490
if [ "${rc:-1}" -ne 0 ]; then
7591
echo "GUARD: every suite passed and the run completed, but the exit status" \
7692
"came back as $rc — the ssh/msys2_shell chain lost it. Reporting the" \
@@ -81,11 +97,11 @@ vm_verdict() {
8197

8298
# Verdict-only mode for the contract test: decide a log WITHOUT a VM.
8399
if [ "${1:-}" = "--verdict" ]; then
84-
if [ $# -ne 3 ]; then
85-
echo "usage: vm-run-tests.sh --verdict <log> <rc>" >&2
100+
if [ $# -lt 3 ] || [ $# -gt 4 ]; then
101+
echo "usage: vm-run-tests.sh --verdict <log> <rc> [full|iteration]" >&2
86102
exit 2
87103
fi
88-
vm_verdict "$2" "$3"
104+
vm_verdict "$2" "$3" "${4:-full}"
89105
exit $?
90106
fi
91107

@@ -227,7 +243,9 @@ fi
227243
#
228244
# So the LOG decides the test outcome (the runner writes it locally; it cannot
229245
# be mangled in transit) and rc decides what the log cannot see. Green requires
230-
# zero reported failures AND the completion marker that scripts/test.sh prints
231-
# as its last statement — a leg that stopped early has a summary but no marker.
232-
vm_verdict "$LOG" "$rc"
246+
# zero reported failures AND, for the full leg, the completion marker that
247+
# scripts/test.sh prints as its last statement — a leg that stopped early has a
248+
# summary but no marker. A named subset of suites prints no marker at all, so it
249+
# is judged on its summaries and its exit status instead.
250+
vm_verdict "$LOG" "$rc" "$([ "$1" = "--par" ] && echo full || echo iteration)"
233251
exit $?

‎tests/test_vm_verdict_contract.sh‎

Lines changed: 17 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -36,9 +36,9 @@ printf ' 7925 passed, 0 failed, 74 skipped\n' > "$WORK/no_marker"
3636
printf 'configure: something exploded\n' > "$WORK/no_summary"
3737

3838
failures=0
39-
expect() { # description log rc want
40-
local description="$1" log="$2" rc="$3" want="$4" got
41-
bash "$RUNNER" --verdict "$WORK/$log" "$rc" >/dev/null 2>&1
39+
expect() { # description log rc want [mode]
40+
local description="$1" log="$2" rc="$3" want="$4" mode="${5:-full}" got
41+
bash "$RUNNER" --verdict "$WORK/$log" "$rc" "$mode" >/dev/null 2>&1
4242
got=$?
4343
if [ "$got" -ne "$want" ]; then
4444
echo "FAIL: $description — verdict $got, expected $want" >&2
@@ -65,8 +65,21 @@ expect "a summary without the completion marker is red" no_marker 0 1
6565
# distinguishable from "ran and failed".
6666
expect "a log with no summary is the no-summary guard" no_summary 0 90
6767

68+
# Iteration mode: `test.sh --suites` finishes after the suites it was named and
69+
# prints no completion marker, because it has no end to reach. Demanding one
70+
# turned a passing subset run into a failure — 23 passed, 1 skipped, rc=0,
71+
# reported red (win.sh test artifact, 2026-09-19). A guard against false greens
72+
# must not invent false reds, so the marker is required of the full leg only.
73+
expect "a subset run with no marker is green in iteration mode" no_marker 0 0 iteration
74+
expect "the same log is still red for the full leg" no_marker 0 1 full
75+
# Without a marker to lean on, rc is the only evidence left that a subset run
76+
# ended badly after its last summary, so iteration mode obeys it.
77+
expect "a non-zero status is red in iteration mode" no_marker 1 1 iteration
78+
# Failures still outrank everything, in either mode.
79+
expect "failures are red in iteration mode too" red 0 1 iteration
80+
6881
if [ "$failures" -gt 0 ]; then
6982
echo "VM verdict contract VIOLATED: $failures case(s)" >&2
7083
exit 1
7184
fi
72-
echo "VM verdict contract passed (8 cases: lost exit status, false green, partial runs)"
85+
echo "VM verdict contract passed (12 cases: lost exit status, false green, partial runs, iteration mode)"

0 commit comments

Comments
 (0)