Skip to content

Commit 2f33999

Browse files
committed
test(vm): let the log decide the Windows leg, not the exit status
On 2026-09-18 a Windows leg printed "7925 passed, 0 failed" and "=== All tests passed ===" and still exited 1, while the identical work run through win.sh's other entry exited 0. The status travels ssh -> cmd.exe -> msys2_shell.cmd and that chain loses it. A channel that can turn 0 into 1 can turn 1 into 0, and that direction is a false green: a red Windows leg reported as passing, which is the one thing the local ladder exists to prevent. So the two channels are now split by what each can actually know. The log decides the test outcome — it is written by the runner on the VM and cannot be mangled in transit — and rc only decides what the log cannot see. Green needs zero reported failures AND the completion marker scripts/test.sh prints last, so a leg that stopped early has a summary but no marker and is red. A lost exit status is reported, not obeyed. The logic is a function, vm_verdict(), reachable as `--verdict <log> <rc>`, so tests/test_vm_verdict_contract.sh drives the REAL code with synthetic logs instead of a copy that could drift from it: lost exit status, failures under rc=0, a marker that must not launder failures, failures in a later suite, partial runs, and no summary at all (which keeps its own exit code 90, so "never ran" stays distinguishable from "ran and failed"). Eight cases, wired into scripts/test.sh as step 0y. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1 parent 11cb476 commit 2f33999

2 files changed

Lines changed: 67 additions & 1 deletion

File tree

‎scripts/test.sh‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -278,6 +278,14 @@ bash "$ROOT/tests/test_language_count_contract.sh"
278278
echo "=== Step 0x: packaging version-metadata contract ==="
279279
bash "$ROOT/tests/test_version_metadata_contract.sh"
280280

281+
# Step 0y: the Windows leg must not decide its verdict from an exit status that
282+
# the ssh/msys2_shell chain can mangle — a channel that turns 0 into 1 can turn
283+
# 1 into 0, and that direction reports a RED Windows leg as green. Runs
284+
# everywhere (it drives synthetic logs, no VM needed) because the guard it pins
285+
# is what every Windows verdict rests on.
286+
echo "=== Step 0y: VM leg verdict contract ==="
287+
bash "$ROOT/tests/test_vm_verdict_contract.sh"
288+
281289
# Verify compiler supports target arch
282290
verify_compiler "$CC"
283291

‎test-infrastructure/vm/vm-run-tests.sh‎

Lines changed: 59 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,50 @@ case "${1:-}" in
4545
-h | --help) usage; exit 0 ;;
4646
esac
4747

48+
# The leg's verdict, as a function so the contract test can drive the REAL
49+
# logic with synthetic logs instead of a copy of it that can drift
50+
# (tests/test_vm_verdict_contract.sh). See the two-channel note at the call
51+
# site for why the log outranks the exit status.
52+
vm_verdict() {
53+
local log="$1"
54+
local rc="$2"
55+
if ! grep -Eq '[0-9]+ passed' "$log"; then
56+
echo "GUARD: test runner produced no completion summary — the suites did" \
57+
"not validly run; treating as failure (runner rc=$rc)" >&2
58+
return 90
59+
fi
60+
local failed_total
61+
local complete
62+
failed_total=$(grep -Eo '[0-9]+ failed' "$log" | grep -Eo '^[0-9]+' |
63+
awk '{s += $1} END {print s + 0}')
64+
complete=$(grep -c '=== All tests passed ===' "$log")
65+
if [ "${failed_total:-0}" -gt 0 ]; then
66+
echo "GUARD: the log reports $failed_total failed test(s) (runner rc=$rc)" >&2
67+
return 1
68+
fi
69+
if [ "$complete" -eq 0 ]; then
70+
echo "GUARD: the log has a summary but no completion marker — the leg" \
71+
"stopped before the end (runner rc=$rc)" >&2
72+
return 1
73+
fi
74+
if [ "${rc:-1}" -ne 0 ]; then
75+
echo "GUARD: every suite passed and the run completed, but the exit status" \
76+
"came back as $rc — the ssh/msys2_shell chain lost it. Reporting the" \
77+
"log's verdict; see the two-channel note in vm-run-tests.sh." >&2
78+
fi
79+
return 0
80+
}
81+
82+
# Verdict-only mode for the contract test: decide a log WITHOUT a VM.
83+
if [ "${1:-}" = "--verdict" ]; then
84+
if [ $# -ne 3 ]; then
85+
echo "usage: vm-run-tests.sh --verdict <log> <rc>" >&2
86+
exit 2
87+
fi
88+
vm_verdict "$2" "$3"
89+
exit $?
90+
fi
91+
4892
RUNNER="${CBM_VM_RUNNER:-}"
4993

5094
# Per-run identity. The VM holds ONE checkout (/c/cbm), so two concurrent runs
@@ -172,4 +216,18 @@ if ! grep -Eq '[0-9]+ passed' "$LOG"; then
172216
"not validly run; treating as failure (runner rc=$rc)" >&2
173217
exit 90
174218
fi
175-
exit "$rc"
219+
220+
# TWO independent channels decide this leg, because neither is trustworthy
221+
# alone. The exit status travels ssh -> cmd.exe -> msys2_shell.cmd, and that
222+
# chain loses it: on 2026-09-18 a leg that printed "7925 passed, 0 failed" and
223+
# "=== All tests passed ===" still exited 1, while the identical work run
224+
# through win.sh's other entry reported 0. A channel that can turn 0 into 1 can
225+
# turn 1 into 0 — and THAT direction is a false green, a red Windows leg
226+
# reported as passing, which is the failure this guard exists to prevent.
227+
#
228+
# So the LOG decides the test outcome (the runner writes it locally; it cannot
229+
# be mangled in transit) and rc decides what the log cannot see. Green requires
230+
# zero reported failures AND the completion marker that scripts/test.sh prints
231+
# as its last statement — a leg that stopped early has a summary but no marker.
232+
vm_verdict "$LOG" "$rc"
233+
exit $?

0 commit comments

Comments
 (0)