Repository navigation
test-windows-guards: test_daemon_stability turns setup failures and timeouts into REGRESSION verdicts on unrelated PRs #2057
Description
Summary
test / test-windows-guards is a required check, and it goes red on pull requests whose diffs cannot touch it. The red always comes from tests/windows/test_daemon_stability.py, and it comes from three mechanisms in the guard's own harness rather than from the product. Until they are fixed the leg is a lottery: a green needs a rerun that changes nothing but the dice, which is exactly what the CI-determinism rule forbids.
Observed (unrelated diffs, same leg, same file)
| PR | diff | run / job |
|---|---|---|
| #1875 | Cypher | job 99662589846 (line 820: SETUP FAIL: ASCII baseline did not index: {nodes: None, edges: None, definition_nodes: 5}) |
| #1876 | Windows Unicode paths | reddened 2026-08-28 |
| #1896 | Swift extraction | reddened 2026-09-01 |
| #2002 | Cypher SKIP/LIMIT | section_start_status_port: "the daemon did not accept the UI configuration; browser was not opened" |
| #2055 | scripts/run-test-wave.py + one contract test |
run 33892540012, job 101150446542: section_crash_recovery ("the daemon did not accept the UI configuration; browser was not opened") and section_cold_storm ("cold-storm client 2 failed (racing daemon spawn): secure CLI coordination could not be created (endpoint)") |
Mechanisms (verified in the tree on 2026-09-03; line numbers as of that audit)
- A setup failure is reported as a regression. The file header of
test_daemon_stability.pydocuments the exit contract0 == all sections green, 1 == regression, 2 == setup error, andscripts/test-windows.ps1(around line 190) treats exit 2 as a documented precondition skip. But every setup-failure branch in the stability guard (return Falseat roughly lines 206 / 218 / 265 / 319) exits 1, while its sibling guards return 2 for the same class of failure. The same daemon-start failure is therefore an invisible skip in one guard and a REGRESSION in this one. That asymmetry is why the red reads like a daemon bug. - A timeout decides a verdict.
_parallel_one_shots(around line 297) runsrun_cli(timeout=120)on bare threads with noexcept; aTimeoutExpiredleavesresults[i] = None, andsection_cold_storm(around line 361) then prints "cold-storm client N failed (racing daemon spawn)" for what was a 120 s timeout. A timeout must never be the thing that decides a test. - The daemon cohort is machine-global.
cbm_daemon_rendezvous_key()insrc/daemon/service.c(around line 144) hashes a compile-time constant, by design. The only relocation hook isCBM_RUNTIME_DIR, and neitherscripts/test-windows.ps1nor anything undertests/windows/sets it. Every section, and every other daemon-using guard on the same runner, shares one rendezvous.
What "fixed" looks like
- Setup-failure branches in the stability guard exit 2, matching the documented contract and the sibling guards; a precondition that cannot be met is a skip with a printed reason, never a regression.
- The cold-storm clients treat
TimeoutExpiredas a setup error with its own message (and their own budget), not as a client failure. - The guards run their daemons under a per-run
CBM_RUNTIME_DIRso the cohort is isolated from anything else on the runner. - Every case that is red-tolerated or skipped carries an at-the-case explanation of WHY plus what was tried (the Memory leak: process grows to 50+ GB virtual memory over hours/days, crashes Windows #581 whitelist shape); an unexplained skip is a board defect.
#1796 ("a refused UI configuration is not a failed daemon") addresses the section_crash_recovery conflation and is being taken as CI-unblocking; it does not cover section_cold_storm or the exit-code contract, which is what this issue tracks.
Activity
DeusData commentedon Sep 5, 2026
Progress: #1796 landed on main as ceba427 — a refused UI configuration is no longer reported as a failed
daemon start, which removes thesection_crash_recoveryred described above. Still open here: the setup-failure exit code (1 instead of 2), the cold-storm 120 s timeout being reported as a client failure, and the machine-global rendezvous key.CaptainMittens commentedon Sep 6, 2026
Still recurring on 2026-09-06, after ceba427 landed. Four
test-windows-guardsreds today, allsection_cold_storm, all on branches that cannot touch it:fix/coverage-range-duplicate-and-eof(#1971)fix-count-failed-readfix/windows-parent-watchdogfeat/atlas-r1Every one printed the same body:
One detail that may narrow mechanism 2 vs 3 in the issue body: none of these four is the timeout path. A
TimeoutExpiredleavesresults[i] = Noneandsection_cold_stormprints(no result). Hereresultis a real completed process that exited non-zero and wrote that line to stderr, so the client ran and was refused. That points at the shared rendezvous of mechanism 3 rather than the 120 s budget of mechanism 2.Second detail: the message carries no validation detail.
src/main.c:2566appendscbm_daemon_ipc_validation_detail()when it is non-empty, and the comment above it (from #1533 / #1574) says a bare(endpoint)is the outcome that left four reporters with nothing to act on.ipc_validation_detail_set("%s", "")atsrc/daemon/ipc.c:1563and:4571clears the buffer on a passing check, so a later failure that sets no detail of its own prints bare. Whatever refuses the endpoint under a six-way race is reaching that path, which means the diagnostic added for #1582 does not cover this case.I have not reproduced it outside CI, and I have no Windows runner to bisect on — this is offered as evidence of recurrence and a narrowing, not a diagnosis.
DeusData commentedon Sep 26, 2026
Thanks, @CaptainMittens, your narrowing was right! The bare
(endpoint)came from a lost runtime-directory creation race on Windows:ERROR_ALREADY_EXISTSin the runtime-dir walk was treated as a failure. That's fixed in a041794, and every refusal on that path now names its component. The per-section privateCBM_RUNTIME_DIRfrom #2026 is in v0.11.0.What remains here is the exit-code contract (#2026 deliberately made only a failed
daemon startexit 2) and the fact that a timeout can still decide a RED. We'll settle both separately.