Skip to content

Commit a700bff

Browse files
tthayerclaude
andcommitted
fix(ci): stop kill-grace from bounding Windows helper spawns
The Windows shard failed Step 0i (parallel suite scheduler contract) with "cleanup failed: hang_after_summary: leader exited leaving live descendants" while its sibling shard passed the same contract. Nothing in the wave was actually leaking a process. run-test-wave.py passed --kill-grace as the subprocess timeout for the two external Windows helpers: taskkill.exe /T /F and the powershell.exe Get-CimInstance descendant probe. Those are different quantities. kill_grace budgets how long a doomed process may take to die; the helpers also have to pay process spawn plus, for PowerShell, CIM startup. The contract fixtures run with --kill-grace 1, and one second is not reliably enough to launch either helper on a loaded runner. The two timeouts then compounded. A timed-out taskkill is reported as "could not prove process-tree cleanup", which raises out of the wave loop; the finally-block cleanup re-enters with the leader already dead, so it falls to the descendant probe, which times out on the same one-second budget and takes its "cannot prove absence -> assume the worst" branch. Phantom descendants, red wave. Give helper invocations their own floor, max(kill_grace, 30). kill_grace still governs every actual death wait, so nothing fails open: the timeout-race contract still refuses with rc=2 over a genuinely surviving descendant, now because PowerShell answered rather than because it timed out. The production path already passed --kill-grace 15 and is unchanged in behaviour. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Tony Thayer-Osborne <tony.thayerosborne@conductorone.com>
1 parent 4e8d01d commit a700bff

1 file changed

Lines changed: 14 additions & 2 deletions

File tree

‎scripts/run-test-wave.py‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,14 @@
2727
SKIPPED = re.compile(r"(?:^|, )(?P<skipped>[0-9]+) skipped")
2828
SLOW_SUITES = frozenset(("incremental", "store_arch", "daemon_runtime"))
2929
POLL_SECONDS = 0.05
30+
# Floor for how long an external Windows helper (taskkill.exe, powershell.exe)
31+
# may take to answer. This is deliberately NOT --kill-grace: that flag budgets
32+
# how long a doomed process may take to die, while this budgets process spawn
33+
# plus CIM startup on a loaded runner, which routinely exceeds a second. Wiring
34+
# the two together made a small kill grace flake the whole wave -- taskkill
35+
# timing out is reported as "could not prove cleanup", and the descendant probe
36+
# that runs afterwards fails closed on its own timeout.
37+
WINDOWS_HELPER_TIMEOUT_SECONDS = 30
3038

3139

3240
@dataclass
@@ -123,6 +131,10 @@ def start_suite(
123131
)
124132

125133

134+
def windows_helper_timeout(kill_grace: int) -> int:
135+
return max(kill_grace, WINDOWS_HELPER_TIMEOUT_SECONDS)
136+
137+
126138
def windows_descendants(pid: int, timeout: int) -> bool:
127139
"""True if any live process still claims `pid` as its parent.
128140
@@ -167,7 +179,7 @@ def terminate_process_tree(active: ActiveSuite, kill_grace: int) -> None:
167179
# how a deliberately-hanging fixture suite reddened a release run.
168180
# taskkill /T cannot walk a tree from a dead PID, so prove cleanup
169181
# the only way still available -- nothing is parented to it.
170-
if windows_descendants(process.pid, kill_grace):
182+
if windows_descendants(process.pid, windows_helper_timeout(kill_grace)):
171183
raise RuntimeError(
172184
f"suite {active.name!r} leader exited leaving live descendants"
173185
)
@@ -185,7 +197,7 @@ def terminate_process_tree(active: ActiveSuite, kill_grace: int) -> None:
185197
stdin=subprocess.DEVNULL,
186198
stdout=subprocess.DEVNULL,
187199
stderr=subprocess.DEVNULL,
188-
timeout=kill_grace,
200+
timeout=windows_helper_timeout(kill_grace),
189201
)
190202
except (OSError, subprocess.TimeoutExpired):
191203
completed = None

0 commit comments

Comments
 (0)