Skip to content

Commit badfc34

Browse files
committed
fix(ci): give project-tests' unittest step a stall dump under its own cap (#2164)
Problem: util/run_unittest_leg.py's default stall dump (1080s) is sized for unittest-ordering's 20-minute step cap. project-tests' step "Run tests/project and the root-level modules under plain unittest" is capped at 15 minutes, so a hang there was killed before the dump naming where it stopped could fire. Fix: that step passes --stall-dump 840, a minute under its cap; the script's docstring says a step with a shorter cap passes its own. Test: TestEveryStepDumpsBeforeItsCap reads every workflow step running the script and checks its effective dump (flag, env, else default) is below its timeout-minutes. It fails on origin/main with 1080 not less than 900.
1 parent 04d319f commit badfc34

3 files changed

Lines changed: 67 additions & 2 deletions

File tree

‎.github/workflows/unit-tests.yml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1345,10 +1345,14 @@ jobs:
13451345
python util/cap_file_size.py python -m pytest -q -n auto --dist load \
13461346
--durations=30 --durations-min=1.0 --junitxml="$RUNNER_TEMP/junit.xml" "${pins[@]}"
13471347
1348+
# #2164: the script's default stall dump sits under `unittest-ordering`'s
1349+
# 20-minute step cap, so this step passes its own, a minute under its
1350+
# 15. tests/project/test_run_unittest_leg_stall_dump.py pins every step
1351+
# running the script to a dump inside its cap.
13481352
- name: Run tests/project and the root-level modules under plain unittest
13491353
if: ${{ needs.changes.outputs.code == 'false' }}
13501354
timeout-minutes: 15
1351-
run: python util/cap_file_size.py python util/run_unittest_leg.py project --verbose
1355+
run: python util/cap_file_size.py python util/run_unittest_leg.py project --verbose --stall-dump 840
13521356

13531357
- name: Run the other tests that read docs/ or Markdown
13541358
if: ${{ needs.changes.outputs.code == 'false' }}

‎tests/project/test_run_unittest_leg_stall_dump.py‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,9 @@
66
builds the suite and cancels it on the way out. The in-process tests mock
77
:mod:`faulthandler`, because this module itself runs inside the ``project`` leg,
88
whose own dump is armed; the subprocess test lets a real one fire.
9+
:class:`TestEveryStepDumpsBeforeItsCap` reads every workflow step that runs the
10+
script, as :func:`tests._workflows.load` parses it: each one's dump has to fire
11+
before its own ``timeout-minutes`` kills it (#2164).
912
1013
The last class pins the decision #1052 recorded against restoring
1114
:data:`sys.modules` between tests: a sibling's write to the ``pcapkit`` region
@@ -17,12 +20,19 @@
1720

1821
import os
1922
import pathlib
23+
import shlex
2024
import subprocess
2125
import sys
2226
import textwrap
2327
import unittest
28+
from typing import TYPE_CHECKING
2429
from unittest import mock
2530

31+
from tests import _workflows
32+
33+
if TYPE_CHECKING:
34+
from typing import Any, Optional
35+
2636
ROOT = pathlib.Path(__file__).resolve().parents[2]
2737

2838
if str(ROOT / 'util') not in sys.path:
@@ -106,6 +116,56 @@ def test_sleeps_past_the_dump(self):
106116
self.assertIn('1 test(s), 0 failure(s), 0 error(s)', proc.stdout)
107117

108118

119+
class TestEveryStepDumpsBeforeItsCap(unittest.TestCase):
120+
"""A workflow step running the leg dumps its stack before its cap kills it (#2164).
121+
122+
The dump is what :func:`run_unittest_leg.main` arms: the step's
123+
``--stall-dump``, else :data:`~run_unittest_leg.STALL_DUMP_ENV` from the
124+
step's, job's or workflow's ``env``, else
125+
:data:`~run_unittest_leg.DEFAULT_STALL_DUMP`. The cap is the step's
126+
``timeout-minutes``, else its job's.
127+
128+
"""
129+
130+
@staticmethod
131+
def _flag(run: 'str') -> 'Optional[float]':
132+
words = shlex.split(run.replace('\\\n', ' '), comments=True)
133+
for i, word in enumerate(words):
134+
if word == '--stall-dump':
135+
return float(words[i + 1])
136+
if word.startswith('--stall-dump='):
137+
return float(word.partition('=')[2])
138+
return None
139+
140+
@staticmethod
141+
def _steps() -> 'list[tuple[str, Any, Any, Any, Any]]':
142+
found = []
143+
for path in sorted(_workflows.WORKFLOWS.glob('*.yml')):
144+
workflow = _workflows.load(path.name)
145+
for job_id, job in (workflow.get('jobs') or {}).items():
146+
for step in job.get('steps') or []:
147+
if 'run_unittest_leg.py' in str(step.get('run', '')):
148+
found.append((f'{path.name} {job_id} {step.get("name")!r}',
149+
step, job, workflow, str(step['run'])))
150+
return found
151+
152+
def test_every_step_dumps_before_its_cap(self):
153+
found = self._steps()
154+
self.assertTrue(found, 'no workflow step runs util/run_unittest_leg.py')
155+
for where, step, job, workflow, run in found:
156+
with self.subTest(step=where):
157+
dump = self._flag(run)
158+
if dump is None:
159+
env = {**(workflow.get('env') or {}), **(job.get('env') or {}),
160+
**(step.get('env') or {})}
161+
dump = float(env.get(leg.STALL_DUMP_ENV) or leg.DEFAULT_STALL_DUMP)
162+
cap = step.get('timeout-minutes', job.get('timeout-minutes'))
163+
self.assertIsNotNone(cap, 'the step has no cap to dump before')
164+
self.assertGreater(dump, 0, 'the dump is disabled')
165+
self.assertLess(dump, float(cap) * 60,
166+
f'the dump at {dump:g}s fires after the {cap}-minute cap')
167+
168+
109169
class TestPeriodicCollect(unittest.TestCase):
110170
"""``--gc-every N`` frees unreachable re-import generations every N tests (#1052)."""
111171

‎util/run_unittest_leg.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -102,7 +102,8 @@
102102
arms :func:`faulthandler.dump_traceback_later`, so a leg still running after
103103
that long writes every thread's stack to stderr once, then carries on. The
104104
default, 1080s, is just under the ``unittest-ordering`` step's 20-minute cap,
105-
so a leg about to be killed says where it was first; ``0`` disables it.
105+
so a leg about to be killed says where it was first; a step with a shorter cap
106+
passes its own (#2164). ``0`` disables it.
106107
107108
Memory: every purge-then-import leaves the previous generation as cyclic
108109
garbage, and the interpreter's own collector intermittently falls behind --

0 commit comments

Comments
 (0)