Skip to content

Commit e416eb9

Browse files
authored
ci: say a superseded Experimental leg was stopped, not that its tests exited 0 (#2077) (#2088)
util/superseded.py's watch exits 0 when it stops a superseded run, so Containment summary closed on 'tests exited 0 after 61 s'. It now reads the tests step's stopped output through env and says the tests were stopped as superseded, with no warning of its own; every other line and warning is unchanged.
1 parent 79934dd commit e416eb9

4 files changed

Lines changed: 205 additions & 4 deletions

File tree

‎.github/workflows/experimental.yml‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -516,13 +516,17 @@ jobs:
516516
# worker xdist replaced (#1753), and a timeout -- exit 124, or 137 with
517517
# no OOM kill after 110 minutes, `--kill-after` ending a teardown SIGINT
518518
# did not; an OOM kill of pytest itself also exits 137. Nothing here
519-
# fails.
519+
# fails. #2077: a run the watcher stopped as superseded exits 0, so the
520+
# watcher's `stopped=true` makes the closing line say the tests were
521+
# stopped, not that they exited 0; its `Superseded` notice already says
522+
# why, so the stop itself gets no warning.
520523
- name: Containment summary
521524
if: ${{ !cancelled() }}
522525
continue-on-error: true
523526
shell: bash
524527
env:
525528
LEG: Python ${{ matrix.python-version }} (experimental, non-blocking)
529+
STOPPED: ${{ steps.tests.outputs.stopped }}
526530
run: |
527531
status="$RUNNER_TEMP/tests-status"
528532
if [ ! -s "$status" ]; then
@@ -558,7 +562,12 @@ jobs:
558562
if [ "$crashed" -gt 0 ] 2>/dev/null; then
559563
warn "tests lost an xdist worker" "${crashed} xdist worker(s) went down, each failing the test it ran, and xdist replaces up to 30; exit ${code}, scope ${scope}; ${last}."
560564
fi
561-
echo "$LEG: tests exited ${code} after ${elapsed} s, scope ${scope}, ${oom} OOM kill(s), ${errors} MemoryError line(s), ${crashed:-0} worker(s) down; ${last}."
565+
if [ "$STOPPED" = true ]; then
566+
ended="were stopped as superseded"
567+
else
568+
ended="exited ${code}"
569+
fi
570+
echo "$LEG: tests ${ended} after ${elapsed} s, scope ${scope}, ${oom} OOM kill(s), ${errors} MemoryError line(s), ${crashed:-0} worker(s) down; ${last}."
562571
563572
# #1660: an experimental leg's report, as unit-tests.yml `test`'s `KTLO
564573
# summary` is a KTLO leg's, and for the same reasons: the same script,

‎docs/source/contributing/workflows.rst‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1017,7 +1017,11 @@ not those that mention one, such as a docstring a traceback prints, and xdist's
10171017
down`` lines. From it and the monitor's last sample, ``Containment summary`` warns
10181018
``No memory cap``, ``tests lost a process to the OOM killer``, ``tests timed out at 110
10191019
minutes``, ``tests raised MemoryError`` or ``tests lost an xdist worker``, each a
1020-
warning of its own. A timeout is exit 124, or 137 with no OOM kill
1020+
warning of its own. Its closing line gives the exit status, but where the watcher
1021+
stopped the run and wrote ``stopped=true`` it reads ``tests were stopped as
1022+
superseded``, not ``tests exited 0``, and adds no warning, since the ``Superseded``
1023+
notice already says why (`#2077
1024+
<https://github.com/JarryShaw/PyPCAPKit/issues/2077>`__). A timeout is exit 124, or 137 with no OOM kill
10211025
after 110 minutes: ``--kill-after``'s SIGKILL to a teardown SIGINT did not end, where an
10221026
OOM kill of pytest exits 137 too. ``Experimental summary``, the job's last step but the job summary's and the
10231027
KTLO summary's script, still warns of every failed step, ``tests`` included. The 2 GiB,

‎tests/project/test_resource_monitor_1740.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -530,8 +530,10 @@ def test_experimental_summary_is_mains_and_runs_last(self) -> None:
530530
self.assertEqual(position(SUMMARY) + 1, position('Experimental summary'))
531531
self.assertEqual(step(SUMMARY).get('if'), '${{ !cancelled() }}')
532532
self.assertIs(step(SUMMARY).get('continue-on-error'), True)
533+
# #2077: and the watcher's `stopped`, so a superseded stop reads as one.
533534
self.assertEqual(step(SUMMARY).get('env'),
534-
{'LEG': 'Python ${{ matrix.python-version }} (experimental, non-blocking)'})
535+
{'LEG': 'Python ${{ matrix.python-version }} (experimental, non-blocking)',
536+
'STOPPED': '${{ steps.tests.outputs.stopped }}'})
535537
self.assertNotIn('tests-status', step('Experimental summary')['run'])
536538

537539

Lines changed: 186 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,186 @@
1+
# -*- coding: utf-8 -*-
2+
"""A superseded Experimental leg's ``Containment summary`` says it was stopped (#2077).
3+
4+
In Experimental Tests run 39898029852, job 120233865230, util/superseded.py's
5+
``watch`` stopped pytest 61 s in, ``main`` having moved on, and exited 0 as it
6+
does for every stop. ``Containment summary`` read that status and closed on
7+
``tests exited 0 after 61 s``, which reads as a pass. #2076 has the watcher
8+
write ``stopped=true`` to ``Run unit tests``'s outputs, so the summary now
9+
reads it and says the tests were stopped as superseded. It adds no warning for
10+
the stop, as the watcher's ``Superseded`` notice already says why, and keeps
11+
every warning it gives for a real cause.
12+
13+
"""
14+
15+
from __future__ import annotations
16+
17+
import os
18+
import pathlib
19+
import re
20+
import shutil
21+
import sys
22+
import tempfile
23+
import threading
24+
import unittest
25+
from typing import TYPE_CHECKING
26+
27+
from tests.project.test_python_matrix_tiers_1633 import workflow_commands
28+
from tests.project.test_python_versions_1660 import ROOT, prose
29+
from tests.project.test_release_gates import NULL, evaluate_expression
30+
from tests.project.test_resource_monitor_1740 import BASH, DEFAULT, SUMMARY, TESTS, sh, step
31+
from tests.project.test_superseded_runs_1938 import OURS, REPO, Handler, Server
32+
33+
if TYPE_CHECKING:
34+
from typing import Any, Dict, List, Optional
35+
36+
LEG = 'Python 3.14t (experimental, non-blocking)'
37+
#: The cited job's ``tests-status``, as its ``Containment summary`` line reports it.
38+
STOPPED_STATUS = ('exit=0\nscope=12G\nprobe=\nelapsed=61\noom_kills=0\nmemory_errors=0\ncrashed_workers=0\n')
39+
STOPPED_LINE = (f'{LEG}: tests were stopped as superseded after 61 s, scope 12G, 0 OOM kill(s), '
40+
'0 MemoryError line(s), 0 worker(s) down; no resource-monitor sample.')
41+
42+
43+
def stopped_env(outputs: 'Dict[str, Any]') -> 'str':
44+
"""``Containment summary``'s ``STOPPED``, as the runner renders it where ``Run unit tests`` wrote ``outputs``."""
45+
expression = step(SUMMARY).get('env', {}).get('STOPPED')
46+
if expression is None: # a step that does not read the output sees none
47+
return ''
48+
value = evaluate_expression(expression, {'steps': {'tests': {'outputs': outputs}}})
49+
# An unset output renders as the empty string in `env:`, as for any `${{ }}` there.
50+
return '' if value is NULL or value is None else str(value)
51+
52+
53+
class Summary(unittest.TestCase):
54+
"""Runs the real ``Containment summary`` script over a ``RUNNER_TEMP`` of its own."""
55+
56+
def setUp(self) -> None:
57+
tmp = tempfile.TemporaryDirectory(prefix='pcapkit-2077-') # pylint: disable=consider-using-with
58+
self.addCleanup(tmp.cleanup)
59+
self.root = pathlib.Path(tmp.name)
60+
self.temp = self.root / 'temp'
61+
self.temp.mkdir()
62+
self.page = self.root / 'summary.md'
63+
self.env = {key: value for key, value in os.environ.items() if not key.startswith('GITHUB_')}
64+
self.env.update(RUNNER_TEMP=str(self.temp), LEG=LEG, GITHUB_STEP_SUMMARY=str(self.page))
65+
66+
def summary(self, outputs: 'Dict[str, Any]', status: 'Optional[str]' = None) -> 'List[str]':
67+
"""The step's output lines, given the tests step's ``outputs`` and, if not left by it, ``tests-status``."""
68+
if status is not None:
69+
(self.temp / 'tests-status').write_text(status, encoding='utf-8')
70+
env = dict(self.env, STOPPED=stopped_env(outputs))
71+
result = sh(step(SUMMARY)['run'], env, BASH)
72+
self.assertEqual(result.returncode, 0, result.stderr)
73+
return result.stdout.splitlines()
74+
75+
def job_summary(self) -> 'str':
76+
return self.page.read_text(encoding='utf-8') if self.page.exists() else ''
77+
78+
79+
@unittest.skipUnless(shutil.which('bash'), 'the step runs under bash, as on the runners')
80+
class TestTheSummaryLine(Summary):
81+
82+
def test_a_stopped_run_says_it_was_stopped_with_no_warning(self) -> None:
83+
lines = self.summary({'stopped': 'true'}, STOPPED_STATUS)
84+
self.assertEqual(lines, [STOPPED_LINE])
85+
self.assertNotIn('exited', '\n'.join(lines))
86+
self.assertEqual(workflow_commands('\n'.join(lines)), [])
87+
self.assertEqual(self.job_summary(), '')
88+
89+
def test_any_other_run_still_gives_its_exit_status(self) -> None:
90+
for outputs in ({}, {'stopped': ''}, {'stopped': 'false'}):
91+
with self.subTest(outputs=outputs):
92+
self.assertEqual(self.summary(outputs, STOPPED_STATUS.replace('exit=0', 'exit=1')), [
93+
f'{LEG}: tests exited 1 after 61 s, scope 12G, 0 OOM kill(s), 0 MemoryError line(s), '
94+
'0 worker(s) down; no resource-monitor sample.'])
95+
96+
def test_a_stopped_run_keeps_its_warnings_for_real_causes(self) -> None:
97+
status = 'exit=0\nscope=none\nprobe=sudo: no\nelapsed=61\noom_kills=1\nmemory_errors=2\ncrashed_workers=3\n'
98+
warnings = {} # type: Dict[str, List[str]]
99+
for stopped in ('true', 'false'):
100+
lines = self.summary({'stopped': stopped}, status)
101+
warnings[stopped] = workflow_commands('\n'.join(lines))
102+
self.assertEqual(len(lines), len(warnings[stopped]) + 1)
103+
self.assertEqual(warnings['true'], warnings['false'])
104+
self.assertEqual([line.split('::')[1].split(' on ')[0] for line in warnings['true']],
105+
['warning title=No memory cap', 'warning title=tests lost a process to the OOM killer',
106+
'warning title=tests raised MemoryError', 'warning title=tests lost an xdist worker'])
107+
108+
def test_the_output_reaches_the_step_through_env(self) -> None:
109+
summary = step(SUMMARY)
110+
self.assertEqual(summary['env']['STOPPED'], '${{ steps.tests.outputs.stopped }}')
111+
self.assertNotIn('${{', summary['run'])
112+
self.assertEqual(step(TESTS)['id'], 'tests')
113+
self.assertEqual(stopped_env({'stopped': 'true'}), 'true')
114+
self.assertEqual(stopped_env({}), '')
115+
116+
117+
@unittest.skipUnless(sys.platform.startswith('linux') and all(map(shutil.which, ('bash', 'timeout', 'prlimit'))),
118+
'the tests step runs coreutils timeout and util-linux prlimit, as on the Ubuntu runners')
119+
class TestTheStepsInTurn(Summary):
120+
"""``Run unit tests`` stopped by the real watcher, then ``Containment summary`` over what it left."""
121+
122+
def setUp(self) -> None:
123+
super().setUp()
124+
bindir, self.output = self.root / 'bin', self.root / 'output'
125+
bindir.mkdir()
126+
(self.temp / 'resource-monitor.log').write_text('', encoding='utf-8')
127+
self.python = bindir / 'python'
128+
for name, body in (('sudo', 'echo "sudo: a password is required" >&2\nexit 1\n'),
129+
('python', 'echo collected\nexec sleep 60\n')):
130+
(bindir / name).write_text('#!/bin/sh\n' + body, encoding='utf-8')
131+
(bindir / name).chmod(0o755)
132+
self.server = Server(('127.0.0.1', 0), Handler)
133+
threading.Thread(target=self.server.serve_forever, daemon=True).start()
134+
self.addCleanup(self.server.server_close)
135+
self.addCleanup(self.server.shutdown)
136+
self.env.update(GITHUB_WORKSPACE=str(ROOT), WORKERS='3', GRAAL_PYTHON_VM_ARGS='', GITHUB_OUTPUT=str(self.output),
137+
GH_TOKEN='token', GITHUB_API_URL=f'http://127.0.0.1:{self.server.server_address[1]}',
138+
GITHUB_EVENT_NAME='push', GITHUB_REF='refs/heads/main', GITHUB_SHA=OURS,
139+
GITHUB_REPOSITORY=REPO, PATH=str(bindir) + os.pathsep + os.environ.get('PATH', ''),
140+
WATCHER='success', WATCHER_PYTHON=sys.executable)
141+
142+
def run_tests(self) -> 'Dict[str, str]':
143+
"""``Run unit tests``, its watcher quickened; the outputs it wrote."""
144+
script = step(TESTS)['run']
145+
quick = script.replace('watch --interval 60 --grace 30 ', 'watch --interval 0.5 --grace 2 --kill-wait 2 ')
146+
self.assertNotEqual(quick, script)
147+
sh(quick, self.env, DEFAULT)
148+
text = self.output.read_text(encoding='utf-8') if self.output.exists() else ''
149+
return dict(line.split('=', 1) for line in text.splitlines() if '=' in line)
150+
151+
def last_line(self, outputs: 'Dict[str, str]') -> 'str':
152+
lines = self.summary(outputs)
153+
self.assertEqual(workflow_commands('\n'.join(lines)), ['::warning title=No memory cap on '
154+
'Python 3.14t (experimental%2C non-blocking)::'
155+
'The probe, sudo -n systemd-run --scope with the '
156+
"tests' flags, failed (sudo: a password is required), "
157+
'so the tests ran without the 12G scope. Experimental '
158+
'legs never block (#1660), so this job still succeeds.'])
159+
return lines[-1]
160+
161+
def test_a_superseded_run_is_reported_as_stopped(self) -> None:
162+
outputs = self.run_tests()
163+
self.assertEqual(outputs, {'stopped': 'true'})
164+
line = self.last_line(outputs)
165+
self.assertRegex(line, '^' + re.escape(LEG) + r': tests were stopped as superseded after \d+ s, scope none, ')
166+
167+
def test_a_current_run_is_reported_by_its_exit_status(self) -> None:
168+
self.server.head = OURS # type: ignore[attr-defined]
169+
self.python.write_text('#!/bin/sh\nsleep 1.5\nexit 1\n', encoding='utf-8')
170+
outputs = self.run_tests()
171+
self.assertEqual(outputs, {})
172+
self.assertRegex(self.last_line(outputs), r': tests exited 1 after \d+ s, scope none, ')
173+
174+
175+
class TestTheDocsSayIt(unittest.TestCase):
176+
177+
def test_the_workflows_page_says_a_stopped_run_was_stopped(self) -> None:
178+
page = prose(ROOT / 'docs' / 'source' / 'contributing' / 'workflows.rst')
179+
for phrase in ('`#2077 <https://github.com/JarryShaw/PyPCAPKit/issues/2077>`__',
180+
'``tests were stopped as superseded``', '``tests exited 0``'):
181+
with self.subTest(phrase=phrase):
182+
self.assertIn(phrase, page)
183+
184+
185+
if __name__ == '__main__':
186+
unittest.main()

0 commit comments

Comments
 (0)