Skip to content

Commit 36c4922

Browse files
authored
ci(experimental): give the GraalPy leg room to finish (#2739) (#2741)
On ab7be2f, job 125752017215 lost no GraalPy worker but hit the 110-minute timeout during the rest: its shards took 6386 s, on a runner 1.46-1.86 times slower than #2713's estimates assumed, and tests/protocols/internet and tests/protocols/application ran 11.5 and 13.9 minutes, their largest worker at 2.25 and 2.48 GiB RSS. - Run unit tests: timeout 110m -> 180m, step cap 120 -> 190, job cap 140 -> 210, about 1.5 times the leg's 7110-7340 s there (estimate); Containment summary's timeout warning and 137 threshold follow. - SHARDS: split those two at test_igmp* and test_tls* by that job's JUnit seconds, so no shard but tests/const takes as many. - Pins of the caps and the timeout in tests/project follow; the #1963 and timeout notes in workflows.rst gain the measurements. Collecting the tier through util/pytest_shards.py on CPython gives the same 3058 node ids, none twice, for the old and new SHARDS. tests/project passes, each module in its own process; the moved pins fail on the base workflow.
1 parent 234d781 commit 36c4922

9 files changed

Lines changed: 86 additions & 66 deletions

‎.github/workflows/experimental.yml‎

Lines changed: 17 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -155,8 +155,8 @@ jobs:
155155
# tests/project/test_ktlo_legs_stay_green_1672.py pins the caps and the
156156
# minutes they leave the rest.
157157
#
158-
# The tests' cap is sized for GraalPy (#1732), whose leg may still end on the
159-
# 110-minute `timeout` in `Run unit tests`, which `Containment summary`
158+
# The tests' cap is sized for GraalPy (#1732, #2739), whose leg may still end on
159+
# the 180-minute `timeout` in `Run unit tests`, which `Containment summary`
160160
# reports as a warning that blocks nothing.
161161
experimental:
162162
name: Python ${{ matrix.python-version }} (experimental)
@@ -169,7 +169,7 @@ jobs:
169169
&& needs.changes.outputs.test-only != 'true' }}
170170
needs: [current, changes, python-versions]
171171
runs-on: ${{ matrix.os || 'ubuntu-latest' }}
172-
timeout-minutes: 140
172+
timeout-minutes: 210
173173
continue-on-error: true
174174
# #1938: `Run unit tests` reads the pull request, as `current` does.
175175
permissions:
@@ -334,14 +334,14 @@ jobs:
334334
# - #1963: util/pytest_shards.py runs pytest once per shard `SHARDS`
335335
# names, a directory or a glob of one directory's entries, then once
336336
# over the rest, so every run starts its workers afresh and none
337-
# outlives its heap. #2713: the shards that lost workers are split by
338-
# their tests' JUnit seconds on GraalPy, each to run in under 5
339-
# minutes; the run over the rest is left whole, as it took about 6.7
340-
# minutes in job 125639681253 and lost no worker. The script runs
337+
# outlives its heap. #2713 split the shards that lost workers in job
338+
# 125639681253 by their tests' JUnit seconds there, and #2739 the two
339+
# that ran 11-14 minutes in job 125752017215 by its; the run over the
340+
# rest is left whole, as it lost no worker in either. The script runs
341341
# inside the timeout, scope and watcher below, so they hold over the
342342
# whole tier; on the timeout's SIGINT no later run starts. Each shard's
343343
# run writes a JUnit file of its own beside junit.xml.
344-
# - `timeout` sends SIGINT at 110 minutes, under the step's cap, so pytest
344+
# - `timeout` sends SIGINT at 180 minutes, under the step's cap, so pytest
345345
# still writes the JUnit XML, and exits 124; SIGKILL 3 minutes on.
346346
# #2020: GNU's `timeout`, which Ubuntu 26.04 installs as `gnutimeout`:
347347
# its `timeout` is uutils' (0.10.0), whose `--kill-after` sends SIGKILL with the
@@ -387,7 +387,7 @@ jobs:
387387
id: tests
388388
if: ${{ steps.install.outcome == 'success' }}
389389
continue-on-error: true
390-
timeout-minutes: 120
390+
timeout-minutes: 190
391391
env:
392392
GRAAL_PYTHON_VM_ARGS: ${{ startsWith(matrix.python-version, 'graalpy') && '--vm.Xmx2g --vm.XX:+ExitOnOutOfMemoryError' || '' }}
393393
# #2571: 3.14t, which installs psutil, takes `logical` as unit-tests.yml `test` does.
@@ -397,8 +397,9 @@ jobs:
397397
${{ startsWith(matrix.python-version, 'graalpy') && 'tests/const
398398
tests/corekit/test_fields_n* tests/corekit/test_[a-h]* tests/corekit/test_i* tests/corekit
399399
tests/foundation/reassembly tests/foundation/test_* tests/foundation
400-
tests/protocols/internet/test_ipv4* tests/protocols/internet/test_ipv6* tests/protocols/internet
401-
tests/protocols/application tests/protocols/link tests/protocols/misc/test_pcapng* tests/protocols/misc
400+
tests/protocols/internet/test_ipv4* tests/protocols/internet/test_ipv6* tests/protocols/internet/test_igmp*
401+
tests/protocols/internet tests/protocols/application/test_tls* tests/protocols/application
402+
tests/protocols/link tests/protocols/misc/test_pcapng* tests/protocols/misc
402403
tests/protocols/transport/test_tcp* tests/protocols/transport tests/protocols/test_t* tests/protocols
403404
tests/toolkit' || '' }}
404405
GH_TOKEN: ${{ github.token }}
@@ -425,7 +426,7 @@ jobs:
425426
fi
426427
# #2020: GNU's `timeout` on any image, `gnutimeout` where it exists.
427428
gnu_timeout=$(command -v gnutimeout || command -v timeout)
428-
set -- prlimit --fsize=2147483648 "$gnu_timeout" --signal=INT --kill-after=3m 110m "$@"
429+
set -- prlimit --fsize=2147483648 "$gnu_timeout" --signal=INT --kill-after=3m 180m "$@"
429430
scoped() {
430431
unit=$1
431432
shift
@@ -528,7 +529,7 @@ jobs:
528529
# summary's (tests/project/test_ktlo_legs_stay_green_1672.py). Each cause
529530
# gets a warning of its own: no 12G scope, an OOM kill, a MemoryError, a
530531
# worker xdist replaced (#1753), and a timeout -- exit 124, or 137 with
531-
# no OOM kill after 110 minutes, `--kill-after` ending a teardown SIGINT
532+
# no OOM kill after 180 minutes, `--kill-after` ending a teardown SIGINT
532533
# did not; an OOM kill of pytest itself also exits 137. Nothing here
533534
# fails. #2077: a run the watcher stopped as superseded exits 0, so the
534535
# watcher's `stopped=true` makes the closing line say the tests were
@@ -566,9 +567,9 @@ jobs:
566567
warn "tests lost a process to the OOM killer" "${oom} OOM kill(s) while the tests ran, scope ${scope}, exit ${code}, ${elapsed} s; ${last}."
567568
fi
568569
if [ "$code" = 124 ]; then
569-
warn "tests timed out at 110 minutes" "timeout sent SIGINT (exit 124), scope ${scope}, ${elapsed} s; ${last}."
570-
elif [ "$code" = 137 ] && [ "$oom" -eq 0 ] 2>/dev/null && [ "$elapsed" -ge 6600 ] 2>/dev/null; then
571-
warn "tests timed out at 110 minutes" "SIGINT did not end them, so timeout sent SIGKILL 3 minutes on (exit 137, no OOM kill), scope ${scope}, ${elapsed} s; ${last}."
570+
warn "tests timed out at 180 minutes" "timeout sent SIGINT (exit 124), scope ${scope}, ${elapsed} s; ${last}."
571+
elif [ "$code" = 137 ] && [ "$oom" -eq 0 ] 2>/dev/null && [ "$elapsed" -ge 10800 ] 2>/dev/null; then
572+
warn "tests timed out at 180 minutes" "SIGINT did not end them, so timeout sent SIGKILL 3 minutes on (exit 137, no OOM kill), scope ${scope}, ${elapsed} s; ${last}."
572573
fi
573574
if [ "$errors" -gt 0 ] 2>/dev/null; then
574575
warn "tests raised MemoryError" "${errors} MemoryError line(s) in the pytest output, exit ${code}, scope ${scope}; ${last}."

‎docs/source/contributing/workflows.rst‎

Lines changed: 28 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -689,7 +689,7 @@ CI Timing Review (:issue:`1538`)
689689
or on dispatch with a ``days`` window and an optional ``concurrency_hours`` runner-pool
690690
measurement. Through the API it lists every completed run in the window of Unit Tests,
691691
Python Compatibility, Lint, CodeQL, GitHub Pages and Experimental Tests (:issue:`1732`, so
692-
its two-hour legs stay measured), a day per query because a filtered listing stops
692+
its long legs stay measured), a day per query because a filtered listing stops
693693
at 1,000 runs (Pages completed 1,154 in the week to 2026-10-09). It then reads the
694694
jobs of an evenly spaced sample -- 150 push and 150 pull-request runs of Unit Tests,
695695
60 runs of each other workflow -- and the summary says how many runs each sample
@@ -1159,8 +1159,8 @@ though the leg's check run still reads ``failure``. A job past its
11591159
``timeout-minutes`` is cancelled, and that page does not say whether the job-level
11601160
guard covers a cancellation. So setup-python, the install, the watcher's setup-python
11611161
and ``requests`` (:issue:`1938`,
1162-
below) and the tests have caps of their own, of 5, 10, 1, 1 and 120 minutes, under
1163-
the job's 140, which leaves 3 for the rest: a step at its cap
1162+
below) and the tests have caps of their own, of 5, 10, 1, 1 and 190 minutes, under
1163+
the job's 210, which leaves 3 for the rest: a step at its cap
11641164
fails rather than cancels, and the guard holds (actions/runner's
11651165
``StepsRunner.RunStepAsync``). The runner's ``has timed out`` error
11661166
stays, changing no conclusion, as ``Process completed with exit code N.`` does.
@@ -1170,7 +1170,7 @@ job, and none on a step. Promoting one whose leg stays green is a hand edit, whi
11701170
:file:`util/python_versions.py` refuses until the build has a wheel, a conda build
11711171
and a ``Compat`` leg.
11721172

1173-
The tests' 120 minutes are sized for GraalPy, the slowest of the legs that share
1173+
The tests' 190 minutes are sized for GraalPy, the slowest of the legs that share
11741174
them: its leg reached 26% of the unit tier in 25 minutes (job 115915976024), where
11751175
3.14t's ran the whole tier in 5.8 (run 38581834697). The experimental legs run the
11761176
``ALL`` test tier alone, as every leg but ``latest`` does (see `Test Tiers`_ below):
@@ -1180,7 +1180,13 @@ passes each module it printed as an ``--ignore``. The
11801180
round-trip sweeps of :issue:`1732`
11811181
took 8% of the unit tier's time in a serial 3.14 run (86 of 1,031 s), and 10-11% on
11821182
the 3.14, 3.15 and 3.14t legs of run 38581834697, so the rest should take GraalPy
1183-
88-113 minutes (estimate).
1183+
88-113 minutes (estimate). That was low. In job 125752017215, on ``ab7be2fb0``, the
1184+
shards below took 6386 s, and the ``timeout``, of 110 minutes at that commit, ended the
1185+
run over the rest, which had taken 404 s in job 125639681253, on a runner 1.46-1.86
1186+
times as fast (see below). With that run at 589-750 s on the slower runner, and
1187+
70-100 s of start-up for each of the two runs the later split adds, the leg comes to
1188+
7110-7340 s there (estimate). So the ``timeout`` is 180 minutes, about 1.5 times that,
1189+
and the step's cap 10 minutes more (:issue:`2739`).
11841190

11851191
GraalPy's leg still loses its runner 18-30 minutes into its tests, with ``The runner
11861192
has received a shutdown signal`` (:issue:`1740`),
@@ -1242,14 +1248,21 @@ warnings:
12421248
sized from that job's ``junit-experimental-graalpy-25.4`` artifact: a shard's
12431249
estimate is its tests' JUnit seconds over the 3 workers, plus 50 s for the 46-57 s
12441250
that each of the job's shards that lost no worker took beyond that, start to start.
1245-
Every shard split this way comes to under 5 minutes, and the tier to about 4750 s
1246-
(estimate). The run over the rest is left whole: in that job it took about 6.7
1247-
minutes and lost no worker. The runs share the step's ``timeout``, scope and
1251+
That put every shard split this way under 5 minutes, and the tier at about 4750 s,
1252+
on that job's runner (estimate). The run over the rest is left whole: in that job it
1253+
took about 6.7 minutes and lost no worker. In job 125752017215, on ``ab7be2fb0``, no
1254+
worker went down, but on a runner whose unchanged shards each took 1.46-1.86 times
1255+
as long, ``tests/protocols/internet`` and ``tests/protocols/application`` ran 11.5
1256+
and 13.9 minutes, their largest worker reaching 2.25 and 2.48 GiB RSS, and the
1257+
other shards of that split 2.3-8.1 (:issue:`2739`). So those two are split again by
1258+
that job's JUnit seconds, at ``test_igmp*`` and ``test_tls*``, which leaves every
1259+
other shard fewer JUnit seconds there than ``tests/const`` took, whose largest
1260+
worker peaked at 2.05 GiB. The runs share the step's ``timeout``, scope and
12481261
watcher, so the caps hold over the whole tier; on ``timeout``'s SIGINT the run in
12491262
progress writes its JUnit file and no later one starts. Each shard's run writes a
12501263
JUnit file of its own beside ``junit.xml``, such as ``junit-tests-protocols.xml``,
12511264
and ``Upload test timings`` takes them all. 3.14t's leg runs pytest once.
1252-
* ``timeout --signal=INT --kill-after=3m 110m`` ends pytest under the step's 120-minute
1265+
* ``timeout --signal=INT --kill-after=3m 180m`` ends pytest under the step's 190-minute
12531266
cap, so it still writes the junit XML, and exits 124; ``prlimit --fsize=2147483648``
12541267
caps any file at 2 GiB. The ``timeout`` is GNU's, ``gnutimeout`` where it exists
12551268
(:issue:`2020`): on Ubuntu 26.04 ``timeout`` is uutils', whose ``--kill-after`` sends SIGKILL
@@ -1269,20 +1282,21 @@ probe's error, the seconds the tests took, the OOM kills while they ran
12691282
(``/proc/vmstat``), the lines in their output that report a raised ``MemoryError`` (#1962),
12701283
not those that mention one, such as a docstring a traceback prints, and xdist's ``node
12711284
down`` lines. From it and the monitor's last sample, ``Containment summary`` warns
1272-
``No memory cap``, ``tests lost a process to the OOM killer``, ``tests timed out at 110
1285+
``No memory cap``, ``tests lost a process to the OOM killer``, ``tests timed out at 180
12731286
minutes``, ``tests raised MemoryError`` or ``tests lost an xdist worker``, each a
12741287
warning of its own. Its closing line gives the exit status, but where the watcher
12751288
stopped the run and wrote ``stopped=true`` it reads ``tests were stopped as
12761289
superseded``, not ``tests exited 0``, and adds no warning, since the ``Superseded``
12771290
notice already says why (`#2077
12781291
<https://github.com/JarryShaw/PyPCAPKit/issues/2077>`__). A timeout is exit 124, or 137 with no OOM kill
1279-
after 110 minutes: ``--kill-after``'s SIGKILL to a teardown SIGINT did not end, where an
1292+
after 180 minutes: ``--kill-after``'s SIGKILL to a teardown SIGINT did not end, where an
12801293
OOM kill of pytest exits 137 too. ``Experimental summary``, the job's last step but the job summary's and the
12811294
KTLO summary's script, still warns of every failed step, ``tests`` included. The 2 GiB,
12821295
``-n 3`` and 12G are estimates for a 16 GB, 4-CPU runner until a run shows them. At 3
1283-
workers rather than 4, GraalPy's tests should take about 117-151 minutes (an inference
1284-
from the 88-113 above, not measured), so its leg may end on the 110-minute timeout as a
1285-
warning: that is expected, blocks nothing, and leaves the caps as they are.
1296+
workers rather than 4, GraalPy's tests were to take about 117-151 minutes (an inference
1297+
from the 88-113 above, not measured), and job 125752017215's came to about 119-122 in
1298+
the shards above (estimate). A leg that still ends on the 180-minute timeout does so as
1299+
a warning that blocks nothing.
12861300

12871301
**The Windows leg.** :file:`experimental.yml`'s ``windows`` job runs the ``ALL`` tier
12881302
on ``windows-latest`` under ``latest``, the newest CPython the conda builds target, as

‎tests/project/test_ci_timing.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -275,7 +275,7 @@ def test_the_real_unit_tests_workflow(self) -> 'None':
275275
with self.subTest(job=name):
276276
experimental = ci_timing.match_job(own, name)
277277
self.assertEqual((experimental['id'], experimental['timeout'],
278-
ci_timing.step_cap(experimental, ci_timing.TESTS_STEP)), ('experimental', 140, 120))
278+
ci_timing.step_cap(experimental, ci_timing.TESTS_STEP)), ('experimental', 210, 190))
279279

280280
def test_every_step_of_interest_names_a_real_step(self) -> 'None':
281281
"""#2230: each alternative of :data:`STEPS_OF_INTEREST` matches a step some workflow has.
@@ -1037,7 +1037,7 @@ class TestNonBlockingLegs(unittest.TestCase):
10371037
"""#1732: a cap finding on a leg nothing waits on is a warning, not HARD."""
10381038

10391039
#: (workflow, job name, its cap in the real workflow file).
1040-
EXPERIMENTAL = ('experimental.yml', 'Python graalpy-25.4 (experimental)', 140)
1040+
EXPERIMENTAL = ('experimental.yml', 'Python graalpy-25.4 (experimental)', 210)
10411041
KTLO = ('unit-tests.yml', 'Python 3.10 (KTLO)', 45)
10421042
BLOCKING = ('unit-tests.yml', 'Python 3.14', 45)
10431043

‎tests/project/test_gnu_timeout_2020.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@
4141

4242
#: How the tests step resolves GNU's ``timeout``, and how it runs it.
4343
RESOLVE = 'gnu_timeout=$(command -v gnutimeout || command -v timeout)\n'
44-
RUN = '"$gnu_timeout" --signal=INT --kill-after=3m 110m "$@"\n'
44+
RUN = '"$gnu_timeout" --signal=INT --kill-after=3m 180m "$@"\n'
4545
#: ``timeout`` run as a command and given an option, which only GNU's honours as GNU does.
4646
OPTIONED = re.compile(r'(?<![\w$"/.-])timeout\s+-')
4747
#: A shell comment, a whole line of one or one after a space.
@@ -197,7 +197,7 @@ def test_the_step_runs_gnutimeout_where_it_exists(self) -> None:
197197
self.assertEqual(result.returncode, 0, result.stderr)
198198
calls = self.calls.read_text(encoding='utf-8').splitlines()
199199
self.assertEqual(len(calls), 1, calls)
200-
self.assertTrue(calls[0].startswith('gnutimeout --signal=INT --kill-after=3m 110m python -m pytest '), calls)
200+
self.assertTrue(calls[0].startswith('gnutimeout --signal=INT --kill-after=3m 180m python -m pytest '), calls)
201201

202202

203203
if __name__ == '__main__':

‎tests/project/test_graalpy_shards_1963.py‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -35,18 +35,20 @@
3535
from util import pytest_shards
3636

3737
#: GraalPy's shards, in the order they run; the rest of the tier runs last. #2713 split the
38-
#: shards that lost workers in job 125639681253 by their JUnit seconds there.
38+
#: shards that lost workers in job 125639681253 by their JUnit seconds there, and #2739
39+
#: split the two that ran 11-14 minutes in job 125752017215 by its.
3940
SHARDS = ('tests/const tests/corekit/test_fields_n* tests/corekit/test_[a-h]* tests/corekit/test_i* tests/corekit '
4041
'tests/foundation/reassembly tests/foundation/test_* tests/foundation '
41-
'tests/protocols/internet/test_ipv4* tests/protocols/internet/test_ipv6* tests/protocols/internet '
42-
'tests/protocols/application tests/protocols/link tests/protocols/misc/test_pcapng* tests/protocols/misc '
42+
'tests/protocols/internet/test_ipv4* tests/protocols/internet/test_ipv6* tests/protocols/internet/test_igmp* '
43+
'tests/protocols/internet tests/protocols/application/test_tls* tests/protocols/application '
44+
'tests/protocols/link tests/protocols/misc/test_pcapng* tests/protocols/misc '
4345
'tests/protocols/transport/test_tcp* tests/protocols/transport tests/protocols/test_t* tests/protocols '
4446
'tests/toolkit')
4547
SCRIPT = ROOT / 'util' / 'pytest_shards.py'
4648
#: The lines that build the command the step's caps run, in order.
4749
PYTEST = 'set -- python -m pytest "$@"\n'
4850
SHARDED = 'set -- /usr/bin/python3 util/pytest_shards.py "$SHARDS" -- "$@"\n'
49-
CAPPED = 'set -- prlimit --fsize=2147483648 "$gnu_timeout" --signal=INT --kill-after=3m 110m "$@"\n'
51+
CAPPED = 'set -- prlimit --fsize=2147483648 "$gnu_timeout" --signal=INT --kill-after=3m 180m "$@"\n'
5052

5153

5254
def rest_ignores(shards: 'str') -> 'list[str]':
@@ -184,7 +186,7 @@ def test_without_shards_pytest_runs_once(self) -> None:
184186

185187
def test_the_timeout_ends_the_run_in_progress_and_starts_no_other(self) -> None:
186188
# pytest ends on SIGINT, which timeout sends to its whole process group, with exit 2.
187-
script = step(TESTS)['run'].replace('--kill-after=3m 110m', '--kill-after=5s 1s')
189+
script = step(TESTS)['run'].replace('--kill-after=3m 180m', '--kill-after=5s 1s')
188190
self.assertNotEqual(script, step(TESTS)['run'])
189191
result = self.run_step("trap 'echo interrupted; exit 2' INT\nsleep 30 &\nwait $!\nexit 0\n", script)
190192
self.assertEqual(result.returncode, 124, result.stderr)

‎tests/project/test_ktlo_legs_stay_green_1672.py‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -173,10 +173,12 @@
173173
#: ``experimental``'s steps with a ``timeout-minutes`` of their own (#1660), by ``id``:
174174
#: setup-python, the install, the watcher's setup-python and requests (#1938) and the
175175
#: tests. #1732: the tests' is GraalPy's unit tier less the round-trip sweeps, 88-113
176-
#: minutes (estimate) from the 26% its leg reached in 25 (job 115915976024).
177-
EXPERIMENTAL_CAPS = {'python': 5, 'install': 10, 'watcher-python': 1, 'watcher': 1, 'tests': 120}
176+
#: minutes (estimate) from the 26% its leg reached in 25 (job 115915976024). #2739: its
177+
#: 180-minute timeout is about 1.5 times the leg in job 125752017215, whose last run is
178+
#: estimated.
179+
EXPERIMENTAL_CAPS = {'python': 5, 'install': 10, 'watcher-python': 1, 'watcher': 1, 'tests': 190}
178180
#: ``experimental``'s own ``timeout-minutes`` (#1732), and ``test``'s.
179-
EXPERIMENTAL_JOB_CAP = 140
181+
EXPERIMENTAL_JOB_CAP = 210
180182
JOB_CAP = 45
181183
#: The job's minutes those caps leave for the rest of an experimental leg: the steps
182184
#: without a cap of their own -- checkout, the parallelism report, the resource

0 commit comments

Comments
 (0)