Skip to content

Commit 8997b44

Browse files
authored
ci(unit-tests): split the PyPy protocols shard so its tail stops stalling (#2604)
- unit-tests.yml: a PyPy leg's SHARDS runs tests/protocols/internet, application and misc on their own ahead of tests/protocols, which then takes the remainder; test-ktlo shares the step by YAML alias. Run whole, tests/protocols peaked at 17.7 GiB at -n 4 against the runner's 16 GB. - test_pypy_worker_memory_2005: pin the new SHARDS, and require each shard to keep unit-tier modules no earlier shard took. - workflows.rst: name the shards and record the measurement. Collect-only on PyPy 7.3.19: same 2717 ids, none duplicated; the split protocols runs peak at 5.7 GiB and take 224 s against 267 s.
1 parent 19e1eca commit 8997b44

3 files changed

Lines changed: 30 additions & 8 deletions

File tree

‎.github/workflows/unit-tests.yml‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -205,10 +205,15 @@ jobs:
205205
# pytest once per directory `SHARDS` names, then once over the rest,
206206
# and every run starts its workers afresh. Each run but the last writes
207207
# a JUnit file of its own, beside junit.xml; the step fails if any did.
208+
#
209+
# #2600: run whole, tests/protocols took the workers past the runner's
210+
# 16 GB, and its tail stalled past tests._support.time_limit's 30 s. So
211+
# its internet, application and misc subpackages run first, each a shard
212+
# of its own, and `tests/protocols` then takes the remainder.
208213
env:
209214
COVERAGE: ${{ matrix.coverage }}
210215
LATEST: ${{ matrix.python-version == needs.python-versions.outputs.latest }}
211-
SHARDS: ${{ startsWith(matrix.python-version, 'pypy') && 'tests/protocols tests/corekit' || '' }}
216+
SHARDS: ${{ startsWith(matrix.python-version, 'pypy') && 'tests/protocols/internet tests/protocols/application tests/protocols/misc tests/protocols tests/corekit' || '' }}
212217
run: |
213218
if [ "$LATEST" = "true" ]; then
214219
if ! skipped=$(python -c "from tests._tiers import project_pin_paths; print(' '.join(project_pin_paths()))"); then

‎docs/source/contributing/workflows.rst‎

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1048,12 +1048,21 @@ extras cap |cryptography|_ below 46, as 45.0.7 is the newest release with a Linu
10481048
On PyPy an xdist worker's RSS grows with every test it runs, rather than in a few
10491049
heavy modules, so a PyPy leg's ``Run unit tests`` hands its pytest command to
10501050
:file:`util/pytest_shards.py`, as GraalPy's leg does (see below), which runs it once per
1051-
directory its ``SHARDS`` names, ``tests/protocols`` and then ``tests/corekit``, and once
1052-
more over the rest, ignoring each (:issue:`2005`).
1051+
directory its ``SHARDS`` names and once more over the rest, ignoring each
1052+
(:issue:`2005`). The directories are ``tests/protocols/internet``,
1053+
``tests/protocols/application``, ``tests/protocols/misc``, ``tests/protocols`` and
1054+
``tests/corekit``, in that order, so ``tests/protocols`` runs after its own
1055+
subpackages and takes only what they left.
10531056
Each run starts its workers afresh. Measured on PyPy 8.0.1 at ``-n 4``, one run
10541057
over the whole ``ALL`` tier took each worker to 5.4-5.9 GB and the workers together
10551058
past 20 GB, beyond the runner's 16 GB; split, the largest run peaked at 2.6 GB a
1056-
worker and 8.5 GB in all, and the tier took 383 s rather than 493 s. A smaller
1059+
worker and 8.5 GB in all, and the tier took 383 s rather than 493 s. With
1060+
:issue:`2543`'s checksum tests, ``tests/protocols`` run whole outgrew the runner as
1061+
well, and on CI the tests at its tail stalled past ``tests._support.time_limit``'s
1062+
30-second deadline, which counts wall-clock time, so it is split (:issue:`2600`).
1063+
Measured on PyPy 7.3.19 at ``-n 4``, its one run peaked at 17.7 GiB and took 267 s,
1064+
its slowest test 16.4 s; split, the largest of its runs peaked at 5.7 GiB, they took
1065+
224 s together, and the slowest test 11.9 s. A smaller
10571066
``-n`` would only hand each worker more tests, and a ``PYPY_GC_MAX`` heap cap turns
10581067
the kills into ``MemoryError``\ s. Each run but the last writes a JUnit file of its
10591068
own beside ``junit.xml``, which ``Upload test timings`` takes too, and the step

‎tests/project/test_pypy_worker_memory_2005.py‎

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,10 @@
99
own script with a stub ``python``, as
1010
:mod:`tests.project.test_sweeps_on_latest_only_1732` does.
1111
12+
#2600: run whole, ``tests/protocols`` outgrew the runner too, so its
13+
``internet``, ``application`` and ``misc`` subpackages are shards of their own
14+
ahead of it, and it takes the remainder.
15+
1216
"""
1317

1418
from __future__ import annotations
@@ -33,8 +37,9 @@
3337

3438
#: The ``test`` job's KTLO twin, which runs its steps by YAML alias.
3539
KTLO = ('unit-tests.yml', 'test-ktlo')
36-
#: The directories a PyPy leg runs on their own, in order.
37-
SHARDS = ['tests/protocols', 'tests/corekit']
40+
#: The directories a PyPy leg runs on their own, in order; #2600 split ``tests/protocols``.
41+
SHARDS = ['tests/protocols/internet', 'tests/protocols/application', 'tests/protocols/misc', 'tests/protocols',
42+
'tests/corekit']
3843
#: The page that says how the legs run.
3944
DOC = ROOT / 'docs' / 'source' / 'contributing' / 'workflows.rst'
4045

@@ -61,10 +66,13 @@ def test_the_ktlo_twin_runs_the_same_step(self) -> None:
6166
self.assertEqual(job(*KTLO)['steps'], job(*JOB)['steps'])
6267

6368
def test_each_shard_is_a_directory_the_tier_keeps(self) -> None:
64-
for shard in SHARDS:
69+
"""#2600: and each keeps unit-tier modules no earlier shard took, so its run collects a test."""
70+
for index, shard in enumerate(SHARDS):
6571
with self.subTest(shard=shard):
6672
self.assertTrue((ROOT / shard).is_dir())
67-
self.assertTrue([module for module in _tiers.all_tier_paths() if module.startswith(shard + '/')])
73+
earlier = tuple(taken + '/' for taken in SHARDS[:index])
74+
self.assertTrue([module for module in _tiers.all_tier_paths() if module.startswith(shard + '/')
75+
and not module.startswith(earlier) and _tiers.is_unit_tier(module)])
6876

6977

7078
@unittest.skipIf(shutil.which('bash') is None, 'the step is a bash script')

0 commit comments

Comments
 (0)