Skip to content

Commit bc4bb35

Browse files
committed
ci: run PyPy's shards through util/pytest_shards.py (#2005, #1963)
- unit-tests.yml `test` / `Run unit tests`: the PyPy leg's inline shard loop becomes `python util/cap_file_size.py python util/pytest_shards.py "$SHARDS" -- python -m pytest "$@"`, #1963's script, on the leg's own interpreter (the #1969 pin requires the cap to exec `python`). Same runs, JUnit names and exit status; later shards also ignore earlier ones, and no run starts after SIGINT. - Pins: 1732's STUB runs the script on the real interpreter and `unsharded` accepts the earlier-shard ignores; 2005's test checks the invocation and each run's JUnit file. - workflows.rst: the PyPy paragraph and the Workflow Scripts row name the script; the `unit-tests.yml:NNNN` citations move 6 lines up (checked by content), here and in releasing.rst.
1 parent 2e735e1 commit bc4bb35

6 files changed

Lines changed: 49 additions & 39 deletions

File tree

‎.github/workflows/unit-tests.yml‎

Lines changed: 9 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -264,12 +264,13 @@ jobs:
264264
# at `-n 4` the `ALL` tier took each worker to 5.4-5.9 GB and them all
265265
# past 20 GB, measured on PyPy 8.0.1, where a 14 GB cap OOM-killed them
266266
# and a 3 GB `PYPY_GC_MAX` raised MemoryError instead. A smaller `-n`
267-
# only hands each worker more tests. So a PyPy leg runs pytest once per
268-
# directory `SHARDS` names, then once more over the rest, ignoring each
269-
# of them, and every run starts its workers afresh: the largest, with
270-
# tests/protocols, peaked at 2.6 GB a worker and 8.5 GB in all, and the
271-
# tier took 383 s rather than 493 s. Each run but the last writes a
272-
# JUnit file of its own, beside junit.xml; the step fails if any run did.
267+
# only hands each worker more tests. So on a PyPy leg util/pytest_shards.py
268+
# runs pytest once per directory `SHARDS` names, then once more over the
269+
# rest, ignoring each of them, and every run starts its workers afresh:
270+
# the largest, with tests/protocols, peaked at 2.6 GB a worker and 8.5 GB
271+
# in all, and the tier took 383 s rather than 493 s. Each run but the
272+
# last writes a JUnit file of its own, beside junit.xml; the step fails
273+
# if any run did.
273274
env:
274275
COVERAGE: ${{ matrix.coverage }}
275276
LATEST: ${{ matrix.python-version == needs.python-versions.outputs.latest }}
@@ -297,15 +298,8 @@ jobs:
297298
COVERAGE_DEBUG=pid,process COVERAGE_DEBUG_FILE="$PWD/coverage-processes.log" \
298299
python util/cap_file_size.py python -m coverage run --rcfile=.github/coverage.toml -m pytest "$@"
299300
elif [ -n "$SHARDS" ]; then
300-
status=0
301-
rest=()
302-
for shard in $SHARDS; do
303-
python util/cap_file_size.py python -m pytest "$@" \
304-
--junitxml="$RUNNER_TEMP/junit-${shard//\//-}.xml" "$shard" || status=$?
305-
rest+=(--ignore="$shard")
306-
done
307-
python util/cap_file_size.py python -m pytest "$@" "${rest[@]}" || status=$?
308-
exit "$status"
301+
# #2005: #1963's script, on the leg's interpreter, under the one cap.
302+
python util/cap_file_size.py python util/pytest_shards.py "$SHARDS" -- python -m pytest "$@"
309303
else
310304
python util/cap_file_size.py python -m pytest "$@"
311305
fi

‎docs/source/contributing/releasing.rst‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@ Editing ``__version__`` by hand without also moving :file:`CITATION.cff`'s
5252
(``:494-514``) asserts the two agree, and ``create-release.yml``'s
5353
``unit-tests`` job (``:581-597``) calls ``unit-tests.yml`` with
5454
``gate-only: true``, which runs the **full** suite rather than the tiered
55-
subset an ordinary push runs (its ``gate`` job, ``unit-tests.yml:1804-1879``).
55+
subset an ordinary push runs (its ``gate`` job, ``unit-tests.yml:1798-1873``).
5656
That gate runs whenever a release will -- a moved ``__version__`` is exactly
5757
what makes ``version_check``'s evidence read "not yet published" -- and every
5858
publishing job requires it to have succeeded (``:621,808,974,1210``), so a
@@ -62,7 +62,7 @@ Either run the script, or move both fields by hand in the same commit.
6262
This is the **only** edit a person makes to get a release started -- the tag,
6363
the Release, and every upload are the workflow's job from here. Other
6464
checks run unconditionally on the same path, though none is a step in *this*
65-
process: the ``changelog`` job (``unit-tests.yml:1647-1673``) fails outright if
65+
process: the ``changelog`` job (``unit-tests.yml:1641-1667``) fails outright if
6666
``CHANGELOG.md`` has drifted from its source entry under
6767
:file:`docs/source/changelog/`, and the release-body step warns, without
6868
failing, if that entry's heading still reads "unreleased"

‎docs/source/contributing/workflows.rst‎

Lines changed: 17 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -376,7 +376,7 @@ the same commit several times over. It also skips ``changes``,
376376
Checks`_ below), which are gated out by their own ``if:`` rather than by having
377377
already run. ``changelog`` carries no
378378
``if:`` at all and runs on every path regardless -- deliberately, per its own
379-
comment (``unit-tests.yml:1635-1641``): ``create-release.yml`` feeds
379+
comment (``unit-tests.yml:1629-1635``): ``create-release.yml`` feeds
380380
``CHANGELOG.md`` to the GitHub Release body, so the release path is exactly
381381
where a drifted file must not go unchecked. Confirmed on run `36210743295
382382
<https://github.com/JarryShaw/PyPCAPKit/actions/runs/36210743295>`__ (a
@@ -689,7 +689,7 @@ grepping every workflow file for its name:
689689
- Where
690690
* - ``Required checks passed``
691691
- job ``required-checks``
692-
- ``unit-tests.yml:1970``
692+
- ``unit-tests.yml:1964``
693693
* - ``Compat Python 3.11``
694694
- job ``compatibility``, matrix leg ``3.11``
695695
- ``python-compatibility.yml:60``, ``.github/python-versions.toml:20``
@@ -711,22 +711,22 @@ which added ``Compat Python 3.15`` and removed ``Compat Python 3.10``. Moving a
711711
between the tiers below means editing the ruleset in the same change.
712712

713713
``Required checks passed`` is defined by exactly one job
714-
(``unit-tests.yml:1970``). ``Compat Python`` is defined by one, the
714+
(``unit-tests.yml:1964``). ``Compat Python`` is defined by one, the
715715
``compatibility`` job in ``python-compatibility.yml`` (``:60``), whose matrix
716716
(``:74``) comes from :file:`.github/python-versions.toml`: the required legs
717717
are its supported tier (``python-versions.toml:20-24``), and its KTLO legs
718718
(``python-versions.toml:16-19``) have names ending in ``(KTLO)``, so no required
719719
context matches one. A comment at
720-
``unit-tests.yml:1882`` mentions the string without defining it.
720+
``unit-tests.yml:1876`` mentions the string without defining it.
721721

722722
``Required checks passed`` is an aggregate, not a single check run. The ruleset
723723
requires it and the ``Compat Python 3.11``-``3.15`` legs, nothing else, as
724-
``unit-tests.yml``'s own comment (``unit-tests.yml:1881-1891``) records. It
724+
``unit-tests.yml``'s own comment (``unit-tests.yml:1875-1885``) records. It
725725
``needs:`` the gating jobs of ``unit-tests.yml`` -- ``test``, ``integration``,
726726
``engine-tests`` and ``pypcap-parity``, plus ``project-tests``, the one run of
727727
the project pins, and ``changes`` for the docs-only path -- and checks each
728728
result explicitly
729-
(``unit-tests.yml:1969-2025``). Rulesets match check names literally, with no
729+
(``unit-tests.yml:1963-2019``). Rulesets match check names literally, with no
730730
wildcard, so requiring the matrix cells one by one would mean editing the
731731
ruleset whenever a matrix changes; the aggregate keeps the required list fixed.
732732
It accepts the gating jobs as ``skipped`` only on a docs-only pull request, where
@@ -735,7 +735,7 @@ It accepts the gating jobs as ``skipped`` only on a docs-only pull request, wher
735735
``python-compatibility.yml``, and a job cannot ``needs:`` a job in another
736736
workflow file. This job runs on Unit Tests' own ``push``/``pull_request``
737737
triggers; the ``gate-only: true`` reusable calls documented above skip it via
738-
``if: ${{ always() && inputs.gate-only != true }}`` (``unit-tests.yml:1972``),
738+
``if: ${{ always() && inputs.gate-only != true }}`` (``unit-tests.yml:1966``),
739739
so it is never produced -- and never expected -- on those paths.
740740

741741
.. _ktlo-tier:
@@ -759,7 +759,7 @@ and keeps its legs' check names, ``Python 3.10 (KTLO)`` and so on. In
759759
with supported legs. Every job holding KTLO legs works the same way:
760760

761761
* Every step that can fail on a KTLO leg has an ``id`` and
762-
``continue-on-error: ${{ matrix.ktlo == true }}`` (``unit-tests.yml:400-417``),
762+
``continue-on-error: ${{ matrix.ktlo == true }}`` (``unit-tests.yml:394-411``),
763763
unless it is masked on every leg, as the timings uploads are. Its ``outcome`` is
764764
then ``failure`` but its ``conclusion`` ``success``, so the job, its check run
765765
and ``required-checks`` read ``success``. A step that needs an earlier one's work
@@ -797,9 +797,10 @@ with the image's Rust and ``libssl-dev``, and none of ``lxml``, so the ``test``
797797
extra carries ``lxml`` on CPython only.
798798

799799
On PyPy an xdist worker's RSS grows with every test it runs, rather than in a few
800-
heavy modules, so a PyPy leg's ``Run unit tests`` runs pytest once per directory its
801-
``SHARDS`` names, ``tests/protocols`` and then ``tests/corekit``, and once more over
802-
the rest, ignoring each (`#2005 <https://github.com/JarryShaw/PyPCAPKit/issues/2005>`__).
800+
heavy modules, so a PyPy leg's ``Run unit tests`` hands its pytest command to
801+
:file:`util/pytest_shards.py`, as GraalPy's leg does (see below), which runs it once per
802+
directory its ``SHARDS`` names, ``tests/protocols`` and then ``tests/corekit``, and once
803+
more over the rest, ignoring each (`#2005 <https://github.com/JarryShaw/PyPCAPKit/issues/2005>`__).
803804
Each run starts its workers afresh. Measured on PyPy 8.0.1 at ``-n 4``, one run
804805
over the whole ``ALL`` tier took each worker to 5.4-5.9 GB and the workers together
805806
past 20 GB, beyond the runner's 16 GB; split, the largest run peaked at 2.6 GB a
@@ -1199,14 +1200,17 @@ does; each has tests under :file:`tests/project/` that run it.
11991200
- ``Report coverage`` in ``unit-tests.yml``'s ``test``, on the coverage leg.
12001201
* - :file:`util/pytest_shards.py`
12011202
- ``Run unit tests`` in ``experimental.yml``'s ``experimental``, on GraalPy's leg
1202-
(`#1963 <https://github.com/JarryShaw/PyPCAPKit/issues/1963>`__).
1203+
(`#1963 <https://github.com/JarryShaw/PyPCAPKit/issues/1963>`__), and in
1204+
``unit-tests.yml``'s ``test`` and ``test-ktlo``, on PyPy's legs (`#2005
1205+
<https://github.com/JarryShaw/PyPCAPKit/issues/2005>`__).
12031206

12041207
The summaries run on the image's ``/usr/bin/python3`` rather than the leg's
12051208
interpreter, which may have failed to set up or be too broken to run anything, so
12061209
they need only the standard library, as ``classify_changes.py`` does on the
12071210
runner's ``python3``. :file:`util/pytest_shards.py` runs on ``/usr/bin/python3``
12081211
too, standard library only, so on GraalPy's leg the leg's interpreter runs pytest
1209-
alone. Both ``changes`` jobs check out on every event, since the script is in the
1212+
alone; on PyPy's legs the leg's interpreter runs the script as well, under the
1213+
file-size cap every test command of ``unit-tests.yml`` takes. Both ``changes`` jobs check out on every event, since the script is in the
12101214
tree. ``create-release.yml``'s ``pypi`` job keeps its own copy of
12111215
the KTLO summary's shell until its shell moves to scripts too; a project pin runs
12121216
that copy and the script on the same steps and requires the same output.

‎tests/project/test_pypy_worker_memory_2005.py‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -91,7 +91,7 @@ def test_a_shard_that_fails_fails_the_step_after_the_rest_ran(self) -> None:
9191
with tempfile.TemporaryDirectory() as temp:
9292
stub = pathlib.Path(temp, 'python')
9393
# The first pytest run fails as pytest does on a failed test; `-c` asks the real interpreter.
94-
stub.write_text('#!/bin/sh\ncase "$1" in -c) exec "$REAL_PYTHON" "$@";; '
94+
stub.write_text('#!/bin/sh\ncase "$1" in -c|util/pytest_shards.py) exec "$REAL_PYTHON" "$@";; '
9595
'util/cap_file_size.py) shift; exec "$@";; esac\n'
9696
'echo run >> "$RUNS"\n'
9797
'[ "$(wc -l < "$RUNS")" -eq 1 ] && exit 1\nexit 0\n', encoding='utf-8')
@@ -112,9 +112,15 @@ class TestTheTimingsAreUploaded(unittest.TestCase):
112112
def test_the_upload_takes_each_runs_junit_file(self) -> None:
113113
upload = next(entry for entry in job(*JOB)['steps'] if entry.get('name') == 'Upload test timings')
114114
self.assertEqual(upload['with']['path'], '${{ runner.temp }}/junit*.xml')
115-
for shard in SHARDS:
115+
runs = run_step_each(pypy_leg())
116+
for shard, run in zip(SHARDS, runs):
116117
with self.subTest(shard=shard):
117-
self.assertIn(f'--junitxml="$RUNNER_TEMP/junit-${{shard//\\//-}}.xml" "$shard"', step()['run'])
118+
self.assertIn(f'--junitxml={RUNNER_TEMP}/junit-{shard.replace("/", "-")}.xml', run)
119+
120+
def test_the_shards_run_through_the_graalpy_script(self) -> None:
121+
"""#2005 reuses #1963's util/pytest_shards.py, on the leg's interpreter, under the one file-size cap."""
122+
self.assertIn('python util/cap_file_size.py python util/pytest_shards.py "$SHARDS" -- '
123+
'python -m pytest "$@"', step()['run'])
118124

119125

120126
class TestTheDocsSayIt(unittest.TestCase):

‎tests/project/test_sweeps_on_latest_only_1732.py‎

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -79,8 +79,10 @@
7979
#: no real interpreter: under the coverage leg one would be measured, be logged as a
8080
#: second ``coverage run --rcfile`` controller, and exec before saving, which fails the
8181
#: #1387 guard (#1969). A PyPy leg runs pytest more than once (#2005), so each run is
82-
#: recorded rather than the last.
83-
STUB = ('#!/bin/sh\ncase "$1" in -c) exec "$REAL_PYTHON" "$@";; util/cap_file_size.py) shift; exec "$@";; esac\n'
82+
#: recorded rather than the last, and util/pytest_shards.py, which starts them there,
83+
#: runs on the real one too.
84+
STUB = ('#!/bin/sh\ncase "$1" in -c|util/pytest_shards.py) exec "$REAL_PYTHON" "$@";; '
85+
'util/cap_file_size.py) shift; exec "$@";; esac\n'
8486
'printf \'%s\\n\' "$@" \'<end>\' >> "$ARGS_FILE"\n')
8587
#: The line :data:`STUB` writes after each run's arguments.
8688
END = '<end>'
@@ -172,8 +174,9 @@ def shards(leg: 'dict[str, Any]') -> 'list[str]':
172174
def unsharded(leg: 'dict[str, Any]', runs: 'list[list[str]]') -> 'list[str]':
173175
"""The arguments each of the ``test`` step's runs on ``leg`` shares, once each run's own are taken off.
174176
175-
#2005: a run per directory :func:`shards` names adds its JUnit file and the
176-
directory, and the last run an ``--ignore`` of each.
177+
#2005: util/pytest_shards.py's run per directory :func:`shards` names adds its JUnit
178+
file, an ``--ignore`` of each earlier shard, and the directory, and the last run an
179+
``--ignore`` of each.
177180
178181
"""
179182
named = shards(leg)
@@ -183,8 +186,8 @@ def unsharded(leg: 'dict[str, Any]', runs: 'list[list[str]]') -> 'list[str]':
183186
common = runs[-1][:len(runs[-1]) - len(rest)]
184187
if runs[-1][len(common):] != rest:
185188
raise AssertionError(f'the last run on {leg} does not end by ignoring {named}: {runs[-1]}')
186-
for shard, run in zip(named, runs):
187-
own = [f'--junitxml={RUNNER_TEMP}/junit-{shard.replace("/", "-")}.xml', shard]
189+
for index, (shard, run) in enumerate(zip(named, runs)):
190+
own = [f'--junitxml={RUNNER_TEMP}/junit-{shard.replace("/", "-")}.xml', *rest[:index], shard]
188191
if run != common + own:
189192
raise AssertionError(f'the {shard} run on {leg} is not the others\' arguments and {own}: {run}')
190193
return common

‎util/pytest_shards.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,9 @@
99
1010
/usr/bin/python3 util/pytest_shards.py "$SHARDS" -- python -m pytest ...
1111
12+
unit-tests.yml's ``Run unit tests`` runs it the same way on PyPy's legs, whose
13+
xdist workers' RSS grows with every test as well (#2005).
14+
1215
``SHARDS`` holds the shards, separated by whitespace, and each is a directory or
1316
a glob of the entries of one directory, such as ``tests/corekit/test_i*``.
1417
The command runs once per shard, then once more over the rest, and each run

0 commit comments

Comments
 (0)