Skip to content

Commit 8cd480d

Browse files
authored
ci(unit-tests): run platform-independent project pins once per push (#1954) (#1956)
- tests/_tiers.py: is_project_pin() and project_pin_paths(), the one rule. A tests/project module is a pin unless its AST, or that of the repository code it runs, imports pcapkit or compares the interpreter version. - unit-tests.yml: project-tests runs the pins under pytest on every pull request and push, but not under gate-only, and required-checks needs it; every test and test-ktlo leg ignores them. The docs-only steps stay. - experimental.yml: lists the pins on the runner's python3 and ignores them. - tests/_dependency_gates.py: models project-tests as its own selection. - Tier-guard tests run the steps against the rule; the 1732, queue, 1938 and 1672 tests, the workflows page and its citations follow the new layout. Closes #1954
1 parent b2cd140 commit 8cd480d

12 files changed

Lines changed: 1108 additions & 83 deletions

‎.github/workflows/experimental.yml‎

Lines changed: 21 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -242,6 +242,20 @@ jobs:
242242
id: checkout
243243
continue-on-error: true
244244

245+
# #1954: the project pins run once per push, in unit-tests.yml's
246+
# `project-tests`, so these legs leave them out, as `test`'s do. The
247+
# runner's own `python3` asks tests/_tiers.py, which needs only the
248+
# standard library, before setup-python puts the leg's interpreter on
249+
# PATH, so GraalPy does not parse the tree. Where it fails, the tests
250+
# run the pins too, and `Experimental summary` names this step.
251+
- name: List the project pins
252+
id: pins
253+
if: ${{ steps.checkout.outcome == 'success' }}
254+
continue-on-error: true
255+
run: |
256+
pins=$(python3 -c "from tests._tiers import project_pin_paths; print(' '.join(project_pin_paths()))")
257+
echo "paths=$pins" >> "$GITHUB_OUTPUT"
258+
245259
- uses: actions/setup-python@v7
246260
id: python
247261
continue-on-error: true
@@ -447,6 +461,7 @@ jobs:
447461
GH_TOKEN: ${{ github.token }}
448462
WATCHER: ${{ steps.watcher.outcome }}
449463
WATCHER_PYTHON: ${{ steps.watcher-python.outputs.python-path }}
464+
PINS: ${{ steps.pins.outputs.paths }}
450465
run: |
451466
tail -n +1 -F "$RUNNER_TEMP/resource-monitor.log" 2>/dev/null &
452467
monitor=$!
@@ -455,8 +470,12 @@ jobs:
455470
--durations=30 --durations-min=1.0 --junitxml="$RUNNER_TEMP/junit.xml" \
456471
--ignore=tests/integration \
457472
--ignore-glob='*_runtime.py' \
458-
--ignore-glob='*_regression.py' \
459-
--ignore-glob='*roundtrip*'
473+
--ignore-glob='*_regression.py'
474+
# #1954: the project pins, from `List the project pins`.
475+
for pin in $PINS; do
476+
set -- "$@" --ignore="$pin"
477+
done
478+
set -- "$@" --ignore-glob='*roundtrip*'
460479
set -- prlimit --fsize=2147483648 timeout --signal=INT --kill-after=3m 110m python -m pytest "$@"
461480
scoped() {
462481
unit=$1

‎.github/workflows/unit-tests.yml‎

Lines changed: 91 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -222,15 +222,29 @@ jobs:
222222
# collects, with `*` spanning `/` (_pytest/main.py:437-461, pytest
223223
# 9.1.1). The modules that import the sweeps' harnesses for their
224224
# packet builders still import them.
225+
#
226+
# #1954, the owner's option 3: the project pins -- the tests/project
227+
# modules whose run does not vary by interpreter, tests/_tiers.py's
228+
# project_pin_paths() -- run once per push, in `project-tests`, so every
229+
# leg passes each as an `--ignore`, as experimental.yml's do. Asked of
230+
# tests/_tiers.py here rather than listed, as `integration` asks for its
231+
# selection. The pins import no pcapkit, so coverage loses nothing.
225232
env:
226233
COVERAGE: ${{ matrix.coverage }}
227234
SWEEPS: ${{ matrix.python-version == needs.python-versions.outputs.latest }}
228235
run: |
236+
if ! pins=$(python -c "from tests._tiers import project_pin_paths; print(' '.join(project_pin_paths()))"); then
237+
echo "::${ANNOTATION_LEVEL} title=Could not compute the project pins::tests._tiers.project_pin_paths() failed -- see the traceback above."
238+
exit 1
239+
fi
229240
set -- -q -n auto --dist load \
230241
--durations=30 --durations-min=1.0 --junitxml="$RUNNER_TEMP/junit.xml" \
231242
--ignore=tests/integration \
232243
--ignore-glob='*_runtime.py' \
233244
--ignore-glob='*_regression.py'
245+
for pin in $pins; do
246+
set -- "$@" --ignore="$pin"
247+
done
234248
if [ "$SWEEPS" != "true" ]; then
235249
set -- "$@" --ignore-glob='*roundtrip*'
236250
fi
@@ -1513,6 +1527,12 @@ jobs:
15131527
# covering none of tests/integration. Adding it back here would buy that
15141528
# nothing, not a failure.
15151529
#
1530+
# The `project` cell keeps the project pins (#1954), which `test` leaves to
1531+
# `project-tests`: this job asks whether each module passes in one
1532+
# interpreter, in file order, with no xdist to isolate it, which is a
1533+
# question about the module and not the Python version, so a pin can fail
1534+
# it as readily as any other. It runs on main's pushes alone, one cell.
1535+
#
15161536
# Not added to ``required-checks`` below: a brand-new job earns a place in
15171537
# the required set once its own track record justifies it, which is the
15181538
# maintainer's call to make separately from landing it.
@@ -1721,23 +1741,32 @@ jobs:
17211741
print(f'code={code}', file=file)
17221742
PY
17231743
1724-
# The docs-only replacement for `test`, running every test that reads
1725-
# docs/ or a Markdown file, since a docs-only diff can still break those:
1726-
# tests/project and the root-level modules, plus the modules elsewhere under
1727-
# tests/ named in the second run step. tests/project/test_unit_tests_queue.py
1728-
# finds every module outside those that names a docs/ or Markdown path
1729-
# and fails until it is listed there, so a new reader cannot go unrun. Runs
1730-
# only when `changes` says the diff is docs-only, so it never duplicates
1731-
# `test` on a code change. It uses
1744+
# #1954, the owner's option 3: the project pins, once per push. They are the
1745+
# tests/project modules whose run does not vary by interpreter -- each reads
1746+
# the repository's text or drives a util/ script, and none imports pcapkit
1747+
# or branches on the Python version, by tests/_tiers.py's
1748+
# is_project_pin() -- so one run on `latest` is every leg's answer, and
1749+
# `test` and experimental.yml leave them out. On every pull request and
1750+
# every push, `required-checks` needs this job to pass; under `gate-only`,
1751+
# `gate`'s full suite runs them instead. tests/_dependency_gates.py reads its
1752+
# pytest step as the `project-pins` selection.
1753+
#
1754+
# On a docs-only diff, it is `test`'s replacement too, running every test
1755+
# that reads docs/ or a Markdown file, since a docs-only diff can still break
1756+
# those: tests/project and the root-level modules, plus the modules
1757+
# elsewhere under tests/ named in the last run step.
1758+
# tests/project/test_unit_tests_queue.py finds every module outside those
1759+
# that names a docs/ or Markdown path and fails until it is listed there, so
1760+
# a new reader cannot go unrun. Those steps run only when `changes` says the
1761+
# diff is docs-only, so they never duplicate `test` on a code change, and
1762+
# they take in the pins, so the pins step is skipped there. They use
17321763
# util/run_unittest_leg.py's `project` leg -- tests/project plus the
17331764
# root-level modules, with the extras `unittest-ordering`'s `project` cell
1734-
# runs green on main -- rather than a pytest invocation of its own, which
1735-
# tests/_dependency_gates.py would then have to model as a new selection
1736-
# shape.
1765+
# runs green on main.
17371766
project-tests:
1738-
name: Project tests (docs-only)
1767+
name: Project tests
17391768
needs: [changes]
1740-
if: ${{ inputs.gate-only != true && needs.changes.outputs.code == 'false' }}
1769+
if: ${{ inputs.gate-only != true }}
17411770
runs-on: ubuntu-latest
17421771
timeout-minutes: 20
17431772

@@ -1754,16 +1783,51 @@ jobs:
17541783
python-version: ${{ steps.python-versions.outputs.latest }}
17551784
cache: pip
17561785

1786+
# `test`'s install line, so a gated pin skips or runs here exactly as it
1787+
# did on `latest`.
17571788
- name: Install package and test dependencies
17581789
run: |
17591790
python -m pip install -U pip setuptools wheel
17601791
python -m pip install -e '.[test,DPKT,crypto,NGAP]'
17611792
1793+
# See the `test` job above for why this step exists.
1794+
- name: Report available parallelism
1795+
run: |
1796+
nproc
1797+
python -c "import os; print('cpu_count', os.cpu_count())"
1798+
1799+
# `test`'s pytest flags, with the pins as the selection. A selection that
1800+
# cannot be computed, or is empty, fails the step rather than letting a
1801+
# bare `pytest` run the whole suite. `ulimit -f` caps any file a test
1802+
# writes at 2 GiB (bash counts 1024-byte blocks), as experimental.yml's
1803+
# `prlimit --fsize` does. The step cap is several times the run's
1804+
# length, under the job's 20.
1805+
- name: Run the project pins
1806+
if: ${{ needs.changes.outputs.code != 'false' }}
1807+
timeout-minutes: 10
1808+
shell: bash
1809+
run: |
1810+
ulimit -f 2097152
1811+
if ! pins=$(python -c "from tests._tiers import project_pin_paths; print(' '.join(project_pin_paths()))"); then
1812+
echo "::error title=Could not compute the project pins::tests._tiers.project_pin_paths() failed -- see the traceback above."
1813+
exit 1
1814+
fi
1815+
if [ -z "$pins" ]; then
1816+
echo "::error title=No project pins::tests._tiers.project_pin_paths() returned nothing. Refusing to run pytest with no arguments, which would run the whole suite."
1817+
exit 1
1818+
fi
1819+
# shellcheck disable=SC2086
1820+
set -- $pins
1821+
python -m pytest -q -n auto --dist load \
1822+
--durations=30 --durations-min=1.0 --junitxml="$RUNNER_TEMP/junit.xml" "$@"
1823+
17621824
- name: Run tests/project and the root-level modules under plain unittest
1825+
if: ${{ needs.changes.outputs.code == 'false' }}
17631826
timeout-minutes: 15
17641827
run: python util/run_unittest_leg.py project --verbose
17651828

17661829
- name: Run the other tests that read docs/ or Markdown
1830+
if: ${{ needs.changes.outputs.code == 'false' }}
17671831
timeout-minutes: 5
17681832
run: python -m unittest -v tests.corekit.test_sentinel_exports_unit tests.utilities.test_config_unit tests.vendor.test_vendor_variables_1726_unit
17691833

@@ -1902,8 +1966,8 @@ jobs:
19021966
# gating leg, and folding it in would widen what merging requires -- a
19031967
# change for the maintainer to decide, not one to slip in here.
19041968
# `test`, `integration`, `engine-tests` and `pypcap-parity` are the gating
1905-
# legs; `changes` and `project-tests` are needed only for the docs-only
1906-
# path below.
1969+
# legs, and `project-tests` gates every diff too, as the one run of the
1970+
# project pins (#1954); `changes` is needed for the docs-only path below.
19071971
#
19081972
# `engine-tests`'s `unsupported` and `not-installable` cells are not a gap
19091973
# here: the cell script exits 0 for a cell that declined exactly as
@@ -1924,13 +1988,13 @@ jobs:
19241988
# `docs/**` and `*.md` files outside the code directories (the `changes`
19251989
# job's classifier), `test`, `integration`, `engine-tests` and
19261990
# `pypcap-parity` are skipped by their own `if:`, and `project-tests` runs
1927-
# instead. A job-level `if:` rather than workflow-level `paths-ignore`,
1928-
# because `paths-ignore` would not run this workflow at all and leave this
1929-
# required context pending forever. So `skipped` is accepted for those
1930-
# legs **only** when `changes` succeeded and said `code == 'false'`, and then
1931-
# `project-tests` must have succeeded. Every other combination keeps the
1932-
# strict rule: `changes` failing, its output missing, or any leg skipped
1933-
# while `code` is `true` fails this gate. `changes` reports `code: true` for
1991+
# the docs readers as well. A job-level `if:` rather than workflow-level
1992+
# `paths-ignore`, because `paths-ignore` would not run this workflow at all
1993+
# and leave this required context pending forever. So `skipped` is accepted
1994+
# for those legs **only** when `changes` succeeded and said
1995+
# `code == 'false'`; `project-tests` must have succeeded either way. Every
1996+
# other combination keeps the strict rule: `changes` failing, its output
1997+
# missing, or any leg skipped while `code` is `true` fails this gate. `changes` reports `code: true` for
19341998
# every event but this workflow's own `pull_request`, so `push` and
19351999
# `workflow_call` runs never take this path.
19362000
required-checks:
@@ -1966,11 +2030,13 @@ jobs:
19662030
failed=1
19672031
fi
19682032
1969-
# Docs-only (code == false): a leg may be skipped, and project-tests
1970-
# is what must have passed. Otherwise every leg must have succeeded.
2033+
# project-tests runs the project pins on every diff (#1954), and the
2034+
# docs readers on a docs-only one, so it must pass on both paths.
2035+
# Docs-only (code == false): a leg may be skipped. Otherwise every leg
2036+
# must have succeeded.
2037+
[ "$PROJECT_TESTS" = "success" ] || { echo "::error title=project-tests did not succeed::result was '$PROJECT_TESTS'; it runs the project pins, and on a docs-only diff the docs readers too, so it has to pass."; failed=1; }
19712038
if [ "$CODE" = "false" ]; then
19722039
allowed="success skipped"
1973-
[ "$PROJECT_TESTS" = "success" ] || { echo "::error title=project-tests did not succeed::result was '$PROJECT_TESTS'; on a docs-only diff it is the test that has to pass."; failed=1; }
19742040
else
19752041
allowed="success"
19762042
fi

‎docs/source/contributing/releasing.rst‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@ Editing ``__version__`` by hand without also moving :file:`CITATION.cff`'s
5252
(``:494-514``) asserts the two agree, and ``create-release.yml``'s
5353
``unit-tests`` job (``:581-597``) calls ``unit-tests.yml`` with
5454
``gate-only: true``, which runs the **full** suite rather than the tiered
55-
subset an ordinary push runs (its ``gate`` job, ``unit-tests.yml:1776-1846``).
55+
subset an ordinary push runs (its ``gate`` job, ``unit-tests.yml:1840-1910``).
5656
That gate runs whenever a release will -- a moved ``__version__`` is exactly
5757
what makes ``version_check``'s evidence read "not yet published" -- and every
5858
publishing job requires it to have succeeded (``:621,808,974,1210``), so a
@@ -62,7 +62,7 @@ Either run the script, or move both fields by hand in the same commit.
6262
This is the **only** edit a person makes to get a release started -- the tag,
6363
the Release, and every upload are the workflow's job from here. Other
6464
checks run unconditionally on the same path, though none is a step in *this*
65-
process: the ``changelog`` job (``unit-tests.yml:1624-1650``) fails outright if
65+
process: the ``changelog`` job (``unit-tests.yml:1644-1670``) fails outright if
6666
``CHANGELOG.md`` has drifted from its source entry under
6767
:file:`docs/source/changelog/`, and the release-body step warns, without
6868
failing, if that entry's heading still reads "unreleased"

0 commit comments

Comments
 (0)