Skip to content

Commit 06e16db

Browse files
authored
ci(release): drop fixed totals and conda-leg installs, smoke-test releases (#1849, #1854, #1855) (#1868)
1 parent 27fa51e commit 06e16db

6 files changed

Lines changed: 330 additions & 11 deletions

File tree

‎.github/workflows/create-release.yml‎

Lines changed: 19 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1338,21 +1338,23 @@ jobs:
13381338
with:
13391339
python-version: ${{ matrix.python-version }}
13401340

1341+
# #1854: nothing below imports pcapkit from this interpreter. ``pip
1342+
# download`` needs pip alone, ``conda build`` installs into its own host
1343+
# environment, the version comes from job outputs, and the smoke test
1344+
# runs its probe under ``-I -S``. So pcapkit and its extras are not
1345+
# installed here: on ``macos-26-intel`` they resolve a cryptography with
1346+
# no Intel wheel, which pip has to compile or fail on.
13411347
- name: Install and Setup
13421348
if: ${{ matrix.os != 'windows-latest' }}
13431349
run: |
13441350
set -x
13451351
13461352
python -m pip install -U pip setuptools wheel
1347-
python -m pip install -e .[all]
1348-
python -m pip install pathlib2 typing_extensions
13491353
13501354
- name: Install and Setup (Windows)
13511355
if: ${{ matrix.os == 'windows-latest' }}
13521356
run: |
13531357
python -m pip install -U pip setuptools wheel
1354-
python -m pip install -e .[all]
1355-
python -m pip install pathlib2 typing_extensions
13561358
13571359
- name: Setup Conda Distribution
13581360
if: ${{ matrix.os != 'windows-latest' }}
@@ -1435,6 +1437,8 @@ jobs:
14351437
;;
14361438
esac
14371439
1440+
echo "host_package=$host_package" >> "$GITHUB_OUTPUT"
1441+
14381442
# #1822: never ``conda convert`` across architectures; see the matrix.
14391443
host_platform="$(basename "$(dirname "$host_package")")"
14401444
if [ "$host_platform" != "$target_platform" ]; then
@@ -1457,6 +1461,16 @@ jobs:
14571461
echo EOF
14581462
} >> "$GITHUB_OUTPUT"
14591463
1464+
# #1855: cron-conda.yml's ``conda-dist`` step of the same name, which says
1465+
# what it checks and why it runs setup-python's interpreter, so a release
1466+
# package is checked as a Conda Update bump's is, before anything below
1467+
# uploads it. Not gated on ``check_leg``, since the GitHub upload is not.
1468+
- name: Smoke-test the vendored packages
1469+
env:
1470+
PCAPKIT_CONDA_PACKAGE: ${{ steps.conda_build.outputs.host_package }}
1471+
run: |
1472+
"${pythonLocation}/bin/python" util/conda_vendor.py --smoke "$PCAPKIT_CONDA_PACKAGE"
1473+
14601474
- name: Upload conda packages to Anaconda
14611475
if: ${{ steps.check_leg.outputs.PCAPKIT_LEG_EXISTS != 'true' }}
14621476
uses: anaconda/actions/upload-package@v0.3.1
@@ -1690,7 +1704,7 @@ jobs:
16901704
# Case 2: every target reconciled by skipping, none of them ran.
16911705
# Quiet -- this is the steady state most runs are in.
16921706
if [ "$all_reconciled" = "true" ] && [ "$all_skipped" = "true" ]; then
1693-
echo "::notice title=Nothing to release::pypcapkit ${PCAPKIT_VERSION} is already fully released -- the v* tag, the conda-*+0 tag, all 7 supported PyPI files and all 10 Anaconda files are present. This run correctly released nothing."
1707+
echo "::notice title=Nothing to release::pypcapkit ${PCAPKIT_VERSION} is already fully released -- the v* tag, the conda-*+0 tag and every PyPI and Anaconda file version_check expects are present. This run correctly released nothing."
16941708
exit 0
16951709
fi
16961710

‎.github/workflows/cron-conda.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -445,19 +445,19 @@ jobs:
445445
with:
446446
python-version: ${{ matrix.python-version }}
447447

448+
# #1854: pcapkit is not installed here; see create-release.yml's step of
449+
# the same name.
448450
- name: Install and Setup
449451
if: ${{ matrix.os != 'windows-latest' }}
450452
run: |
451453
set -x
452454
453455
python -m pip install -U pip setuptools wheel
454-
python -m pip install -e .[all]
455456
456457
- name: Install and Setup (Windows)
457458
if: ${{ matrix.os == 'windows-latest' }}
458459
run: |
459460
python -m pip install -U pip setuptools wheel
460-
python -m pip install -e .[all]
461461
462462
# #1827: pip evaluates conda/requirements.txt's platform markers here,
463463
# on this leg's own runner, so Intel macOS fetches its own cryptography

‎docs/source/contributing/releasing.rst‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -84,7 +84,7 @@ a workaround for another:
8484
of *that* tree (``:327``). Every downstream tag and release name is built
8585
from that output -- ``v${{ needs.version_check.outputs.PCAPKIT_VERSION }}``
8686
is ``github``'s ``tag_name`` at ``:719`` and the release ``pypi`` and
87-
``conda`` upload to at ``:1117`` and ``:1479``, always from
87+
``conda`` upload to at ``:1117`` and ``:1493``, always from
8888
``PCAPKIT_VERSION``, never from ``github.ref_name`` -- so the version
8989
actually released is whatever ``__version__`` says on the tagged commit,
9090
not the name of the tag that was pushed. The two agree only if the tag
@@ -162,12 +162,12 @@ the default branch, and ``tag`` pulled ``main`` before tagging.
162162
naming another commit skips the gate and every publisher
163163
(`#1748 <https://github.com/JarryShaw/PyPCAPKit/issues/1748>`__):
164164
``Release status`` posts a ``::notice`` if that release is complete, or a
165-
``::warning`` naming both commits and what is missing (``:1624-1634``), and
165+
``::warning`` naming both commits and what is missing (``:1638-1648``), and
166166
the run stays green. A ``v*`` push of such a version still fails with the
167167
check's ``::error``. A partial release is finished by re-running the run that
168168
tagged it.
169169
* ``pypi`` and ``conda`` attach their files with ``gh release upload``
170-
(``:1110-1147``, ``:1474-1509``), not ``action-gh-release``, which calls Update
170+
(``:1110-1147``, ``:1488-1523``), not ``action-gh-release``, which calls Update
171171
a release before every upload (3.0.2's ``src/github.ts:631-683``). A file
172172
the release already has is left alone
173173
(`#1640 <https://github.com/JarryShaw/PyPCAPKit/issues/1640>`__): the action
@@ -296,7 +296,7 @@ the release test gate, ask for a second approval and race the first run's
296296
uploads from another concurrency group, so ``version_check`` skips a ``push``
297297
whose ``github.actor_id`` is ``vars.RELEASE_APP_BOT_ID`` (``:290-294``), and
298298
``release_status`` reports it with ``::notice title=Nothing to release``
299-
(``:1586-1588``). A tag pushed by a person runs as before. Whether a tag made
299+
(``:1600-1602``). A tag pushed by a person runs as before. Whether a tag made
300300
through the Releases API fires ``push``, and with the bot as actor, is
301301
unmeasured; if not, the guard never matches and the cost is one approval to
302302
reject.
Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
# -*- coding: utf-8 -*-
2+
"""The conda legs do not install pcapkit into setup-python's interpreter (#1854).
3+
4+
``conda-dist`` in :file:`.github/workflows/cron-conda.yml` and ``conda`` in
5+
:file:`.github/workflows/create-release.yml` ran ``pip install -e .[all]``, and
6+
the release leg ``pip install pathlib2 typing_extensions`` too, for no later
7+
step: ``pip download`` needs pip alone, ``conda build`` installs into its own
8+
host environment, the version comes from job outputs, and ``--smoke`` runs its
9+
probe under ``-I -S``. On ``macos-26-intel`` the extras resolve a cryptography
10+
with no Intel wheel, which pip has to compile or fail on. The jobs are read
11+
with :func:`yaml.safe_load`.
12+
13+
"""
14+
15+
from __future__ import annotations
16+
17+
import pathlib
18+
import re
19+
import shlex
20+
import unittest
21+
from typing import TYPE_CHECKING
22+
23+
try:
24+
import yaml
25+
except ImportError: # pragma: no cover
26+
yaml = None # type: ignore[assignment]
27+
28+
if TYPE_CHECKING:
29+
from typing import Any
30+
31+
WORKFLOWS = pathlib.Path(__file__).resolve().parents[2] / '.github' / 'workflows'
32+
#: The conda jobs, as (workflow, job).
33+
JOBS = (('cron-conda.yml', 'conda-dist'), ('create-release.yml', 'conda'))
34+
#: What each install step still upgrades: ``pip download`` runs on this pip.
35+
BUILD_TOOLS = ['python', '-m', 'pip', 'install', '-U', 'pip', 'setuptools', 'wheel']
36+
#: The backports #1821 removed from ``cron-conda.yml``.
37+
BACKPORTS = {'pathlib2', 'typing_extensions', 'typing-extensions'}
38+
#: A step running pcapkit from setup-python's interpreter, which would need it installed.
39+
USES_PCAPKIT = re.compile(r'\bimport\s+pcapkit\b|-m\s+pcapkit\b|\bpcapkit-(?:cli|vendor)\b')
40+
41+
42+
def commands(step: 'dict[str, Any]') -> 'list[str]':
43+
"""The step's ``run`` lines that are not blank or comments."""
44+
return [line.strip() for line in str(step.get('run', '')).splitlines()
45+
if line.strip() and not line.lstrip().startswith('#')]
46+
47+
48+
def installs(step: 'dict[str, Any]') -> 'list[list[str]]':
49+
"""Each ``python -m pip install`` command of the step, split as the shell splits it."""
50+
found = []
51+
for line in commands(step):
52+
if re.search(r'\bpip\s+install\b', line):
53+
words = shlex.split(line)
54+
if words[1:4] != ['-m', 'pip', 'install']:
55+
raise AssertionError(f'not run as `python -m pip install`, so not read here: {line}')
56+
found.append(words)
57+
return found
58+
59+
60+
@unittest.skipIf(yaml is None, 'PyYAML, from the test extra, is not installed')
61+
class TestTheCondaLegsInstallOnlyTheBuildTools(unittest.TestCase):
62+
"""setup-python's interpreter gets pip, setuptools and wheel, and nothing to import."""
63+
64+
def steps(self) -> 'list[tuple[str, str, dict[str, Any]]]':
65+
found = []
66+
for name, job_id in JOBS:
67+
spec = yaml.safe_load((WORKFLOWS / name).read_text(encoding='utf-8'))['jobs'][job_id]
68+
found.extend((name, job_id, step) for step in spec['steps'])
69+
return found
70+
71+
def test_no_leg_installs_the_project(self) -> None:
72+
for name, job_id, step in self.steps():
73+
for words in installs(step):
74+
targets = [word for word in words[4:] if not word.startswith('-')]
75+
with self.subTest(workflow=name, job=job_id, step=step.get('name'), command=words):
76+
self.assertFalse([word for word in targets if word == '.' or word.startswith('.[')])
77+
78+
def test_no_leg_installs_the_backports(self) -> None:
79+
for name, job_id, step in self.steps():
80+
for words in installs(step):
81+
with self.subTest(workflow=name, job=job_id, step=step.get('name'), command=words):
82+
self.assertFalse(BACKPORTS.intersection(words))
83+
84+
def test_each_install_step_upgrades_the_build_tools_alone(self) -> None:
85+
for name, job_id in JOBS:
86+
steps = [step for found, job, step in self.steps() if (found, job) == (name, job_id)
87+
and str(step.get('name', '')).startswith('Install and Setup')]
88+
with self.subTest(workflow=name, job=job_id):
89+
self.assertEqual(len(steps), 2, 'the Linux/macOS step and the Windows one')
90+
for step in steps:
91+
with self.subTest(workflow=name, job=job_id, step=step['name']):
92+
self.assertEqual(installs(step), [BUILD_TOOLS])
93+
94+
def test_no_step_runs_pcapkit_from_that_interpreter(self) -> None:
95+
"""What would need the install back: an import, ``-m pcapkit`` or an entry point."""
96+
for name, job_id, step in self.steps():
97+
with self.subTest(workflow=name, job=job_id, step=step.get('name')):
98+
self.assertFalse([line for line in commands(step) if USES_PCAPKIT.search(line)])
99+
100+
101+
if __name__ == '__main__':
102+
unittest.main()
Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
# -*- coding: utf-8 -*-
2+
"""The release conda leg smoke-tests its package as Conda Update does (#1855).
3+
4+
#1821 added ``util/conda_vendor.py --smoke`` to ``conda-dist`` in
5+
:file:`.github/workflows/cron-conda.yml`, after ``conda build`` and before any
6+
upload. ``conda`` in :file:`.github/workflows/create-release.yml`, which
7+
publishes the release packages, had no such step, and its build step did not
8+
write the ``host_package`` output the step reads. The jobs are read with
9+
:func:`yaml.safe_load`, and each build step is run with a stand-in ``conda``,
10+
as ``test_conda_native_platforms_1822.py`` runs it.
11+
12+
"""
13+
14+
from __future__ import annotations
15+
16+
import os
17+
import pathlib
18+
import re
19+
import stat
20+
import subprocess # nosec: B404
21+
import tempfile
22+
import unittest
23+
from typing import TYPE_CHECKING
24+
25+
from tests._support import scale_timeout
26+
from tests.project.test_conda_native_platforms_1822 import BASH, CONDA
27+
28+
try:
29+
import yaml
30+
except ImportError: # pragma: no cover
31+
yaml = None # type: ignore[assignment]
32+
33+
if TYPE_CHECKING:
34+
from typing import Any
35+
36+
WORKFLOWS = pathlib.Path(__file__).resolve().parents[2] / '.github' / 'workflows'
37+
#: The conda jobs, as (workflow, job); the first is the one the release mirrors.
38+
CRON, RELEASE = ('cron-conda.yml', 'conda-dist'), ('create-release.yml', 'conda')
39+
#: The steps that publish what the leg built.
40+
UPLOADS = ('Upload conda packages to Anaconda', 'Upload package to GitHub Release')
41+
42+
43+
def steps(workflow: 'str', job_id: 'str') -> 'list[dict[str, Any]]':
44+
return yaml.safe_load((WORKFLOWS / workflow).read_text(encoding='utf-8'))['jobs'][job_id]['steps']
45+
46+
47+
def smoke_steps(found: 'list[dict[str, Any]]') -> 'list[int]':
48+
"""The index of each step that runs ``util/conda_vendor.py --smoke``."""
49+
return [index for index, step in enumerate(found)
50+
if re.search(r'\butil/conda_vendor\.py\s+--smoke\b', str(step.get('run', '')))]
51+
52+
53+
@unittest.skipIf(yaml is None, 'PyYAML, from the test extra, is not installed')
54+
class TestTheReleaseLegSmokeTests(unittest.TestCase):
55+
"""The release leg checks its package before publishing it."""
56+
57+
def test_the_smoke_test_runs_after_the_build_and_before_every_upload(self) -> None:
58+
for workflow, job_id in (CRON, RELEASE):
59+
found = steps(workflow, job_id)
60+
smoke = smoke_steps(found)
61+
with self.subTest(workflow=workflow, job=job_id):
62+
self.assertEqual(len(smoke), 1, f'{workflow} {job_id} runs no smoke test')
63+
build = next(index for index, step in enumerate(found) if step.get('id') == 'conda_build')
64+
self.assertGreater(smoke[0], build)
65+
uploads = [index for index, step in enumerate(found) if step.get('name') in UPLOADS]
66+
self.assertEqual(len(uploads), len(UPLOADS))
67+
self.assertLess(smoke[0], min(uploads))
68+
69+
def test_it_is_conda_updates_step(self) -> None:
70+
"""The same interpreter, package and gate: setup-python's, ``host_package``, none."""
71+
cron = steps(*CRON)
72+
release = steps(*RELEASE)
73+
expected = cron[smoke_steps(cron)[0]]
74+
self.assertEqual(expected['env'], {'PCAPKIT_CONDA_PACKAGE': '${{ steps.conda_build.outputs.host_package }}'})
75+
self.assertIn('"${pythonLocation}/bin/python" util/conda_vendor.py --smoke', expected['run'])
76+
found = smoke_steps(release)
77+
self.assertEqual(len(found), 1, 'the release leg runs no smoke test')
78+
self.assertEqual(release[found[0]], expected)
79+
80+
81+
@unittest.skipIf(yaml is None, 'PyYAML, from the test extra, is not installed')
82+
@unittest.skipIf(not BASH, 'needs bash on PATH')
83+
class TestEachBuildStepNamesItsPackage(unittest.TestCase):
84+
"""``host_package``, which the smoke step reads, is the package the step built."""
85+
86+
def outputs(self, workflow: 'str', job_id: 'str') -> 'tuple[str, str]':
87+
"""The build step's ``$GITHUB_OUTPUT`` on ``ubuntu-latest``, and its log."""
88+
step = next(step for step in steps(workflow, job_id) if step.get('id') == 'conda_build')
89+
script = step['run'].replace('${{ matrix.os }}', 'ubuntu-latest').replace('${{ matrix.python-version }}',
90+
'3.14')
91+
self.assertNotIn('${{', script)
92+
with tempfile.TemporaryDirectory() as temp:
93+
scratch = pathlib.Path(temp)
94+
(scratch / 'bin').mkdir()
95+
conda = scratch / 'bin' / 'conda'
96+
conda.write_text(CONDA, encoding='utf-8')
97+
conda.chmod(conda.stat().st_mode | stat.S_IXUSR)
98+
output = scratch / 'output'
99+
output.write_text('', encoding='utf-8')
100+
result = subprocess.run(
101+
[BASH, '-e', '-c', script], cwd=temp, # nosec: B603
102+
env=dict(os.environ, PATH=f'{scratch / "bin"}{os.pathsep}{os.environ["PATH"]}',
103+
TMPDIR=temp, GITHUB_OUTPUT=str(output), FAKE_SUBDIR='linux-64',
104+
PCAPKIT_VERSION='1.0', PCAPKIT_BUILD='0'),
105+
capture_output=True, text=True, check=False, timeout=scale_timeout(60))
106+
return output.read_text(encoding='utf-8'), result.stdout + result.stderr
107+
108+
def test_host_package_is_the_package_built(self) -> None:
109+
for workflow, job_id in (CRON, RELEASE):
110+
written, log = self.outputs(workflow, job_id)
111+
with self.subTest(workflow=workflow, job=job_id):
112+
host = re.findall(r'^host_package=(.*)$', written, re.MULTILINE)
113+
self.assertEqual(len(host), 1, f'no host_package output\n{written}{log}')
114+
packages = re.search(r'^packages<<EOF\n(.*?)^EOF$', written, re.MULTILINE | re.DOTALL)
115+
self.assertIsNotNone(packages, log)
116+
self.assertEqual(host, packages.group(1).splitlines()) # type: ignore[union-attr]
117+
self.assertEqual(pathlib.PurePosixPath(host[0]).parent.name, 'linux-64')
118+
119+
120+
if __name__ == '__main__':
121+
unittest.main()

0 commit comments

Comments
 (0)