Skip to content

Commit 3a7a1cd

Browse files
committed
ci(util): fetch with requests and poll memory with psutil in the scripts (#1894)
- util/ci_timing.py: fetch through a requests session; an HTTPError's status decides the retry and the 404 note, the Link header is parsed as requests parses it, and the token staying off a cross-host redirect is tested against loopback servers. ci-timing.yml installs requests and lxml. - util/update_python_versions.py: fetch() uses requests and parse_images() reads the runner-images table with markdown-it-py, so an escaped pipe stays in its cell; both import lazily, so `publish` still runs on a bare python3. The detect job installs both. - examples/generators/pcapng.py: download with requests, keeping the retry, Retry-After and offline fallback; a timeout mid-body is still retried. - util/vendor_test_gate.py: sample a run's process group with psutil, and with ps where psutil is not installed. - util/ci_timing.py parse_junit() and util/pyshark_encap_map.py's PDML read: lxml with a hardened parser instead of ElementTree, no fallback; the tests that parse are gated on HAS_LXML. - pyproject.toml: the test extra carries psutil (CPython only), markdown-it-py and lxml (not on GraalPy). - Pipfile [dev-packages]: the same, with the test extra's markers, pinned by a test. Closes #1894
1 parent c9b4823 commit 3a7a1cd

19 files changed

Lines changed: 1153 additions & 175 deletions

‎.github/workflows/ci-timing.yml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -54,11 +54,15 @@ jobs:
5454
id: python-versions
5555
run: python3 util/python_versions.py --github-output
5656

57-
# Nothing is installed: the script is standard library only.
5857
- uses: actions/setup-python@v7
5958
with:
6059
python-version: ${{ steps.python-versions.outputs.latest }}
6160

61+
# The script fetches with requests and parses the JUnit artifacts with lxml
62+
# (#1894), and needs nothing else outside the standard library.
63+
- name: Install requests and lxml
64+
run: python -m pip install requests lxml
65+
6266
# The newest earlier run that uploaded a report, so a week whose run failed
6367
# on a HARD threshold is still next week's baseline. A first run, or one
6468
# after 90 days without a report, has none and says so.

‎.github/workflows/cron-python-versions.yml‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -57,9 +57,11 @@ jobs:
5757
python-version: ${{ steps.python-versions.outputs.latest }}
5858

5959
# PyYAML, from the `test` extra, for the workflow readers and the audit;
60-
# neither imports pcapkit.
61-
- name: Install PyYAML
62-
run: python -m pip install PyYAML
60+
# neither imports pcapkit. requests fetches the sources and markdown-it-py
61+
# reads the runner-images table (#1894); only `update` imports either, so
62+
# `pr`'s `publish` runs without them.
63+
- name: Install PyYAML, requests and markdown-it-py
64+
run: python -m pip install PyYAML requests markdown-it-py
6365

6466
# Fetches the sources, rewrites the files, and writes the proposal to
6567
# $RUNNER_TEMP/proposal. `publish` is false when nothing changed, or when

‎.github/workflows/cron-vendor.yml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -264,8 +264,9 @@ jobs:
264264
# zero once it has decided; if this step fails or times out instead, the
265265
# tree may hold modules no run passed with, and the next step discards
266266
# every regenerated module. The test extra is installed here because only
267-
# this step needs it. The script's budget bounds its runs together, and
268-
# `timeout-minutes` sits above it as the backstop. Run serially, pytest
267+
# this step needs it; it carries psutil too, which the script samples a
268+
# run's memory with (#1894). The script's budget bounds its runs together,
269+
# and `timeout-minutes` sits above it as the backstop. Run serially, pytest
269270
# peaked at 0.7 GB resident over tests/const, measured locally, so the
270271
# memory cap leaves headroom on this runner; files are capped at 2 GiB, as
271272
# in experimental.yml.

‎.github/workflows/unit-tests.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -472,8 +472,8 @@ jobs:
472472

473473
# The Scapy extra is there for examples/generators/pcap.py and legacy.py,
474474
# which build their captures with scapy; examples/generators/pcapng.py needs
475-
# nothing beyond the standard library and pcapkit itself, which it uses to
476-
# parse each fixture back. DPKT joins it so the dpkt-gated methods
475+
# requests (#1894), from the test extra, and pcapkit itself, which it uses
476+
# to parse each fixture back. DPKT joins it so the dpkt-gated methods
477477
# living under tests/integration/ (test_engine_parity.py,
478478
# test_engine_runtime.py) actually run here instead of skipping -- see
479479
# #729; same reasoning as the `test` job above.

‎Pipfile‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,10 @@ libpcap = {version = ">=1.11.0b29",markers = "python_version >= '3.10'"}
4444
pypcapfile = "*"
4545
beautifulsoup4 = {extras = ["html5lib"],version = "*"}
4646
requests = {extras = ["socks"],version = "*"}
47+
# The markers match the test extra in pyproject.toml (#1894).
48+
psutil = {version = "*",markers = "platform_python_implementation == 'CPython'"}
49+
lxml = {version = "*",markers = "platform_python_implementation != 'GraalVM'"}
50+
markdown-it-py = "*"
4751
autopep8 = "*"
4852
bandit = "*"
4953
isort = "*"

‎examples/generators/pcapng.py‎

Lines changed: 37 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -6,13 +6,14 @@
66
out of the repository. This module puts them back on any machine, without a
77
checkout of anything private, by one of these routes:
88
9-
* **Downloaded** -- fetched from a public source, with the source URL and the
10-
SHA-256 of the exact bytes pinned below. The digest is checked after every
11-
download and a mismatch is a hard error, never a warning. An HTTP 429 or
12-
5xx, or a timeout, is retried a few times with backoff (see
13-
:func:`_download`). If the network is still unavailable the download
14-
degrades to a synthesised stand-in, and both the console output and the
15-
return value say so rather than pretending otherwise.
9+
* **Downloaded** -- fetched from a public source with :mod:`requests`, with the
10+
source URL and the SHA-256 of the exact bytes pinned below. The digest is
11+
checked after every download and a mismatch is a hard error, never a warning.
12+
An HTTP 429 or 5xx, or a timeout, is retried a few times with backoff (see
13+
:func:`_download`). If the network is still unavailable, or :mod:`requests`
14+
is not installed, the download degrades to a synthesised stand-in, and both
15+
the console output and the return value say so rather than pretending
16+
otherwise.
1617
* **Synthesised** -- written byte by byte by :class:`_Blocks` below, with no
1718
network access whatsoever, from packet payloads carried in the committed
1819
``examples/captures/dhcp.pcapng`` fixture. Generation is deterministic: the same input
@@ -162,8 +163,6 @@
162163
import struct
163164
import sys
164165
import time
165-
import urllib.error
166-
import urllib.request
167166
from typing import TYPE_CHECKING, NamedTuple
168167

169168
if TYPE_CHECKING:
@@ -836,11 +835,22 @@ def _transient(exc: 'OSError') -> 'bool':
836835
server's final answer, and an unreachable host fails at once rather than
837836
after a wait, which keeps an offline run as quick as it was.
838837
838+
:mod:`requests` raises a :exc:`~requests.Timeout` for a timeout while it
839+
connects or waits for the headers, but a timeout while the body is read
840+
reaches here as a :exc:`~requests.ConnectionError` wrapping urllib3's
841+
:exc:`~urllib3.exceptions.ReadTimeoutError`.
842+
839843
"""
840-
if isinstance(exc, urllib.error.HTTPError):
841-
return exc.code == 429 or 500 <= exc.code <= 599
842-
reason = exc.reason if isinstance(exc, urllib.error.URLError) else exc
843-
return isinstance(reason, (socket.timeout, TimeoutError))
844+
import requests # pylint: disable=import-outside-toplevel
845+
import urllib3 # pylint: disable=import-outside-toplevel
846+
847+
if isinstance(exc, requests.HTTPError):
848+
code = getattr(exc.response, 'status_code', None)
849+
return code is not None and (code == 429 or 500 <= code <= 599)
850+
if isinstance(exc, (requests.Timeout, socket.timeout, TimeoutError)):
851+
return True
852+
reason = exc.args[0] if isinstance(exc, requests.ConnectionError) and exc.args else None
853+
return isinstance(reason, urllib3.exceptions.ReadTimeoutError)
844854

845855

846856
def _retry_after(exc: 'OSError', now: 'float') -> 'Optional[float]':
@@ -857,8 +867,8 @@ def _retry_after(exc: 'OSError', now: 'float') -> 'Optional[float]':
857867
The wait, or :data:`None` if there is no header or it does not parse.
858868
859869
"""
860-
headers = getattr(exc, 'headers', None)
861-
value = None if headers is None else headers.get('Retry-After')
870+
response = getattr(exc, 'response', None)
871+
value = None if response is None else response.headers.get('Retry-After')
862872
if value is None:
863873
return None
864874
value = value.strip()
@@ -893,14 +903,19 @@ def _download(url: 'str') -> 'bytes':
893903
Raises:
894904
OSError: The last failure, once the retries are over or for a failure
895905
that is not retried -- :func:`_materialise` then falls back to a
896-
stand-in (#1658).
906+
stand-in (#1658). Every :exc:`requests.RequestException` is one.
907+
ImportError: :mod:`requests` is not installed, which
908+
:func:`_materialise` treats as it does an unreachable host.
897909
898910
"""
911+
import requests # pylint: disable=import-outside-toplevel
912+
899913
attempt, waited = 1, 0.0
900914
while True:
901915
try:
902-
with urllib.request.urlopen(url, timeout=TIMEOUT) as response: # nosec: B310
903-
return response.read()
916+
response = requests.get(url, timeout=TIMEOUT)
917+
response.raise_for_status()
918+
return response.content
904919
except OSError as exc:
905920
if attempt >= DOWNLOAD_ATTEMPTS or not _transient(exc):
906921
raise
@@ -946,8 +961,10 @@ def _materialise(fixture: '_Fixture', path: 'pathlib.Path') -> 'tuple[bytes, str
946961

947962
try:
948963
data = _download(fixture.url)
949-
except (urllib.error.URLError, OSError, ValueError) as exc:
950-
print(f' ! {fixture.url} is unreachable ({exc});')
964+
except (ImportError, OSError, ValueError) as exc:
965+
why = ('cannot be fetched without requests' if isinstance(exc, ImportError)
966+
else 'is unreachable')
967+
print(f' ! {fixture.url} {why} ({exc});')
951968
print(f' ! synthesising a stand-in for {fixture.name} instead -- the '
952969
'upstream capture was NOT used')
953970
return fixture.build(), 'synthesised (download unavailable)'

‎pyproject.toml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -363,6 +363,22 @@ test = [
363363
# Test-only, as ``pcapkit`` never imports ``yaml``. Unpinned, since
364364
# ``safe_load`` is all they use.
365365
"PyYAML",
366+
# What util/vendor_test_gate.py samples a test run's memory with (#1894),
367+
# here because cron-vendor.yml's gate step installs this extra and nothing
368+
# else. CPython only: PyPI has wheels for every CPython leg, free-threaded
369+
# 3.14t included, and none for PyPy or GraalPy, where the gate falls back to
370+
# ``ps``. Unpinned, like the entries above.
371+
"psutil; platform_python_implementation == 'CPython'",
372+
# What util/update_python_versions.py reads the runner-images README's table
373+
# with (#1894), for the tests under ``tests/project/`` that run it on the
374+
# saved README. A pure-Python wheel; unpinned, like the entries above.
375+
"markdown-it-py",
376+
# The strict XML parser util/ci_timing.py reads JUnit with and
377+
# util/pyshark_encap_map.py reads PDML with (#1894). PyPI has wheels for
378+
# CPython and PyPy but none for GraalPy, which reports its implementation as
379+
# ``GraalVM``; the tests that parse skip there, and the scripts do not fall
380+
# back to a laxer parser. Unpinned, like the entries above.
381+
"lxml; platform_python_implementation != 'GraalVM'",
366382
]
367383

368384
# setuptools specific

‎tests/_dependency_gates.py‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -250,6 +250,15 @@
250250
# The theme ``HAS_FURO`` asks for, whose table wrapper the #1861 transform
251251
# test runs. Only the ``docs`` extra carries it.
252252
'furo': ('furo',),
253+
# The strict XML parser ``HAS_LXML`` asks for (#1894): util/ci_timing.py
254+
# reads JUnit and util/pyshark_encap_map.py reads PDML with it, and neither
255+
# falls back to a laxer parser. The ``test`` extra carries it with a
256+
# ``platform_python_implementation != 'GraalVM'`` marker, because PyPI has
257+
# no wheel for GraalPy, so the gate skips on experimental.yml's GraalPy leg.
258+
# That is no :data:`DEPENDENCY_GATE_EXCLUSIONS` entry: this module does not
259+
# model markers and does not read experimental.yml, so every job it reads
260+
# satisfies the gate, and an entry would describe a gap that is not there.
261+
'lxml': ('lxml',),
253262
}
254263

255264
#: Flags whose condition is not a distribution, with what it is instead. A flag

0 commit comments

Comments
 (0)