Skip to content

Commit 7f01d49

Browse files
committed
fix(conda): vendor only the needed dependencies and ship their data files (#1814)
- conda/requirements.txt: dependencies plus the all extra and what they require, for the supported CPython tier. The capture engines and the crawlers' requests and beautifulsoup4 go, as on PyPI since #910; cryptography and pycrate, with cffi and pycparser, come in; the pathlib2 and typing-extensions lines were dead on every conda build. - util/conda_vendor.py derives that list from pyproject.toml and the installed metadata, replacing util/conda-dist.py. --check reports a stale list; --smoke checks a built package's vendored files. - pyproject.toml: package-data ships every file under pcapkit/_extern but bytecode. Only modules reached the wheel before, so chardet could not detect, emoji had no JSON and certifi no CA bundle. - cron-conda.yml: Conda Update installs .[all] and runs the script; each conda-dist leg smoke-tests its package before the upload. - tests/project/test_conda_vendor_1814.py pins the package-data rule, the derivation, the committed list and the smoke. Clean sdist and wheel: unchanged but for the files added and removed here, with no new warning. tests/project passes. Closes #1814
1 parent b7488c6 commit 7f01d49

9 files changed

Lines changed: 696 additions & 80 deletions

File tree

‎.github/workflows/create-release.yml‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1221,10 +1221,10 @@ jobs:
12211221
- ubuntu-latest
12221222
# - windows-latest # skip for now
12231223
# #1633: the supported tier only, from .github/python-versions.toml
1224-
# through ``version_check`` (#1660). There is no KTLO 3.10 leg because it
1225-
# can only fail: conda/requirements.txt pins soupsieve==3.0.3
1226-
# (Requires-Python >=3.11.5), and util/conda-dist.py pins the newest
1227-
# release of every dependency, so that floor only rises.
1224+
# through ``version_check`` (#1660). There is no KTLO 3.10 leg: the list
1225+
# util/conda_vendor.py derives is for the supported tier, so it leaves
1226+
# out typing-extensions, which pcapkit/utilities/compat.py imports below
1227+
# 3.11, and pins the newest release of each dependency, a rising floor.
12281228
python-version: ${{ fromJSON(needs.version_check.outputs.PYTHON_CONDA) }}
12291229
steps:
12301230
# #1634: this run's commit, not ``ref: conda-<version>+0``, then the check

‎.github/workflows/cron-conda.yml‎

Lines changed: 24 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -156,28 +156,25 @@ jobs:
156156
with:
157157
python-version: ${{ steps.python-versions.outputs.latest }}
158158

159-
# `.[all,dev]`, not `.[all]` alone: `util/conda-dist.py` below only
160-
# *refreshes* the version pin of every package already named in
161-
# `conda/requirements.txt` -- via `importlib.metadata.version(name)`,
162-
# which raises unless that distribution is installed here -- and that
163-
# file names `dpkt`, `scapy`, `pyshark`, `requests` and `beautifulsoup4`
164-
# with no marker. #910 narrowed `all` to core addons only, so those five
165-
# come from `dev` instead; without it this step would install
166-
# cleanly and the next one would crash on the first missing name.
159+
# `.[all]`, which is what `util/conda_vendor.py` below vendors (#1814):
160+
# it derives `conda/requirements.txt` from `dependencies` and the `all`
161+
# extra, walks what they require through the installed metadata, and
162+
# pins each package at the version installed here. So this install is
163+
# the step that resolves the newest releases, and the script itself
164+
# reads nothing from the network.
167165
- name: Install and Setup
168166
run: |
169167
set -x
170168
171169
python -m pip install -U pip setuptools wheel packaging
172-
python -m pip install -e .[all,dev]
173-
#python -m pip install pathlib2 typing_extensions
170+
python -m pip install -e .[all]
174171
175172
- name: Setup Conda Distribution
176173
run: |
177174
export PCAPKIT_DEVMODE=1
178175
export PCAPKIT_VERBOSE=1
179176
180-
python util/conda-dist.py
177+
python util/conda_vendor.py
181178
182179
- name: Verify Changed files
183180
uses: tj-actions/verify-changed-files@v17
@@ -530,6 +527,8 @@ jobs:
530527
;;
531528
esac
532529
530+
echo "host_package=$host_package" >> "$GITHUB_OUTPUT"
531+
533532
host_platform="$(basename "$(dirname "$host_package")")"
534533
if [ -n "$target_platform" ] && [ "$target_platform" != "$host_platform" ]; then
535534
conda convert "$host_package" -o "$out_dir" -p "$target_platform"
@@ -550,6 +549,20 @@ jobs:
550549
echo EOF
551550
} >> "$GITHUB_OUTPUT"
552551
552+
# #1814: the package declares no run requirements, so the packages it
553+
# vendors in `pcapkit/_extern` are all it has, and they once shipped as
554+
# modules alone. This unpacks the package built for this runner and
555+
# checks that every vendored distribution has every file its RECORD lists
556+
# and imports from there, with no site-packages on the path, before
557+
# anything is uploaded. setup-python's interpreter, not conda's: it is
558+
# the matrix version the package was built for, which conda's base is
559+
# not on 3.15.
560+
- name: Smoke-test the vendored packages
561+
env:
562+
PCAPKIT_CONDA_PACKAGE: ${{ steps.conda_build.outputs.host_package }}
563+
run: |
564+
"${pythonLocation}/bin/python" util/conda_vendor.py --smoke "$PCAPKIT_CONDA_PACKAGE"
565+
553566
- name: Upload conda packages to Anaconda
554567
uses: anaconda/actions/upload-package@v0.3.1
555568
with:

‎MANIFEST.in‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,7 @@ recursive-include docs/source/changelog *.rst
8787
# Nothing here names ``pcapkit/_extern``. ``conda/pypcapkit/build.sh`` and
8888
# ``bld.bat`` create it and then run ``pip install .``, which builds the wheel
8989
# straight from the tree, so the conda recipe builds no sdist; its modules
90-
# reach that wheel through package discovery. A line naming it would warn in
91-
# every other build, and ``include`` of a directory matched nothing even in the
92-
# conda one (#1794).
90+
# reach that wheel through package discovery, and every other file under it
91+
# through the ``package-data`` entry in ``pyproject.toml`` (#1814). A line naming
92+
# it would warn in every other build, and ``include`` of a directory matched
93+
# nothing even in the conda one (#1794).

‎conda/requirements.txt‎

Lines changed: 4 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,25 +1,9 @@
11
aenum==3.1.17
2-
appdirs==1.4.4
3-
beautifulsoup4==4.15.0
4-
certifi==2026.7.22
2+
cffi==2.1.1
53
chardet==7.6.0
6-
charset-normalizer==3.5.2
4+
cryptography==50.0.2
75
dictdumper==0.8.4.post6
8-
dpkt==1.9.8
96
emoji==2.16.0
10-
html5lib==1.1
11-
idna==3.20
12-
lxml==6.1.3
13-
packaging==26.3
14-
pathlib2; python_version < "3.5"
15-
pyshark==0.6
16-
pysocks==1.7.1
17-
requests==2.34.2
18-
scapy==2.8.0
19-
six==1.17.0
20-
soupsieve==3.0.3
7+
pycparser==3.11
8+
pycrate==0.8.1
219
tbtrim==0.3.1
22-
termcolor==3.3.0
23-
typing-extensions==4.16.0; python_version < "3.11"
24-
urllib3==2.8.0
25-
webencodings==0.6.1

‎pyproject.toml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -368,6 +368,21 @@ exclude = [
368368
"sample*",
369369
]
370370

371+
# Only the conda recipe creates ``pcapkit/_extern``: ``conda/pypcapkit/build.sh``
372+
# and ``bld.bat`` install the packages ``conda/requirements.txt`` names there,
373+
# then run ``pip install .``, which builds the wheel from the tree. Package
374+
# discovery reaches their modules alone, so every other file -- ``emoji``'s JSON,
375+
# ``chardet``'s models, compiled extensions, each ``dist-info`` -- stays behind
376+
# without this entry (#1814). Elsewhere ``pcapkit._extern`` is no package at all,
377+
# and setuptools ignores both entries without a warning; ``MANIFEST.in`` could not
378+
# do this, since a line naming a missing directory warns in every such build. The
379+
# bytecode ``pip install --target`` compiles stays out, as from any other wheel.
380+
[tool.setuptools.package-data]
381+
"pcapkit._extern" = [ "**/*" ]
382+
383+
[tool.setuptools.exclude-package-data]
384+
"pcapkit._extern" = [ "*.pyc" ]
385+
371386
[tool.setuptools.dynamic]
372387
version = { attr="pcapkit.__version__" }
373388

0 commit comments

Comments
 (0)