Skip to content

Commit d905712

Browse files
authored
chore: drop stale build leftovers and move the examples' READMEs to Markdown (#1773) (#1780)
- setup.py: drop the bdist_wheel override, which converted dist_dir/pcapkit after the wheel was written; build_py and install already convert the tree bdist_wheel archives. test_setup.py loses its now-dead bdist_wheel fake. - setup.py: the f2format and poseur failures name f-string and positional-only parameter backports, not assignment expressions. - Makefile: `profile` delegates to examples/legacy_smoke (moved in 78e28d8). - MANIFEST.in: drop `prune test` (no such directory); add `recursive-include examples *.md`. `global-include *.rst` now matches nothing shipped (sdist identical without it); left for a later pass. - examples/{benchmark,legacy_smoke}/README.rst -> README.md; .gitignore, benchmark.py, the tier test and the documentation conventions page follow. Sdist listings differ only by the renames and the new test; wheels match. test_build_leftovers_1773.py fails with each fix reverted; tests/project green. Closes #1773
1 parent a1f573b commit d905712

13 files changed

Lines changed: 877 additions & 722 deletions

File tree

‎.gitignore‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ examples/captures/*
1818
# PCAP-NG packet blocks their captured octets back. Regenerating them on every such
1919
# change is work with no reader, and the tier guard in `tests/_tiers.py` asks git
2020
# which captures are tracked rather than hardcoding a list, so dropping them here
21-
# is the whole change -- see `examples/legacy_smoke/README.rst` for how to rebuild
21+
# is the whole change -- see `examples/legacy_smoke/README.md` for how to rebuild
2222
# them locally.
2323
!examples/captures/in.pcap
2424
!examples/captures/dhcp.pcapng

‎MANIFEST.in‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,14 +2,15 @@ global-exclude .DS_Store
22

33
global-include *.rst
44
global-include *.py
5+
# Neither global-include reaches Markdown, which the examples' READMEs are (#719).
6+
recursive-include examples *.md
57

68
prune .eggs
79
prune .github
810
prune .venv
911
prune docs
1012
prune examples/captures
1113
prune temp
12-
prune test
1314

1415
# Belt-and-braces, both of them: setuptools ships these two whether or not this
1516
# file mentions them, so neither line is load-bearing. An earlier version of this

‎Makefile‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -193,8 +193,10 @@ mypy:
193193
bandit:
194194
$(RUN) bandit $(BANDIT_FLAGS) pcapkit
195195

196+
# Profile one extraction and render its call graph; the tools the recipe needs
197+
# are listed in examples/legacy_smoke/README.md.
196198
profile:
197-
$(MAKE) -C test profile
199+
$(MAKE) -C examples/legacy_smoke profile
198200

199201
mypy-types:
200202
pipenv run mypy --install-types

‎docs/source/contributing/conventions/documentation.rst‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -289,10 +289,10 @@ Format and Mechanics
289289
ruled this on :issue:`719`, correcting a blanket *always* ``.rst`` that had been in
290290
circulation: the Sphinx documentation is reST, and the other documents --
291291
the READMEs included -- are Markdown where that applies. ``CONTRIBUTING.md``'s own
292-
*Documentation* section records the same split. One trap arrived with the ruling:
293-
:file:`MANIFEST.in` reaches the two READMEs under :file:`examples/` through
294-
``global-include *.rst`` and has no ``*.md`` equivalent, so renaming one without
295-
adding coverage drops it from every source distribution.
292+
*Documentation* section records the same split. Neither ``global-include`` in
293+
:file:`MANIFEST.in` reaches Markdown, so the two READMEs under :file:`examples/`
294+
ship through ``recursive-include examples *.md``, and any other Markdown file
295+
ships only if setuptools or an ``include`` line adds it.
296296
* **Prose names the real defining module, not a re-export.** One strand of the
297297
consistency sweep on :doc:`the roadmap </contributing/pep>`. The sentinels are the
298298
worked example: all four are defined in :mod:`pcapkit.corekit.sentinels`, and

‎examples/benchmark/README.md‎

Lines changed: 470 additions & 0 deletions
Large diffs are not rendered by default.

‎examples/benchmark/README.rst‎

Lines changed: 0 additions & 536 deletions
This file was deleted.

‎examples/benchmark/benchmark.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@
2222
:data:`ROUNDS` timed extractions of the same capture per engine, timed with
2323
:func:`time.perf_counter_ns`, first sample discarded as a warm-up, reported as
2424
milliseconds per packet. The deliberate departures are listed in
25-
:file:`README.rst` under "Departures from the legacy methodology".
25+
:file:`README.md` under "Departures from the legacy methodology".
2626
2727
Two properties matter more than the numbers themselves.
2828

‎examples/legacy_smoke/README.md‎

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
# Demonstration Scripts
2+
3+
A tour of `pcapkit` by example: each script here does one thing and prints or dumps
4+
the result, so you can get a thorough view of the library either on how to use it or
5+
on what it can do. They are demos rather than tests -- `pytest` does not collect
6+
this directory (`testpaths = ["tests"]`) and nothing in CI runs them, so they are
7+
also where API drift goes unnoticed the longest. Run them after changing anything
8+
public.
9+
10+
## Running them
11+
12+
Every script reads its capture with a path relative to its own directory, so **run
13+
them from inside** `examples/legacy_smoke/`, not from the repository root:
14+
15+
```shell
16+
cd examples/legacy_smoke
17+
python test_basic.py
18+
```
19+
20+
The captures live in `../captures/`. Only `in.pcap` and `dhcp.pcapng` are
21+
committed; the rest are generated, so build them once before running anything else:
22+
23+
```shell
24+
python ../generators/make_samples.py # or: make samples, from the repository root
25+
```
26+
27+
## The scripts
28+
29+
| Script | What it demonstrates |
30+
| --- | --- |
31+
| [test_basic](test_basic.py) | the smallest useful call -- extract `in.pcap` to a tree-view text file |
32+
| [test_extractor](test_extractor.py) | `pcapkit.extract` dumping the same capture as plist, JSON and tree |
33+
| [test_file](test_file.py) | reading from an already-open binary file object rather than a path |
34+
| [test_api](test_api.py) | most of the keyword surface at once: per-frame files, reassembly and flow tracing |
35+
| [test_tcp](test_tcp.py) | extraction of TCP packets |
36+
| [test_ipv6](test_ipv6.py) | extraction of IPv6 packets, one output file per frame |
37+
| [test_pcapng](test_pcapng.py) | extraction of a PCAP-NG capture, which uses a different engine |
38+
| [test_http](test_http.py) | iterating frames manually and testing `pcapkit.HTTP in frame` |
39+
| [test_reassembly](test_reassembly.py) | TCP payload reassembly, writing the reassembled datagrams out |
40+
| [test_ip_reasm](test_ip_reasm.py) | IPv4 fragment reassembly |
41+
| [test_ipv6_reasm](test_ipv6_reasm.py) | IPv6 fragment reassembly |
42+
| [test_analyse](test_analyse.py) | analysis of the application layer *after* reassembly, printing the recovered HTTP messages |
43+
| [test_trace](test_trace.py) | tracing TCP flows and dumping the flow index |
44+
| [test_engine](test_engine.py) | the same capture through each extraction engine, side by side |
45+
| [test_time](test_time.py) | timing each engine, reported as milliseconds per packet |
46+
| [test_profile](test_profile.py) | cProfile stats for one extraction; `make profile` renders the call graph |
47+
| [test_perf](test_perf.py) | the same comparison under the `pyperf` benchmark harness |
48+
| [test_stream](test_stream.py) | extracting a live capture streamed from `tcpdump` on stdin |
49+
| [test_stream_askpass](test_stream_askpass.py) | the same, with `sudo -A` for a non-interactive password |
50+
51+
`_engine_support.py` is not a demo -- it is the shared helper the engine demos use
52+
to work out whether an engine can run on this host.
53+
54+
## What needs more than pcapkit
55+
56+
Most of these run with nothing but `pcapkit`. These do not:
57+
58+
- **test_engine, test_time, test_perf** exercise the `dpkt`, `scapy` and
59+
`pyshark` engines. Those are optional packages, and `pyshark` additionally
60+
drives Wireshark's `tshark` binary. An engine that is missing is reported as
61+
skipped, with the reason, and the rest still run -- see `_engine_support.py`. Two
62+
things worth knowing:
63+
64+
- `pyshark` 0.6 does not work on Python 3.14 at all. It asks
65+
`asyncio.get_event_loop_policy().get_event_loop()` for a loop on a thread that
66+
has none, which 3.14 no longer creates implicitly, so every extraction through it
67+
raises `RuntimeError: There is no current event loop in thread 'MainThread'`
68+
before `tshark` is even reached. That is a pyshark bug, not a pcapkit one, and
69+
there is nothing to configure around it.
70+
- When an engine's *package* is missing, `pcapkit.extract` does not fail. It warns
71+
and falls back to pcapkit's own parser, so the extraction succeeds under the wrong
72+
engine. The demos name the driver that actually ran, so a fallback is visible
73+
rather than being reported as a healthy result.
74+
75+
- **test_perf** needs the `pyperf` harness, which `pcapkit` does not depend on:
76+
`python -m pip install pyperf`. Without it the script says so and stops.
77+
`test_time.py` does the same comparison with no extra dependency.
78+
- **test_profile**'s `make profile` target needs `gprof2dot`, `graphviz` and
79+
`snakeviz` to render the call graph. The script itself only needs `pcapkit`.
80+
- **test_stream, test_stream_askpass** run `sudo tcpdump` against a live interface,
81+
so they need root and cannot run unattended. `INTERFACE` at the top of each is
82+
`en0` (macOS); on Linux it is usually `eth0` or `wlan0`. They buffer the live
83+
capture into a temporary file -- never into `../captures/`, which holds generated
84+
captures the test suite pins.
85+
86+
## Regenerating the reference reports
87+
88+
Four files in `../captures/` are reports rather than inputs -- `out.json`,
89+
`out.plist`, `out.txt` and `pcapng.txt` -- and they are produced by two of the
90+
scripts here. **Git does not track them**, so a fresh clone does not have them. Build
91+
them the way you would any other generated capture, and again after a change to how
92+
`pcapkit` renders one:
93+
94+
```shell
95+
cd examples/legacy_smoke && make fixtures
96+
```
97+
98+
That is the whole recipe. It runs, equivalently:
99+
100+
```shell
101+
TZ=UTC python test_extractor.py # ../captures/out.json, out.plist, out.txt <- in.pcap
102+
TZ=UTC python test_pcapng.py # ../captures/pcapng.txt <- dhcp.pcapng
103+
```
104+
105+
Both *inputs* are committed, so this works on a fresh clone with no `make samples`
106+
first. Use `make fixtures PYTHON=../../.venv/bin/python` to pick a specific
107+
interpreter.
108+
109+
These four were tracked once, and #685 is why they are not any more. Nothing reads a
110+
report back -- no test asserts on one, and the docs only quote them as prose -- so a
111+
parser change that alters how a frame renders leaves the committed copy quietly wrong,
112+
with no failure anywhere to say so. `pcapng.txt` sat recording `packet -> NIL` for
113+
four blocks that had started carrying their captured octets, and the staleness was
114+
found by reading the file rather than by anything breaking. Generating them on demand
115+
means the copy on your disk describes the tree you are standing in, which was the only
116+
thing that made them worth looking at.
117+
118+
`TZ=UTC` **is not decoration.** `pcapkit` renders frame timestamps in the host's
119+
local zone, so an unpinned run rewrites every timestamp line in all four files with
120+
wherever it happened to be run -- which is how these reports came to disagree with
121+
each other while they were still tracked, the PCAP three having been generated at
122+
UTC-05:00 and `pcapng.txt` at UTC+08:00. Pinning UTC makes the output reproducible on
123+
any host, so two runs can be diffed against each other. It also happens to be the only
124+
zone in which PCAP-NG's `timestamp_epoch` is the true UNIX epoch:
125+
`PCAPNG._read_timestamp` adds the zone's UTC offset to the value it returns, so a
126+
report generated anywhere else bakes in that offset.

‎examples/legacy_smoke/README.rst‎

Lines changed: 0 additions & 156 deletions
This file was deleted.

‎setup.py‎

Lines changed: 5 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ def refactor(path: "str") -> "None":
4747
)
4848
except subprocess.CalledProcessError as error:
4949
logger.error(
50-
"Failed to perform assignment expression backport compiling. "
50+
"Failed to perform f-string backport compiling. "
5151
"Please consider manually install `bpc-f2format` and try again."
5252
)
5353
sys.exit(error.returncode)
@@ -70,7 +70,7 @@ def refactor(path: "str") -> "None":
7070
)
7171
except subprocess.CalledProcessError as error:
7272
logger.error(
73-
"Failed to perform assignment expression backport compiling. "
73+
"Failed to perform positional-only parameter backport compiling. "
7474
"Please consider manually install `bpc-poseur` and try again."
7575
)
7676
sys.exit(error.returncode)
@@ -156,24 +156,9 @@ def run(self) -> "None":
156156
"setuptools version is too old to support PyBPC conversion in install."
157157
)
158158

159-
try:
160-
from setuptools.command.bdist_wheel import bdist_wheel
161-
162-
class pcapkit_bdist_wheel(bdist_wheel):
163-
"""Modified bdist_wheel to run PyBPC conversion."""
164-
165-
def run(self) -> "None":
166-
super(pcapkit_bdist_wheel, self).run()
167-
logger.info("running bdist_wheel")
168-
169-
# PyBPC compatibility enforcement
170-
refactor(os.path.join(self.dist_dir, "pcapkit")) # type: ignore[arg-type]
171-
172-
cmdclass["bdist_wheel"] = pcapkit_bdist_wheel
173-
except ImportError:
174-
logger.warning(
175-
"setuptools version is too old to support PyBPC conversion in bdist_wheel."
176-
)
159+
# No bdist_wheel override: ``bdist_wheel`` runs ``build_py`` and ``install``
160+
# above, so the tree it archives is already converted before the wheel is
161+
# written, and ``dist_dir`` holds only the finished wheels (#1773).
177162

178163
setup(
179164
cmdclass=cmdclass,

0 commit comments

Comments
 (0)