Skip to content

Commit 6aeba80

Browse files
authored
refactor(toolkit): revert pypcapfile's agreement-only code (#1708) (#1721)
Third-party engines are best effort (#1687), so the pypcapfile code that exists only to match the default engine comes out: - revert #1682, #1600, #1575 and #1535, and the pypcapfile hunks of #1655, #1608 and #1605: reading loopback and raw-IP link types, the frame layer bound, the VLAN and tunnel walks, keying tunnelled TCP by the innermost header, stepping over AH and extension headers, and checking options with the default engine's parsers - keep the helpers the dpkt and scapy adapters import, unchanged, until their own reverts drop them; their docstrings no longer say pypcapfile asks the default engine's parsers too - re-land #1576, #1577, #1590 and #1591 as guards in _network and _transport, #1513's once-per-capture warning for TCP straight over IPv6, and #1667's packet2dict fix: an undecoded frame is reported as its octets, not as hex text twice as long - keep #1569's record reader and #1690's packet2timestamp - delete the three test modules those reverts added, the pypcapfile timestamp agreement class, and #1682's link-type warning test; the re-landed guards' tests assert constants - drop what the reverts removed from the docs, READ_LINKTYPES included, and say that TCP behind a VLAN tag, in a tunnel or behind an extension header (an IPv6 one, or AH in IPv4) is now left out without a warning The pypcapfile modules, test_new_engine_parity_runtime.py, test_engine_runtime.py, the timestamp module and the project guards pass on 3.11 (pypcapfile 0.12.0) and 3.14.
1 parent 27ccb5f commit 6aeba80

20 files changed

Lines changed: 455 additions & 2714 deletions

‎docs/source/contributing/pep.rst‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -563,13 +563,15 @@ both of the following are worth knowing before reaching for them:
563563
sends the loader to ``find_library("pcap")``. No compiler and no
564564
:file:`pcap.h` are needed; a ``libpcap.so.1`` is.
565565

566-
Both engines support less than the ``default`` engine does, deliberately and
567-
noisily: `pypcap`_ performs no protocol dissection, so it disables reassembly
568-
*and* flow tracing; `pypcapfile`_ has no IPv6 decoder, so it disables IPv6
569-
reassembly while keeping IPv4 and TCP. Each gap is announced through an
566+
Both engines support less than the ``default`` engine does, deliberately:
567+
`pypcap`_ performs no protocol dissection, so it disables reassembly *and* flow
568+
tracing; `pypcapfile`_ has no IPv6 decoder, so it disables IPv6 reassembly while
569+
keeping IPv4 and TCP. Each gap is announced through an
570570
:class:`~pcapkit.utilities.warnings.AttributeWarning` or an outright exception
571-
rather than by silently returning nothing --
572-
:doc:`/pcapkit/foundation/engines/index` tabulates them.
571+
rather than by silently returning nothing, save that `pypcapfile`_ leaves TCP
572+
behind a VLAN tag, in a tunnel or behind an extension header (an IPv6 one, or AH
573+
in IPv4) out without one -- :doc:`/pcapkit/foundation/engines/index` tabulates
574+
them.
573575

574576
Adding a further engine does not mean adding handler methods to
575577
:class:`~pcapkit.foundation.extraction.Extractor`, as the thread describes. A new engine subclasses

‎docs/source/ext.rst‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -420,7 +420,9 @@ The available engines and the file formats they read:
420420
whatsoever, so they support neither reassembly nor flow tracing; `PyPCAPFile`_
421421
has no IPv6 decoder, so it supports IPv4 and TCP reassembly but not IPv6. What
422422
matters is that the gap is *announced* -- each engine warns, or raises, for the
423-
capability it cannot provide, rather than silently producing an empty result. See
423+
capability it cannot provide, rather than silently producing an empty result,
424+
save that `PyPCAPFile`_ leaves TCP behind a VLAN tag, in a tunnel or behind an
425+
extension header (an IPv6 one, or AH in IPv4) out without a warning. See
424426
:doc:`pcapkit/foundation/engines/index` for the full table.
425427

426428
.. note::

‎docs/source/index.rst‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -147,8 +147,10 @@ Speed is not free. Every third-party engine supports **less** than the
147147
dissection at all, so they offer neither reassembly nor flow tracing, and
148148
:class:`~pcapkit.foundation.engines.pypcapfile.PyPCAPFile` has no IPv6 decoder,
149149
so IPv6 reassembly is unavailable. Each gap is announced with a warning or an
150-
exception rather than silently returning nothing;
151-
:doc:`pcapkit/foundation/engines/index` tabulates them.
150+
exception rather than silently returning nothing, save that
151+
:class:`~pcapkit.foundation.engines.pypcapfile.PyPCAPFile` leaves TCP behind a
152+
VLAN tag, in a tunnel or behind an extension header (an IPv6 one, or AH in IPv4)
153+
out without one; :doc:`pcapkit/foundation/engines/index` tabulates them.
152154

153155
Every engine also answers a preflight check --
154156
:meth:`~pcapkit.foundation.engines.engine.EngineBase.unsupported_reason`, which

‎docs/source/pcapkit/foundation/engines/3rdparty.rst‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -399,9 +399,7 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`.
399399
TCP and UDP and nothing else. IPv6 reassembly is therefore unavailable and is
400400
disabled -- with an :class:`~pcapkit.utilities.warnings.AttributeWarning` --
401401
when requested; IPv4 and TCP reassembly and TCP flow tracing remain
402-
available. The IP packet of a BSD loopback or raw-IP frame, which it does
403-
not decode, is read by :mod:`pcapkit.toolkit.pypcapfile` instead
404-
(:issue:`1667`). PCAP-NG is rejected with a
402+
available. PCAP-NG is rejected with a
405403
:exc:`~pcapkit.utilities.exceptions.FormatError`. `PyPCAPFile`_ 0.12.0 drops
406404
every record of a big-endian savefile, every record from a zero-length one
407405
on, and a truncated final record, so the engine reads the records itself,

‎docs/source/pcapkit/foundation/engines/index.rst‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,8 @@ class hierarchy of :mod:`pcapkit.foundation.engines`:
7070
click PyPCAPFile "/pcapkit/foundation/engines/3rdparty.html#pcapkit.foundation.engines.pypcapfile.PyPCAPFile"
7171

7272
Not every engine can do everything :class:`~pcapkit.foundation.extraction.Extractor`
73-
offers, and the ones that cannot say so rather than quietly doing less:
73+
offers, and the ones that cannot say so rather than quietly doing less, unless
74+
their row says otherwise:
7475

7576
+-----------------------------------------------------------------+---------------------------------------------------------------+
7677
| Engine | Gap, and how it is surfaced |
@@ -92,11 +93,12 @@ offers, and the ones that cannot say so rather than quietly doing less:
9293
| :class:`~pcapkit.foundation.engines.pypcapfile.PyPCAPFile` | no IPv6 decoder, hence no IPv6 reassembly -- disabled with an |
9394
| | :class:`~pcapkit.utilities.warnings.AttributeWarning`, and |
9495
| | :func:`~pcapkit.toolkit.pypcapfile.ipv6_reassembly` raises; |
95-
| | TCP over IPv6, tunnelled in IP (e.g. 6in4) or not, is left |
96-
| | out of TCP reassembly and flow tracing, with an |
96+
| | TCP over IPv6 straight over Ethernet is left out of TCP |
97+
| | reassembly and flow tracing, with an |
9798
| | :class:`~pcapkit.utilities.warnings.AttributeWarning` once |
98-
| | per capture; IPv4 behind 802.1Q/802.1ad VLAN tags is read as |
99-
| | it is untagged; PCAP savefiles only, otherwise |
99+
| | per capture, and TCP behind a VLAN tag, in a tunnel or behind |
100+
| | an extension header (an IPv6 one, or AH in IPv4) without one; |
101+
| | PCAP savefiles only, otherwise |
100102
| | :exc:`~pcapkit.utilities.exceptions.FormatError` |
101103
+-----------------------------------------------------------------+---------------------------------------------------------------+
102104
| :class:`~pcapkit.foundation.engines.dpkt.DPKT`, | no PCAP output, since each frame reaches the writer as a |

‎docs/source/pcapkit/protocols/internet/internet.rst‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -37,8 +37,9 @@ A chain of IPv6 extension headers is dissected to at most
3737
a row, whether it follows IPv6 or another layer such as IPv4. What follows is
3838
kept as :class:`~pcapkit.protocols.misc.raw.Raw`, with one
3939
:exc:`~pcapkit.utilities.warnings.ProtocolWarning` (:issue:`1604`). The
40-
``pypcapfile``, ``dpkt`` and ``scapy`` toolkits stop at the same depth. The
41-
``pyshark`` engine leaves the walk to its own parser, which does not stop
42-
there, so it may still find the upper layer past it (:issue:`1609`).
40+
``dpkt`` and ``scapy`` toolkits stop at the same depth. The ``pyshark`` engine
41+
leaves the walk to its own parser, which does not stop there, so it may still
42+
find the upper layer past it (:issue:`1609`). The ``pypcapfile`` toolkit walks
43+
no extension header chain, so it finds no upper layer behind one.
4344

4445
.. autodata:: pcapkit.protocols.internet.internet.EXTENSION_HEADER_LIMIT

‎docs/source/pcapkit/protocols/protocol.rst‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -116,9 +116,9 @@ there back out, as they do one the TCP parser rejected (:issue:`1518`). A
116116
protocol constructed from its own octets is not counted itself, so the count
117117
starts at the first layer it dissects. One chain of extension headers is also bounded on
118118
its own, by :data:`~pcapkit.protocols.internet.internet.EXTENSION_HEADER_LIMIT`.
119-
The ``pyshark`` engine dissects with its own parser, which does not apply this
120-
bound (:issue:`1609`). The ``dpkt``, ``scapy`` and ``pypcapfile`` toolkits
121-
count the layers they walk as the default engine does, and stop at it.
119+
The ``pyshark`` and ``pypcapfile`` engines dissect with their own parsers,
120+
which do not apply this bound (:issue:`1609`). The ``dpkt`` and ``scapy``
121+
toolkits count the layers they walk as the default engine does, and stop at it.
122122

123123
.. autodata:: pcapkit.protocols.protocol.FRAME_LAYER_LIMIT
124124

‎docs/source/pcapkit/toolkit/3rdparty.rst‎

Lines changed: 7 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -184,25 +184,15 @@ cannot be used for it.
184184
:data:`None` -- which would be indistinguishable from "this frame carries no
185185
IPv6 fragment".
186186

187-
Nor does it decode a VLAN tag or a tunnel. The adapters read an IPv4 packet
188-
behind 802.1Q and 802.1ad tags as `PyPCAPFile`_ reads an untagged one, and
189-
TCP tunnelled in IP (e.g. 4in4, 4in6) from its innermost IPv4 packet, keyed by
190-
that packet's addresses as the default engine keys it, while TCP over IPv6,
191-
tunnelled (e.g. 6in4, 6in6) or not, is left out of
187+
TCP over IPv6 is therefore left out of
192188
:func:`~pcapkit.toolkit.pypcapfile.tcp_reassembly` and
193189
:func:`~pcapkit.toolkit.pypcapfile.tcp_traceflow`, with an
194-
:class:`~pcapkit.utilities.warnings.AttributeWarning` once per capture.
195-
196-
Nor does it decode a link layer but Ethernet. The adapters read the IP packet
197-
of a BSD loopback (``NULL``, ``LOOP``) or raw-IP (``RAW``, ``IPV4``,
198-
``IPV6``) frame themselves, as the default engine dissects it (see
199-
:data:`~pcapkit.toolkit.pypcapfile.READ_LINKTYPES`, :issue:`1667`). Every
200-
other link type is left unread, as the default engine leaves it, and the
201-
engine warns of it with an :class:`~pcapkit.utilities.warnings.AttributeWarning`.
202-
Nor do the adapters read deeper than the default engine dissects: each VLAN
203-
tag, IP header, AH and IPv6 extension header is one of the
204-
:data:`~pcapkit.protocols.protocol.FRAME_LAYER_LIMIT` layers of a frame, and
205-
nothing past the last of them is read (:issue:`1609`).
190+
:class:`~pcapkit.utilities.warnings.AttributeWarning` once per capture when
191+
the IPv6 packet is straight over Ethernet. `PyPCAPFile`_ decodes no link
192+
layer but Ethernet, and no VLAN tag or tunnel, so a frame of any other link
193+
type is left undecoded, with the engine's own warning, and TCP behind a VLAN
194+
tag, in a tunnel or behind an extension header (an IPv6 one, or AH in IPv4)
195+
is left out without one.
206196

207197
.. autofunction:: pcapkit.toolkit.pypcapfile.ipv4_reassembly
208198

@@ -226,9 +216,6 @@ Auxiliary Functions
226216
Internal Definitions
227217
--------------------
228218

229-
.. autodata:: pcapkit.toolkit.pypcapfile.READ_LINKTYPES
230-
:no-value:
231-
232219
.. autodata:: pcapkit.toolkit.pypcapfile.TCP_MIN_HEADER_LEN
233220

234221
.. autodata:: pcapkit.toolkit.pypcapfile.IPV4_FLAG_DF

‎pcapkit/foundation/engines/pypcapfile.py‎

Lines changed: 19 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -44,18 +44,15 @@ class PyPCAPFile(EngineBase['PCAPFilePacket']):
4444
* reads PCAP savefiles only, raising
4545
:exc:`~pcapkit.utilities.exceptions.FormatError` on PCAP-NG,
4646
* reads the savefile's records itself, into `PyPCAPFile`_'s own packet
47-
objects, as `PyPCAPFile`_ 0.12.0 drops records the ``default`` engine
48-
reads -- see :meth:`_read_records`,
47+
objects, as `PyPCAPFile`_ 0.12.0 drops some of them -- see
48+
:meth:`_read_records`,
4949
* disables IPv6 reassembly, warning as it does so, while leaving IPv4 and
50-
TCP reassembly and TCP flow tracing in place,
50+
TCP reassembly and TCP flow tracing in place, and
5151
* leaves TCP over IPv6 out of TCP reassembly and flow tracing, with one
52-
:exc:`~pcapkit.utilities.warnings.AttributeWarning` per capture -- see
53-
:mod:`pcapkit.toolkit.pypcapfile` -- and
54-
* leaves the frames of a link type `PyPCAPFile`_ has no decoder for
55-
undecoded, from which :mod:`pcapkit.toolkit.pypcapfile` reads the IP
56-
packet of the BSD loopback and raw-IP ones itself, as the ``default``
57-
engine dissects it (see
58-
:data:`~pcapkit.toolkit.pypcapfile.READ_LINKTYPES`).
52+
:exc:`~pcapkit.utilities.warnings.AttributeWarning` per capture when the
53+
IPv6 packet is straight over Ethernet, and TCP behind a VLAN tag, in a
54+
tunnel or behind an extension header (an IPv6 one, or AH in IPv4) out
55+
without one -- see :mod:`pcapkit.toolkit.pypcapfile`.
5956
6057
The engine stops decoding at the network layer rather than descending into
6158
the transport layer. `PyPCAPFile`_ decoders *replace* the payload bytes of
@@ -189,10 +186,8 @@ def run(self) -> 'None':
189186
support such operations.
190187
* if IPv6 reassembly is enabled, as :mod:`pcapfile` has no IPv6
191188
decoder.
192-
* if the savefile's link layer type is left unread: neither
193-
:mod:`pcapfile` has a decoder for it nor
194-
:mod:`pcapkit.toolkit.pypcapfile` reads it (see
195-
:meth:`_get_decoder`).
189+
* if :mod:`pcapfile` has no decoder for the savefile's link layer
190+
type, in which case frames are left undecoded.
196191
197192
Raises:
198193
FormatError: If the file format is not supported, i.e., not a PCAP
@@ -309,36 +304,29 @@ def _get_decoder(self, linktype: 'int') -> 'Optional[Callable[..., Any]]':
309304
The decoder class, or :data:`None` when :mod:`pcapfile` has none.
310305
311306
Warns:
312-
AttributeWarning: If no decoder is available and the link layer type
313-
is none of the :data:`~pcapkit.toolkit.pypcapfile.READ_LINKTYPES`,
314-
as frames will then be left hexlified -- see :meth:`run` -- and
315-
no reassembly or flow tracing is possible. The frames of those
316-
link types are left undecoded too, but reassembly and flow
317-
tracing read the IP packet in them, as the ``default`` engine
318-
does, so nothing is lost there to warn of.
307+
AttributeWarning: If no decoder is available, as frames will then be
308+
left hexlified -- see :meth:`run` -- and no reassembly or flow
309+
tracing is possible.
319310
320311
"""
321-
from pcapkit.toolkit.pypcapfile import READ_LINKTYPES # isort:skip
322-
323312
try:
324313
decoder = self._expkg.linklayer.clookup(linktype)
325314
except IndexError: # malformed entry in ``pcapfile.linklayer.__LL_TYPES__``
326315
decoder = None
327316

328317
if not callable(decoder):
329-
if linktype not in READ_LINKTYPES:
330-
warn(f'unrecognised link layer protocol: {self._dlink!r}; frames will be left '
331-
'undecoded and all analysis functions ignored', AttributeWarning,
332-
stacklevel=stacklevel())
318+
warn(f'unrecognised link layer protocol: {self._dlink!r}; frames will be left '
319+
'undecoded and all analysis functions ignored', AttributeWarning,
320+
stacklevel=stacklevel())
333321
return None
334322
return decoder
335323

336324
def _read_records(self, file: 'NamedStream',
337325
header: 'PCAPFileHeader') -> 'Iterator[PCAPFilePacket]':
338-
"""Read the savefile's records, as the ``default`` engine reads them.
326+
"""Read the savefile's records, every one of them.
339327
340328
:func:`pcapfile.savefile._read_a_packet` (pypcapfile 0.12.0) drops
341-
records the ``default`` engine reads, in two ways:
329+
records in two ways:
342330
343331
* It picks the byte order of each record header by testing
344332
``hdrp[0].byteorder == 'big'`` -- :class:`bytes` against :class:`str`,
@@ -354,9 +342,8 @@ def _read_records(self, file: 'NamedStream',
354342
builds with ``layers=0`` -- the frame hexlified, pointing at the
355343
savefile's own header -- so :meth:`_decode` and
356344
:mod:`pcapkit.toolkit.pypcapfile` see what :mod:`pcapfile` hands them.
357-
It stops where :class:`~pcapkit.protocols.misc.pcap.frame.Frame` stops,
358-
at a tail too short for a record header, and keeps a record whose capture
359-
length runs past the data as that does: the capture length as declared,
345+
It stops at a tail too short for a record header, and keeps a record
346+
whose capture length runs past the data: the capture length as declared,
360347
the octets present as the frame.
361348
362349
Args:

‎pcapkit/toolkit/dpkt.py‎

Lines changed: 6 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -649,9 +649,8 @@ def _ipv4_header_accepted(header: 'bytes') -> 'bool':
649649
650650
Note:
651651
A header of version 4 with no options is not parsed, as the parser
652-
checks nothing else of one; any other is asked of the parser itself, as
653-
the pypcapfile engine asks it (see
654-
:func:`pcapkit.toolkit.pypcapfile._default_accepts`).
652+
checks nothing else of one; any other is asked of the parser itself
653+
(see :func:`pcapkit.toolkit.pypcapfile._default_accepts`).
655654
656655
"""
657656
return header[0] == 0x45 or _default_accepts('IPv4', bytes(header))
@@ -1023,11 +1022,10 @@ def _tcp_carrier(packet: 'Packet') -> 'Optional[_Segment]':
10231022
(:func:`pcapkit.toolkit.pcap.tcp_segment`), as `DPKT`_ decodes one, and
10241023
an IPv6 packet's extension headers do not come between, as it dissects
10251024
them itself. Behind a header that is a layer of its own, it does not, so
1026-
the parser's answer stands there, as for the pypcapfile engine. The TCP
1027-
endpoints are the innermost IP layer's addresses, as for the default
1028-
engine; the outer ones are the tunnel's (:issue:`1581`). Each flag is
1029-
read from the flags octet alone, as the default engine reads it
1030-
(:issue:`1618`).
1025+
the parser's answer stands there. The TCP endpoints are the innermost IP
1026+
layer's addresses, as for the default engine; the outer ones are the
1027+
tunnel's (:issue:`1581`). Each flag is read from the flags octet alone,
1028+
as the default engine reads it (:issue:`1618`).
10311029
10321030
.. _DPKT: https://dpkt.readthedocs.io
10331031

0 commit comments

Comments
 (0)