Skip to content

Commit 71df9d4

Browse files
committed
fix(engines): take a scapy packet's link type and clock from its interface (#1503, #1517, #1549)
- A Simple Packet Block has no timestamp, and scapy left the packet with the time it was built; the engine's PCAP-NG reader now sets it to 0, as the default engine reads it (#1503). - The flow tracer looked the link type up by the first layer's name, so a raw IPv4 packet (`IP`) raised MissingKeyError. Both readers now attach the interface's link type with `attach_linktype`, and `tcp_traceflow` uses it; only a packet built by hand is still looked up by name, with no default (#1517). - scapy ignores `if_tsoffset`; the PCAP-NG reader now reads option 14 of each interface itself and adds it to every timestamp on it (#1549). - The overrides hold on scapy 2.5, 2.7 and 2.8: `_check_interface_id` passes its result through (2.8 skips a block on a false one), the offsets are kept by the reader rather than in scapy's interface tuple, and a missing hook raises VersionError. - Document `packet2frame`, `attach_resolution` and `attach_linktype`; delete the #1503 Gap row of the engine agreement table. New runtime and unit tests fail on main; scapy, trace and agreement selections pass on scapy 2.7.0 and 2.8.0.
1 parent ea883a6 commit 71df9d4

7 files changed

Lines changed: 613 additions & 35 deletions

File tree

‎docs/source/pcapkit/toolkit/3rdparty.rst‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,12 @@ Auxiliary Functions
3333

3434
.. autofunction:: pcapkit.toolkit.scapy.packet2dict
3535

36+
.. autofunction:: pcapkit.toolkit.scapy.packet2frame
37+
38+
.. autofunction:: pcapkit.toolkit.scapy.attach_resolution
39+
40+
.. autofunction:: pcapkit.toolkit.scapy.attach_linktype
41+
3642
DPKT Tools
3743
==========
3844

‎pcapkit/foundation/engines/scapy.py‎

Lines changed: 97 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -37,10 +37,11 @@
3737
warnings.filterwarnings('ignore', category=CryptographyDeprecationWarning)
3838
3939
"""
40+
import struct
4041
from typing import TYPE_CHECKING, cast
4142

4243
from pcapkit.foundation.engines.engine import EngineBase
43-
from pcapkit.utilities.exceptions import stacklevel
44+
from pcapkit.utilities.exceptions import VersionError, stacklevel
4445
from pcapkit.utilities.logging import get_logger
4546
from pcapkit.utilities.warnings import AttributeWarning, warn
4647

@@ -58,37 +59,89 @@
5859
#: :data:`pcapkit.utilities.logging.logger`.
5960
logger = get_logger(__name__)
6061

62+
#: The private methods of :class:`scapy.utils.PcapNgReader` that
63+
#: :func:`_reader_type` overrides.
64+
_READER_HOOKS = ('_read_block', '_read_block_shb', '_read_block_idb', '_check_interface_id')
65+
66+
67+
def _read_tsoffset(options: 'bytes', endian: 'str') -> 'int':
68+
"""Read the ``if_tsoffset`` of a PCAP-NG Interface Description Block (#1549).
69+
70+
Read here rather than by `Scapy`_'s ``_read_options``, whose result differs
71+
by version: keyed by option code in 2.7 and 2.8, and in 2.5 by name, with
72+
only ``tsresol`` and ``comment`` kept.
73+
74+
Args:
75+
options: The block's options area.
76+
endian: :mod:`struct` byte order of the section, ``'<'`` or ``'>'``.
77+
78+
Returns:
79+
The offset (option 14), a signed count of seconds; 0 without one.
80+
81+
.. _Scapy: https://scapy.net
82+
83+
"""
84+
while len(options) >= 4:
85+
code, length = struct.unpack(f'{endian}HH', options[:4])
86+
if not code: # opt_endofopt
87+
break
88+
if code == 14 and length == 8 and len(options) >= 12:
89+
return cast('int', struct.unpack(f'{endian}q', options[4:12])[0])
90+
options = options[4 + length + (-length % 4):]
91+
return 0
92+
93+
6194
def _reader_type() -> 'Type[PcapReader]':
62-
"""A :class:`scapy.utils.PcapReader` that stamps each packet with its resolution.
95+
"""A :class:`scapy.utils.PcapReader` that stamps each packet with its interface.
6396
6497
`Scapy`_'s readers convert a record's timestamp to a :class:`~decimal.Decimal`
6598
and drop the resolution it was counted in, and an exact count of microseconds
6699
comes out with the same digits whether the capture counted microseconds or
67-
nanoseconds. The PCAP frame record of a packet needs that resolution, so these
68-
subclasses attach it with :func:`pcapkit.toolkit.scapy.attach_resolution`:
69-
from the global header of a PCAP, and from the interface of each PCAP-NG
70-
packet block, which :meth:`~scapy.utils.RawPcapNgReader._read_block` is the
71-
last place to see.
100+
nanoseconds. They drop the link type too, once it has picked the packet's
101+
first layer, and a layer does not name its link type: a raw IPv4 packet's is
102+
:class:`~scapy.layers.inet.IP` (#1517). The PCAP frame record and the flow
103+
tracing input of a packet need both, so these subclasses attach them with
104+
:func:`pcapkit.toolkit.scapy.attach_resolution` and
105+
:func:`pcapkit.toolkit.scapy.attach_linktype`: from the global header of a
106+
PCAP, and from the interface of each PCAP-NG packet block, which
107+
:meth:`~scapy.utils.RawPcapNgReader._read_block` is the last place to see.
72108
73109
The PCAP-NG subclass also starts each section with no interfaces, as the
74110
PCAP-NG specification defines a section: an interface ID counts the Interface
75111
Description Blocks of its own section only. `Scapy`_'s reader keeps appending
76112
to one table across Section Header Blocks, so a later section's interface 0
77113
resolved to the first section's, with its link type and resolution (#1522).
78114
115+
And it dates two kinds of packet as the default engine does. A Simple Packet
116+
Block has no timestamp, and `Scapy`_ leaves its packet with the time it was
117+
built, so it is set to 0 (#1503). `Scapy`_ ignores an interface's
118+
``if_tsoffset``, the seconds to add to each of its timestamps, so the
119+
subclass reads it and adds it (#1549).
120+
79121
Built when called, so that importing this module does not import `Scapy`_,
80122
and so that it always subclasses the :class:`scapy.utils.PcapReader` of the
81123
moment.
82124
83125
Returns:
84126
A reader type, which opens a PCAP-NG file as its PCAP-NG alternative.
85127
128+
Raises:
129+
VersionError: If :class:`scapy.utils.PcapNgReader` lacks a method in
130+
:data:`_READER_HOOKS`, which these subclasses override.
131+
86132
.. _Scapy: https://scapy.net
87133
88134
"""
89-
from scapy.utils import PcapNgReader, PcapReader # isort:skip
135+
from scapy.utils import EDecimal, PcapNgReader, PcapReader # isort:skip
136+
137+
from pcapkit.toolkit.scapy import attach_linktype, attach_resolution # isort:skip
90138

91-
from pcapkit.toolkit.scapy import attach_resolution # isort:skip
139+
# The PCAP-NG hooks overridden below are private to Scapy. One renamed away
140+
# would leave its fix unapplied without a word, so a missing one is an error.
141+
missing = [name for name in _READER_HOOKS if not callable(getattr(PcapNgReader, name, None))]
142+
if missing:
143+
raise VersionError(f"scapy.utils.PcapNgReader has no {', '.join(missing)}, which the Scapy "
144+
"engine's reader overrides; this Scapy version is not supported")
92145

93146
class _Closing:
94147
# ``sniff`` closes the reader at the end of the file, and the engine closes
@@ -105,28 +158,63 @@ class _PcapReader(_Closing, PcapReader):
105158
def read_packet(self, *args: 'Any', **kwargs: 'Any') -> 'ScapyPacket':
106159
packet = super().read_packet(*args, **kwargs)
107160
attach_resolution(packet, 1_000_000_000 if self.nano else 1_000_000)
161+
attach_linktype(packet, self.linktype)
108162
return packet
109163

110164
class _PcapNgReader(_Closing, PcapNgReader):
111165
# the metaclass makes this ``_PcapReader``'s alternative in turn
112166
alternative = _PcapReader
113167

114168
_resolution = 1_000_000
169+
# each set from the packet block last read, before ``read_packet`` reads it
170+
_linktype: 'int'
171+
_timed: 'bool'
172+
_offset: 'int'
173+
_interface: 'int'
174+
#: ``if_tsoffset`` of each interface of the section, by interface ID.
175+
#: Kept here: what Scapy keeps of an interface differs by version.
176+
_offsets: 'list[int]'
115177

116178
def _read_block_shb(self, *args: 'Any', **kwargs: 'Any') -> 'None':
117179
super()._read_block_shb(*args, **kwargs)
118180
# a new section: its interface IDs restart from 0 (#1522)
119181
self.interfaces = []
182+
self._offsets = []
183+
184+
def _read_block_idb(self, block: 'bytes', *args: 'Any', **kwargs: 'Any') -> 'None':
185+
super()._read_block_idb(block, *args, **kwargs)
186+
# appended once Scapy has appended the interface, so the IDs match
187+
self._offsets.append(_read_tsoffset(block[8:], self.endian))
188+
189+
def _check_interface_id(self, intid: 'int') -> 'Any':
190+
# Each packet block's reader checks its interface ID here, a Simple
191+
# Packet Block's as 0, and the metadata it returns does not keep it.
192+
# Scapy 2.8 skips the block when this returns false, and before 2.8
193+
# it raises instead, so the result must be passed through. A block
194+
# it rejects is never returned, so its ID is never read.
195+
self._interface = intid
196+
return super()._check_interface_id(intid)
120197

121198
def _read_block(self, *args: 'Any', **kwargs: 'Any') -> 'Any':
122199
block = super()._read_block(*args, **kwargs)
123200
if block is not None:
124201
self._resolution = block[1].tsresol
202+
self._linktype = block[1].linktype
203+
# ``None`` for a Simple Packet Block, which has no timestamp (#1503)
204+
self._timed = block[1].tshigh is not None
205+
self._offset = self._offsets[self._interface]
125206
return block
126207

127208
def read_packet(self, *args: 'Any', **kwargs: 'Any') -> 'ScapyPacket':
128209
packet = super().read_packet(*args, **kwargs)
129210
attach_resolution(packet, self._resolution)
211+
attach_linktype(packet, self._linktype)
212+
if not self._timed:
213+
# no timestamp at all, which the default engine reads as 0, where
214+
# Scapy leaves the time the packet was built (#1503)
215+
packet.time = EDecimal(0)
216+
elif self._offset:
217+
packet.time += self._offset # if_tsoffset (#1549)
130218
return packet
131219

132220
return _PcapReader

‎pcapkit/toolkit/scapy.py‎

Lines changed: 47 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@
6666
from scapy.packet import Packet
6767

6868
__all__ = [
69-
'attach_resolution', 'packet2chain', 'packet2dict', 'packet2frame',
69+
'attach_linktype', 'attach_resolution', 'packet2chain', 'packet2dict', 'packet2frame',
7070
'ipv4_reassembly', 'ipv6_reassembly', 'tcp_reassembly', 'tcp_traceflow'
7171
]
7272

@@ -75,6 +75,10 @@
7575
#: second, by :func:`attach_resolution`.
7676
RESOLUTION_ATTR = '__pcapkit_resolution__'
7777

78+
#: Attribute a packet's data link type is stashed under, as the capture's
79+
#: ``LINKTYPE_*`` number, by :func:`attach_linktype`.
80+
LINKTYPE_ATTR = '__pcapkit_linktype__'
81+
7882

7983
def attach_resolution(packet: 'Packet', resolution: 'int') -> 'None':
8084
"""Stash the timestamp resolution of a packet's capture on the packet.
@@ -88,6 +92,18 @@ def attach_resolution(packet: 'Packet', resolution: 'int') -> 'None':
8892
setattr(packet, RESOLUTION_ATTR, resolution)
8993

9094

95+
def attach_linktype(packet: 'Packet', linktype: 'int') -> 'None':
96+
"""Stash the data link type of a packet's capture on the packet.
97+
98+
Args:
99+
packet: Scapy packet.
100+
linktype: ``LINKTYPE_*`` number: the PCAP global header's, or the
101+
PCAP-NG interface's the packet was captured on.
102+
103+
"""
104+
setattr(packet, LINKTYPE_ATTR, linktype)
105+
106+
91107
def packet2chain(packet: 'Packet') -> 'str':
92108
"""Fetch Scapy packet protocol chain.
93109
@@ -408,27 +424,36 @@ def tcp_traceflow(packet: 'Packet', *, count: 'int' = -1) -> 'TF_TCP_Packet | No
408424
ip = cast('IP', packet['IP']) if 'IP' in packet else cast('IPv6', packet['IPv6'])
409425
tcp = cast('TCP', packet['TCP'])
410426

411-
# NOTE: no default here, deliberately. ``get()`` with no default
412-
# raises on an unresolvable name rather than minting one (#775).
413-
# NULL and RAW are genuine DLTs -- BSD loopback and raw IP
414-
# framing, respectively -- each meant to go with its own handler
415-
# protocol class, so neither is an honest stand-in for "unknown link
416-
# type" and this must not paper over the miss with either. An
417-
# IP-rooted Scapy packet's ``(IP()/TCP()).name`` is ``'IP'``, which is
418-
# not a LinkType member name, and raises. Note the asymmetry, which
419-
# is not a choice made here: an IPv6-rooted packet's name uppercases to
420-
# ``'IPV6'``, which *is* a member (``LinkType.IPV6``, 229), so it
421-
# resolves silently -- to a DLT the caller never chose. Only the v4 name
422-
# happens to miss. The ``KeyError``
423-
# from :meth:`LinkType.get` is caught and re-raised as
424-
# :exc:`~pcapkit.utilities.exceptions.MissingKeyError` -- this
425-
# package's own house exception for a lookup miss -- rather than
426-
# letting it escape this public function.
427-
name = packet.name.upper()
428-
try:
429-
protocol = Enum_LinkType.get(name)
430-
except KeyError:
431-
raise MissingKeyError(name) from None
427+
# NOTE: the link type of the interface the packet was read from, which
428+
# :class:`~pcapkit.foundation.engines.scapy.Scapy` attaches with
429+
# :func:`attach_linktype` -- not one inferred from the first layer,
430+
# which a raw IPv4 interface's packets share with no link type (#1517).
431+
linktype = getattr(packet, LINKTYPE_ATTR, None)
432+
if linktype is not None:
433+
protocol = Enum_LinkType.get(linktype)
434+
else:
435+
# A packet built by hand carries none, so it is looked up by its
436+
# first layer's name -- with no default, deliberately. ``get()``
437+
# with no default raises on an unresolvable name rather than minting
438+
# one (#775). NULL and RAW are genuine DLTs -- BSD loopback and raw
439+
# IP framing, respectively -- each meant to go with its own handler
440+
# protocol class, so neither is an honest stand-in for "unknown link
441+
# type" and this must not paper over the miss with either. An
442+
# IP-rooted Scapy packet's ``(IP()/TCP()).name`` is ``'IP'``, which
443+
# is not a LinkType member name, and raises. Note the asymmetry,
444+
# which is not a choice made here: an IPv6-rooted packet's name
445+
# uppercases to ``'IPV6'``, which *is* a member (``LinkType.IPV6``,
446+
# 229), so it resolves silently -- to a DLT the caller never chose.
447+
# Only the v4 name happens to miss. The ``KeyError`` from
448+
# :meth:`LinkType.get` is caught and re-raised as
449+
# :exc:`~pcapkit.utilities.exceptions.MissingKeyError` -- this
450+
# package's own house exception for a lookup miss -- rather than
451+
# letting it escape this public function.
452+
name = packet.name.upper()
453+
try:
454+
protocol = Enum_LinkType.get(name)
455+
except KeyError:
456+
raise MissingKeyError(name) from None
432457

433458
data = TF_TCP_Packet( # type: ignore[type-var]
434459
protocol=protocol, # data link type

‎tests/foundation/engines/test_engine_agreement_runtime.py‎

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -394,10 +394,6 @@ class TestEngineAgreement(harness.RoundTripBase):
394394
('INPUTS', 'DATAGRAMS'), ('extra, default has none', 'datagrams != '),
395395
('test.pcapng/scapy/ipv4', 'test.pcapng/scapy/datagrams',
396396
'many_interfaces.pcapng/scapy/ipv4', 'many_interfaces.pcapng/scapy/datagrams')),
397-
Gap(1503, 'the scapy engine dates a Simple Packet Block, which carries no timestamp, with '
398-
'the wall-clock time it was read (scapy leaves Packet.time at construction; '
399-
'pcapkit/toolkit/scapy.py:186 forwards it), where default reports 0',
400-
'RECORDS', 'frame 4: timestamp ', ('test.pcapng/scapy/timestamps',)),
401397
)
402398

403399
def setUp(self) -> None:

0 commit comments

Comments
 (0)