Skip to content

Commit 5fcf07d

Browse files
committed
fix(engines): the dpkt engine applies the first PCAP-NG interface's linktype and timestamp resolution to every frame
dpkt.pcapng.Reader reads only the first Interface Description Block, so every packet got interface 0's link type and if_tsresol, and a later section's interfaces were never read. The DPKT engine now reads PCAP-NG with its own PCAPNGReader: SHBs and IDBs are parsed by pcapkit's PCAPNG protocol per section, packet blocks are still decoded by dpkt, and each packet takes the link type, if_tsresol and if_tsoffset of the interface it names, with the default engine's timestamp arithmetic. Closes #1379
1 parent 970fed9 commit 5fcf07d

4 files changed

Lines changed: 324 additions & 12 deletions

File tree

‎docs/source/pcapkit/foundation/engines/3rdparty.rst‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -69,8 +69,9 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`.
6969
On PCAP-NG, `DPKT`_ yields **Enhanced Packet Blocks and (obsolete) Packet
7070
Blocks only**. ``dpkt.pcapng.Reader.__iter__`` skips every other block type
7171
without a warning (``# just ignore other blocks``, as of ``dpkt`` 1.9.8), and
72-
that includes the **Simple Packet Block**, which carries a packet just as an
73-
Enhanced Packet Block does. Such packets are silently missing from this
72+
:class:`~pcapkit.foundation.engines.dpkt.PCAPNGReader`, which this engine
73+
reads PCAP-NG with, keeps that block set. It includes the **Simple Packet
74+
Block**, which carries a packet just as an Enhanced Packet Block does. Such packets are silently missing from this
7475
engine's output, and the frames after them are numbered as if they had never
7576
been captured.
7677

@@ -83,6 +84,15 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`.
8384
`DPKT`_, not of :mod:`pcapkit`, so read a capture that may contain Simple
8485
Packet Blocks with the default engine instead.
8586

87+
.. note::
88+
89+
``dpkt.pcapng.Reader`` reads only the **first** Interface Description Block
90+
of a file, and applies its link type and timestamp resolution to every
91+
packet. This engine therefore reads PCAP-NG with
92+
:class:`~pcapkit.foundation.engines.dpkt.PCAPNGReader` instead, which
93+
resolves each packet's link type, ``if_tsresol`` and ``if_tsoffset`` from
94+
the interface it names, in its own section (:issue:`1379`).
95+
8696
.. autoclass:: pcapkit.foundation.engines.dpkt.DPKT
8797
:no-members:
8898
:show-inheritance:
@@ -96,6 +106,10 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`.
96106
.. autoattribute:: _expkg
97107
.. autoattribute:: _extmp
98108

109+
.. autoclass:: pcapkit.foundation.engines.dpkt.PCAPNGReader
110+
:members: datalink
111+
:show-inheritance:
112+
99113
PyShark Support
100114
===============
101115

‎pcapkit/foundation/engines/dpkt.py‎

Lines changed: 150 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -10,36 +10,175 @@
1010
.. _DPKT: https://dpkt.readthedocs.io
1111
1212
"""
13+
import decimal
1314
from typing import TYPE_CHECKING, cast
1415

16+
from pcapkit.const.pcapng.block_type import BlockType as Enum_BlockType
17+
from pcapkit.const.pcapng.option_type import OptionType as Enum_OptionType
1518
from pcapkit.const.reg.linktype import LinkType as Enum_LinkType
1619
from pcapkit.foundation.engines.engine import EngineBase
20+
from pcapkit.utilities.compat import localcontext
1721
from pcapkit.utilities.exceptions import FormatError, stacklevel
1822
from pcapkit.utilities.logging import get_logger
1923
from pcapkit.utilities.warnings import AttributeWarning, DPKTWarning, warn
2024

21-
__all__ = ['DPKT']
25+
__all__ = ['DPKT', 'PCAPNGReader']
2226

2327
if TYPE_CHECKING:
24-
from typing import Optional, Type, Union
28+
from typing import IO, Iterator, Literal, Optional, Type, Union
2529

2630
from dpkt.dpkt import Packet as DPKTPacket
2731
from dpkt.pcap import Reader as PCAPReader
28-
from dpkt.pcapng import Reader as PCAPNGReader
2932

33+
from pcapkit.foundation.engines.pcapng import Context
3034
from pcapkit.foundation.extraction import Extractor
35+
from pcapkit.protocols.data.misc.pcapng import \
36+
InterfaceDescriptionBlock as Data_InterfaceDescriptionBlock
37+
from pcapkit.protocols.data.misc.pcapng import SectionHeaderBlock as Data_SectionHeaderBlock
3138

32-
Reader = Union[PCAPReader, PCAPNGReader]
39+
Reader = Union[PCAPReader, 'PCAPNGReader']
3340

3441
#: logging.Logger: Module-level logger, a child of the package-wide
3542
#: :data:`pcapkit.utilities.logging.logger`.
3643
logger = get_logger(__name__)
3744

3845

46+
class PCAPNGReader:
47+
"""PCAP-NG reader for the DPKT engine, aware of every interface.
48+
49+
:class:`dpkt.pcapng.Reader` reads only the first Interface Description
50+
Block (IDB) of a file, and applies its link type and timestamp resolution
51+
to every packet (:issue:`1379`). This reader walks the blocks itself
52+
instead: it parses each Section Header Block and IDB with :mod:`pcapkit`'s
53+
own :class:`~pcapkit.protocols.misc.pcapng.PCAPNG`, keeps the interfaces
54+
of the current section, and resolves every packet's link type,
55+
``if_tsresol`` and ``if_tsoffset`` from the interface it names. Packet
56+
blocks are still decoded by `DPKT`_.
57+
58+
As :class:`dpkt.pcapng.Reader` does, it yields Enhanced Packet Blocks and
59+
(obsolete) Packet Blocks only, and skips every other block.
60+
61+
.. _DPKT: https://dpkt.readthedocs.io
62+
63+
Args:
64+
file: Source PCAP-NG stream, positioned at its first Section Header Block.
65+
66+
"""
67+
68+
def __init__(self, file: 'IO[bytes]') -> 'None':
69+
#: Source PCAP-NG stream.
70+
self._file = file
71+
#: Context of the current section.
72+
self._ctx = cast('Context', None)
73+
#: Section index number.
74+
self._sect = 0
75+
#: Link type of the packet last yielded.
76+
self._linktype = cast('Enum_LinkType', Enum_LinkType.NULL)
77+
#: Underlying block iterator.
78+
self._iter = self._read()
79+
80+
def datalink(self) -> 'Enum_LinkType':
81+
"""Link type of the interface the packet last yielded was captured on."""
82+
return self._linktype
83+
84+
def __iter__(self) -> 'PCAPNGReader':
85+
return self
86+
87+
def __next__(self) -> 'tuple[float, bytes]':
88+
return next(self._iter)
89+
90+
def _read(self) -> 'Iterator[tuple[float, bytes]]':
91+
"""Read blocks until the next packet block, and yield its timestamp and data."""
92+
from pcapkit.foundation.engines.pcapng import Context
93+
from pcapkit.protocols.misc.pcapng import PCAPNG as P_PCAPNG
94+
95+
while True:
96+
head = self._file.read(8)
97+
if len(head) < 8:
98+
return
99+
100+
# the SHB block type is a palindrome, so it reads the same in
101+
# either byte order; any other block uses the section's order
102+
byteorder: 'Literal["big", "little"]'
103+
if head[:4] == b'\x0a\x0d\x0d\x0a':
104+
head += self._file.read(4)
105+
byteorder = 'little' if head[8:] == b'\x4d\x3c\x2b\x1a' else 'big'
106+
else:
107+
byteorder = self._ctx.section.byteorder
108+
block_type = int.from_bytes(head[:4], byteorder)
109+
length = int.from_bytes(head[4:8], byteorder)
110+
if length < 12:
111+
raise FormatError(f'PCAP-NG: [Block {block_type}] invalid block length: {length}')
112+
113+
buf = head + self._file.read(length - len(head))
114+
if len(buf) < length:
115+
return
116+
117+
if block_type == Enum_BlockType.Section_Header_Block:
118+
self._sect += 1
119+
shb = P_PCAPNG(buf, num=0, sct=self._sect, ctx=None)
120+
self._ctx = Context(cast('Data_SectionHeaderBlock', shb.info))
121+
elif block_type == Enum_BlockType.Interface_Description_Block:
122+
idb = P_PCAPNG(buf, num=0, sct=self._sect, ctx=self._ctx)
123+
self._ctx.interfaces.append(cast('Data_InterfaceDescriptionBlock', idb.info))
124+
elif block_type in (Enum_BlockType.Enhanced_Packet_Block, Enum_BlockType.Packet_Block):
125+
yield self._read_packet(buf, block_type, byteorder == 'little')
126+
127+
def _read_packet(self, buf: 'bytes', block_type: 'int',
128+
little: 'bool') -> 'tuple[float, bytes]':
129+
"""Decode a packet block with `DPKT`_, against the interface it names.
130+
131+
Args:
132+
buf: Whole block, as read from the file.
133+
block_type: Block type, an EPB or an (obsolete) Packet Block.
134+
little: Whether the enclosing section is little-endian.
135+
136+
Returns:
137+
Timestamp in seconds since the UNIX epoch, and the packet data.
138+
139+
Raises:
140+
FormatError: If the section describes no interface of the ID the
141+
block names.
142+
143+
"""
144+
import dpkt # isort:skip
145+
146+
if block_type == Enum_BlockType.Enhanced_Packet_Block:
147+
tag = 'EPB'
148+
pkt = (dpkt.pcapng.EnhancedPacketBlockLE(buf) if little
149+
else dpkt.pcapng.EnhancedPacketBlock(buf))
150+
else:
151+
tag = 'Packet'
152+
pkt = dpkt.pcapng.PacketBlockLE(buf) if little else dpkt.pcapng.PacketBlock(buf)
153+
154+
iface_id = pkt.iface_id # pylint: disable=no-member
155+
if iface_id >= len(self._ctx.interfaces):
156+
raise FormatError(f'PCAP-NG: [{tag}] invalid interface ID: {iface_id}')
157+
interface = self._ctx.interfaces[iface_id]
158+
159+
tsresol = interface.options.get(Enum_OptionType.if_tsresol)
160+
tsoffset = interface.options.get(Enum_OptionType.if_tsoffset)
161+
resolution = 1_000_000 if tsresol is None else tsresol.resolution
162+
offset = 0 if tsoffset is None else tsoffset.offset
163+
164+
# same arithmetic as :meth:`PCAPNG._read_timestamp
165+
# <pcapkit.protocols.misc.pcapng.PCAPNG._read_timestamp>`
166+
ticks = (pkt.ts_high << 32) | pkt.ts_low
167+
with localcontext(prec=64):
168+
timestamp = decimal.Decimal(ticks) / resolution + offset
169+
170+
self._linktype = interface.linktype
171+
return float(timestamp), pkt.pkt_data
172+
173+
39174
class DPKT(EngineBase['DPKTPacket']):
40175
"""DPKT engine support.
41176
42-
On PCAP-NG, `DPKT`_'s reader yields Enhanced Packet Blocks and (obsolete)
177+
A PCAP-NG file is read by :class:`PCAPNGReader` rather than by
178+
:class:`dpkt.pcapng.Reader`, so each packet takes the link type and the
179+
timestamp resolution and offset of the interface it was captured on.
180+
181+
Like `DPKT`_'s reader, it yields Enhanced Packet Blocks and (obsolete)
43182
Packet Blocks only, and skips every other block without a warning -- the
44183
Simple Packet Block included. A packet carried in a Simple Packet Block is
45184
therefore missing from this engine's output; the default engine reads it.
@@ -89,7 +228,8 @@ def run(self) -> 'None':
89228
90229
This method assigns :attr:`self._expkg <DPKT._expkg>`
91230
as :mod:`dpkt` and :attr:`self._extmp <DPKT._extmp>`
92-
as an iterator from :class:`dpkt.pcap.Reader`.
231+
as an iterator from :class:`dpkt.pcap.Reader`, or from
232+
:class:`PCAPNGReader` for a PCAP-NG file.
93233
94234
The global header is parsed and dumped first, by
95235
:meth:`self.extractor.record_header <pcapkit.foundation.extraction.Extractor.record_header>`.
@@ -139,7 +279,7 @@ def run(self) -> 'None':
139279
reader = dpkt.pcap.Reader(ext._ifile)
140280
elif ext.magic_number in PCAPNG.MAGIC_NUMBER:
141281
logger.debug('dpkt: reading %s as PCAP-NG', ext._ifnm)
142-
reader = dpkt.pcapng.Reader(ext._ifile)
282+
reader = PCAPNGReader(ext._ifile)
143283
else:
144284
raise FormatError(f'unsupported file format: {ext.magic_number!r}')
145285

@@ -163,10 +303,11 @@ def read_frame(self) -> 'DPKTPacket':
163303
ext = self._extractor
164304

165305
reader = self._extmp
166-
linktype = Enum_LinkType.get(reader.datalink())
167306

168-
# fetch DPKT packet
307+
# fetch DPKT packet; a PCAP-NG reader reports the link type of the
308+
# interface this packet was captured on, so ask only after reading it
169309
timestamp, pkt = cast('tuple[float, bytes]', next(reader))
310+
linktype = Enum_LinkType.get(reader.datalink())
170311
protocol = self._get_protocol(linktype)
171312
try:
172313
packet = protocol(pkt) # type: DPKTPacket

0 commit comments

Comments
 (0)