Skip to content

Commit ac02fa9

Browse files
committed
fix(engines): the dpkt engine applies the first PCAP-NG interface's linktype and timestamp resolution to every frame
dpkt.pcapng.Reader reads only the first Interface Description Block, so every packet got interface 0's link type and if_tsresol, and a later section's interfaces were never read. The DPKT engine now reads PCAP-NG with its own PCAPNGReader: SHBs and IDBs are parsed by pcapkit's PCAPNG protocol per section, packet blocks are still decoded by dpkt, and each packet takes the link type, if_tsresol and if_tsoffset of the interface it names, with the default engine's timestamp arithmetic. Closes #1379
1 parent 739e3da commit ac02fa9

4 files changed

Lines changed: 323 additions & 12 deletions

File tree

‎docs/source/pcapkit/foundation/engines/3rdparty.rst‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -69,8 +69,9 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`.
6969
On PCAP-NG, `DPKT`_ yields **Enhanced Packet Blocks and (obsolete) Packet
7070
Blocks only**. ``dpkt.pcapng.Reader.__iter__`` skips every other block type
7171
without a warning (``# just ignore other blocks``, as of ``dpkt`` 1.9.8), and
72-
that includes the **Simple Packet Block**, which carries a packet just as an
73-
Enhanced Packet Block does. Such packets are silently missing from this
72+
:class:`~pcapkit.foundation.engines.dpkt.PCAPNGReader`, which this engine
73+
reads PCAP-NG with, keeps that block set. It includes the **Simple Packet
74+
Block**, which carries a packet just as an Enhanced Packet Block does. Such packets are silently missing from this
7475
engine's output, and the frames after them are numbered as if they had never
7576
been captured.
7677

@@ -83,6 +84,15 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`.
8384
`DPKT`_, not of :mod:`pcapkit`, so read a capture that may contain Simple
8485
Packet Blocks with the default engine instead.
8586

87+
.. note::
88+
89+
``dpkt.pcapng.Reader`` reads only the **first** Interface Description Block
90+
of a file, and applies its link type and timestamp resolution to every
91+
packet. This engine therefore reads PCAP-NG with
92+
:class:`~pcapkit.foundation.engines.dpkt.PCAPNGReader` instead, which
93+
resolves each packet's link type, ``if_tsresol`` and ``if_tsoffset`` from
94+
the interface it names, in its own section (:issue:`1379`).
95+
8696
.. autoclass:: pcapkit.foundation.engines.dpkt.DPKT
8797
:no-members:
8898
:show-inheritance:
@@ -96,6 +106,10 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`.
96106
.. autoattribute:: _expkg
97107
.. autoattribute:: _extmp
98108

109+
.. autoclass:: pcapkit.foundation.engines.dpkt.PCAPNGReader
110+
:members: datalink
111+
:show-inheritance:
112+
99113
PyShark Support
100114
===============
101115

‎pcapkit/foundation/engines/dpkt.py‎

Lines changed: 149 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -10,36 +10,174 @@
1010
.. _DPKT: https://dpkt.readthedocs.io
1111
1212
"""
13+
import decimal
1314
from typing import TYPE_CHECKING, cast
1415

16+
from pcapkit.const.pcapng.block_type import BlockType as Enum_BlockType
17+
from pcapkit.const.pcapng.option_type import OptionType as Enum_OptionType
1518
from pcapkit.const.reg.linktype import LinkType as Enum_LinkType
1619
from pcapkit.foundation.engines.engine import EngineBase
1720
from pcapkit.utilities.exceptions import FormatError, stacklevel
1821
from pcapkit.utilities.logging import get_logger
1922
from pcapkit.utilities.warnings import AttributeWarning, DPKTWarning, warn
2023

21-
__all__ = ['DPKT']
24+
__all__ = ['DPKT', 'PCAPNGReader']
2225

2326
if TYPE_CHECKING:
24-
from typing import Optional, Type, Union
27+
from typing import IO, Iterator, Literal, Optional, Type, Union
2528

2629
from dpkt.dpkt import Packet as DPKTPacket
2730
from dpkt.pcap import Reader as PCAPReader
28-
from dpkt.pcapng import Reader as PCAPNGReader
2931

32+
from pcapkit.foundation.engines.pcapng import Context
3033
from pcapkit.foundation.extraction import Extractor
34+
from pcapkit.protocols.data.misc.pcapng import \
35+
InterfaceDescriptionBlock as Data_InterfaceDescriptionBlock
36+
from pcapkit.protocols.data.misc.pcapng import SectionHeaderBlock as Data_SectionHeaderBlock
3137

32-
Reader = Union[PCAPReader, PCAPNGReader]
38+
Reader = Union[PCAPReader, 'PCAPNGReader']
3339

3440
#: logging.Logger: Module-level logger, a child of the package-wide
3541
#: :data:`pcapkit.utilities.logging.logger`.
3642
logger = get_logger(__name__)
3743

3844

45+
class PCAPNGReader:
46+
"""PCAP-NG reader for the DPKT engine, aware of every interface.
47+
48+
:class:`dpkt.pcapng.Reader` reads only the first Interface Description
49+
Block (IDB) of a file, and applies its link type and timestamp resolution
50+
to every packet (:issue:`1379`). This reader walks the blocks itself
51+
instead: it parses each Section Header Block and IDB with :mod:`pcapkit`'s
52+
own :class:`~pcapkit.protocols.misc.pcapng.PCAPNG`, keeps the interfaces
53+
of the current section, and resolves every packet's link type,
54+
``if_tsresol`` and ``if_tsoffset`` from the interface it names. Packet
55+
blocks are still decoded by `DPKT`_.
56+
57+
As :class:`dpkt.pcapng.Reader` does, it yields Enhanced Packet Blocks and
58+
(obsolete) Packet Blocks only, and skips every other block.
59+
60+
.. _DPKT: https://dpkt.readthedocs.io
61+
62+
Args:
63+
file: Source PCAP-NG stream, positioned at its first Section Header Block.
64+
65+
"""
66+
67+
def __init__(self, file: 'IO[bytes]') -> 'None':
68+
#: Source PCAP-NG stream.
69+
self._file = file
70+
#: Context of the current section.
71+
self._ctx = cast('Context', None)
72+
#: Section index number.
73+
self._sect = 0
74+
#: Link type of the packet last yielded.
75+
self._linktype = cast('Enum_LinkType', Enum_LinkType.NULL)
76+
#: Underlying block iterator.
77+
self._iter = self._read()
78+
79+
def datalink(self) -> 'Enum_LinkType':
80+
"""Link type of the interface the packet last yielded was captured on."""
81+
return self._linktype
82+
83+
def __iter__(self) -> 'PCAPNGReader':
84+
return self
85+
86+
def __next__(self) -> 'tuple[float, bytes]':
87+
return next(self._iter)
88+
89+
def _read(self) -> 'Iterator[tuple[float, bytes]]':
90+
"""Read blocks until the next packet block, and yield its timestamp and data."""
91+
from pcapkit.foundation.engines.pcapng import Context
92+
from pcapkit.protocols.misc.pcapng import PCAPNG as P_PCAPNG
93+
94+
while True:
95+
head = self._file.read(8)
96+
if len(head) < 8:
97+
return
98+
99+
# the SHB block type is a palindrome, so it reads the same in
100+
# either byte order; any other block uses the section's order
101+
byteorder: 'Literal["big", "little"]'
102+
if head[:4] == b'\x0a\x0d\x0d\x0a':
103+
head += self._file.read(4)
104+
byteorder = 'little' if head[8:] == b'\x4d\x3c\x2b\x1a' else 'big'
105+
else:
106+
byteorder = self._ctx.section.byteorder
107+
block_type = int.from_bytes(head[:4], byteorder)
108+
length = int.from_bytes(head[4:8], byteorder)
109+
if length < 12:
110+
raise FormatError(f'PCAP-NG: [Block {block_type}] invalid block length: {length}')
111+
112+
buf = head + self._file.read(length - len(head))
113+
if len(buf) < length:
114+
return
115+
116+
if block_type == Enum_BlockType.Section_Header_Block:
117+
self._sect += 1
118+
shb = P_PCAPNG(buf, num=0, sct=self._sect, ctx=None)
119+
self._ctx = Context(cast('Data_SectionHeaderBlock', shb.info))
120+
elif block_type == Enum_BlockType.Interface_Description_Block:
121+
idb = P_PCAPNG(buf, num=0, sct=self._sect, ctx=self._ctx)
122+
self._ctx.interfaces.append(cast('Data_InterfaceDescriptionBlock', idb.info))
123+
elif block_type in (Enum_BlockType.Enhanced_Packet_Block, Enum_BlockType.Packet_Block):
124+
yield self._read_packet(buf, block_type, byteorder == 'little')
125+
126+
def _read_packet(self, buf: 'bytes', block_type: 'int',
127+
little: 'bool') -> 'tuple[float, bytes]':
128+
"""Decode a packet block with `DPKT`_, against the interface it names.
129+
130+
Args:
131+
buf: Whole block, as read from the file.
132+
block_type: Block type, an EPB or an (obsolete) Packet Block.
133+
little: Whether the enclosing section is little-endian.
134+
135+
Returns:
136+
Timestamp in seconds since the UNIX epoch, and the packet data.
137+
138+
Raises:
139+
FormatError: If the section describes no interface of the ID the
140+
block names.
141+
142+
"""
143+
import dpkt # isort:skip
144+
145+
if block_type == Enum_BlockType.Enhanced_Packet_Block:
146+
tag = 'EPB'
147+
pkt = (dpkt.pcapng.EnhancedPacketBlockLE(buf) if little
148+
else dpkt.pcapng.EnhancedPacketBlock(buf))
149+
else:
150+
tag = 'Packet'
151+
pkt = dpkt.pcapng.PacketBlockLE(buf) if little else dpkt.pcapng.PacketBlock(buf)
152+
153+
iface_id = pkt.iface_id # pylint: disable=no-member
154+
if iface_id >= len(self._ctx.interfaces):
155+
raise FormatError(f'PCAP-NG: [{tag}] invalid interface ID: {iface_id}')
156+
interface = self._ctx.interfaces[iface_id]
157+
158+
tsresol = interface.options.get(Enum_OptionType.if_tsresol)
159+
tsoffset = interface.options.get(Enum_OptionType.if_tsoffset)
160+
resolution = 1_000_000 if tsresol is None else tsresol.resolution
161+
offset = 0 if tsoffset is None else tsoffset.offset
162+
163+
# same arithmetic as :meth:`PCAPNG._read_timestamp
164+
# <pcapkit.protocols.misc.pcapng.PCAPNG._read_timestamp>`
165+
ticks = (pkt.ts_high << 32) | pkt.ts_low
166+
with decimal.localcontext(prec=64):
167+
timestamp = decimal.Decimal(ticks) / resolution + offset
168+
169+
self._linktype = interface.linktype
170+
return float(timestamp), pkt.pkt_data
171+
172+
39173
class DPKT(EngineBase['DPKTPacket']):
40174
"""DPKT engine support.
41175
42-
On PCAP-NG, `DPKT`_'s reader yields Enhanced Packet Blocks and (obsolete)
176+
A PCAP-NG file is read by :class:`PCAPNGReader` rather than by
177+
:class:`dpkt.pcapng.Reader`, so each packet takes the link type and the
178+
timestamp resolution and offset of the interface it was captured on.
179+
180+
Like `DPKT`_'s reader, it yields Enhanced Packet Blocks and (obsolete)
43181
Packet Blocks only, and skips every other block without a warning -- the
44182
Simple Packet Block included. A packet carried in a Simple Packet Block is
45183
therefore missing from this engine's output; the default engine reads it.
@@ -89,7 +227,8 @@ def run(self) -> 'None':
89227
90228
This method assigns :attr:`self._expkg <DPKT._expkg>`
91229
as :mod:`dpkt` and :attr:`self._extmp <DPKT._extmp>`
92-
as an iterator from :class:`dpkt.pcap.Reader`.
230+
as an iterator from :class:`dpkt.pcap.Reader`, or from
231+
:class:`PCAPNGReader` for a PCAP-NG file.
93232
94233
The global header is parsed and dumped first, by
95234
:meth:`self.extractor.record_header <pcapkit.foundation.extraction.Extractor.record_header>`.
@@ -139,7 +278,7 @@ def run(self) -> 'None':
139278
reader = dpkt.pcap.Reader(ext._ifile)
140279
elif ext.magic_number in PCAPNG.MAGIC_NUMBER:
141280
logger.debug('dpkt: reading %s as PCAP-NG', ext._ifnm)
142-
reader = dpkt.pcapng.Reader(ext._ifile)
281+
reader = PCAPNGReader(ext._ifile)
143282
else:
144283
raise FormatError(f'unsupported file format: {ext.magic_number!r}')
145284

@@ -163,10 +302,11 @@ def read_frame(self) -> 'DPKTPacket':
163302
ext = self._extractor
164303

165304
reader = self._extmp
166-
linktype = Enum_LinkType.get(reader.datalink())
167305

168-
# fetch DPKT packet
306+
# fetch DPKT packet; a PCAP-NG reader reports the link type of the
307+
# interface this packet was captured on, so ask only after reading it
169308
timestamp, pkt = cast('tuple[float, bytes]', next(reader))
309+
linktype = Enum_LinkType.get(reader.datalink())
170310
protocol = self._get_protocol(linktype)
171311
try:
172312
packet = protocol(pkt) # type: DPKTPacket

0 commit comments

Comments
 (0)