Skip to content

Commit bdf5c99

Browse files
authored
fix(protocols): report a rebuilt frame's messages once and record cuts on the layer (#2711) (#2738)
- Rebuilding an outer layer rebuilt its payload by from_data, whose own parse reported what the octets carry, and then parsed them again itself, so Ethernet over a cut IGMPv3 reported its messages twice. Under rebuilding, ProtocolBase.__post_init__ drops the payload's parse warnings with hold_warnings, and beholder builds the errors it absorbs quiet through CaughtByCaller and logs none; the enclosing layer's parse reports them once. The owner ruled on #2711 that this carve-out from the handled-refusals rule stands, as protocol-family-grouping.rst now records. - An IGMPv3 Report parsed on its own and cut on a field boundary before its counted records read INVALID. ListField now takes a count: a list whose octets hold fewer items than it counts reports the shortfall, and Schema.unpack records it as a short read, the items absent never made up; the fields after it find the record in packet, as #2727 gives them. Report.records counts num_group_records for a message parsed on its own, so such a cut reads TRUNCATED whatever the context parsed before; inside a frame the count sizes nothing. A count no IP payload could carry raises ProtocolError. - A message cut inside the length its context gives it read TRUNCATED in its frame but INVALID rebuilt on its own. Schema.unpack records a cut in a payload of a declared length; keep_short_read records a frame cut on a layer only where Schema._frame_end_field finds its last field sized by what is left of the length it was given, so a layer whose header declares its end (IPv4, IPv6 and its extension headers, UDP, an HTTP/2 frame) records none at or past it, nor one ending in a list, as SCTP's chunks, whose buffer holds only what its items took. The TSO IPv4 payload is sized that way, and ICMPv4 reads a recorded cut as TRUNCATED, as GRE and IGMP do. - Tests that pinned no record for those cuts now pin the record. The new modules fail without the fix; the protocols, corekit, foundation, utilities and project modules pass. Sample captures give the same octets and messages; cut frames gain __short_read__ on Ethernet, IPv4 and UDP. Closes #2711
1 parent 21ec0a6 commit bdf5c99

21 files changed

Lines changed: 1127 additions & 34 deletions

‎docs/source/contributing/conventions/protocol-family-grouping.rst‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -229,7 +229,10 @@ What stays loud:
229229
mode, installs the hooks.
230230
* **Any other error class.** An error the fallback absorbs is as loud as it was
231231
constructed, and :func:`~pcapkit.corekit.decorators.beholder` logs it at
232-
:data:`logging.WARNING` as well.
232+
:data:`logging.WARNING` as well. Inside a payload
233+
:meth:`Protocol._make_payload <pcapkit.protocols.protocol.Protocol._make_payload>`
234+
rebuilds, it is quiet, as the owner ruled on :issue:`2711`: the layer the payload
235+
is rebuilt for parses the same octets again and reports it there, once.
233236
* **A refusal built under an inner** ``CaughtByCaller()``. A block naming no class,
234237
such as the one :meth:`~pcapkit.corekit.enum.EnumRegistry.register` enters, makes
235238
errors loud again inside the fallback's.

‎docs/source/contributing/pep.rst‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -969,7 +969,10 @@ offload heuristic on :issue:`2470`.
969969
ICMP and IGMP go by the original length as well (:issue:`2689`). A GRE or IGMP
970970
header the data ends inside is
971971
:attr:`~pcapkit.corekit.checksum.internet.ChecksumStatus.TRUNCATED` too
972-
(:issue:`2689`).
972+
(:issue:`2689`). Each of these cuts is recorded on the message's own header, so
973+
the message rebuilt on its own reads
974+
:attr:`~pcapkit.corekit.checksum.internet.ChecksumStatus.TRUNCATED` as well, an
975+
ICMP one included (:issue:`2711`).
973976
* **DCCP covers what its Checksum Coverage gives** (:rfc:`4340#section-9.2`), over
974977
a pseudo-header that carries the whole packet's length, as TCP's does, so a
975978
fragment is

‎pcapkit/corekit/decorators.py‎

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@
3030
import traceback
3131
from typing import TYPE_CHECKING, cast
3232

33-
from pcapkit.utilities.contextvars import beholding
33+
from pcapkit.utilities.contextvars import beholding, rebuilding
3434
from pcapkit.utilities.exceptions import (CaughtByCaller, ProtocolError, StreamEOFError,
3535
StructError, stacklevel)
3636
from pcapkit.utilities.logging import DEVMODE, VERBOSE, get_logger
@@ -124,6 +124,13 @@ def beholder(
124124
exception is logged here at :data:`logging.WARNING`, and is as loud as it
125125
was constructed.
126126
127+
Inside a payload :meth:`Protocol._make_payload
128+
<pcapkit.protocols.protocol.Protocol._make_payload>` rebuilds, the block
129+
names every :exc:`Exception`, which is what the ``except`` below catches,
130+
and nothing is logged here: the layer the payload is rebuilt for parses
131+
the same octets again, outside the rebuild, and its own :func:`beholder`
132+
reports the error there (:issue:`2711`).
133+
127134
The recovery runs after the block has ended. Under the outermost
128135
:func:`beholder` of a parse, a
129136
:exc:`~pcapkit.utilities.exceptions.ProtocolError` it raises therefore
@@ -155,7 +162,8 @@ def beholder(
155162
156163
A :exc:`~pcapkit.utilities.exceptions.ProtocolError` the decorated
157164
function builds is quiet, and one this decorator absorbs is not logged
158-
here either (:issue:`2588`).
165+
here either (:issue:`2588`). Inside a payload being rebuilt, so is every
166+
other error (:issue:`2711`).
159167
160168
:param func: decorated function
161169
:meta decorator:
@@ -173,13 +181,16 @@ def behold(*args: 'P.args', **kwargs: 'P.kwargs') -> 'R_beholder':
173181
except IndexError:
174182
length = None
175183

184+
# NOTE: A rebuilt payload's parse is repeated by the layer it is rebuilt
185+
# for, which reports what this one would (#2711).
186+
rebuilt = rebuilding.get()
176187
try:
177188
# call method
178189
token = beholding.set(True)
179190
try:
180191
# NOTE: The whole call is quiet for a ProtocolError, which the ``except`` below
181192
# records; the recovery runs after it, loud unless an outer block is active (#2588).
182-
with CaughtByCaller(ProtocolError):
193+
with CaughtByCaller(Exception if rebuilt else ProtocolError):
183194
return func(*args, **kwargs)
184195
finally:
185196
beholding.reset(token)
@@ -192,7 +203,7 @@ def behold(*args: 'P.args', **kwargs: 'P.kwargs') -> 'R_beholder':
192203

193204
# NOTE: Keyed on the class, not on quietness: a quiet end-of-file StructError is still
194205
# logged here, and a ProtocolError built loud has logged at CRITICAL already (#2588).
195-
if not isinstance(exc, ProtocolError):
206+
if not isinstance(exc, ProtocolError) and not rebuilt:
196207
logger.warning(str(exc), stack_info=DEVMODE, stacklevel=stacklevel())
197208

198209
if VERBOSE:

‎pcapkit/corekit/fields/collections.py‎

Lines changed: 38 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -76,10 +76,23 @@ class ListField(FieldBase[List[_TL]], Generic[_TL]):
7676
item_type: Field type of the contained items.
7777
callback: Callback function to be called upon
7878
:meth:`self.__call__ <pcapkit.corekit.fields.field.FieldBase.__call__>`.
79+
count: Number of items a field before this one counts, for a list
80+
running to the end of the data; if a callable is given, it should
81+
return an integer value and accept the current packet as its only
82+
argument. A negative value, the default, counts none.
7983
8084
This field is used to represent a list of fields, as in the case of lists of
8185
constant-length-field items in a protocol.
8286
87+
A list whose octets hold fewer items than ``count`` ended with the data
88+
before the rest, and reports them as :attr:`shortfall`, so that
89+
:meth:`Schema.unpack <pcapkit.protocols.schema.schema.Schema.unpack>`
90+
records the cut; the items absent are not made up.
91+
92+
.. versionchanged:: 1.6.0
93+
94+
Takes ``count`` (:issue:`2711`).
95+
8396
"""
8497

8598
@property
@@ -92,9 +105,20 @@ def optional(self) -> 'bool':
92105
"""Field is optional."""
93106
return True
94107

108+
@property
109+
def count(self) -> 'int':
110+
"""Number of items counted; negative where none are."""
111+
return self._count
112+
113+
@property
114+
def shortfall(self) -> 'int':
115+
"""Number of the items counted that the octets last unpacked did not hold."""
116+
return self._shortfall
117+
95118
def __init__(self, length: 'int | Callable[[dict[str, Any]], int]' = lambda _: -1,
96119
item_type: 'Optional[FieldBase]' = None,
97-
callback: 'Callable[[Self, dict[str, Any]], None]' = lambda *_: None) -> 'None':
120+
callback: 'Callable[[Self, dict[str, Any]], None]' = lambda *_: None,
121+
count: 'int | Callable[[dict[str, Any]], int]' = -1) -> 'None':
98122
#self._name = '<list>'
99123
if not hasattr(self, '_name'):
100124
self._name = f'<{type(self).__name__[:-5].lower()}>'
@@ -108,6 +132,12 @@ def __init__(self, length: 'int | Callable[[dict[str, Any]], int]' = lambda _: -
108132
self._length = length
109133
self._template = '0s'
110134

135+
self._count_callback = None
136+
if not isinstance(count, int):
137+
self._count_callback, count = count, -1
138+
self._count = count
139+
self._shortfall = 0
140+
111141
def __call__(self, packet: 'dict[str, Any]') -> 'Self':
112142
"""Update field attributes.
113143
@@ -126,6 +156,8 @@ def __call__(self, packet: 'dict[str, Any]') -> 'Self':
126156
if new_self._length_callback is not None:
127157
new_self._length = new_self._length_callback(packet)
128158
new_self._template = f'{new_self._length}s'
159+
if new_self._count_callback is not None:
160+
new_self._count = new_self._count_callback(packet)
129161
return new_self
130162

131163
def pack(self, value: 'Optional[list[_TL]]', packet: 'dict[str, Any]') -> 'bytes':
@@ -306,6 +338,11 @@ def _unpack_items(self, file: 'IO[bytes]', packet: 'dict[str, Any]', is_schema:
306338
data = field.unpack(buffer, packet)
307339

308340
temp.append(data)
341+
342+
# NOTE: Items are read to the end of the octets, so those counted past them are
343+
# not there to read: the data ended first, and the field reads short (#2711).
344+
if self._count >= 0:
345+
self._shortfall = max(self._count - len(temp), 0)
309346
return temp
310347

311348

‎pcapkit/corekit/schema/shortread.py‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,8 @@
2222
import io
2323
from typing import TYPE_CHECKING, cast
2424

25+
from pcapkit.utilities.contextvars import frame_length
26+
2527
if TYPE_CHECKING:
2628
from typing import Any, Optional, TypeVar
2729

@@ -53,10 +55,26 @@ def keep_short_read(protocol: 'ProtocolBase') -> 'None':
5355
and ``from_data(info.to_dict())`` cuts its rebuild back too; it is absent for
5456
a header that was captured whole.
5557
58+
Where the capture cut the frame short, as
59+
:data:`~pcapkit.utilities.contextvars.frame_length` gives its lengths, a
60+
layer whose message runs to the end of the frame records the cut too, on the
61+
field it runs to the end in, through :meth:`Schema._note_short_read
62+
<pcapkit.protocols.schema.schema.Schema._note_short_read>`, so its rebuild
63+
reads as cut, as the layer did inside the frame (:issue:`2711`). A layer
64+
whose own header declares where its message ends records none, whether the
65+
cut falls at that end or past it; see :meth:`Schema._frame_end_field
66+
<pcapkit.protocols.schema.schema.Schema._frame_end_field>`.
67+
5668
"""
5769
header = cast('Optional[Schema]', getattr(protocol, '__header__', None))
5870
if header is None:
5971
return
72+
lengths = frame_length.get()
73+
if lengths is not None and lengths[1] > lengths[0]:
74+
name = header._frame_end_field() # pylint: disable=protected-access
75+
if name is not None:
76+
octets = header.__buffer__[name]
77+
header._note_short_read(name, len(octets) + lengths[1] - lengths[0], octets) # pylint: disable=protected-access
6078
record = header.__dict__.get('__short_read__')
6179
if record is not None:
6280
protocol._info.__update__({'__short_read__': record}) # pylint: disable=protected-access

‎pcapkit/protocols/internet/icmpv4.py‎

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -97,7 +97,10 @@ class ICMPv4(ICMP[Data_ICMPv4, Schema_ICMPv4],
9797
9898
The checksum is computed when :meth:`make` is given none, and verified
9999
on read under :data:`~pcapkit.utilities.config.CHECKSUM_VERIFY`,
100-
unless the enclosing IP layer is a fragment (:issue:`2470`).
100+
unless the enclosing IP layer is a fragment (:issue:`2470`). A message
101+
whose header records a cut reads
102+
:attr:`~pcapkit.corekit.checksum.internet.ChecksumStatus.TRUNCATED`
103+
(:issue:`2711`).
101104
102105
"""
103106

@@ -202,9 +205,16 @@ def read(self, length: 'Optional[int]' = None, *,
202205
# NOTE: The checksum covers the whole message, with no pseudo-header (RFC 792);
203206
# an IP fragment carries only part of one.
204207
if config.CHECKSUM_VERIFY:
205-
from pcapkit.corekit.checksum.internet import verify_message # isort: skip # pylint: disable=import-outside-toplevel
208+
from pcapkit.corekit.checksum.internet import ( # isort: skip # pylint: disable=import-outside-toplevel
209+
ChecksumStatus, verify_message)
210+
# NOTE: A message whose header records a cut leaves part of what the
211+
# checksum covers uncaptured, as for GRE and IGMP (#2711).
212+
if '__short_read__' in schema.__dict__:
213+
status = ChecksumStatus.TRUNCATED
214+
else:
215+
status = verify_message(self.alias, self._data, schema.checksum, length=length)
206216
icmp.__update__([
207-
('checksum_status', verify_message(self.alias, self._data, schema.checksum, length=length)),
217+
('checksum_status', status),
208218
])
209219

210220
# NOTE: The quoted original datagram stays raw in the message body,

‎pcapkit/protocols/link/gre.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -111,7 +111,8 @@ class GRE(Link[Data_GRE, Schema_GRE],
111111
as :attr:`~pcapkit.corekit.checksum.internet.ChecksumStatus.TRUNCATED`,
112112
as :data:`~pcapkit.utilities.contextvars.frame_length` tells it
113113
(:issue:`2653`), and so does a header the data ends inside
114-
(:issue:`2689`).
114+
(:issue:`2689`). The header records such a cut, so the message rebuilt on
115+
its own reads as cut too (:issue:`2711`).
115116
116117
"""
117118

‎pcapkit/protocols/protocol.py‎

Lines changed: 25 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1424,6 +1424,13 @@ def __post_init__(self, file: 'Optional[IO[bytes] | bytes]' = None,
14241424
For construction arguments, please refer to
14251425
:meth:`self.make <pcapkit.protocols.protocol.Protocol.make>`.
14261426
1427+
Note:
1428+
Inside a payload :meth:`_make_payload` rebuilds, the warnings of the
1429+
parse are dropped, as the layer the payload is rebuilt for parses
1430+
the same octets again and reports them there (:issue:`2711`). Those
1431+
of a parse that raises are reported as the error leaves, unless
1432+
:func:`~pcapkit.corekit.decorators.beholder` replaces the layer.
1433+
14271434
"""
14281435
if file is None:
14291436
_data = self.pack(**kwargs)
@@ -1439,8 +1446,20 @@ def __post_init__(self, file: 'Optional[IO[bytes] | bytes]' = None,
14391446
self._data = _data
14401447
#: io.BytesIO: Source packet stream.
14411448
self._file = io.BytesIO(self._data)
1442-
#: pcapkit.protocols.data.data.Data: Parsed packet data.
1443-
self._info = self.unpack(length, **kwargs)
1449+
if not rebuilding.get():
1450+
#: pcapkit.protocols.data.data.Data: Parsed packet data.
1451+
self._info = self.unpack(length, **kwargs)
1452+
return
1453+
1454+
# NOTE: The enclosing layer parses these octets again and reports what
1455+
# they carry; as in ``_parse_next_layer``, a raise keeps them (#2711).
1456+
with hold_warnings() as held:
1457+
try:
1458+
self._info = self.unpack(length, **kwargs)
1459+
except Exception:
1460+
held.discard = beholding.get()
1461+
raise
1462+
held.discard = True
14441463

14451464
def __init_subclass__(cls, /, schema: 'Optional[Type[_ST]]' = None,
14461465
data: 'Optional[Type[_PT]]' = None,
@@ -2561,8 +2580,10 @@ def _make_payload(cls, data: 'Data') -> 'ProtocolBase':
25612580
parses its own octets again. So rebuilding a frame stops at
25622581
:data:`~pcapkit.utilities.config.FRAME_LAYER_LIMIT` where parsing it did, rather than that many
25632582
layers below each layer of it (:issue:`1610`). Nor does the payload's
2564-
parse warn of the limit: this protocol's own parse goes over the same
2565-
octets, so the rebuilt frame warns of it once, as the parsed one did.
2583+
parse report anything, the limit included: this protocol's own parse
2584+
goes over the same octets, so the rebuilt frame reports each warning,
2585+
and each error a next-layer fallback absorbs, once, as the parsed one
2586+
did (:issue:`2711`).
25662587
25672588
Args:
25682589
data: protocol data

‎pcapkit/protocols/schema/internet/igmp.py‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -43,8 +43,10 @@ class IGMP(Schema):
4343
#: Group address.
4444
group: 'IPv4Address' = IPv4AddressField()
4545
#: Payload, empty for a standards-conformant IGMPv1/v2 message, since the
46-
#: 8-octet header is the whole message.
47-
payload: 'bytes' = PayloadField()
46+
#: 8-octet header is the whole message. It is the rest of the IP payload,
47+
#: which the checksum covers too, so a cut inside it is recorded
48+
#: (:issue:`2711`).
49+
payload: 'bytes' = PayloadField(length=lambda pkt: pkt['__length__'])
4850

4951
if TYPE_CHECKING:
5052
def __init__(self, type: 'Enum_Type', max_resp_time: 'int', checksum: 'bytes', # pylint: disable=unused-argument,super-init-not-called,multiple-statements,redefined-builtin

‎pcapkit/protocols/schema/internet/igmpv3.py‎

Lines changed: 42 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,9 @@
1111
from pcapkit.corekit.fields.misc import ConditionalField, PayloadField, SchemaField, SwitchField
1212
from pcapkit.corekit.fields.numbers import EnumField, UInt8Field, UInt16Field
1313
from pcapkit.corekit.fields.strings import BitField, BytesField
14+
from pcapkit.protocols.schema.internet.ipv6 import IP_MAX_LENGTH
1415
from pcapkit.protocols.schema.schema import Schema, schema_final
16+
from pcapkit.utilities.contextvars import frame_depth
1517
from pcapkit.utilities.exceptions import ProtocolError
1618

1719
__all__ = ['IGMPv3', 'Query', 'QueryExtension', 'Report', 'GroupRecord']
@@ -90,6 +92,43 @@ def query_extension_present(pkt: 'dict[str, Any]') -> 'bool':
9092
return pkt['__length__'] >= 4
9193

9294

95+
def report_record_count(pkt: 'dict[str, Any]') -> 'int':
96+
"""Count callback for :attr:`Report.records`.
97+
98+
Args:
99+
pkt: Packet data.
100+
101+
Returns:
102+
:attr:`Report.num_group_records`, for a message parsed on its own;
103+
otherwise ``-1``, counting none: inside a frame, the layer before the
104+
message or the frame's own lengths say where it ends, and on the
105+
packing path ``'records'`` is already in ``pkt``, as no count is
106+
without :attr:`Report.num_group_records`.
107+
108+
No layer before a message parsed on its own gives its length, and none
109+
tells a cut from the end of the message, so the count does: records it
110+
counts past the data are a cut, however the cut falls, on a field boundary
111+
too, and :meth:`Schema.unpack
112+
<pcapkit.protocols.schema.schema.Schema.unpack>` records it, the records
113+
present alone being read (:issue:`2711`).
114+
115+
Raises:
116+
ProtocolError: If the records counted, at the 8 fixed octets of each,
117+
behind the 8 octets ahead of them, reach past the
118+
:data:`~pcapkit.protocols.schema.internet.ipv6.IP_MAX_LENGTH` octets
119+
an IP payload can hold, so the count is no cut but one the message
120+
cannot carry (:issue:`2711`).
121+
122+
"""
123+
count = pkt.get('num_group_records') # type: Optional[int]
124+
if count is None or 'records' in pkt or frame_depth.get():
125+
return -1
126+
# NOTE: The 8 octets ahead of the records count against the IP payload too.
127+
if 8 + 8 * count > IP_MAX_LENGTH:
128+
raise ProtocolError(f'IGMPv3: {count} group records cannot fit in one IP payload')
129+
return count
130+
131+
93132
@schema_final
94133
class QueryExtension(Schema):
95134
"""Header schema for the IGMPv3 Query's source-list extension.
@@ -230,7 +269,8 @@ class Report(Schema):
230269
:class:`~pcapkit.protocols.schema.internet.hip.LocatorSetParameter`).
231270
:meth:`post_process` keeps only the first :attr:`num_group_records`
232271
records and gives the rest to :attr:`payload`, which is
233-
:rfc:`3376#section-4.2.11`'s "Additional Data".
272+
:rfc:`3376#section-4.2.11`'s "Additional Data". Where the data ends
273+
before the records counted, see :func:`report_record_count`.
234274
235275
"""
236276

@@ -247,6 +287,7 @@ class Report(Schema):
247287
records: 'list[GroupRecord]' = ListField(
248288
length=lambda pkt: pkt['__length__'],
249289
item_type=SchemaField(schema=GroupRecord),
290+
count=report_record_count,
250291
)
251292
#: Payload; empty for a standards-conformant message with
252293
#: :attr:`num_group_records` matching the octets actually present.

0 commit comments

Comments
 (0)