Skip to content

Commit 234d781

Browse files
authored
fix(internet)!: read a cut after an IPv6 option's type octet as a short read (#2727) (#2737)
In HOPOPT and IPv6-Opts, a capture cut leaving only the type octet of a CALIPSO, SMF_DPD or MPL option logged "FieldValueError: ... invalid <X> option length: 0", at CRITICAL and again by beholder inside IPv6: the callbacks sizing the option refused the zero the cut left for Opt Data Len, before the short read could record the cut. - Schema.unpack gives the fields after the one the data ends inside the short read, as __short_read__ in their packet, a forward match included, and clears it as each unpack starts, so a nested schema finds only its own. - calipso_pad_len, mpl_opt_pad_len, smf_i_dpd_id_len and smf_dpd_data_selector leave an Opt Data Len the data ends before to the short read. #2694's rule then keeps CALIPSO and SMF_DPD, and refuses MPL quietly, as its parser refuses the zero, as Router Alert's does. - An Opt Data Len on the wire too short raises ProtocolError, not FieldValueError, so a header refused for it is refused quietly. The four helpers' documented Raises class moves: the breaking change. - Schema._note_short_read takes packet as an optional argument, so a caller with none keeps the record on the schema alone. - The utilities-or-corekit page says how a callback tells a cut. - Tests sweep every option of both headers at every cut, alone and in IPv6; the pins of the FieldValueError move to ProtocolError. The new tests fail on ab7be2f. protocols, corekit and project pass module by module.
1 parent 6ae240d commit 234d781

9 files changed

Lines changed: 570 additions & 62 deletions

File tree

‎docs/source/contributing/conventions/utilities-or-corekit.rst‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,14 @@ through them, never through an ``unpack`` override of its own or a length callab
8484
clamped to the octets present, which hides the cut from the record. Ruled in review
8585
of :pr:`2682` (:issue:`2680`), and applied to option areas on :issue:`2694`.
8686

87+
A field callback tells a value the cut left as zero from one read through the same
88+
record: :meth:`Schema.unpack <pcapkit.protocols.schema.schema.Schema.unpack>` gives
89+
it to the fields after the one the data ends inside, as ``__short_read__`` in their
90+
packet. A callback that would refuse such a zero leaves it to the short read, and
91+
refuses a value read with a :exc:`~pcapkit.utilities.exceptions.ProtocolError`, as
92+
the ``Opt Data Len`` helpers of
93+
:mod:`pcapkit.protocols.schema.internet.ipv6_option` do (:issue:`2727`).
94+
8795
**A helper in one protocol's own schema module moves there too once code outside that
8896
protocol uses it and it is generic enough to reuse**, so that another implementor can
8997
find it; a helper only its own protocol calls stays beside the schema. Ruled on

‎pcapkit/protocols/schema/internet/ipv6_option.py‎

Lines changed: 77 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@
2323
from pcapkit.corekit.fields.ipaddress import IPv4AddressField, IPv6AddressField
2424
from pcapkit.corekit.fields.misc import NoValueField, SchemaField
2525
from pcapkit.corekit.fields.strings import BytesField
26-
from pcapkit.utilities.exceptions import FieldValueError
26+
from pcapkit.utilities.exceptions import FieldValueError, ProtocolError
2727

2828
__all__ = [
2929
'rpl_opt_sub_tlv_len', 'mpl_opt_seed_id_len', 'smf_i_dpd_id_len',
@@ -38,6 +38,23 @@
3838
from pcapkit.protocols.schema.schema import EnumMeta, Schema
3939

4040

41+
def _opt_data_len_cut(pkt: 'dict[str, Any]') -> 'bool':
42+
"""Whether the data ends before an option's ``Opt Data Len``.
43+
44+
Args:
45+
pkt: Option unpacked schema.
46+
47+
Returns:
48+
Whether the field :meth:`Schema.unpack
49+
<pcapkit.protocols.schema.schema.Schema.unpack>` gives as the one the
50+
data ends inside is ``len``, so that its value is the zero a capture cut
51+
leaves rather than a length read (:issue:`2727`).
52+
53+
"""
54+
record = pkt.get('__short_read__')
55+
return record is not None and record[0] == 'len'
56+
57+
4158
def rpl_opt_sub_tlv_len(pkt: 'dict[str, Any]') -> 'int':
4259
"""Return RPL option sub-TLV length.
4360
@@ -87,17 +104,27 @@ def smf_i_dpd_id_len(pkt: 'dict[str, Any]', *, prefix: 'str') -> 'int':
87104
prefix: Name of the extension header, leading the error message.
88105
89106
Returns:
90-
SMF I-DPD identifier length.
107+
SMF I-DPD identifier length, or zero if the data ends before
108+
``Opt Data Len``, which leaves the option to the short read.
91109
92110
Raises:
93-
FieldValueError: If ``Opt Data Len`` on the wire is too short to hold
111+
ProtocolError: If ``Opt Data Len`` on the wire is too short to hold
94112
the TaggerID it declares, which would otherwise underflow the
95113
identifier length below zero.
96114
115+
.. versionchanged:: 1.6.0
116+
117+
A length the data ends before is left to the short read, and one too
118+
short raises :exc:`~pcapkit.utilities.exceptions.ProtocolError` rather
119+
than :exc:`~pcapkit.utilities.exceptions.FieldValueError`
120+
(:issue:`2727`).
121+
97122
"""
98123
length = pkt['len'] - (1 if pkt['info']['type'] == 0 else (pkt['info']['len'] + 2))
99124
if length < 0:
100-
raise FieldValueError(f'{prefix}: invalid SMF I-DPD option length: {pkt["len"]}')
125+
if _opt_data_len_cut(pkt):
126+
return 0
127+
raise ProtocolError(f'{prefix}: invalid SMF I-DPD option length: {pkt["len"]}')
101128
return length
102129

103130

@@ -120,8 +147,8 @@ def smf_dpd_data_selector(pkt: 'dict[str, Any]', *, prefix: 'str',
120147
wrapped ``SMFHashBasedDPDOption`` instance.
121148
122149
Raises:
123-
FieldValueError: If ``Opt Data Len`` is ``0``, or if no schema is
124-
registered for the DPD mode.
150+
ProtocolError: If ``Opt Data Len`` is ``0`` on the wire.
151+
FieldValueError: If no schema is registered for the DPD mode.
125152
126153
Note:
127154
The field is sized ``Opt Data Len + 2`` rather than ``Opt Data Len``.
@@ -133,11 +160,26 @@ def smf_dpd_data_selector(pkt: 'dict[str, Any]', *, prefix: 'str',
133160
:class:`~pcapkit.corekit.fields.collections.OptionField` would then
134161
mis-count against the option area.
135162
163+
Where the data ends before ``Opt Data Len``, the forward match reads
164+
it as zero, and the option is the short read of the schema chosen by
165+
the mode bit, likewise zero.
166+
167+
.. versionchanged:: 1.6.0
168+
169+
A length the data ends before is left to the short read, and a zero
170+
on the wire raises :exc:`~pcapkit.utilities.exceptions.ProtocolError`
171+
rather than :exc:`~pcapkit.utilities.exceptions.FieldValueError`
172+
(:issue:`2727`).
173+
136174
"""
137175
# NOTE: With ``Opt Data Len`` 0 the option has no data octet, so the mode
138-
# bit forward-matched above belongs to whatever follows the option.
139-
if pkt['test']['len'] == 0:
140-
raise FieldValueError(f'{prefix}: invalid SMF DPD option length: 0')
176+
# bit forward-matched above belongs to whatever follows the option. Unless
177+
# the data ends inside the forward match before ``Opt Data Len``, the
178+
# option's second octet, which then reads as zero (#2727).
179+
record = pkt.get('__short_read__')
180+
cut = record is not None and record[0] == 'test' and record[1] < 2
181+
if pkt['test']['len'] == 0 and not cut:
182+
raise ProtocolError(f'{prefix}: invalid SMF DPD option length: 0')
141183

142184
mode = Enum_SMFDPDMode.get(pkt['test']['mode'])
143185
schema: 'Optional[Type[Schema]]' = base.registry[mode]
@@ -268,18 +310,28 @@ def calipso_pad_len(pkt: 'dict[str, Any]', *, prefix: 'str') -> 'int':
268310
prefix: Name of the extension header, leading the error message.
269311
270312
Returns:
271-
CALIPSO option padding length.
313+
CALIPSO option padding length, or zero if the data ends before
314+
``Opt Data Len``, which leaves the option to the short read.
272315
273316
Raises:
274-
FieldValueError: If ``Opt Data Len`` on the wire is too short to hold
317+
ProtocolError: If ``Opt Data Len`` on the wire is too short to hold
275318
the fixed header and the compartment bitmap declared by
276319
``cmpt_len``, which would otherwise underflow the padding length
277320
below zero.
278321
322+
.. versionchanged:: 1.6.0
323+
324+
A length the data ends before is left to the short read, and one too
325+
short raises :exc:`~pcapkit.utilities.exceptions.ProtocolError` rather
326+
than :exc:`~pcapkit.utilities.exceptions.FieldValueError`
327+
(:issue:`2727`).
328+
279329
"""
280330
length = pkt['len'] - 8 - pkt['cmpt_len'] * 4
281331
if length < 0:
282-
raise FieldValueError(f'{prefix}: invalid CALIPSO option length: {pkt["len"]}')
332+
if _opt_data_len_cut(pkt):
333+
return 0
334+
raise ProtocolError(f'{prefix}: invalid CALIPSO option length: {pkt["len"]}')
283335
return length
284336

285337

@@ -291,16 +343,26 @@ def mpl_opt_pad_len(pkt: 'dict[str, Any]', *, prefix: 'str') -> 'int':
291343
prefix: Name of the extension header, leading the error message.
292344
293345
Returns:
294-
MPL option padding length.
346+
MPL option padding length, or zero if the data ends before
347+
``Opt Data Len``, which leaves the option to the short read.
295348
296349
Raises:
297-
FieldValueError: If ``Opt Data Len`` on the wire is too short to hold
350+
ProtocolError: If ``Opt Data Len`` on the wire is too short to hold
298351
the fixed header and the Seed-ID declared by ``flags.type``, which
299352
would otherwise underflow the padding length below zero.
300353
354+
.. versionchanged:: 1.6.0
355+
356+
A length the data ends before is left to the short read, and one too
357+
short raises :exc:`~pcapkit.utilities.exceptions.ProtocolError` rather
358+
than :exc:`~pcapkit.utilities.exceptions.FieldValueError`
359+
(:issue:`2727`).
360+
301361
"""
302362
length = pkt['len'] - 2 - (0 if pkt['flags']['type'] == 0
303363
else mpl_opt_seed_id_len(pkt, prefix=prefix))
304364
if length < 0:
305-
raise FieldValueError(f'{prefix}: invalid MPL option length: {pkt["len"]}')
365+
if _opt_data_len_cut(pkt):
366+
return 0
367+
raise ProtocolError(f'{prefix}: invalid MPL option length: {pkt["len"]}')
306368
return length

‎pcapkit/protocols/schema/schema.py‎

Lines changed: 37 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -738,13 +738,32 @@ def unpack(cls, data: 'bytes | IO[bytes]',
738738
remainder itself rather than the same number of octets from beyond
739739
it, and record a cut inside it (:issue:`2685`).
740740
741+
The fields after the one the data ends inside find it in ``packet``
742+
as ``__short_read__``: the ``(name, octets read)`` pair recorded as
743+
the short read, or that of a
744+
:class:`~pcapkit.corekit.fields.misc.ForwardMatchField` the data
745+
ends inside first, which the record leaves out as it consumes
746+
nothing. So a field callback can tell a value the end of the data
747+
left as zero from one read, and leave such a value to the short
748+
read rather than refuse it. The key is cleared as unpacking starts,
749+
so a nested schema finds only its own.
750+
751+
.. versionchanged:: 1.6.0
752+
753+
The field the data ends inside is given to the fields after it, as
754+
``__short_read__`` in ``packet`` (:issue:`2727`).
755+
741756
"""
742757
# force cast arg type since decorator changed their signatures
743758
if TYPE_CHECKING:
744759
data = cast('IO[bytes]', data)
745760
length = cast('int', length)
746761
packet = cast('dict[str, Any]', packet)
747762

763+
# NOTE: ``packet`` may be an enclosing schema's, copied into a nested
764+
# context, so its short read is not this schema's (#2727).
765+
packet.pop('__short_read__', None)
766+
748767
self = cls.__new__(cls)
749768
for field in self.__fields__.values():
750769
field = field(packet)
@@ -779,7 +798,7 @@ def unpack(cls, data: 'bytes | IO[bytes]',
779798

780799
byte = data.read(length)
781800
self.__buffer__[field.name] = byte
782-
self._note_short_read(field.name, length, byte)
801+
self._note_short_read(field.name, length, byte, packet)
783802

784803
packet[field.name] = byte
785804
packet['__length__'] -= length
@@ -801,7 +820,7 @@ def unpack(cls, data: 'bytes | IO[bytes]',
801820
byte = data.read(min(length, sys.maxsize))
802821
self.__buffer__[field.name] = byte
803822
if not isinstance(field, (ForwardMatchField, OptionField)):
804-
self._note_short_read(field.name, length, byte)
823+
self._note_short_read(field.name, length, byte, packet)
805824

806825
value = field.unpack(byte, packet.copy())
807826
if isinstance(field, SchemaField) and isinstance(value, Schema):
@@ -843,6 +862,12 @@ def unpack(cls, data: 'bytes | IO[bytes]',
843862
# preceding octet as their first.
844863
data.seek(-len(byte), io.SEEK_CUR)
845864
self.__buffer__[field.name] = b''
865+
866+
# NOTE: Left out of the record, but the values it matched past the
867+
# end of the data are zeros all the same, so the fields after it
868+
# are told (#2727).
869+
if len(byte) < length:
870+
packet.setdefault('__short_read__', (field.name, len(byte)))
846871
elif isinstance(field, OptionField) and field.option_padding > 0:
847872
# NOTE: The option list ended before the declared area did, and
848873
# ``option_padding`` is the rest of that area; give back as much
@@ -856,10 +881,10 @@ def unpack(cls, data: 'bytes | IO[bytes]',
856881
self.__buffer__[field.name] = byte[:kept]
857882
packet['__length__'] -= consumed
858883
if kept == len(byte):
859-
self._note_short_read(field.name, length, byte)
884+
self._note_short_read(field.name, length, byte, packet)
860885
else:
861886
if isinstance(field, OptionField):
862-
self._note_short_read(field.name, length, byte)
887+
self._note_short_read(field.name, length, byte, packet)
863888
packet['__length__'] -= length
864889

865890
if packet['__length__'] < 0:
@@ -1187,17 +1212,24 @@ def __delattr__(self, name: 'str') -> 'None':
11871212
return
11881213
return super().__delattr__(name)
11891214

1190-
def _note_short_read(self, name: 'str', length: 'int', byte: 'bytes') -> 'None':
1215+
def _note_short_read(self, name: 'str', length: 'int', byte: 'bytes',
1216+
packet: 'Optional[dict[str, Any]]' = None) -> 'None':
11911217
"""Record the first field the data ends inside, for :meth:`_cut_short_read`.
11921218
11931219
Args:
11941220
name: Field name.
11951221
length: Octets the field declares.
11961222
byte: Octets actually read for it.
1223+
packet: Unpacked data, where the fields after it find the record,
1224+
unless a forward match the data ended inside is there first
1225+
(see :meth:`unpack`). Without it, the record is kept on the
1226+
schema alone.
11971227
11981228
"""
11991229
if len(byte) < length and '__short_read__' not in self.__dict__:
12001230
self.__dict__['__short_read__'] = (name, len(byte))
1231+
if packet is not None:
1232+
packet.setdefault('__short_read__', self.__dict__['__short_read__'])
12011233

12021234
def _cut_short_read(self) -> 'None':
12031235
"""Cut the packed buffers back to the octets a short read consumed.

‎tests/protocols/internet/test_ipv6_extension_unit.py‎

Lines changed: 17 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1422,7 +1422,7 @@ def _assert_option_schema_helpers_and_post_process_branches(self, schema: types.
14221422
from pcapkit.corekit.fields.ipaddress import IPv4AddressField, IPv6AddressField
14231423
from pcapkit.corekit.fields.misc import NoValueField, SchemaField
14241424
from pcapkit.corekit.fields.strings import BytesField
1425-
from pcapkit.utilities.exceptions import FieldValueError
1425+
from pcapkit.utilities.exceptions import FieldValueError, ProtocolError
14261426

14271427
self.assertEqual(schema.mpl_opt_seed_id_len({'flags': {'type': 0}}), 0)
14281428
self.assertEqual(schema.mpl_opt_seed_id_len({'flags': {'type': 1}}), 2)
@@ -1439,10 +1439,11 @@ def _assert_option_schema_helpers_and_post_process_branches(self, schema: types.
14391439
self.assertEqual(schema.smf_i_dpd_id_len({'len': 7, 'info': {'type': 2, 'len': 3}}), 2)
14401440
# ``Opt Data Len`` too short to hold the TaggerID it declares must raise
14411441
# rather than drive the identifier length negative -- c.f. #438, where the
1442-
# unguarded subtraction reached :func:`struct.calcsize` as ``'-1s'``.
1443-
with self.assertRaisesRegex(FieldValueError, 'invalid SMF I-DPD option length'):
1442+
# unguarded subtraction reached :func:`struct.calcsize` as ``'-1s'``. A
1443+
# ``ProtocolError``, so that a header refused for it is refused quietly (#2727).
1444+
with self.assertRaisesRegex(ProtocolError, 'invalid SMF I-DPD option length'):
14441445
schema.smf_i_dpd_id_len({'len': 0, 'info': {'type': 0, 'len': 0}})
1445-
with self.assertRaisesRegex(FieldValueError, 'invalid SMF I-DPD option length'):
1446+
with self.assertRaisesRegex(ProtocolError, 'invalid SMF I-DPD option length'):
14461447
schema.smf_i_dpd_id_len({'len': 0, 'info': {'type': 2, 'len': 3}})
14471448

14481449
for mode in (SMFDPDMode.I_DPD, SMFDPDMode.H_DPD):
@@ -2110,14 +2111,15 @@ def _assert_identification_based_dpd_option_rejects_underflowing_length(self, pr
21102111
identifier length to ``-1``. That reached :func:`struct.calcsize` as the
21112112
template ``'-1s'`` and raised a bare ``struct.error: bad char in struct
21122113
format`` -- not one of pcapkit's own exception types, and uncatchable
2113-
through :mod:`pcapkit.utilities.exceptions`.
2114+
through :mod:`pcapkit.utilities.exceptions`. It is a ``ProtocolError``, so
2115+
that a header refused for it is refused quietly (#2727).
21142116
21152117
"""
2116-
from pcapkit.utilities.exceptions import FieldValueError
2118+
from pcapkit.utilities.exceptions import ProtocolError
21172119

21182120
raw = bytes.fromhex('3b00080000000000')
21192121

2120-
with self.assertRaisesRegex(FieldValueError, 'invalid SMF (I-)?DPD option length'):
2122+
with self.assertRaisesRegex(ProtocolError, 'invalid SMF (I-)?DPD option length'):
21212123
with time_limit():
21222124
protocol_cls(raw, extension=True)
21232125

@@ -2143,17 +2145,18 @@ def _assert_calipso_option_rejects_underflowing_length(self, protocol_cls: type)
21432145
length to ``-8``. That reached :func:`struct.calcsize` as the template
21442146
``'-8s'`` and raised a bare ``struct.error: bad char in struct
21452147
format`` -- not one of pcapkit's own exception types, and uncatchable
2146-
through :mod:`pcapkit.utilities.exceptions`.
2148+
through :mod:`pcapkit.utilities.exceptions`. It is a ``ProtocolError``, so
2149+
that a header refused for it is refused quietly (#2727).
21472150
21482151
"""
2149-
from pcapkit.utilities.exceptions import FieldValueError
2152+
from pcapkit.utilities.exceptions import ProtocolError
21502153

21512154
# next(1) hdr_ext_len(1)=1 -> 16-octet extension header; option
21522155
# type(1)=CALIPSO(0x07) len(1)=0, then domain(4) cmpt_len(1) level(1)
21532156
# checksum(2) all zero, with four trailing zero octets.
21542157
raw = bytes.fromhex('3b0007000000000000000000000000')
21552158

2156-
with self.assertRaisesRegex(FieldValueError, 'invalid CALIPSO option length'):
2159+
with self.assertRaisesRegex(ProtocolError, 'invalid CALIPSO option length'):
21572160
with time_limit():
21582161
protocol_cls(raw, extension=True)
21592162

@@ -2178,17 +2181,18 @@ def _assert_mpl_option_rejects_underflowing_length(self, protocol_cls: type) ->
21782181
:func:`struct.calcsize` as the template ``'-2s'`` and raised a bare
21792182
``struct.error: bad char in struct format`` -- not one of pcapkit's
21802183
own exception types, and uncatchable through
2181-
:mod:`pcapkit.utilities.exceptions`.
2184+
:mod:`pcapkit.utilities.exceptions`. It is a ``ProtocolError``, so that a
2185+
header refused for it is refused quietly (#2727).
21822186
21832187
"""
2184-
from pcapkit.utilities.exceptions import FieldValueError
2188+
from pcapkit.utilities.exceptions import ProtocolError
21852189

21862190
# next(1) hdr_ext_len(1)=0 -> 8-octet extension header; option
21872191
# type(1)=MPL_Option(0x6d) len(1)=0, flags(1)=0 (Seed-ID type 0),
21882192
# seq(1)=0, with two trailing zero octets.
21892193
raw = bytes.fromhex('3b006d0000000000')
21902194

2191-
with self.assertRaisesRegex(FieldValueError, 'invalid MPL option length'):
2195+
with self.assertRaisesRegex(ProtocolError, 'invalid MPL option length'):
21922196
with time_limit():
21932197
protocol_cls(raw, extension=True)
21942198

0 commit comments

Comments
 (0)