Skip to content

Commit 342fe7e

Browse files
authored
feat(link): decode radiotap TLV items a producer cut short (#2655) (#2690)
radiotap.org lets a producer leave off the octets it does not fill at the end of a TLV item, and a parser reads them as zeros. Such an item stayed raw. - _read_tlv_data unpacks data short of its schema, or of the vendor header, at the schema's full length, rounded up to a whole list item, so Schema.unpack zero-fills it and records the cut; the decoded data keeps it as __short_read__. Data past a fixed layout stays raw. - _make_tlv_data packs through _pack_tlv_data, which hands the record back to the schema, so Schema.pack writes the octets as captured. A malformed record, or vendor data after a cut header, raises ProtocolError. - Both unpacks share the CaughtByCaller(FieldValueError) guard, so a cut the zero-pad limits refuse stays raw, quietly. - The module docstring, radiotap.rst, the TLV data model and make()'s Raises describe it. - Tests: U-SIG, S1G, EHT and vendor items cut at a field boundary and inside a field, each against its zero-padded control; from_data and make() round trips; quietness; the padding-limit refusal; a registered layout. The #2576 pins on a short S1G and a ragged EHT now expect them decoded.
1 parent 832f54f commit 342fe7e

5 files changed

Lines changed: 416 additions & 39 deletions

File tree

‎docs/source/pcapkit/protocols/link/radiotap.rst‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -84,9 +84,14 @@ decoded the same way.
8484
octets after it, kept whole. Without either, the next word continues the
8585
namespace, each bit numbered 32 past the same bit of the word before.
8686
* **TLVs.** Bit 28 makes the rest of the region a list of type-length-value
87-
items. An item's data is decoded where its type has a schema whose size it
88-
matches, and kept raw otherwise; a vendor item is split into its header and
89-
raw data.
87+
items. An item's data is decoded where its type has a schema it does not run
88+
past, and kept raw otherwise; a vendor item is split into its header and raw
89+
data. radiotap.org lets a producer leave out the octets that end an item, which
90+
are then zeros, so an item short of its schema, or of the vendor header, is
91+
decoded as though they were there. Its decoded data records the cut as
92+
``__short_read__``, as :meth:`Schema.unpack <pcapkit.protocols.schema.schema.Schema.unpack>`
93+
does, so :meth:`Radiotap.make <pcapkit.protocols.link.radiotap.Radiotap.make>`
94+
writes the octets that were captured.
9095
* **What is not decoded.** A radiotap-namespace bit no field is defined for --
9196
18, FreeBSD's XChannel, which radiotap.org lists as unofficial, and every one
9297
numbered 32 or more, the TLV-only types included -- or a field that runs

‎pcapkit/protocols/data/link/radiotap.py‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -448,8 +448,10 @@ class TLV(Data):
448448
type: 'FieldType'
449449
#: Length of :attr:`data`, in octets, less the padding after it.
450450
length: 'int'
451-
#: The data, decoded where the type defines a layout and :attr:`length`
452-
#: matches it; the raw octets otherwise.
451+
#: The data, decoded where the type defines a layout :attr:`length` does
452+
#: not run past; the raw octets otherwise. Data short of the layout is
453+
#: decoded with the octets it lacks as zeros, and records the cut as
454+
#: ``__short_read__``.
453455
data: 'RadiotapField | VendorTLV | bytes'
454456

455457
if TYPE_CHECKING:

‎pcapkit/protocols/link/radiotap.py‎

Lines changed: 93 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -89,9 +89,14 @@
8989
octets after it, kept whole. Without either, the next word continues the
9090
namespace, each bit numbered 32 past the same bit of the word before.
9191
* **TLVs.** Bit 28 makes the rest of the region a list of type-length-value
92-
items. An item's data is decoded where its type has a schema whose size it
93-
matches, and kept raw otherwise; a vendor item is split into its header and
94-
raw data.
92+
items. An item's data is decoded where its type has a schema it does not run
93+
past, and kept raw otherwise; a vendor item is split into its header and raw
94+
data. radiotap.org lets a producer leave out the octets that end an item, which
95+
are then zeros, so an item short of its schema, or of the vendor header, is
96+
decoded as though they were there. Its decoded data records the cut as
97+
``__short_read__``, as :meth:`Schema.unpack <pcapkit.protocols.schema.schema.Schema.unpack>`
98+
does, so :meth:`Radiotap.make <pcapkit.protocols.link.radiotap.Radiotap.make>`
99+
writes the octets that were captured.
95100
* **What is not decoded.** A radiotap-namespace bit no field is defined for --
96101
18, FreeBSD's XChannel, which radiotap.org lists as unofficial, and every one
97102
numbered 32 or more, the TLV-only types included -- or a field that runs
@@ -497,8 +502,10 @@ def make(self,
497502
fit in 32 bits; if ``fields`` is neither :class:`bytes` nor
498503
fields; if the fields do not match the bits of ``present``,
499504
a word but the last of which ends the chain, or a value does
500-
not fit its subfield; or if ``it_len`` is given and does not
501-
fit in 16 bits.
505+
not fit its subfield; if a TLV item's ``__short_read__`` names
506+
no field and an octet count, or cuts short a vendor header that
507+
data follows; or if ``it_len`` is given and does not fit in 16
508+
bits.
502509
503510
"""
504511
for name, value in (('it_version', it_version), ('it_pad', it_pad)):
@@ -2053,66 +2060,123 @@ def _read_tlv_data(self, tlv_type: 'Enum_FieldType', data: 'bytes') -> 'Data_Fie
20532060
20542061
Returns:
20552062
A vendor TLV for the Vendor Namespace type, the field for a type
2056-
whose schema ``data`` fills exactly, and the raw octets otherwise.
2063+
whose schema ``data`` does not run past, and the raw octets
2064+
otherwise. Data short of its schema, or of the vendor header, is
2065+
read with the octets it lacks as zeros, and the result records the
2066+
cut as ``__short_read__``, which :meth:`_make_tlv_data` replays;
2067+
it stays raw if the zero-padding limits of
2068+
:mod:`pcapkit.utilities.config` refuse the zeros.
20572069
20582070
"""
2071+
# NOTE: radiotap.org reads the octets a producer leaves off an item as zeros, so short data is unpacked
2072+
# at its schema's full length, to the end of a list item, and ``Schema.unpack`` records the cut (#2655).
2073+
layout = None # type: Optional[tuple[FieldParser, FieldConstructor, Type[Schema_Field]]]
20592074
if tlv_type == Enum_FieldType.Vendor_Namespace:
2060-
size, _ = self._field_size(Schema_VendorTLV)
2061-
if len(data) < size:
2075+
field_schema = Schema_VendorTLV # type: Type[Schema]
2076+
length, _ = self._field_size(field_schema)
2077+
octets = data[:length]
2078+
else:
2079+
layout = self._lookup_field(tlv_type)
2080+
if layout is None:
20622081
return data
2063-
vendor = cast('Schema_VendorTLV',
2064-
Schema_VendorTLV.unpack(data[:size], size, None)) # type: ignore[call-arg,misc]
2065-
return Data_VendorTLV(oui=vendor.oui, subtype=vendor.subtype, presence_type=vendor.presence_type,
2066-
reserved=vendor.reserved, data=data[size:])
2067-
2068-
layout = self._lookup_field(tlv_type)
2069-
if layout is None:
2070-
return data
2071-
parser, _, field_schema = layout
2072-
size, rest = self._field_size(field_schema)
2073-
if len(data) < size or (not rest and len(data) != size):
2074-
return data
2075-
# NOTE: a list taking the rest refuses octets that are not whole items
2076-
# of it; the refusal is handled here, so it is quiet.
2082+
field_schema = layout[2]
2083+
size, rest = self._field_size(field_schema)
2084+
if not rest and len(data) > size:
2085+
return data
2086+
length = max(len(data), size)
2087+
if rest and len(data) > size:
2088+
tail = next(field for field in field_schema.__fields__.values() if field.length < 0)
2089+
item = tail._item_type if isinstance(tail, ListField) else None # pylint: disable=protected-access
2090+
if item is not None and item.length > 0:
2091+
length += -(len(data) - size) % item.length
2092+
octets = data
2093+
2094+
# NOTE: a limit on zero-padding, or a list taking the rest, may refuse
2095+
# what it is given; the refusal is handled here, so it is quiet.
20772096
with CaughtByCaller(FieldValueError):
20782097
try:
2079-
schema = cast('Schema_Field', field_schema.unpack(data, len(data), None)) # type: ignore[call-arg,misc]
2098+
schema = cast('Schema', field_schema.unpack(octets, length, None)) # type: ignore[call-arg,misc]
20802099
except FieldValueError:
20812100
return data
2082-
return parser(tlv_type, schema)
2101+
if layout is None:
2102+
vendor = cast('Schema_VendorTLV', schema)
2103+
entry = Data_VendorTLV(oui=vendor.oui, subtype=vendor.subtype, presence_type=vendor.presence_type,
2104+
reserved=vendor.reserved, data=data[length:]) # type: Data_Field | Data_VendorTLV
2105+
else:
2106+
entry = layout[0](tlv_type, cast('Schema_Field', schema))
2107+
2108+
record = schema.__dict__.get('__short_read__')
2109+
if record is not None:
2110+
entry.__update__({'__short_read__': record})
2111+
return entry
20832112

20842113
def _make_tlv_data(self, tlv_type: 'Enum_FieldType',
20852114
data: 'Data_Field | Data_VendorTLV | Mapping[str, Any] | bytes') -> 'bytes':
20862115
"""Encode a TLV item's data, the inverse of :meth:`_read_tlv_data`.
20872116
20882117
Args:
20892118
tlv_type: Type of the item.
2090-
data: The item's data: raw octets, written as they are, or decoded.
2119+
data: The item's data: raw octets, written as they are, or decoded,
2120+
cut back as its ``__short_read__`` says when it has one.
20912121
20922122
Returns:
20932123
The item's data.
20942124
20952125
Raises:
20962126
ProtocolError: If decoded data is given for a type with no layout,
2097-
or does not encode.
2127+
or does not encode; if its ``__short_read__`` is not a field
2128+
name and an octet count; or if a vendor TLV whose header is cut
2129+
short has data.
20982130
20992131
"""
21002132
if isinstance(data, (bytes, bytearray)):
21012133
return bytes(data)
21022134
if tlv_type == Enum_FieldType.Vendor_Namespace:
21032135
oui, vendor_data = self._make_vendor_data(tlv_type, Schema_VendorTLV, data)
21042136
values = {'reserved': 0, **data}
2105-
return Schema_VendorTLV(
2137+
header = self._pack_tlv_data(tlv_type, Schema_VendorTLV(
21062138
oui=oui,
21072139
subtype=self._make_subfield(tlv_type, Schema_VendorTLV, values, 'subtype'),
21082140
presence_type=self._make_subfield(tlv_type, Schema_VendorTLV, values, 'presence_type'),
21092141
reserved=self._make_subfield(tlv_type, Schema_VendorTLV, values, 'reserved'),
2110-
).pack() + vendor_data
2142+
), data)
2143+
if vendor_data and len(header) < self._field_size(Schema_VendorTLV)[0]:
2144+
raise ProtocolError(f'{self.alias}: invalid VendorTLV.data: {vendor_data!r}, '
2145+
'the vendor header before it is cut short')
2146+
return header + vendor_data
21112147
layout = self._lookup_field(tlv_type)
21122148
if layout is None:
21132149
raise ProtocolError(f'{self.alias}: invalid TLV.data: {tlv_type!r} defines no layout, '
21142150
'so its data must be bytes')
2115-
return layout[1](tlv_type, data).pack()
2151+
return self._pack_tlv_data(tlv_type, layout[1](tlv_type, data), data)
2152+
2153+
def _pack_tlv_data(self, tlv_type: 'Enum_FieldType', schema: 'Schema',
2154+
data: 'Data_Field | Data_VendorTLV | Mapping[str, Any]') -> 'bytes':
2155+
"""Pack a TLV item's schema, cut back as :meth:`_read_tlv_data` recorded.
2156+
2157+
Args:
2158+
tlv_type: Type of the item, for the error message.
2159+
schema: The schema made from ``data``.
2160+
data: The item's decoded data, whose ``__short_read__``, if any,
2161+
names the field its octets ended inside and how many of that
2162+
field's octets there were.
2163+
2164+
Returns:
2165+
The packed schema, without the octets past the cut.
2166+
2167+
Raises:
2168+
ProtocolError: If ``__short_read__`` is not a field of ``schema``
2169+
and a count of octets.
2170+
2171+
"""
2172+
record = data.get('__short_read__')
2173+
if record is not None:
2174+
if not (isinstance(record, (tuple, list)) and len(record) == 2 and record[0] in schema.__fields__
2175+
and isinstance(record[1], int) and not isinstance(record[1], bool) and record[1] >= 0):
2176+
raise ProtocolError(f'{self.alias}: invalid {tlv_type.name}.__short_read__: {record!r}')
2177+
# NOTE: ``Schema.pack`` cuts its buffers back to a record it holds.
2178+
schema.__dict__['__short_read__'] = tuple(record)
2179+
return schema.pack()
21162180

21172181
def _make_vendor_data(self, field_type: 'Enum_FieldType', schema: 'Type[Schema]',
21182182
entry: 'Mapping[str, Any]') -> 'tuple[bytes, bytes]':

‎tests/protocols/link/test_radiotap_fields_2576_unit.py‎

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -377,8 +377,11 @@ def item(tlv_type: int, data: bytes) -> bytes:
377377
'presence_type': 5, 'reserved': 0,
378378
'data': b'\xaa\xbb'}))
379379
self.assertEqual(got[4], ('TSFT', 8, {'type': 0, 'mactime': 0x8887868584838281}))
380-
# S1G cut short by its producer, and a padding item, stay raw
381-
self.assertEqual(got[5:], [('S1G', 4, s1g[:4]), ('TLV_fields_in_radiotap', 0, b'')])
380+
# S1G cut short by its producer decodes with the zeros it left out (#2655),
381+
# and a padding item stays raw
382+
self.assertEqual(got[5:], [('S1G', 4, {'type': 32, 'known': 1, 'data1': 2, 'data2': 0,
383+
'__short_read__': ('data2', 0)}),
384+
('TLV_fields_in_radiotap', 0, b'')])
382385
self._assert_round_trips(parsed, raw)
383386

384387
def test_a_tlv_past_the_region_is_unknown(self) -> None:
@@ -580,8 +583,8 @@ def test_each_field_type_pairs_a_schema_with_its_methods(self) -> None:
580583
self.assertTrue(callable(getattr(Radiotap, f'_read_field_{name}', None)))
581584
self.assertTrue(callable(getattr(Radiotap, f'_make_field_{name}', None)))
582585

583-
def test_a_ragged_eht_tlv_stays_raw_and_quiet(self) -> None:
584-
"""An EHT item whose user words do not end on a word stays raw, and nothing is logged."""
586+
def test_a_ragged_eht_tlv_decodes_quietly(self) -> None:
587+
"""An EHT item whose last user word is cut short decodes it padded with zeros (#2655), and nothing is logged."""
585588
import logging
586589

587590
records = [] # type: list[logging.LogRecord]
@@ -595,7 +598,9 @@ def test_a_ragged_eht_tlv_stays_raw_and_quiet(self) -> None:
595598
parsed = self._parse(raw)
596599
finally:
597600
logger.removeHandler(handler)
598-
self.assertEqual([(item.type.name, item.data) for item in parsed.info.decoded[28].tlvs], [('EHT', data)])
601+
self.assertEqual([(item.type.name, item.data) for item in parsed.info.decoded[28].tlvs],
602+
[('EHT', {'type': 34, 'known': 0, 'data': tuple(range(9)), 'user_info': (0xA2A1,),
603+
'__short_read__': ('user_info', 2)})])
599604
self.assertEqual(records, [])
600605
self._assert_round_trips(parsed, raw)
601606

0 commit comments

Comments
 (0)