Skip to content

Commit d1c7bb5

Browse files
authored
fix(reassembly): a datagram too long for its length field never completes (#1585) (#1598)
1 parent 588b24c commit d1c7bb5

8 files changed

Lines changed: 285 additions & 43 deletions

File tree

‎docs/source/pcapkit/foundation/reassembly/ip/index.rst‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -94,10 +94,17 @@ Because completion frees the buffer (:rfc:`791#section-3.2`), a fragment that
9494
arrives after its datagram has completed opens a new buffer, which is later
9595
reported as a separate incomplete datagram; this is intended (:issue:`1507`).
9696

97-
Two departures from the procedure above:
97+
Three departures from the procedure above:
9898

9999
* Only the final fragment's partial last ``RCVBT`` block is set. A non-final
100100
fragment carries a multiple of 8 octets, so one ending mid-block was cut or
101101
is malformed, and setting its last block would complete the datagram with
102102
octets nobody sent (:issue:`1567`).
103103
* A datagram longer than the data buffer is never complete (:issue:`1566`).
104+
* Nor is a datagram longer than its length field can declare: 65,535 octets
105+
with its header, all of which IPv4's Total Length counts and all but the
106+
fixed 40 octets of which IPv6's Payload Length counts. Linux rejects such a
107+
datagram too (:issue:`1585`). A fragment with Total Length 0, as TCP
108+
segmentation offload and BIG TCP leave one, declares no length
109+
(:issue:`1555`), so a datagram it carries past that bound grows the data
110+
buffer instead, and only the buffer bounds it.

‎docs/source/pcapkit/foundation/reassembly/ip/ipv4.rst‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -155,9 +155,13 @@ Terminology
155155
:meth:`IP._detect_conflicts <pcapkit.foundation.reassembly.ip.IP._detect_conflicts>`.
156156
``RCVBT`` records receipt in 8-octet blocks, which is coarser than
157157
the octet a conflict needs; ``TDL`` is what recovers the exact
158-
extent for the one block that can be partially real -- the final
159-
fragment's own tail -- so a conflict here is never wider than the
160-
octets that genuinely disagreed, even inside that block.
158+
extent for the one block marked received that can be partially real
159+
-- the final fragment's own tail -- so a conflict here is never wider
160+
than the octets that genuinely disagreed, even inside that block.
161+
Any other partial last block -- a non-final, truncated or clipped
162+
fragment's -- is left unmarked (:issue:`1567`), and its octets are
163+
not compared: a later fragment that overwrites them records no
164+
conflict (:issue:`1585`).
161165

162166
.. note::
163167

‎docs/source/pcapkit/foundation/reassembly/ip/ipv6.rst‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -198,6 +198,10 @@ Terminology
198198
:meth:`IP._detect_conflicts <pcapkit.foundation.reassembly.ip.IP._detect_conflicts>`.
199199
``RCVBT`` records receipt in 8-octet blocks, which is coarser than
200200
the octet a conflict needs; ``TDL`` is what recovers the exact
201-
extent for the one block that can be partially real -- the final
202-
fragment's own tail -- so a conflict here is never wider than the
203-
octets that genuinely disagreed, even inside that block.
201+
extent for the one block marked received that can be partially real
202+
-- the final fragment's own tail -- so a conflict here is never wider
203+
than the octets that genuinely disagreed, even inside that block.
204+
Any other partial last block -- a non-final, truncated or clipped
205+
fragment's -- is left unmarked (:issue:`1567`), and its octets are
206+
not compared: a later fragment that overwrites them records no
207+
conflict (:issue:`1585`).

‎pcapkit/foundation/reassembly/data/ip.py‎

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -128,14 +128,17 @@ class Datagram(DeferredPacket, Info, Generic[_AT]):
128128
#:
129129
#: :attr:`Buffer.RCVBT <pcapkit.foundation.reassembly.data.ip.Buffer.RCVBT>`
130130
#: only records receipt in 8-octet blocks, coarser than the octet
131-
#: granularity a conflict needs. Every fragment but the last is required
132-
#: to be block-aligned (and :attr:`Packet.fo` is *always* a multiple of 8,
133-
#: being wire-encoded in 8-octet units), so the only block that can be
134-
#: partially real is the one holding the final fragment's own tail -- and
135-
#: a range reported here never extends past
131+
#: granularity a conflict needs, and an octet is compared only if its
132+
#: block is marked. :attr:`Packet.fo` is *always* a multiple of 8, being
133+
#: wire-encoded in 8-octet units, and only the final fragment marks a
134+
#: partial last block, so the only marked block that can be partially real
135+
#: is the one holding the final fragment's own tail -- and a range reported
136+
#: here never extends past
136137
#: :attr:`Buffer.TDL <pcapkit.foundation.reassembly.data.ip.Buffer.TDL>`
137138
#: for exactly that reason, even though a whole ``RCVBT`` block straddling
138-
#: it reads as "received". See
139+
#: it reads as "received". Any other partial last block -- a non-final,
140+
#: truncated or clipped fragment's -- is left unmarked, so a later fragment
141+
#: that overwrites its octets records no conflict. See
139142
#: :meth:`IP._detect_conflicts <pcapkit.foundation.reassembly.ip.IP._detect_conflicts>`.
140143
conflict: 'tuple[tuple[int, int], ...]'
141144

‎pcapkit/foundation/reassembly/ip.py‎

Lines changed: 72 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,11 @@
2929

3030
__all__ = ['IP']
3131

32+
#: The most octets a 16-bit length field declares -- IPv4's Total Length, the
33+
#: header included, and IPv6's Payload Length, the fixed header excluded -- and
34+
#: so the length the data buffer is preallocated to.
35+
_MAX_LENGTH = 65535
36+
3237

3338
class IP(ReassemblyBase[Packet[_AT], Datagram[_AT], BufferID, Buffer[_AT]], Generic[_AT]): # pylint: disable=abstract-method
3439
"""Reassembly for IP payload.
@@ -44,7 +49,7 @@ class IP(ReassemblyBase[Packet[_AT], Datagram[_AT], BufferID, Buffer[_AT]], Gene
4449
does not: it is left out of the runs and ends the prefix, but once
4550
the total length is known the contiguous payload keeps its real
4651
octets, with only the rest of the hole zero-filled. A datagram
47-
longer than the data buffer is never complete
52+
longer than its length field can declare is never complete
4853
store: if store reassembled datagram in memory, i.e.,
4954
:attr:`self._dtgram <pcapkit.foundation.reassembly.reassembly.Reassembly._dtgram>`
5055
(if not, datagram will be discarded after callback)
@@ -83,6 +88,23 @@ def _rectify_header(self, header: 'bytes', proto: 'TransType') -> 'bytes': # py
8388
"""
8489
return header
8590

91+
def _header_overhead(self, length: 'int') -> 'int':
92+
"""Count the header octets that a datagram's length field includes.
93+
94+
The base implementation counts them all, which is what IPv4 wants: its
95+
Total Length is the header's length plus the data's
96+
(:rfc:`791#section-3.1`). IPv6 overrides this, because its Payload
97+
Length leaves out the fixed header (:rfc:`8200#section-3`).
98+
99+
Args:
100+
length: Length of the datagram's header.
101+
102+
Returns:
103+
How many of those octets the length field counts.
104+
105+
"""
106+
return length
107+
86108
def reassembly(self, info: 'Packet[_AT]') -> 'None':
87109
"""Reassembly procedure.
88110
@@ -138,7 +160,7 @@ def reassembly(self, info: 'Packet[_AT]') -> 'None':
138160
RCVBT=bytearray(8192), # Fragment Received Bit Table
139161
index=[], # index record
140162
header=header, # header buffer
141-
datagram=bytearray(65535), # data buffer
163+
datagram=bytearray(_MAX_LENGTH), # data buffer
142164
timestamp=TS, # first-arriving fragment's clock reading
143165
conflict=[], # conflicting octet ranges
144166
)
@@ -166,9 +188,12 @@ def reassembly(self, info: 'Packet[_AT]') -> 'None':
166188
# anything, and so does this, whatever **MF** and the offset say: an
167189
# unfragmented one is then whole rather than ``-IHL`` octets long
168190
# (:issue:`1555`). Linux's BIG TCP zeroes the field precisely because
169-
# the datagram outgrows the 65535 octets it can declare, so the buffer
170-
# grows to hold such a datagram rather than clipping it, ``RCVBT`` in
171-
# step with it.
191+
# the datagram, header included, outgrows the 65535 octets it can
192+
# declare -- and such a datagram never completes in a buffer of that
193+
# length (:meth:`_is_complete`). So the buffer grows past it, to hold
194+
# the datagram rather than clip it, ``RCVBT`` in step with it; it grows
195+
# by an octet even when the data alone would fit, since its growing is
196+
# what lifts the bound.
172197
#
173198
# With nothing declared, a snaplen cut cannot be seen in the length. A
174199
# non-final fragment's cut still shows, as a partial last block, which
@@ -179,9 +204,10 @@ def reassembly(self, info: 'Packet[_AT]') -> 'None':
179204
if TL == 0:
180205
length = len(info.payload)
181206
extent = FO + length
182-
if extent > len(buf.datagram):
183-
buf.datagram.extend(bytes(extent - len(buf.datagram)))
184-
buf.RCVBT.extend(bytes((extent + 7) // 8 - len(buf.RCVBT)))
207+
if self._header_overhead(IHL) + extent > _MAX_LENGTH:
208+
size = max(extent, _MAX_LENGTH + 1, len(buf.datagram))
209+
buf.datagram.extend(bytes(size - len(buf.datagram)))
210+
buf.RCVBT.extend(bytes((size + 7) // 8 - len(buf.RCVBT)))
185211
else:
186212
length = TL - IHL
187213
held = max(min(len(info.payload), length, len(buf.datagram) - FO), 0)
@@ -243,25 +269,35 @@ def reassembly(self, info: 'Packet[_AT]') -> 'None':
243269
self.submit(self._buffer.pop(BUFID), bufid=BUFID, checked=True)
244270
)
245271

246-
@staticmethod
247-
def _is_complete(buf: 'Buffer[_AT]') -> 'bool':
272+
def _is_complete(self, buf: 'Buffer[_AT]') -> 'bool':
248273
"""Tell whether a buffer holds its whole datagram.
249274
250275
Arguments:
251276
buf: buffer to check
252277
253278
Returns:
254-
Whether the total data length is known, fits the data buffer, and
255-
every 8-octet block up to it is marked received.
256-
257-
A datagram longer than the data buffer cannot be held, so it is never
258-
complete. Checking ``RCVBT`` alone does not show that: slicing it past
259-
its end stops silently, so the blocks past the buffer would read as
260-
received (:issue:`1566`).
279+
Whether the total data length is known, the datagram is no longer
280+
than it may be, and every 8-octet block up to it is marked
281+
received.
282+
283+
A datagram is no longer than ``_MAX_LENGTH`` with its header, as
284+
:meth:`_header_overhead` counts it: IPv4's Total Length counts the
285+
whole header, and IPv6's Payload Length every header but the fixed one.
286+
A longer datagram cannot be declared, so it is never complete, and
287+
Linux's ``ip_frag_reasm`` and ``ip6_frag_reasm`` reject it too
288+
(:issue:`1585`). A fragment with Total Length 0 declares no length, so
289+
a buffer it has grown past that bounds the datagram by its own length
290+
alone (:issue:`1555`). Checking ``RCVBT`` does not show either bound:
291+
slicing it past its end stops silently, so the blocks past the buffer
292+
would read as received (:issue:`1566`).
261293
262294
"""
263295
TDL = buf.TDL
264-
return 0 <= TDL <= len(buf.datagram) and all(buf.RCVBT[:(TDL + 7) // 8])
296+
if len(buf.datagram) > _MAX_LENGTH:
297+
limit = len(buf.datagram)
298+
else:
299+
limit = _MAX_LENGTH - self._header_overhead(len(buf.header))
300+
return 0 <= TDL <= limit and all(buf.RCVBT[:(TDL + 7) // 8])
265301

266302
@staticmethod
267303
def _detect_conflicts(rcvbt: 'bytearray', tdl: 'int', datagram: 'bytearray', payload: 'bytearray',
@@ -291,23 +327,30 @@ def _detect_conflicts(rcvbt: 'bytearray', tdl: 'int', datagram: 'bytearray', pay
291327
already genuinely received.
292328
293329
:rfc:`791` marks receipt in 8-octet blocks (``RCVBT``), coarser than
294-
the octet granularity a conflict needs: every fragment but the last is
295-
required to be a multiple of 8 octets, and a fragment's ``FO`` is
296-
*always* a multiple of 8 -- it is wire-encoded in 8-octet units -- so a
297-
non-final fragment's range is always exactly block-aligned. The only
298-
block that can be *partially* real is therefore the one holding the
299-
final fragment's own tail: the ``RCVBT`` update in :meth:`reassembly`
300-
sets that block's bit across its full 8 octets even though only the
301-
octets up to ``tdl`` were ever actually written, the rest still being
302-
``datagram``'s zero-fill.
330+
the octet granularity a conflict needs, so an octet is compared only if
331+
its block is marked. A fragment's ``FO`` is *always* a multiple of 8 --
332+
it is wire-encoded in 8-octet units -- and the ``RCVBT`` update in
333+
:meth:`reassembly` marks the blocks a fragment fills whole. Only the
334+
final fragment, held in full, also marks its partial last block, across
335+
its full 8 octets even though only the octets up to ``tdl`` were ever
336+
actually written, the rest still being ``datagram``'s zero-fill.
303337
304338
So once ``tdl`` is known, an octet at or past it is excluded here
305339
regardless of its block's bit -- comparing it would manufacture a
306340
conflict against a byte nothing ever really sent, over a distinction
307341
:meth:`~pcapkit.foundation.reassembly.ip.IP.submit` does not need
308342
anyway, since it never reports a payload past ``tdl``. Before ``tdl``
309-
is known (``tdl < 0``), every set ``rcvbt`` bit came from a non-final,
310-
block-aligned fragment and is exact on its own, with nothing to clip.
343+
is known (``tdl < 0``), every set ``rcvbt`` bit came from a non-final
344+
fragment's whole block and is exact on its own, with nothing to clip.
345+
346+
Any other partial last block -- a non-final, truncated or clipped
347+
fragment's -- is written but left clear (:issue:`1567`), so its octets
348+
are deliberately not compared: a later fragment that overwrites them
349+
records no conflict, whether or not it agrees (:issue:`1585`). Telling
350+
which octets of a clear block were written would take receipt state per
351+
octet, which ``RCVBT`` does not keep. Linux never compares a non-final
352+
fragment's partial block either: it trims an IPv4 one to a multiple of
353+
8 (``ip_frag_queue``) and drops an IPv6 one (``ip6_frag_queue``).
311354
312355
"""
313356
conflicts = [] # type: list[tuple[int, int]]

‎pcapkit/foundation/reassembly/ipv6.py‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -158,3 +158,20 @@ def _rectify_header(self, header: 'bytes', proto: 'TransType') -> 'bytes':
158158
if header[offset] != Enum_TransType.IPv6_Frag:
159159
return header
160160
return header[:offset] + bytes((int(proto),)) + header[offset + 1:]
161+
162+
def _header_overhead(self, length: 'int') -> 'int':
163+
"""Count the header octets that the Payload Length includes.
164+
165+
The Payload Length counts every octet after the fixed IPv6 header
166+
(:rfc:`8200#section-3`), so a reassembled packet's is its unfragmentable
167+
part's extension headers plus its data (:rfc:`8200#section-4.5`). The
168+
header here is the unfragmentable part, its Fragment header excluded.
169+
170+
Args:
171+
length: Length of the datagram's header.
172+
173+
Returns:
174+
How many of those octets the Payload Length counts.
175+
176+
"""
177+
return max(length - _IPV6_HDR_LEN, 0)

‎tests/foundation/reassembly/test_ip_last_fragment_offset_1565_unit.py‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,9 +82,12 @@ def test_a_lone_fragment_at_the_largest_offset_is_buffered(self) -> None:
8282
self.assertEqual(datagram.payload, bytes(LAST) + tail)
8383

8484
def test_the_last_fragment_at_the_largest_offset_completes_its_datagram(self) -> None:
85+
# IPv6 only: an IPv4 datagram reaching octet 65,528 is too long for its
86+
# Total Length, header included, so it never completes (#1585).
8587
from pcapkit.foundation.reassembly.data.data import Completion
88+
from pcapkit.foundation.reassembly.ipv6 import IPv6
8689

87-
for cls in self._classes():
90+
for cls in (IPv6,):
8891
for size in (1, 7):
8992
for strict in (True, False):
9093
with self.subTest(cls=cls.__name__, size=size, strict=strict):

0 commit comments

Comments
 (0)