fix(reassembly): abandon an IPv6 datagram on an overlapping fragment (#2547)

docs/source/pcapkit/foundation/reassembly/ip/index.rst

@@ -106,6 +106,11 @@ Departures from the procedure above:
 * An IPv6 atomic fragment, one with fragment offset zero and **M** clear,
   flushes nothing: it is reassembled in isolation and the fragments buffered
   under its ``BUFID`` are kept, per :rfc:`6946#section-4` (:issue:`2270`).
+* An IPv6 fragment that overlaps another, other than an exact duplicate,
+  abandons its datagram rather than overwriting the octets it overlaps, per
+  :rfc:`8200#section-4.5`; the datagram is reported as
+  :attr:`~pcapkit.foundation.reassembly.data.data.Completion.OVERLAP`
+  (:issue:`2547`).
 * ``TDL`` starts at ``-1`` rather than ``0``, so that a datagram carrying no
   data can complete (:issue:`1359`).
 * Only the final fragment's partial last ``RCVBT`` block is set. A non-final

docs/source/pcapkit/foundation/reassembly/ip/ip.rst

@@ -12,9 +12,11 @@ origin, using the :rfc:`791` procedure described in :doc:`index`.
    :no-members:
    :show-inheritance:
 
+   .. autoattribute:: __discard_overlap__
    .. automethod:: reassembly
    .. automethod:: submit
    .. automethod:: _detect_conflicts
+   .. automethod:: _find_overlap
 
 Data Models
 -----------

docs/source/pcapkit/foundation/reassembly/ip/ipv4.rst

@@ -146,6 +146,7 @@ Terminology
            |                         |                  |--> (tuple) (first, last), absolute and
            |                         |                               inclusive
            |                         |                  |--> ...
+           |                         |--> 'extent' : (list) always empty: IPv4 keeps no extents
            |--> (tuple) BUFID ...
 
        .. note::

docs/source/pcapkit/foundation/reassembly/ip/ipv6.rst

@@ -17,6 +17,7 @@ origin. Please refer to :doc:`ip` for more information.
    .. autoattribute:: __callback_fn__
       :no-value:
    .. autoattribute:: __timeout__
+   .. autoattribute:: __discard_overlap__
 
 Terminology
 -----------
@@ -108,7 +109,7 @@ Terminology
            |     |                  |--> (tuple) (first, last), absolute and inclusive
            |     |                  |--> ...
            |--> (Info) data
-           |     |--> 'completed' : (Completion) PARTIAL or TIMEOUT --> incomplete
+           |     |--> 'completed' : (Completion) PARTIAL, TIMEOUT or OVERLAP --> incomplete
            |     |--> 'id' : (Info) original packet identifier
            |     |            |--> 'src' --> (IPv6Address) ipv6.src
            |     |            |--> 'dst' --> (IPv6Address) ipv6.dst
@@ -148,17 +149,19 @@ Terminology
 
        .. note::
 
-          ``completed`` and ``conflict`` are independent signals: a datagram
-          can be :attr:`~pcapkit.foundation.reassembly.data.data.Completion.COMPLETE`
-          and still carry a non-empty ``conflict``. IPv6 reassembly shares
-          IPv4's resolution of an overlapping fragment's disagreement, the one
-          :rfc:`791` gives -- "this procedure will use the more recently
-          arrived copy in the data buffer" -- the opposite resolution from
-          TCP's first-write-wins (:rfc:`9293#section-3.10`), rather than
-          abandoning the datagram as :rfc:`8200#section-4.5` has a receiving
-          node do. So a contested range never leaves a hole on its own, and
-          ``conflict`` is what lets a caller tell a clean datagram from a
-          contested one.
+          A fragment that overlaps another, other than an exact duplicate,
+          abandons its datagram, as :rfc:`8200#section-4.5` has a receiving
+          node do, rather than overwriting the octets it overlaps as IPv4's
+          :rfc:`791` procedure does. The datagram is reported as
+          :attr:`Completion.OVERLAP <pcapkit.foundation.reassembly.data.data.Completion.OVERLAP>`,
+          its ``payload`` the fragments merged before the overlapping one, and
+          ``conflict`` where that fragment disagreed with them -- empty if it
+          repeated their octets. So a
+          :attr:`~pcapkit.foundation.reassembly.data.data.Completion.COMPLETE`
+          IPv6 datagram never carries a conflict. An exact duplicate, with the
+          same offset, length, **M** flag and data, is dropped as the RFC
+          permits, its frame number kept in ``index``. A later fragment of an
+          abandoned datagram opens a new buffer (:issue:`2547`).
 
    reasm.ipv6.buffer
        Data structure for internal buffering when performing reassembly algorithms
@@ -190,6 +193,9 @@ Terminology
            |                         |                  |--> (tuple) (first, last), absolute and
            |                         |                               inclusive
            |                         |                  |--> ...
+           |                         |--> 'extent' : (list) merged fragments, sorted and disjoint
+           |                         |                |--> (tuple) (start, stop, mf), stop exclusive
+           |                         |                |--> ...
            |--> (tuple) BUFID ...
 
        .. note::

pcapkit/foundation/reassembly/data/data.py

@@ -63,7 +63,7 @@ class Completion(EnumLookup, StrEnum):
     where that name is the stdlib ``enum.StrEnum``.
 
     Warning:
-        Being a :class:`str` whose :attr:`PARTIAL` and :attr:`TIMEOUT` members are
+        Being a :class:`str` whose members other than :attr:`COMPLETE` are
         **falsy** makes this a non-empty string that tests false, so ``bool(x)``
         and ``bool(str(x))`` disagree. That is deliberate -- the truthiness above
         is what lets ``if datagram.completed:`` read as a boolean test -- but code
@@ -91,11 +91,20 @@ class Completion(EnumLookup, StrEnum):
     #: discarded, so no further fragment will ever be added to this datagram.
     TIMEOUT = auto()
 
+    #: Reassembly was **abandoned** because a fragment overlapped another of the
+    #: same datagram without being an exact duplicate of it, which
+    #: :rfc:`8200#section-4.5` discards an IPv6 datagram for. The overlapping
+    #: fragment's octets are not in the payload; ``conflict`` still lists where
+    #: they disagreed with the octets kept.
+    #:
+    #: .. versionadded:: 1.6.0
+    OVERLAP = auto()
+
     def __bool__(self) -> 'bool':
         """Whether the datagram was reassembled in whole.
 
-        Only :attr:`COMPLETE` is truthy; both :attr:`PARTIAL` and
-        :attr:`TIMEOUT` describe an incomplete datagram.
+        Only :attr:`COMPLETE` is truthy; every other member describes an
+        incomplete datagram.
 
         Note:
             This override is what a :class:`str` base does *not* give -- every

pcapkit/foundation/reassembly/data/ip.py

@@ -123,15 +123,19 @@ class Datagram(DeferredPacket, Info, Generic[_AT]):
     #: fragment overlapped octets already buffered but did not repeat them.
     #: Empty when the datagram never saw a contested octet.
     #:
-    #: :rfc:`791` resolves the disagreement itself: "In the case that two or
-    #: more fragments contain the same data either identically or through a
-    #: partial overlap, this procedure will use the more recently arrived copy
-    #: in the data buffer and datagram delivered." So :attr:`payload` always
-    #: holds whichever fragment arrived *last* over a contested range -- the
-    #: opposite resolution from TCP's first-write-wins
+    #: For IPv4, :rfc:`791` resolves the disagreement itself: "In the case that
+    #: two or more fragments contain the same data either identically or
+    #: through a partial overlap, this procedure will use the more recently
+    #: arrived copy in the data buffer and datagram delivered." So
+    #: :attr:`payload` always holds whichever fragment arrived *last* over a
+    #: contested range -- the opposite resolution from TCP's first-write-wins
     #: (:rfc:`9293#section-3.10`) -- and this field is what lets a caller tell
     #: a clean datagram from a contested one, since a resolved conflict does
-    #: not, on its own, leave a hole for ``completed`` to report.
+    #: not, on its own, leave a hole for ``completed`` to report. IPv6 instead
+    #: abandons the datagram on the overlapping fragment, without writing it
+    #: (:rfc:`8200#section-4.5`), so there a conflict comes only with
+    #: :attr:`Completion.OVERLAP
+    #: <pcapkit.foundation.reassembly.data.data.Completion.OVERLAP>`.
     #:
     #: :attr:`Buffer.RCVBT <pcapkit.foundation.reassembly.data.ip.Buffer.RCVBT>`
     #: only records receipt in 8-octet blocks, coarser than the octet
@@ -190,6 +194,15 @@ class Buffer(Info, Generic[_AT]):
     #: :attr:`Datagram.conflict <pcapkit.foundation.reassembly.data.ip.Datagram.conflict>`
     #: verbatim when the buffer is submitted.
     conflict: 'list[tuple[int, int]]'
+    #: Extent of each fragment merged into this buffer, as ``(start, stop,
+    #: mf)``: its first octet, one past its last, and its **MF** (**M**) flag,
+    #: sorted by ``start``. Kept only where an overlapping fragment abandons
+    #: the datagram, i.e. for IPv6, whose merged fragments therefore never
+    #: overlap; empty for IPv4. See
+    #: :attr:`IP.__discard_overlap__ <pcapkit.foundation.reassembly.ip.IP.__discard_overlap__>`.
+    #:
+    #: .. versionadded:: 1.6.0
+    extent: 'list[tuple[int, int, bool]]'
 
     if TYPE_CHECKING:
-        def __init__(self, TDL: 'int', RCVBT: 'bytearray', index: 'list[int]', header: 'bytes', datagram: 'bytearray', timestamp: 'float', conflict: 'list[tuple[int, int]]') -> 'None': ...  # pylint: disable=unused-argument,super-init-not-called,multiple-statements,line-too-long,redefined-builtin
+        def __init__(self, TDL: 'int', RCVBT: 'bytearray', index: 'list[int]', header: 'bytes', datagram: 'bytearray', timestamp: 'float', conflict: 'list[tuple[int, int]]', extent: 'list[tuple[int, int, bool]]') -> 'None': ...  # pylint: disable=unused-argument,super-init-not-called,multiple-statements,line-too-long,redefined-builtin

pcapkit/foundation/reassembly/ip.py

@@ -14,22 +14,29 @@
 uses the elder one.
 
 """
+import bisect
+import sys
 from typing import TYPE_CHECKING, Generic
 
 from pcapkit.foundation.reassembly.data.data import Completion
 from pcapkit.foundation.reassembly.data.ip import (_AT, Buffer, BufferID, Datagram, DatagramID,
                                                    Deferred, Packet)
 from pcapkit.foundation.reassembly.reassembly import ReassemblyBase
 from pcapkit.protocols.schema.internet.ipv6 import IP_MAX_LENGTH
+from pcapkit.utilities.logging import get_logger
 
 if TYPE_CHECKING:
-    from typing import Type
+    from typing import Optional, Type
 
     from pcapkit.const.reg.transtype import TransType
     from pcapkit.protocols.internet.ip import IP as IP_Protocol
 
 __all__ = ['IP']
 
+#: logging.Logger: Module-level logger, a child of the package-wide
+#: :data:`pcapkit.utilities.logging.logger`.
+logger = get_logger(__name__)
+
 
 class IP(ReassemblyBase[Packet[_AT], Datagram[_AT], BufferID, Buffer[_AT]], Generic[_AT]):  # pylint: disable=abstract-method
     """Reassembly for IP payload.
@@ -71,6 +78,12 @@ class IP(ReassemblyBase[Packet[_AT], Datagram[_AT], BufferID, Buffer[_AT]], Gene
        IPv6) clear is reported even when its BUFID is already buffered,
        rather than dropped (:issue:`2270`).
 
+    .. versionchanged:: 1.6.0
+
+       A subclass setting :attr:`__discard_overlap__` abandons a datagram on an
+       overlapping fragment instead of overwriting, and
+       :class:`~pcapkit.foundation.reassembly.ipv6.IPv6` does (:issue:`2547`).
+
     """
     if TYPE_CHECKING:
         protocol: 'Type[IP_Protocol]'
@@ -82,6 +95,16 @@ class IP(ReassemblyBase[Packet[_AT], Datagram[_AT], BufferID, Buffer[_AT]], Gene
     #: buffer intact (:issue:`2270`).
     __isolate_atomic__: 'bool' = False
 
+    #: bool: Whether a fragment overlapping another of its datagram abandons
+    #: that datagram, reported as
+    #: :attr:`Completion.OVERLAP <pcapkit.foundation.reassembly.data.data.Completion.OVERLAP>`,
+    #: rather than overwriting the octets it overlaps. IPv4 overwrites them
+    #: (:rfc:`791#section-3.2`); IPv6 overrides this, since
+    #: :rfc:`8200#section-4.5` discards the datagram (:issue:`2547`).
+    #:
+    #: .. versionadded:: 1.6.0
+    __discard_overlap__: 'bool' = False
+
     ##########################################################################
     # Methods.
     ##########################################################################
@@ -183,7 +206,8 @@ def reassembly(self, info: 'Packet[_AT]') -> 'None':
         # would cover octets 0 to 65527 only, so a fragment reaching octet 65528,
         # e.g. at the largest Fragment Offset, would index past it (#1565)
         buf: 'Buffer[_AT]'
-        if isolated or BUFID not in self._buffer:
+        fresh = isolated or BUFID not in self._buffer
+        if fresh:
             buf = Buffer(
                 TDL=-1,                              # Total Data Length
                 RCVBT=bytearray(8192),              # Fragment Received Bit Table
@@ -192,14 +216,12 @@ def reassembly(self, info: 'Packet[_AT]') -> 'None':
                 datagram=bytearray(IP_MAX_LENGTH),  # data buffer
                 timestamp=TS,                       # first-arriving fragment's clock reading
                 conflict=[],                        # conflicting octet ranges
+                extent=[],                          # merged fragments' extents
             )
             if not isolated:
                 self._buffer[BUFID] = buf
         else:
             buf = self._buffer[BUFID]
-            # put header into header buffer
-            if not FO:
-                buf.__update__(header=header)
 
         # append packet index
         buf.index.append(info.num)
@@ -254,6 +276,31 @@ def reassembly(self, info: 'Packet[_AT]') -> 'None':
         if conflicts:
             buf.conflict.extend(conflicts)
 
+        # :rfc:`8200#section-4.5` abandons a datagram, and discards every
+        # fragment received for it, once one fragment overlaps another; it lets
+        # an implementation drop an exact duplicate instead, which this does.
+        # The span is the length the fragment declares, not the octets it
+        # holds. A fresh buffer has no extent to overlap, so the ``pop`` below
+        # never reaches an isolated one.
+        if self.__discard_overlap__:
+            span = (FO, FO + max(length, 0), MF)
+            other = self._find_overlap(buf.extent, span[0], span[1])
+            if other is not None:
+                if other == span and buf.datagram[start:stop] == payload:
+                    return
+                logger.debug('%s: abandoning buffer %s on frame %d, which overlaps '
+                             'octets %d-%d', self.name, BUFID, info.num, other[0], other[1] - 1)
+                self._dtgram.extend(
+                    self.submit(self._buffer.pop(BUFID), bufid=BUFID, overlap=True)
+                )
+                return
+            if span[1] > span[0]:
+                bisect.insort(buf.extent, span)
+
+        # put header into header buffer, once the fragment is known to be kept
+        if not FO and not fresh:
+            buf.__update__(header=header)
+
         # :rfc:`791` is explicit that an overlapping fragment's data "will use
         # the more recently arrived copy in the data buffer" -- the opposite of
         # TCP's first-write-wins (:rfc:`9293#section-3.10`) -- so the arriving
@@ -410,8 +457,39 @@ def _detect_conflicts(rcvbt: 'bytearray', tdl: 'int', datagram: 'bytearray', pay
             index = run_stop
         return conflicts
 
+    @staticmethod
+    def _find_overlap(extent: 'list[tuple[int, int, bool]]', start: 'int',
+                      stop: 'int') -> 'Optional[tuple[int, int, bool]]':
+        """Find a merged fragment that an arriving one overlaps.
+
+        Arguments:
+            extent: this buffer's
+                :attr:`~pcapkit.foundation.reassembly.data.ip.Buffer.extent`,
+                sorted and pairwise disjoint.
+            start: the arriving fragment's first octet, i.e. its ``FO``.
+            stop: one past the arriving fragment's last octet.
+
+        Returns:
+            The extent of a merged fragment sharing an octet with the arriving
+            one, or :data:`None` if none does.
+
+        Since the extents are disjoint, only the last one starting at or before
+        ``start`` and the first one starting after it can reach the arriving
+        fragment.
+
+        """
+        if stop <= start:
+            return None
+        after = bisect.bisect_right(extent, (start, sys.maxsize, True))
+        if after and extent[after - 1][1] > start:
+            return extent[after - 1]
+        if after < len(extent) and extent[after][0] < stop:
+            return extent[after]
+        return None
+
     def submit(self, buf: 'Buffer[_AT]', *, bufid: 'tuple[_AT, _AT, int, TransType]',  # type: ignore[override] # pylint: disable=arguments-differ
-               checked: 'bool' = False, timeout: 'bool' = False) -> 'list[Datagram[_AT]]':
+               checked: 'bool' = False, timeout: 'bool' = False,
+               overlap: 'bool' = False) -> 'list[Datagram[_AT]]':
         """Submit reassembled payload.
 
         Arguments:
@@ -425,6 +503,11 @@ def submit(self, buf: 'Buffer[_AT]', *, bufid: 'tuple[_AT, _AT, int, TransType]'
                 :attr:`Completion.TIMEOUT <pcapkit.foundation.reassembly.data.data.Completion.TIMEOUT>`
                 from
                 :attr:`Completion.PARTIAL <pcapkit.foundation.reassembly.data.data.Completion.PARTIAL>`
+            overlap: whether this buffer is being submitted because an
+                overlapping fragment abandoned it (:attr:`__discard_overlap__`),
+                reported as :attr:`Completion.OVERLAP
+                <pcapkit.foundation.reassembly.data.data.Completion.OVERLAP>`
+                and never as complete
 
         Returns:
             Reassembled packets.
@@ -437,13 +520,13 @@ def submit(self, buf: 'Buffer[_AT]', *, bufid: 'tuple[_AT, _AT, int, TransType]'
         datagram = buf.datagram
         conflict = tuple(buf.conflict)
 
-        flag = checked or self._is_complete(buf)
+        flag = not overlap and (checked or self._is_complete(buf))
         ret = []  # type: list[Datagram[_AT]]
 
         # How completely this datagram came out, and why it stopped. Derived once,
         # so the two branches below cannot disagree about it.
         completion = Completion.COMPLETE if flag else (
-            Completion.TIMEOUT if timeout else Completion.PARTIAL
+            Completion.OVERLAP if overlap else Completion.TIMEOUT if timeout else Completion.PARTIAL
         )
 
         # if datagram is incomplete

pcapkit/foundation/reassembly/ipv6.py

@@ -96,6 +96,15 @@ class IPv6(IP):
        isolation, without flushing a buffer that shares its BUFID
        (:issue:`2270`).
 
+    .. versionchanged:: 1.6.0
+
+       A fragment overlapping another of its datagram, other than an exact
+       duplicate, abandons that datagram, which is reported as
+       :attr:`Completion.OVERLAP
+       <pcapkit.foundation.reassembly.data.data.Completion.OVERLAP>`, per
+       :rfc:`8200#section-4.5`; 1.5.x overwrote the octets it overlapped, as
+       :rfc:`791` has IPv4 do (:issue:`2547`).
+
     """
 
     ##########################################################################
@@ -133,6 +142,18 @@ class IPv6(IP):
     #: discarded (:issue:`2270`).
     __isolate_atomic__ = True
 
+    #: bool: A fragment that overlaps another of its datagram abandons that
+    #: datagram: :rfc:`8200#section-4.5` says "reassembly of that packet must be
+    #: abandoned and all the fragments that have been received for that packet
+    #: must be discarded". It also lets an implementation "drop exact duplicate
+    #: fragments while keeping the other fragments belonging to the same
+    #: packet", which this one does: a fragment with the same offset, length,
+    #: **M** flag and data as one already merged changes nothing but
+    #: :attr:`Datagram.index <pcapkit.foundation.reassembly.data.ip.Datagram.index>`.
+    #: A later fragment of an abandoned datagram opens a new buffer, as one
+    #: does after a datagram completes (:issue:`2547`).
+    __discard_overlap__ = True
+
     ##########################################################################
     # Methods.
     ##########################################################################