feat(foundation): reassemble fragmented DTLS handshake messages (#2440)

- add foundation.reassembly.dtls.DTLS, keyed by UDP 4-tuple and DTLS epoch
- DTLS.read decodes the fragment completing a handshake message into
  ClientHello/ServerHello
- overlapping fragments are accepted (RFC 6347 4.2.3)
- opt-in only; Extractor-level wiring left as a follow-up
- isort-clean reassembly and DTLS modules
- DTLS.read reassembles only the datagram's first record (Known Limitations)

Tests: tests/foundation/reassembly/ -- 197 passed, 897 subtests.

Author: JarryShaw

docs/source/changelog/1.5.0.rst

@@ -958,7 +958,7 @@ Changed
   :rfc:`791` specifies the opposite resolution, and records the disagreement
   instead (:pr:`482`).
 * renames with no compatibility alias left behind: ``HoleDiscriptor`` is spelled
-  :class:`.HoleDescriptor` and its package alias ``TCP_HoleDiscriptor`` is
+  :class:`~pcapkit.foundation.reassembly.data.tcp.HoleDescriptor` and its package alias ``TCP_HoleDiscriptor`` is
   ``TCP_HoleDescriptor`` (:pr:`350`).
 * subclass registration is **opt-in** for :class:`.Engine`, :class:`.Reassembly`,
   :class:`.TraceFlow` and ``dumpkit``'s :class:`.Dumper` (:issue:`514`). Each registers

docs/source/contributing/pep.rst

@@ -246,7 +246,7 @@ without keys.
 **Not done: reassembling a handshake message DTLS has fragmented across
 several records** (:rfc:`6347#section-4.2.3`, :issue:`2440`). That needs a
 :class:`~pcapkit.foundation.reassembly.reassembly.Reassembly` subclass keyed
-on the UDP 5-tuple plus epoch. A fragmented message's body is kept as that one
+on the UDP 4-tuple plus epoch. A fragmented message's body is kept as that one
 fragment's raw octets -- see
 :attr:`DTLSHandshake.is_fragmented <pcapkit.protocols.data.application.dtls.DTLSHandshake.is_fragmented>`
 -- rather than decoded from a slice of the message. An unfragmented message is

docs/source/pcapkit/foundation/reassembly/dtls.rst

@@ -0,0 +1,122 @@
+=================================
+DTLS Handshake-Message Reassembly
+=================================
+
+.. module:: pcapkit.foundation.reassembly.dtls
+
+:mod:`pcapkit.foundation.reassembly.dtls` contains
+:class:`~pcapkit.foundation.reassembly.dtls.DTLS` only, which reconstructs a
+DTLS handshake message split across several records back to origin
+(:rfc:`6347#section-4.2.3`).
+
+.. autoclass:: pcapkit.foundation.reassembly.dtls.DTLS
+   :no-members:
+   :show-inheritance:
+
+   .. autoproperty:: last
+
+   .. automethod:: reassembly
+   .. automethod:: submit
+   .. automethod:: _datagram_from_fragment
+   .. automethod:: _received_runs
+   .. automethod:: _update_hole_descriptors
+
+   .. autoattribute:: __protocol_name__
+   .. autoattribute:: __timeout__
+
+Algorithm
+=========
+
+.. seealso::
+
+   Holes are inclusive byte offsets into the message. They never wrap, and
+   the upper bound is known from the start: every fragment carries the
+   message's total ``length`` (:rfc:`6347#section-4.2.3`).
+
+A DTLS handshake record may carry more than one message fragment --
+:rfc:`6347#section-4.2.3` allows several handshake messages (or fragments of
+them) to be coalesced into one record, and :meth:`SSL._read_tls_handshake_messages
+<pcapkit.protocols.application.ssl.SSL._read_tls_handshake_messages>`
+reads however many fit rather than assuming exactly one. A fragment names the
+message it belongs to (``message_seq``), the message's total length
+(``length``), and its own byte range within that message (``fragment_offset``,
+``fragment_length``). A message is reassembled by punching the arriving
+fragment's range out of the message's hole list; it is whole once no hole
+remains.
+
+Terminology
+===========
+
+.. glossary::
+
+   reasm.dtls.packet
+       Data structure for **DTLS handshake-fragment reassembly**
+       (:meth:`DTLS.reassembly <pcapkit.foundation.reassembly.dtls.DTLS.reassembly>`)
+       is :class:`~pcapkit.foundation.reassembly.data.dtls.Packet` -- one
+       fragment's own ``bufid``, ``message_seq``, ``fragment_offset``,
+       ``fragment_length`` and raw ``payload``, built by
+       :meth:`DTLS.read <pcapkit.protocols.application.dtls.DTLS.read>` from
+       a single :class:`~pcapkit.protocols.data.application.dtls.DTLSHandshake`.
+
+   reasm.dtls.datagram
+       Data structure for **reassembled DTLS handshake message** (element of
+       :attr:`DTLS.datagram <pcapkit.foundation.reassembly.reassembly.Reassembly.datagram>`)
+       is :class:`~pcapkit.foundation.reassembly.data.dtls.Datagram` --
+       ``completed``, the original packet identifier, the packet numbers
+       that built it, and ``body``, the reassembled payload (a contiguous
+       :obj:`bytes`, or -- under ``strict=True`` and an incomplete message --
+       a :obj:`tuple` of the octet runs actually received).
+
+   reasm.dtls.buffer
+       Data structure for internal buffering
+       (:attr:`DTLS._buffer <pcapkit.foundation.reassembly.reassembly.Reassembly._buffer>`)
+       is :class:`~pcapkit.foundation.reassembly.data.dtls.Buffer` -- every
+       message under reconstruction for one
+       :data:`~pcapkit.foundation.reassembly.data.dtls.BufferID` (one UDP
+       association at one DTLS epoch), each keyed by its
+       :attr:`~pcapkit.foundation.reassembly.data.dtls.Packet.message_seq` and
+       held as a :class:`~pcapkit.foundation.reassembly.data.dtls.Fragment` --
+       its hole descriptor list
+       (:class:`~pcapkit.foundation.reassembly.data.dtls.HoleDescriptor`) and
+       its zero-filled payload buffer.
+
+Data Models
+===========
+
+.. module:: pcapkit.foundation.reassembly.data.dtls
+.. currentmodule:: pcapkit.foundation.reassembly.dtls
+
+.. autoclass:: pcapkit.foundation.reassembly.data.dtls.Packet
+   :members:
+   :show-inheritance:
+
+.. autodata:: pcapkit.foundation.reassembly.data.dtls.BufferID
+
+.. autoclass:: pcapkit.foundation.reassembly.data.dtls.DatagramID
+   :members:
+   :show-inheritance:
+
+.. autoclass:: pcapkit.foundation.reassembly.data.dtls.Datagram
+   :members:
+   :show-inheritance:
+
+.. autoclass:: pcapkit.foundation.reassembly.data.dtls.HoleDescriptor
+   :members:
+   :show-inheritance:
+
+.. autoclass:: pcapkit.foundation.reassembly.data.dtls.Fragment
+   :members:
+   :show-inheritance:
+
+.. autoclass:: pcapkit.foundation.reassembly.data.dtls.Buffer
+   :members:
+   :show-inheritance:
+
+Type Variables
+==============
+
+.. data:: _AT
+   :module: pcapkit.foundation.reassembly.data.dtls
+   :type: ipaddress.IPv4Address | ipaddress.IPv6Address
+
+   IP address type.

docs/source/pcapkit/foundation/reassembly/index.rst

@@ -5,8 +5,8 @@ Packets Reassembly
 .. module:: pcapkit.foundation.reassembly
 
 :mod:`pcapkit.foundation.reassembly` implements datagram reassembly
-of IP and TCP packets, based on the algorithms described in :rfc:`791`
-and :rfc:`815`.
+of IP packets, TCP segments and DTLS handshake messages, based on the
+algorithms described in :rfc:`791`, :rfc:`815` and :rfc:`6347`.
 
 .. seealso::
 
@@ -19,6 +19,7 @@ and :rfc:`815`.
    reassembly
    ip/index
    tcp
+   dtls
 
 All reassembly classes are implemented as :class:`~pcapkit.foundation.reassembly.reassembly.ReassemblyBase`
 subclasses, which are responsible for processing extracted packets and
@@ -30,7 +31,7 @@ diagram of the class hierarchy of :mod:`pcapkit.foundation.reassembly`:
    flowchart LR
        A{{ReassemblyMeta}} -.->|metaclass| B(ReassemblyBase)
 
-       B --> IP & TCP
+       B --> IP & TCP & DTLS
        IP --> IPv4 & IPv6
 
        B --> C(Reassembly)
@@ -45,6 +46,7 @@ diagram of the class hierarchy of :mod:`pcapkit.foundation.reassembly`:
        click IPv4 "ip/ipv4.html#pcapkit.foundation.reassembly.ipv4.IPv4"
        click IPv6 "ip/ipv6.html#pcapkit.foundation.reassembly.ipv6.IPv6"
        click TCP "tcp.html#pcapkit.foundation.reassembly.tcp.TCP"
+       click DTLS "dtls.html#pcapkit.foundation.reassembly.dtls.DTLS"
 
 Auxiliary Data
 ==============

pcapkit/all.py

@@ -82,6 +82,7 @@
     'ReassemblyManager',                                    # Reassembly Manager
     'IPv4_Reassembly', 'IPv6_Reassembly',                   # IP Reassembly
     'TCP_Reassembly',                                       # TCP Reassembly
+    'DTLS_Reassembly',                                      # DTLS Reassembly
 
     # pcapkit.foundation.registry
     'register_protocol',

pcapkit/foundation/__init__.py

@@ -20,7 +20,7 @@
 __all__ = [
     'Extractor',
 
-    'IPv4_Reassembly', 'IPv6_Reassembly', 'TCP_Reassembly',
+    'IPv4_Reassembly', 'IPv6_Reassembly', 'TCP_Reassembly', 'DTLS_Reassembly',
     'ReassemblyManager',
 
     'TCP_TraceFlow',

pcapkit/foundation/reassembly/__init__.py

@@ -6,24 +6,25 @@
 .. module:: pcapkit.foundation.reassembly
 
 :mod:`pcapkit.foundation.reassembly`, based on algorithms described
-in :rfc:`791` and :rfc:`815`, implements datagram reassembly
-of IP and TCP packets.
+in :rfc:`791`, :rfc:`815` and :rfc:`6347`, implements datagram
+reassembly of IP packets, TCP segments and DTLS handshake messages.
 
 """
-# Base Class for Reassembly
-from pcapkit.foundation.reassembly.reassembly import Reassembly
+# Reassembly for DTLS handshake messages
+from pcapkit.foundation.reassembly.dtls import DTLS as DTLS_Reassembly
 from pcapkit.foundation.reassembly.ip import IP as IP_Reassembly
-
 # Reassembly for IP
 from pcapkit.foundation.reassembly.ipv4 import IPv4 as IPv4_Reassembly
 from pcapkit.foundation.reassembly.ipv6 import IPv6 as IPv6_Reassembly
-
+# Base Class for Reassembly
+from pcapkit.foundation.reassembly.reassembly import Reassembly
 # Reassembly for TCP
 from pcapkit.foundation.reassembly.tcp import TCP as TCP_Reassembly
 
 __all__ = [
     'IPv4_Reassembly', 'IPv6_Reassembly',   # IP Reassembly
     'TCP_Reassembly',                       # TCP Reassembly
+    'DTLS_Reassembly',                      # DTLS Reassembly
     'ReassemblyManager',                    # Reassembly Manager, defined below
 ]
 

pcapkit/foundation/reassembly/data/__init__.py

@@ -20,6 +20,15 @@
 from pcapkit.foundation.reassembly.data.tcp import HoleDescriptor as TCP_HoleDescriptor
 from pcapkit.foundation.reassembly.data.tcp import Packet as TCP_Packet
 
+# DTLS handshake-message reassembly
+from pcapkit.foundation.reassembly.data.dtls import Buffer as DTLS_Buffer
+from pcapkit.foundation.reassembly.data.dtls import BufferID as DTLS_BufferID
+from pcapkit.foundation.reassembly.data.dtls import Datagram as DTLS_Datagram
+from pcapkit.foundation.reassembly.data.dtls import DatagramID as DTLS_DatagramID
+from pcapkit.foundation.reassembly.data.dtls import Fragment as DTLS_Fragment
+from pcapkit.foundation.reassembly.data.dtls import HoleDescriptor as DTLS_HoleDescriptor
+from pcapkit.foundation.reassembly.data.dtls import Packet as DTLS_Packet
+
 __all__ = [
     'ReassemblyData', 'Completion', 'Deferred',
 
@@ -28,4 +37,7 @@
 
     'TCP_Packet', 'TCP_DatagramID', 'TCP_Datagram', 'TCP_Buffer',
     'TCP_Fragment', 'TCP_HoleDescriptor', 'TCP_BufferID',
+
+    'DTLS_Packet', 'DTLS_DatagramID', 'DTLS_Datagram', 'DTLS_Buffer',
+    'DTLS_Fragment', 'DTLS_HoleDescriptor', 'DTLS_BufferID',
 ]