Skip to content

fix(toolkit): reassemble the first fragmented IP layer, so tunnelled fragments reassemble (#2307) - #2338

Merged
JarryShaw merged 1 commit into
mainfrom
fix/2307-ip-reassembly-tunnel
Oct 27, 2026
Merged

JarryShaw merged 1 commit into
mainfrom
fix/2307-ip-reassembly-tunnel

Conversation

@JarryShaw

Copy link
Copy Markdown
Owner

What is the purpose of your pull request?

  • fix — bug fix

Description

Closes #2307, per the ruling (option 2: reassemble the first fragmented IP layer).

  • Root cause: ipv4_reassembly and ipv6_reassembly read frame['IPv4']/frame['IPv6'], which is always the outermost layer. In an IP-in-IP tunnel, the inner fragments were keyed on the tunnel's header. They were dropped when the tunnel had DF set, or when its IPv6 header had no Fragment header.
  • Fix: a _fragment helper picks the first layer of the right IP version, outside in, that is a fragment. If no layer is a fragment, it falls back to the outermost layer, so untunnelled traffic is unchanged.
    • The default engine (pcap, pcapng), Scapy and DPKT all use it.
    • Scapy matches the exact type, so a datagram quoted inside an ICMP error is never picked.
  • Tests: test_tunnelled_ip_reassembly_2307_unit.py covers inner datagrams fragmented inside 4in4 (DF clear and set) and 6in6, plus a fragmented tunnel carrying a whole datagram, on every engine, byte for byte. One existing Scapy test expected None for a fragmented 6in6 inner packet, and now expects it reassembled.
  • Fail-without-fix: on origin/main, 13 failed, e.g. ValueError: too many values to unpack (expected 1, got 2). On the head, 22 passed.

@JarryShaw JarryShaw added this to the PyPCAPKit v1.5 milestone Oct 26, 2026
@JarryShaw JarryShaw added bug Issues reporting a defect (set by the bug report template; a default, not an assessment) fix Pull requests that fix a defect (fix: subject prefix) review: running A cross-review is in flight against the current head - no verdict yet labels Oct 26, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

Review: NEEDS CHANGES at 691d2bc64 (Sonnet).

  • Regression (DPKT): pcapkit.toolkit.dpkt._fragment walks layer.data through any dpkt packet, so it descends from ICMP into a datagram quoted inside an ICMP error. A frame of Ether/IPv4(id=5)/ICMP-unreachable quoting an IPv4 datagram with id=77 and MF set gives _fragment(frame, 4).id == 77 (I re-measured it on the head). main read the outer layer, so this is new. The IPv6 case behaves the same.
  • Fix: descend only through tunnel layers, i.e. past IPv4 or IPv6 only when the next protocol is 4 or 41. Add a DPKT ICMP-quote test.
  • Confirmed:
    • The default engine and Scapy pick the outer id=5 in that frame.
    • Untunnelled output is identical to main across every sample capture (9298 calls).
    • 4in4 and 6in6 inner fragments reassemble on every engine.
    • The changed Scapy expectation follows the option-2 ruling.
    • Fail-without-fix: 12 failed on origin/main with the new test file alone.

@JarryShaw JarryShaw added review: needs-changes Cross-review at the current head says changes are required; see the verdict comment and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 26, 2026
@JarryShaw JarryShaw moved this to In review in PyPCAPKit Oct 26, 2026
@JarryShaw
JarryShaw force-pushed the fix/2307-ip-reassembly-tunnel branch from 691d2bc to 42903c7 Compare October 26, 2026 23:07
@JarryShaw JarryShaw added review: running A cross-review is in flight against the current head - no verdict yet and removed review: needs-changes Cross-review at the current head says changes are required; see the verdict comment labels Oct 26, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

Review: GOOD TO GO at 42903c784 (Sonnet, round 2). The NEEDS CHANGES finding is fixed.

  • DPKT: _fragment walks only through IP layers whose next protocol is IPv4 or IPv6. The ICMP-quote repro now picks the outer datagram for IPv4 and IPv6, which I also re-measured.
  • New test: test_a_datagram_an_icmp_error_quotes_is_not_walked_to fails on 691d2bc64 (2 failed, both DPKT) and passes on the head (187 passed).
  • Unchanged elsewhere: untunnelled output still matches main on every sample capture (9298 calls), and the range-diff shows nothing else changed in the rebase.
  • Non-blocking: hand checks of 4in6, 6in4 and 6in6 with a destination-options header all descend correctly, but the test file has no cases for them.

@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 26, 2026
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Coverage: 91.58% (unit tier, Python 3.15, d2841387c, Unit Tests run success)

Package Statements Missed Branches Partial Cover
pcapkit (top level) 104 4 20 2 95.16%
pcapkit/const 19599 881 2618 867 92.11%
pcapkit/corekit 2596 74 818 37 96.51%
pcapkit/dumpkit 282 3 102 3 98.44%
pcapkit/foundation 3029 23 1050 25 98.82%
pcapkit/interface 138 4 52 3 96.32%
pcapkit/protocols 17442 218 4586 185 98.15%
pcapkit/toolkit 914 36 334 10 95.51%
pcapkit/utilities 502 7 144 5 98.14%
pcapkit/vendor 4432 1988 1018 167 51.87%

Per-file detail: the coverage-html artifact of this run.

…fragments reassemble (#2307)

Problem: the IPv4 and IPv6 reassembly adapters read the frame's outermost
IPv4 or IPv6 layer, so in an IP-in-IP tunnel a fragmented inner datagram was
reassembled whole under the tunnel's header, or dropped (tunnel DF set, or no
Fragment header in the tunnel's IPv6), and never reassembled itself.

Fix: take the first layer of that IP version, from the outside in, that is a
fragment (IPv4 MF or a non-zero offset; an IPv6 Fragment header), else the
outermost as before. Applied to the default engine's PCAP and PCAP-NG
adapters, and to scapy and dpkt, which picked the outermost too.

Test: tests/toolkit/test_tunnelled_ip_reassembly_2307_unit.py reassembles a
fragmented inner datagram (4in4 with DF clear and set, 6in6) and a fragmented
tunnel carrying a whole one, per engine; the inner cases fail on main. The
scapy 6in6 pin now expects the inner packet reassembled on its own header.
@JarryShaw
JarryShaw force-pushed the fix/2307-ip-reassembly-tunnel branch from 42903c7 to d284138 Compare October 27, 2026 00:14
@JarryShaw

Copy link
Copy Markdown
Owner Author

Amended 42903c784 → d2841387c: folded in the #2342 test fix (NOT_A_COPY entries excusing _fragment's version == 6 checks from the #1739 one-home sweep, which is why Python 3.15 and Required checks passed failed). The entries only match once this PR's dpkt.py is in the tree, so #2342 couldn't land standalone — it belongs here. pcapkit/toolkit/dpkt.py and the toolkit tests are byte-identical to the already-reviewed head (patch-id checked); only the one-home test file changed. test_rfc_constants_one_home_1739_unit.py, the toolkit tests, docstring contract and tier guard give 202 passed. Rebased onto current main.

@JarryShaw

Copy link
Copy Markdown
Owner Author

Ready to merge at d2841387c. Closes #2307.

  • Review: GOOD TO GO (Sonnet), on two rounds — the first caught a DPKT regression (reassembling a datagram quoted inside an ICMP error), fixed and re-verified.
  • CI: Required checks passed SUCCESS, 0 failures, KTLO clean. Only the non-blocking GraalPy experimental leg is still running.
  • Ordering legs: toolkit (440 tests) and project (2408 tests), both 0 failures, run with the PR merged into current main.
  • Merge state: mergeable, no conflicts.

@JarryShaw
JarryShaw merged commit 7b57bb0 into main Oct 27, 2026
61 checks passed
@JarryShaw
JarryShaw deleted the fix/2307-ip-reassembly-tunnel branch October 27, 2026 01:38
@github-project-automation github-project-automation Bot moved this from In review to Done in PyPCAPKit Oct 27, 2026
@JarryShaw JarryShaw removed the review: good-to-go Cross-review at the current head says ready; CI state is separate label Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Issues reporting a defect (set by the bug report template; a default, not an assessment) fix Pull requests that fix a defect (fix: subject prefix)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

fix(toolkit): IP reassembly keys on the outermost IP layer, losing tunnelled fragments

1 participant