Skip to content

fix(toolkit): scapy packet2dict dumps nested packets as {} and mutates its input - #1287

Merged
JarryShaw merged 1 commit into
mainfrom
fix/scapy-packet2dict
Oct 7, 2026
Merged

JarryShaw merged 1 commit into
mainfrom
fix/scapy-packet2dict

Conversation

@JarryShaw

@JarryShaw JarryShaw commented Oct 7, 2026 •

Copy link
Copy Markdown
Owner
  • Searched for similar pull requests
  • Followed the coding style (make pylint, make mypy, make isort) — ran pylint/mypy/isort on toolkit/scapy.py: nothing new on the changed lines; the file's existing mypy/isort findings are unchanged
  • make test passes, and a test case covers the change — ran the modules listed below, not the full suite
  • Added a changelog entry — N/A, added centrally

What is the purpose of your pull request?

  • fix — corrects a defect

Description of your pull request and other information

packet2dict now converts field values that are scapy Packets, including inside lists and tuples (IP/IPv6/MH options, DNS qd/an/ns, SCTP params, TCP-AO values). Each layer's dict is built from a copy of packet.fields, so the input packet is not mutated.

Defaults: none are added. A layer carries exactly the fields set on it, so a dissected layer carries only what was read from the wire. A hand-built nested packet with no fields set still dumps as {}, for example IPOption_NOP().

  • Before: in repro D5, options-ipv4.pcap dumps with 14 {}. In D6, p.fields goes from [] to ['IP'], and then p.copy() raises KeyError: 'IP'.
  • After: options-ipv4.pcap dumps with 0 {}, and in D6 p.fields stays [].
  • Scapy-engine JSON of all of examples/captures/*: 288 empty {} before, 1 after. The one left is dhcp.pcapng Section Header 1/options, which is a genuinely empty pcapng option list, not a packet object.
  • Over all 1614 fixture packets, every layer and every nested packet has exactly the keys in fields plus the payload name: 0 mismatches.
  • New tests/toolkit/test_scapy_packet2dict_unit.py, 7 tests:
    • With this fix: 7 passed, 10 subtests passed.
    • Against main: 6 failed, 3 passed.
    • Against a variant that fills in defaults: the two ICMP-echo key tests fail, one fixture-free and one reading little_endian.pcap.
  • Also run:
    • test_scapy_unit: 6 passed.
    • engines/test_scapy_engine: 2 passed.
    • tests/project: 389 passed, 1 skipped.
    • util/run_unittest_leg.py toolkit: 275 tests, 0 failures.

Closes #1259
Closes #1260

@JarryShaw JarryShaw added fix Pull requests that fix a defect (fix: subject prefix) review: running A cross-review is in flight against the current head - no verdict yet labels Oct 7, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

Cross-review verdict on 6a9808546: NEEDS CHANGES (reviewed on Sonnet; the author ran on Opus).

  • Must fix: building each layer from default_fields + fields adds values to dissected packets that were never on the wire. Diffing all 1605 fixture packets against the old logic:
    • ICMP echo replies gain ts_ori, ts_rx, ts_tx, gw, ptr, addr_mask and similar fields (3 packets).
    • DNS gains length (17 packets), and NTPHeader gains ref_id and id.
    • The ts_* defaults are time-derived, so dumping the same capture now gives different output from run to run. test_unset_nested_fields_fall_back_to_defaults locks this in.
  • Suggested fix: use packet.fields for dissected layers, and fall back to defaults only for hand-built nested packets. Add a test that a dissected packet gains no keys.
  • Confirmed: 280 packet objects are now expanded, as intended. The input packet is no longer mutated at any depth. A 400-layer packet and PacketListField both work. The new tests fail without the fix. The toolkit leg passes: 273 tests, 0 failures.

@JarryShaw JarryShaw added review: needs-changes Cross-review at the current head says changes are required; see the verdict comment and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 7, 2026
@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Coverage: 88.88% (unit tier, Python 3.14, c308af6b7, Unit Tests run failure)

Package Statements Missed Branches Partial Cover
pcapkit (top level) 104 4 20 4 93.55%
pcapkit/const 18757 1034 2342 845 90.24%
pcapkit/corekit 1878 91 580 22 94.34%
pcapkit/dumpkit 142 0 42 0 100.00%
pcapkit/foundation 2442 102 852 39 94.38%
pcapkit/interface 112 7 40 5 92.11%
pcapkit/protocols 15777 190 4036 165 98.18%
pcapkit/toolkit 495 73 150 1 83.57%
pcapkit/utilities 429 4 122 4 98.55%
pcapkit/vendor 4409 2359 1006 158 42.84%

Per-file detail: the coverage-html artifact of this run.

…s its input

packet2dict now converts field values that are scapy packets, including
those inside lists and tuples (IP/IPv6/MH options, DNS records, SCTP
params, TCP-AO values), and builds each layer's dict from a copy of
packet.fields, so the given packet is left unchanged and no default
values are added to dissected layers.

Closes #1259
Closes #1260
@JarryShaw
JarryShaw force-pushed the fix/scapy-packet2dict branch from 6a98085 to c308af6 Compare October 7, 2026 18:25
@JarryShaw JarryShaw added review: running A cross-review is in flight against the current head - no verdict yet and removed review: needs-changes Cross-review at the current head says changes are required; see the verdict comment labels Oct 7, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

Cross-review verdict on c308af6b7: GOOD TO GO (Sonnet reviewed, round 2; the author ran on Opus).

  • Fixed: no layer gains a key it only has by default. A dissected ICMP echo now carries only chksum, code, id, seq, type, unused, and IP options expand to dicts. I re-ran both checks myself.
  • Fixtures: I compared all 1579 frames against main. The only differences are 280 packet objects that are now expanded, adding 891 keys taken from those packets' own fields. No keys were lost and no values changed. Two runs produce byte-identical JSON.
  • No mutation: no frame's fields, payload or bytes change after two calls.
  • Tests: the new tests fail on both main and the defaults version. The fixture test skips cleanly when its capture is missing. The toolkit leg passed 275 tests with 0 failures.
  • Nit: the mutation test checks only each layer's own fields, not nested option packets.

@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 7, 2026
@JarryShaw
JarryShaw merged commit 3cffc93 into main Oct 7, 2026
38 of 40 checks passed
@JarryShaw
JarryShaw deleted the fix/scapy-packet2dict branch October 7, 2026 19:09
@JarryShaw JarryShaw removed the review: good-to-go Cross-review at the current head says ready; CI state is separate label Oct 7, 2026
@JarryShaw JarryShaw added this to the PyPCAPKit v1.5 milestone Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix Pull requests that fix a defect (fix: subject prefix)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

fix(toolkit): packet2dict mutates the scapy packet it is given fix(toolkit): scapy engine dumps nested packet objects (IP options, DNS records) as {}

1 participant