Skip to content

Commit 2b075ca

Browse files
committed
fix(protocols): a next-layer hit resolves its descriptor instead of memoising the class (#1559)
ProtocolBase._lookup_next_layer wrote the class a registered ModuleDescriptor resolved to back over the descriptor. After importlib.reload of the module, dispatch kept reaching the pre-reload class; parsing mutated registries shared by the whole process; and whether re-registering a shipped code warned depended on what had been parsed. - Every lookup now resolves through ModuleDescriptor.klass, which reads sys.modules, and nothing is written back. The docstring, which already said a memo would go stale, now matches the code. - The ethernet schema callback and ModuleDescriptor.klass docstrings no longer describe the write-back; the resolution count is re-measured. - Six assertions that pinned the write-back now check the entry stays a ModuleDescriptor; a new test covers reload, registry immutability and the register warning. Cost: ~150 ns per hit, ~0.1% of extract() on http.pcap; not measurable in wall clock. Narrow test selections and the protocols unittest leg pass.
1 parent cf6e78a commit 2b075ca

8 files changed

Lines changed: 189 additions & 41 deletions

File tree

‎pcapkit/corekit/module.py‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -59,14 +59,14 @@ def klass(self) -> 'Type[_T]':
5959
The module is read from :data:`sys.modules` first, and
6060
:func:`importlib.import_module` is entered only when it is not
6161
loaded yet. That matters because this property is on a *per-frame*
62-
dispatch path: a next layer code nobody registered falls back to a
63-
:class:`ModuleDescriptor` for
64-
:class:`~pcapkit.protocols.misc.raw.Raw` which
65-
:meth:`ProtocolBase._lookup_next_layer
62+
dispatch path: :meth:`ProtocolBase._lookup_next_layer
6663
<pcapkit.protocols.protocol.ProtocolBase._lookup_next_layer>`
67-
deliberately does not write back, so every unrecognised frame
68-
resolves the same descriptor again -- 48 of the 52 resolutions an
69-
extraction of :file:`many_interfaces.pcapng` performs.
64+
deliberately writes no resolved class back, so every next layer
65+
lookup resolves its descriptor again -- the one registered for the
66+
code, or the fallback for :class:`~pcapkit.protocols.misc.raw.Raw`
67+
when nobody registered it. An extraction of
68+
:file:`many_interfaces.pcapng` performs 296 resolutions, 48 of them
69+
fallbacks (:issue:`1559`).
7070
:func:`~importlib.import_module` keeps real per-call work for an
7171
already-imported module (locks, :class:`~importlib.machinery.ModuleSpec`
7272
checks, the ``fromlist`` walk), so each repeat costs ~436 ns

‎pcapkit/protocols/protocol.py‎

Lines changed: 14 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -2154,35 +2154,29 @@ def _lookup_next_layer(registry: 'DefaultDict[int, ModuleDescriptor[ProtocolBase
21542154
The lookup itself is :meth:`self._lookup_registry
21552155
<ProtocolBase._lookup_registry>`, so a miss does not grow the shared
21562156
registry. What this adds is the next-layer-specific resolution step:
2157-
a registered code may hold a
2157+
the entry found -- registered, or the fallback -- may be a
21582158
:class:`~pcapkit.corekit.module.ModuleDescriptor` rather than a
2159-
class, and importing it is written back so the import happens once.
2159+
class, and it is resolved to the class it names.
21602160
2161-
That write-back is deliberately confined to a *hit*. Memoising the
2162-
fallback's resolution under ``proto`` would be exactly the insertion
2161+
Nothing is written back, on a hit or a miss. A miss has no key to
2162+
write under, and recording one would be exactly the insertion
21632163
:meth:`self._lookup_registry <ProtocolBase._lookup_registry>` exists
2164-
to avoid.
2165-
2166-
So a miss resolves its fallback descriptor again on every frame, and
2167-
what keeps that affordable is :attr:`ModuleDescriptor.klass
2164+
to avoid. A hit memoised under ``proto`` would retain a class that
2165+
:func:`importlib.reload` then makes stale, and would make parsing
2166+
mutate a registry shared by the whole process (:issue:`1559`). The
2167+
same holds for a memo in ``registry``'s default factory or in a cache
2168+
beside the registry (:issue:`421`, :issue:`425`, :issue:`555`).
2169+
2170+
So every lookup resolves its descriptor again, and what keeps that
2171+
affordable is :attr:`ModuleDescriptor.klass
21682172
<pcapkit.corekit.module.ModuleDescriptor.klass>` reading
21692173
:data:`sys.modules` instead of re-entering
2170-
:func:`importlib.import_module` (:issue:`574`). Memoising the
2171-
resolved class instead, whether under ``proto``, in ``registry``'s
2172-
default factory, or in a cache beside the registry, would retain a
2173-
class that :func:`importlib.reload` then makes stale (:issue:`421`,
2174-
:issue:`425`, :issue:`555`).
2174+
:func:`importlib.import_module` (:issue:`574`).
21752175
21762176
"""
21772177
protocol = ProtocolBase._lookup_registry(registry, proto)
21782178
if isinstance(protocol, ModuleDescriptor):
2179-
klass = protocol.klass
2180-
# a descriptor can also come back from the default factory, and that
2181-
# one has no key to memoise under -- writing it back would recreate
2182-
# the insertion-on-miss this exists to avoid
2183-
if proto in registry:
2184-
registry[proto] = klass # update mapping upon import
2185-
return klass
2179+
return protocol.klass
21862180
return protocol
21872181

21882182
def _decode_next_layer(self, dict_: '_PT', proto: 'int', length: 'Optional[int]' = None, *,

‎pcapkit/protocols/schema/link/ethernet.py‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -39,9 +39,9 @@ def callback_payload(self: 'PayloadField', packet: 'dict[str, Any]') -> 'None':
3939
saw would be recorded as though somebody had registered it, and
4040
:meth:`Link.register <pcapkit.protocols.link.link.Link.register>` would
4141
afterwards report it as an overwrite. The helper reads the fallback
42-
without recording it, and for a code that really is registered as a
43-
:class:`~pcapkit.corekit.module.ModuleDescriptor` it writes the resolved
44-
class back, so the import happens once.
42+
without recording it, and resolves a
43+
:class:`~pcapkit.corekit.module.ModuleDescriptor` without writing the
44+
class back, so the registry is left as registered (:issue:`1559`).
4545
4646
"""
4747
from pcapkit.protocols.link.ethernet import Ethernet # pylint: disable=import-outside-toplevel

‎tests/protocols/internet/test_internet_unit.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -184,7 +184,8 @@ def __init__(self, file_: bytes, length: int, **kwargs: object) -> None:
184184
proto._sigterm = False
185185
module = DummyInternet._import_next_layer(proto, TransType.get(251), length=4, payload=b'mod!')
186186
self.assertEqual(module.data, b'mod!')
187-
self.assertNotIsInstance(DummyInternet.__dict__['__proto__'][TransType.get(251)], ModuleDescriptor)
187+
# resolved for the dispatch, not written back over the descriptor (#1559)
188+
self.assertIsInstance(DummyInternet.__dict__['__proto__'][TransType.get(251)], ModuleDescriptor)
188189

189190
from_header = DummyInternet._import_next_layer(proto, TransType.get(250))
190191
self.assertEqual(from_header.file, b'header-payload')

‎tests/protocols/internet/test_ipv6_unit.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -233,7 +233,8 @@ def test_ipv6_decode_next_layer_without_extensions_and_import_branches(self) ->
233233
})
234234
imported = proto._import_next_layer(custom_proto, payload=b'module')
235235
self.assertIsInstance(imported, Raw)
236-
self.assertIs(proto.__proto__[custom_proto], Raw)
236+
# resolved for the dispatch, not written back over the descriptor (#1559)
237+
self.assertIsInstance(proto.__proto__[custom_proto], ModuleDescriptor)
237238

238239
def test_ipv6_remaining_decode_and_import_edges(self) -> None:
239240
from collections import defaultdict

‎tests/protocols/link/test_link_unit.py‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -399,9 +399,9 @@ def test_link_schema_callbacks_resolve_payload_fields(self) -> None:
399399
callback_payload(descriptor_field, {'type': custom_type})
400400
self.assertIs(descriptor_field.protocol, Raw)
401401

402-
# Resolving a registered descriptor memoises the imported class,
403-
# which the callback used to resolve afresh on every frame.
404-
self.assertIs(registry[custom_type], Raw)
402+
# Resolving a registered descriptor leaves the descriptor in place:
403+
# a memoised class would go stale on a reload (#1559).
404+
self.assertIsInstance(registry[custom_type], ModuleDescriptor)
405405

406406
# An *unregistered* EtherType still resolves to Raw, and must not be
407407
# recorded on the way: the registry is Link.__proto__, shared by
Lines changed: 151 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
# -*- coding: utf-8 -*-
2+
"""GitHub issue #1559: a next-layer hit is resolved, not memoised.
3+
4+
:meth:`ProtocolBase._lookup_next_layer
5+
<pcapkit.protocols.protocol.ProtocolBase._lookup_next_layer>` used to write the
6+
class a registered :class:`~pcapkit.corekit.module.ModuleDescriptor` resolved to
7+
back over that descriptor. Three consequences, one test each:
8+
9+
* after :func:`importlib.reload` of the protocol's module, dispatch kept reaching
10+
the pre-reload class, so a parsed layer failed :func:`isinstance` against the
11+
live one -- what the method's own docstring said a memo would do;
12+
* parsing mutated registries shared by the whole process, so what a registry
13+
held depended on what had been parsed before;
14+
* so whether re-registering a shipped code warned depended on it too.
15+
16+
Every lookup now resolves through :attr:`ModuleDescriptor.klass
17+
<pcapkit.corekit.module.ModuleDescriptor.klass>`, which reads
18+
:data:`sys.modules`. Each test reloads or parses on a private import, under
19+
:func:`~tests._support.isolate_modules`, so neither a reloaded module nor a
20+
mutated registry outlives it.
21+
22+
"""
23+
24+
import importlib
25+
import struct
26+
import sys
27+
import unittest
28+
import warnings
29+
from unittest import mock
30+
31+
from tests._support import isolate_modules
32+
33+
#: Module of the class the reload tests dispatch to.
34+
UDP_MODULE = 'pcapkit.protocols.transport.udp'
35+
36+
#: An HTTP request, so that a TCP segment to port 80 dispatches once more.
37+
HTTP_REQUEST = b'GET / HTTP/1.1\r\nHost: x\r\n\r\n'
38+
39+
40+
def _ipv4(proto: 'int', body: 'bytes') -> 'bytes':
41+
"""An IPv4 packet carrying ``body`` as next layer ``proto``."""
42+
return struct.pack('!BBHHHBBH4s4s', 0x45, 0, 20 + len(body), 0, 0x4000, 64, proto, 0,
43+
bytes([10, 0, 0, 1]), bytes([10, 0, 0, 2])) + body
44+
45+
46+
def _udp(body: 'bytes') -> 'bytes':
47+
"""A UDP datagram between two ports no registry dispatches on."""
48+
return struct.pack('!HHHH', 12345, 9999, 8 + len(body), 0) + body
49+
50+
51+
def _tcp(dstport: 'int', body: 'bytes') -> 'bytes':
52+
"""A TCP segment, PSH/ACK, to ``dstport``."""
53+
return struct.pack('!HHIIBBHHH', 12345, dstport, 1, 0, 5 << 4, 0x18, 65535, 0, 0) + body
54+
55+
56+
class NextLayerHitNotMemoisedTests(unittest.TestCase):
57+
def setUp(self) -> None:
58+
isolate_modules(self)
59+
60+
def _parse_ipv4_udp(self) -> 'object':
61+
"""Parse an IPv4/UDP packet and return its UDP layer."""
62+
from pcapkit.protocols.internet.ipv4 import IPv4
63+
64+
packet = _ipv4(17, _udp(b'hello'))
65+
return IPv4(packet, len(packet)).payload
66+
67+
def test_a_reload_after_a_hit_dispatches_to_the_reloaded_class(self) -> None:
68+
"""The issue's reproduction, and the same through a real parse."""
69+
from pcapkit.const.reg.transtype import TransType
70+
from pcapkit.protocols.internet.internet import Internet
71+
from pcapkit.protocols.protocol import ProtocolBase
72+
73+
registry = Internet.__proto__
74+
module = importlib.import_module(UDP_MODULE)
75+
before = ProtocolBase._lookup_next_layer(registry, TransType.UDP)
76+
self.assertIs(before, module.UDP)
77+
self.assertIs(type(self._parse_ipv4_udp()), module.UDP)
78+
79+
with warnings.catch_warnings():
80+
warnings.simplefilter('ignore')
81+
importlib.reload(module)
82+
self.assertIsNot(module.UDP, before) # the reload really minted a class
83+
84+
self.assertIs(ProtocolBase._lookup_next_layer(registry, TransType.UDP), module.UDP)
85+
self.assertIsInstance(self._parse_ipv4_udp(), module.UDP)
86+
87+
# and popping the module, which replaces the module object outright
88+
reloaded = module.UDP
89+
sys.modules.pop(UDP_MODULE)
90+
fresh = importlib.import_module(UDP_MODULE)
91+
self.assertIsNot(fresh.UDP, reloaded)
92+
self.assertIs(ProtocolBase._lookup_next_layer(registry, TransType.UDP), fresh.UDP)
93+
94+
def test_a_parse_leaves_the_shared_registries_as_declared(self) -> None:
95+
"""Ethernet/IPv4/TCP/HTTP hits each registry once; none of them changes."""
96+
from pcapkit.const.reg.ethertype import EtherType
97+
from pcapkit.const.reg.transtype import TransType
98+
from pcapkit.corekit.module import ModuleDescriptor
99+
from pcapkit.protocols.application.http import HTTP
100+
from pcapkit.protocols.internet.internet import Internet
101+
from pcapkit.protocols.link.ethernet import Ethernet
102+
from pcapkit.protocols.link.link import Link
103+
from pcapkit.protocols.transport.tcp import TCP
104+
105+
hits = (
106+
('Link', Link.__proto__, EtherType.Internet_Protocol_version_4),
107+
('Internet', Internet.__proto__, TransType.TCP),
108+
('TCP', TCP.__proto__, 80),
109+
)
110+
before = {name: dict(registry) for name, registry, _ in hits}
111+
for name, registry, code in hits:
112+
with self.subTest(registry=name, when='before'):
113+
self.assertIsInstance(registry[code], ModuleDescriptor)
114+
115+
frame = bytes(6) + bytes([2, 0, 0, 0, 0, 1]) + b'\x08\x00' + _ipv4(6, _tcp(80, HTTP_REQUEST))
116+
parsed = Ethernet(frame, len(frame))
117+
self.assertIsInstance(parsed.payload.payload.payload, HTTP) # every hit really dispatched
118+
119+
for name, registry, code in hits:
120+
with self.subTest(registry=name, when='after'):
121+
self.assertIsInstance(registry[code], ModuleDescriptor)
122+
self.assertEqual(dict(registry), before[name])
123+
124+
def test_registering_a_dispatched_code_warns_alike_before_and_after_a_parse(self) -> None:
125+
"""Whether ``register`` warns no longer depends on what was parsed.
126+
127+
``register`` warns when it displaces an incumbent other than the class
128+
it is handed, and a shipped descriptor counts as different from the
129+
class it names. While a hit wrote the class back, re-registering UDP
130+
warned before any parse and was silent after one had reached UDP.
131+
132+
"""
133+
from pcapkit.const.reg.transtype import TransType
134+
from pcapkit.protocols.internet.internet import Internet
135+
from pcapkit.protocols.transport.udp import UDP
136+
137+
def overwrites() -> 'list[str]':
138+
with mock.patch.dict(Internet.__proto__), warnings.catch_warnings(record=True) as caught:
139+
warnings.simplefilter('always')
140+
Internet.register(TransType.UDP, UDP)
141+
return [str(record.message) for record in caught
142+
if 'already registered' in str(record.message)]
143+
144+
before = overwrites()
145+
self.assertEqual(len(before), 1)
146+
self.assertIs(type(self._parse_ipv4_udp()), UDP)
147+
self.assertEqual(overwrites(), before)
148+
149+
150+
if __name__ == '__main__':
151+
unittest.main()

‎tests/protocols/test_protocol_base_unit.py‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -261,7 +261,8 @@ class Other(Raw):
261261
)
262262
analyzed_module = DummyProtocol.analyze(4, b'module-body', alias=4)
263263
self.assertIsInstance(analyzed_module, Raw)
264-
self.assertIs(DummyProtocol.__proto__[4], Raw)
264+
# resolved for the dispatch, not written back over the descriptor (#1559)
265+
self.assertIsInstance(DummyProtocol.__proto__[4], ModuleDescriptor)
265266

266267
DummyProtocol.__proto__ = collections.defaultdict(lambda: Raw, {1: Raw})
267268
analyzed = DummyProtocol.analyze(1, b'raw-body', alias=1)
@@ -335,7 +336,7 @@ def __index__(cls) -> int:
335336
{5: ModuleDescriptor('pcapkit.protocols.misc.raw', 'Raw')},
336337
)
337338
self.assertIsInstance(proto_module._import_next_layer(5, None), Raw)
338-
self.assertIs(proto_module.__proto__[5], Raw)
339+
self.assertIsInstance(proto_module.__proto__[5], ModuleDescriptor) # not written back (#1559)
339340

340341
proto_stop = DummyProtocol(packet=b'abpayload', _protocol='dummyprotocol')
341342
self.assertTrue(proto_stop._check_term_threshold())
@@ -419,8 +420,8 @@ def test_lookup_next_layer_reads_the_fallback_without_recording_it(self) -> None
419420
:meth:`~pcapkit.protocols.protocol.ProtocolBase.register` call.
420421
421422
Resolving a :class:`~pcapkit.corekit.module.ModuleDescriptor` for a code
422-
that *is* registered still writes back, since that is memoisation of an
423-
import rather than a new entry.
423+
that *is* registered does not write back either: the resolved class
424+
would go stale on :func:`importlib.reload` (GitHub issue #1559).
424425
425426
"""
426427
DummyProtocol, _, _ = self._make_protocol_class()
@@ -439,9 +440,9 @@ def test_lookup_next_layer_reads_the_fallback_without_recording_it(self) -> None
439440
self.assertNotIn(99, registry)
440441
self.assertEqual(set(registry), {7})
441442

442-
# A hit resolves the descriptor once and keeps the resolved class.
443+
# A hit resolves the descriptor and keeps the descriptor.
443444
self.assertIs(DummyProtocol._lookup_next_layer(registry, 7), Raw)
444-
self.assertIs(registry[7], Raw)
445+
self.assertIsInstance(registry[7], ModuleDescriptor)
445446

446447
# A class registered directly is returned as it is.
447448
registry[8] = Raw

0 commit comments

Comments
 (0)