Skip to content

perf(protocols): import the packages above a protocol's on first use (#2730) - #2757

Merged
JarryShaw merged 1 commit into
mainfrom
perf/2730-lazy-protocol-packages
Nov 4, 2026
Merged

perf(protocols): import the packages above a protocol's on first use (#2730)#2757
JarryShaw merged 1 commit into
mainfrom
perf/2730-lazy-protocol-packages

Conversation

@JarryShaw

@JarryShaw JarryShaw commented Nov 4, 2026

Copy link
Copy Markdown
Owner

What is the purpose of your pull request?

  • perf — changes performance, not behaviour

Description of your pull request and other information

pcapkit/protocols, each layer package, misc/pcap, misc/checksum and pcapkit/__init__ now import their exports on first use, through the shared lazy_exports. Each protocol package __init__ keeps its literal imports, as ruled on #2753.

  • import pcapkit median: 640 → 568 ms. Protocol modules loaded at import: 232 → 129. The remaining cost is mostly foundation.registry's wrapper modules importing their classes at module level; that can be a follow-up.
  • A new pcapkit.protocols.registry builds __proto__ in full on first read, and foundation reads it at call time. So name-clash warnings, protochain lookups ('S_Tag' in a C-Tag frame) and next-layer dispatch behave as before. A cold-process test guards the Shim6 entry that ipv6_ext adds on import.
  • Incidental names (os, sys, ROOT, …) and the stray loop variable name no longer appear in dir(pcapkit). The private alias foundation.registry.protocols.protocol_registry is gone; read pcapkit.protocols.__proto__ instead.
  • Twelve plain-unittest legs and 581 pytest modules pass. A clean-venv docs build has 0 warnings. The new module fails on base.

Closes #2730

@JarryShaw
perf(protocols): import the packages above a protocol's on first use (#…
080c8ed 
…2730)

* pcapkit.protocols, its layer packages and misc.pcap/misc.checksum name
  each protocol class under TYPE_CHECKING and resolve it through
  lazy_exports; pcapkit re-exports pcapkit.protocols the same way. A
  protocol's own package keeps its literal imports (#2753 layout).
* pcapkit.protocols.__proto__ moves to the new pcapkit.protocols.registry
  module, built the first time it is reached; the foundation registry,
  reassembly and traceflow modules read it at call time instead of import.
* protocol.py stays imported first from pcapkit.protocols, since defining
  Protocol imports Raw and NoPayload, which import it back.
* tests: a sweep and cold-import checks of the lazy packages, the name
  registry, next-layer dispatch, selectors, pickling and the Shim6 entry
  ipv6_ext registers on import; conftest warms every protocol module;
  subclass walks reach every protocol first; the registry tests read
  pcapkit.protocols.__proto__; the apptype test expects SCTP's unloaded.
* docs: the ruling on protocol-family-grouping, and the registry module.

import pcapkit: median 640 ms -> 568 ms; protocol modules loaded 232 -> 129.
@JarryShaw JarryShaw added perf Pull requests that improve performance (perf: subject prefix) review: running A cross-review is in flight against the current head - no verdict yet labels Nov 4, 2026
@JarryShaw JarryShaw moved this to In review in PyPCAPKit Nov 4, 2026
@github-actions github-actions Bot added this to the PyPCAPKit v1.6 milestone Nov 4, 2026
@JarryShaw JarryShaw mentioned this pull request Nov 4, 2026
Closed
@JarryShaw

JarryShaw commented Nov 4, 2026

Copy link
Copy Markdown
Owner Author

GOOD TO GO at 080c8eda5, from a Sonnet review.

  • The public surface matches base, apart from incidental names that dir() no longer lists (os, sys, ROOT, …). These still resolve through getattr, and none was documented in v1.5.1.
  • A cold read of __proto__ gives the same 62 entries as base, and a clashing user class still warns. Five dispatch chains (Ethernet → IPv4 → IGMPv2, VLAN, 802.1ad, IPv6-Ext → Shim6, IPv6 → MH) match base. 25 cold processes, each racing 12 threads on the first read, never saw a half-built registry.
  • Six captures extract to the same digests as base, and a pickle written by base loads. The edited tests are adapted, not weakened, and the lazy tests run in cold child processes, so the conftest warm-up cannot mask them.
  • import pcapkit measured 766 → 678 ms on a busy host (the PR body's 640 → 568 ms is the same delta). Project tests, Lint and deploy-pages pass.

@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate and removed review: running A cross-review is in flight against the current head - no verdict yet labels Nov 4, 2026
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Coverage: 94.44% (unit tier, Python 3.15, 080c8eda5, Unit Tests run success)

Package Statements Missed Branches Partial Cover
pcapkit (top level) 106 4 20 2 95.24%
pcapkit/const 20918 175 2908 156 98.57%
pcapkit/corekit 2890 76 928 37 96.83%
pcapkit/dumpkit 286 3 102 3 98.45%
pcapkit/foundation 3507 27 1216 32 98.75%
pcapkit/interface 138 4 52 3 96.32%
pcapkit/protocols 22741 155 5908 127 99.00%
pcapkit/toolkit 961 26 342 8 96.62%
pcapkit/utilities 559 7 150 5 98.31%
pcapkit/vendor 5590 2286 1280 214 55.75%

Per-file detail: the coverage-html artifact of this run.

@JarryShaw JarryShaw mentioned this pull request Nov 4, 2026
Open
1 of 5 tasks
@JarryShaw
JarryShaw merged commit c211805 into main Nov 4, 2026
70 checks passed
@github-project-automation github-project-automation Bot moved this from In review to Done in PyPCAPKit Nov 4, 2026
@JarryShaw
JarryShaw deleted the perf/2730-lazy-protocol-packages branch November 4, 2026 14:00
@JarryShaw JarryShaw removed the review: good-to-go Cross-review at the current head says ready; CI state is separate label Nov 4, 2026
@JarryShaw JarryShaw mentioned this pull request Nov 4, 2026
Closed
JarryShaw added a commit that referenced this pull request Nov 4, 2026
@JarryShaw
docs(changelog): fold #2757 (#2449)
ed294b4 
- Folded into the lazy-import entry (#2757): pcapkit.protocols, its layer
  packages, misc.pcap, misc.checksum and pcapkit import their protocol
  classes on first use, pcapkit.protocols.registry builds __proto__ on first
  read, and dir(pcapkit) lists none of the incidental names 1.5.1 listed
  (os, sys, ROOT, warnings, tbtrim), measured on a v1.5.1 export; the ruling
  into the conventions entry.
- No mention of foundation.registry.protocols.protocol_registry, an import
  alias 1.5.1 neither exported nor documented, nor of an import time.
- CHANGELOG.md regenerated; util/changelog_md.py --check passes.
This was referenced Nov 4, 2026
Closed
Closed
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

perf Pull requests that improve performance (perf: subject prefix)

Projects

PyPCAPKit
Status: Done

Milestone

PyPCAPKit v1.6

Development

Successfully merging this pull request may close these issues.

perf(protocols): import the protocol, schema and data packages lazily

1 participant

@JarryShaw