Skip to content

Commit e85ef22

Browse files
committed
fix(corekit): share the pickle reductions in pcapkit.utilities.pickle, protocol 2's for Generic classes (#2714)
Below protocol 2, object.__reduce_ex__ hands an instance to copyreg._reduce_ex, whose MRO walk stops at the first non-heap base and calls it on the instance. GraalPy builds typing.Generic as a built-in, so the walk stopped at Generic and raised "typing.Generic() takes no arguments" for every Info, Schema and reassembly engine. - pcapkit.utilities.pickle holds the reductions classes had each written: ReduceAsProtocol2 (protocol 2's under every protocol), ReduceToName (a singleton's global name), ReduceToItems (a call on the item pairs) and reduce_to (a per-instance reducer). - Info, Schema, ReassemblyBase, Deferred and FrameRecord take ReduceAsProtocol2, and Deferred drops a __getstate__ matching Python's default; Sentinel and multidict's missing marker take ReduceToName; OrderedMultiDict and MultiInfo take ReduceToItems; the unregistered enum members of EnumRegistry and EnumField use reduce_to. - New tests stand in GraalPy's walk for copyreg._reduce_ex on CPython; the MH timestamp and sentinel tests drop their GraalPy protocol-2 floor, and the TCP datagram pickle test runs every protocol. Pickling never resolved the deferred packet: pytest's failure report renders the datagram with repr, which does, so the tests that pickle one datagram under each protocol found it resolved after protocol 0 failed.
1 parent 596dcfa commit e85ef22

21 files changed

Lines changed: 704 additions & 280 deletions

‎docs/source/pcapkit/corekit/fields/numbers.rst‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,11 @@ Internal Definitions
6565

6666
.. autofunction:: pcapkit.corekit.fields.numbers._rebuild_unregistered_member
6767

68-
.. autofunction:: pcapkit.corekit.fields.numbers._reduce_unregistered_member
68+
.. versionremoved:: 1.6.0
69+
70+
``pcapkit.corekit.fields.numbers._reduce_unregistered_member``, with no
71+
alias; :func:`pcapkit.utilities.pickle.reduce_to` installs the reduction it
72+
built (:issue:`2714`).
6973

7074
.. autoclass:: pcapkit.corekit.fields.numbers._CapturedInt
7175
:no-members:

‎docs/source/pcapkit/utilities/index.rst‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,10 @@ Module Utilities
77
:mod:`pcapkit.utilities` holds the cross-cutting services :mod:`pcapkit` is
88
built with: user-refined exceptions and warnings, logging, the tunable limits
99
of :mod:`~pcapkit.utilities.config`, a compatibility layer, timestamp
10-
conversion, the :mod:`context variables <pcapkit.utilities.contextvars>` its
11-
modules share, and wrappers around third-party libraries. It imports nothing from
12-
:mod:`pcapkit` outside itself; see :ref:`utilities-or-corekit`.
10+
conversion, pickle reductions, the
11+
:mod:`context variables <pcapkit.utilities.contextvars>` its modules share, and
12+
wrappers around third-party libraries. It imports nothing from :mod:`pcapkit`
13+
outside itself; see :ref:`utilities-or-corekit`.
1314

1415
.. toctree::
1516
:maxdepth: 2
@@ -20,6 +21,7 @@ modules share, and wrappers around third-party libraries. It imports nothing fro
2021
config
2122
chardet
2223
datetime
24+
pickle
2325
contextvars
2426

2527
Version Compatibility
Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
Pickle Reductions
2+
=================
3+
4+
.. module:: pcapkit.utilities.pickle
5+
6+
:mod:`pcapkit.utilities.pickle` holds the :mod:`pickle` reductions several
7+
:mod:`pcapkit` classes share, so each is written once (:issue:`2714`).
8+
:class:`ReduceAsProtocol2` lets a :class:`typing.Generic` subclass or a slotted
9+
:class:`dict` pickle under protocol 0 and 1, on GraalPy as on CPython;
10+
:class:`ReduceToName` reduces a singleton to the module-level name holding it;
11+
:class:`ReduceToItems` reduces a multi-valued mapping to a call of its class on its
12+
item pairs; and :func:`reduce_to` makes one instance reduce to a call of a given
13+
function. A pickle a class wrote before it took one of these still loads, and one
14+
it writes with it loads where the class still reduces by its own.
15+
16+
.. autoclass:: pcapkit.utilities.pickle.ReduceAsProtocol2
17+
:members: __reduce_ex__
18+
19+
.. autoclass:: pcapkit.utilities.pickle.ReduceToName
20+
:members: __reduce__
21+
22+
.. autoclass:: pcapkit.utilities.pickle.ReduceToItems
23+
:members: __reduce_ex__
24+
25+
.. autofunction:: pcapkit.utilities.pickle.reduce_to
26+
27+
.. autofunction:: pcapkit.utilities.pickle._reduce_to

‎pcapkit/corekit/enum.py‎

Lines changed: 7 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -96,16 +96,16 @@
9696
an unknown value does not mint one (:issue:`775`).
9797
9898
"""
99-
import functools
10099
from typing import TYPE_CHECKING
101100

102101
from aenum import extend_enum
103102

104103
from pcapkit.corekit.sentinels import NO_DEFAULT
105104
from pcapkit.utilities.exceptions import BaseError, CaughtByCaller, EnumKeyError, EnumValueError
105+
from pcapkit.utilities.pickle import reduce_to
106106

107107
if TYPE_CHECKING:
108-
from typing import Any, Callable, Type
108+
from typing import Any, Type
109109

110110
from typing_extensions import Self
111111

@@ -730,11 +730,10 @@ def _make_reducible(cls, obj: 'Any', *args: 'Any') -> 'Any':
730730
lookup table grows. The rebuilt member is itself reducible, since the
731731
method it comes back through installs this again.
732732
733-
The override goes on the *instance* rather than on the type, which both
734-
:mod:`pickle` and :mod:`copy` honour: each fetches ``__reduce_ex__``
735-
with :func:`getattr` on the object. :func:`functools.partial` rather
736-
than a closure keeps the member itself out of its own reducer, so being
737-
reducible does not put it in a reference cycle.
733+
The override goes on the *instance* rather than on the type, through
734+
:func:`~pcapkit.utilities.pickle.reduce_to`, which :mod:`pickle` and
735+
:mod:`copy` both honour, and which keeps the member out of its own
736+
reducer.
738737
739738
Note:
740739
This is opt-in per registry rather than done by
@@ -757,8 +756,7 @@ def _make_reducible(cls, obj: 'Any', *args: 'Any') -> 'Any':
757756
``obj``, for use as a ``return`` expression.
758757
759758
"""
760-
obj.__reduce_ex__ = functools.partial(_reduce_unregistered_member, cls, args)
761-
return obj
759+
return reduce_to(obj, _rebuild_unregistered_member, cls, args)
762760

763761

764762
def _rebuild_unregistered_member(namespace: 'Type[EnumRegistry]',
@@ -783,25 +781,3 @@ def _rebuild_unregistered_member(namespace: 'Type[EnumRegistry]',
783781
return namespace._unregistered_member(*args) # pylint: disable=protected-access
784782

785783

786-
def _reduce_unregistered_member( # pylint: disable=unused-argument
787-
namespace: 'Type[EnumRegistry]', args: 'tuple[Any, ...]',
788-
protocol: 'int') -> 'tuple[Callable[..., Any], tuple[Any, ...]]':
789-
"""The ``__reduce_ex__`` :meth:`EnumRegistry._make_reducible` installs.
790-
791-
Bound to its first two arguments with :func:`functools.partial`, so that
792-
the reducer holds the ingredients of the member rather than the member
793-
itself.
794-
795-
Args:
796-
namespace: The registry class the member is an instance of.
797-
args: The arguments to rebuild the member from.
798-
protocol: The :mod:`pickle` protocol version, as passed by
799-
:mod:`pickle` and :mod:`copy`. Unused -- the reduction is the same
800-
for every protocol -- but accepted because both call this with it.
801-
802-
Returns:
803-
The callable and arguments :mod:`pickle` and :mod:`copy` should rebuild
804-
the member through.
805-
806-
"""
807-
return (_rebuild_unregistered_member, (namespace, args))

‎pcapkit/corekit/fields/numbers.py‎

Lines changed: 13 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,15 @@
11
# -*- coding: utf-8 -*-
2-
"""numerical field class"""
2+
"""numerical field class
3+
4+
.. versionremoved:: 1.6.0
5+
6+
``pcapkit.corekit.fields.numbers._reduce_unregistered_member``, with no
7+
alias; :func:`pcapkit.utilities.pickle.reduce_to` installs the reduction it
8+
built (:issue:`2714`).
9+
10+
"""
311

412
import enum
5-
import functools
613
import math
714
from typing import TYPE_CHECKING, Generic, TypeVar, Union, cast
815

@@ -11,6 +18,7 @@
1118
from pcapkit.corekit.fields.field import Field
1219
from pcapkit.corekit.sentinels import NO_VALUE
1320
from pcapkit.utilities.exceptions import BaseError, FieldValueError, IntError, ProtocolError
21+
from pcapkit.utilities.pickle import reduce_to
1422

1523
__all__ = [
1624
'NumberField',
@@ -958,14 +966,9 @@ def _unregistered_member(namespace: 'Type[StdlibEnum] | Type[AenumEnum]',
958966
# NOTE: ``Enum.__reduce_ex__`` reduces a member to ``(cls, (value,))``,
959967
# i.e. to the one lookup this member is deliberately absent from, so
960968
# pickle and copy would both raise on it without this. Overriding it
961-
# per instance -- which pickle and copy both honour, since both fetch
962-
# it with getattr on the object rather than on its type -- rebuilds an
963-
# equivalent unregistered member instead. functools.partial rather
964-
# than a closure keeps this off obj itself, so the member does not
965-
# become part of a reference cycle merely by being reducible.
966-
obj.__reduce_ex__ = functools.partial(
967-
_reduce_unregistered_member, namespace, value, name, attrs)
968-
return obj
969+
# per instance, which pickle and copy both honour, rebuilds an
970+
# equivalent unregistered member instead.
971+
return reduce_to(obj, _rebuild_unregistered_member, namespace, value, name, attrs)
969972

970973

971974
def _rebuild_unregistered_member(namespace: 'Type[StdlibEnum] | Type[AenumEnum]',
@@ -996,31 +999,6 @@ def _rebuild_unregistered_member(namespace: 'Type[StdlibEnum] | Type[AenumEnum]'
996999
namespace, value, name, **attrs)
9971000

9981001

999-
def _reduce_unregistered_member( # pylint: disable=unused-argument
1000-
namespace: 'Type[StdlibEnum] | Type[AenumEnum]',
1001-
value: 'Any', name: 'str', attrs: 'dict[str, Any]',
1002-
protocol: 'int') -> 'tuple[Callable[..., Any], tuple[Any, ...]]':
1003-
"""The ``__reduce_ex__`` :meth:`EnumField._unregistered_member` installs.
1004-
1005-
Bound to its first four arguments with :func:`functools.partial`, so that
1006-
the reducer holds the ingredients of the member rather than the member
1007-
itself.
1008-
1009-
Args:
1010-
namespace: The registry class the member is an instance of.
1011-
value: The member's ``_value_``.
1012-
name: The member's ``_name_``.
1013-
attrs: The extra attributes the member carries.
1014-
protocol: The :mod:`pickle` protocol version, ignored -- the reduction
1015-
is the same for all of them, and :mod:`copy` passes 4 here.
1016-
1017-
Returns:
1018-
A two-tuple of :func:`_rebuild_unregistered_member` and its arguments.
1019-
1020-
"""
1021-
return (_rebuild_unregistered_member, (namespace, value, name, attrs))
1022-
1023-
10241002
class PortEnumField(EnumField):
10251003
"""Enumerated value for transport-layer port fields.
10261004

‎pcapkit/corekit/infoclass.py‎

Lines changed: 3 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@
2727
from pcapkit.corekit.multidict import MultiDict, OrderedMultiDict, iter_multi_items
2828
from pcapkit.utilities.compat import Mapping, UnionType, final, get_annotations
2929
from pcapkit.utilities.exceptions import InfoError, UnsupportedCall, stacklevel
30+
from pcapkit.utilities.pickle import ReduceAsProtocol2, ReduceToItems
3031
from pcapkit.utilities.warnings import InfoWarning, warn
3132

3233
if TYPE_CHECKING:
@@ -318,7 +319,7 @@ def __new__(cls, name: 'str', bases: 'tuple[type, ...]', attrs: 'dict[str, Any]'
318319
return super().__new__(cls, name, bases, attrs, **kwargs) # type: ignore[return-value]
319320

320321

321-
class Info(Mapping[str, VT], Generic[VT], metaclass=InfoMeta):
322+
class Info(ReduceAsProtocol2, Mapping[str, VT], Generic[VT], metaclass=InfoMeta):
322323
"""Turn dictionaries into :obj:`object` like instances.
323324
324325
* :class:`Info` objects are :class:`~collections.abc.Mapping` instances,
@@ -975,7 +976,7 @@ def _immutable(self, *args: 'Any', **kwargs: 'Any') -> 'NoReturn':
975976
__setstate__ = _immutable
976977

977978

978-
class MultiInfo(_MultiInfo[KT, VT], MultiDict[KT, VT]):
979+
class MultiInfo(ReduceToItems, _MultiInfo[KT, VT], MultiDict[KT, VT]):
979980
"""An immutable :class:`~pcapkit.corekit.multidict.MultiDict` that is an :class:`Info`.
980981
981982
It holds every value of a key, as a
@@ -1065,9 +1066,6 @@ def listvalues(self) -> 'Iterator[list[VT]]':
10651066
"""
10661067
return (values for (_, values) in self.lists())
10671068

1068-
def __reduce_ex__(self, protocol: 'Any') -> 'tuple[type, tuple[list[tuple[KT, VT]]]]':
1069-
return type(self), (list(self.items(multi=True)),)
1070-
10711069

10721070
class OrderedMultiInfo(_MultiInfo[KT, VT], OrderedMultiDict[KT, VT]):
10731071
"""An immutable :class:`~pcapkit.corekit.multidict.OrderedMultiDict` that is an :class:`Info`.

‎pcapkit/corekit/multidict.py‎

Lines changed: 6 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -15,9 +15,10 @@
1515
from typing import TYPE_CHECKING, Generic, TypeVar, overload
1616

1717
from pcapkit.utilities.exceptions import MissingKeyError, UnsupportedCall
18+
from pcapkit.utilities.pickle import ReduceToItems, ReduceToName
1819

1920
if TYPE_CHECKING:
20-
from typing import Any, Iterable, Iterator, Mapping, NoReturn, Optional, SupportsIndex
21+
from typing import Any, Iterable, Iterator, Mapping, NoReturn, Optional
2122

2223
from typing_extensions import Literal, Self
2324

@@ -68,13 +69,12 @@ def unlink(self, omd: 'OrderedMultiDict[_KT, _VT]') -> 'None':
6869
omd._last_bucket = self.prev # pylint: disable=protected-access
6970

7071

71-
class _Missing:
72+
class _Missing(ReduceToName):
73+
_name = '_missing'
74+
7275
def __repr__(self) -> 'str':
7376
return "no value"
7477

75-
def __reduce__(self) -> 'str':
76-
return "_missing"
77-
7878

7979
_missing = _Missing()
8080

@@ -470,7 +470,7 @@ def __repr__(self) -> 'str':
470470
return f"{type(self).__name__}({list(self.items(multi=True))!r})"
471471

472472

473-
class OrderedMultiDict(MultiDict[_KT, _VT]):
473+
class OrderedMultiDict(ReduceToItems, MultiDict[_KT, _VT]):
474474
"""Works like a regular :class:`MultiDict` but preserves the
475475
order of the fields. To convert the ordered multi dict into a
476476
list you can use the :meth:`items` method and pass it ``multi=True``.
@@ -537,9 +537,6 @@ def deepcopy(self, memo: 'Optional[dict]' = None) -> 'OrderedMultiDict[_KT, _VT]
537537
"""
538538
return self.__class__(copy.deepcopy(list(self.items(multi=True)), memo=memo))
539539

540-
def __reduce_ex__(self, protocol: 'SupportsIndex') -> 'Any':
541-
return type(self), (list(self.items(multi=True)),)
542-
543540
def __getstate__(self) -> 'Any':
544541
return list(self.items(multi=True))
545542

‎pcapkit/corekit/sentinels.py‎

Lines changed: 7 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@
4545
from typing import TYPE_CHECKING, cast
4646

4747
from pcapkit.utilities.compat import final
48+
from pcapkit.utilities.pickle import ReduceToName
4849

4950
__all__ = ['Sentinel', 'NULL', 'NO_VALUE', 'NO_DEFAULT']
5051

@@ -197,7 +198,7 @@ def register(cls, subclass: 'type[Any]') -> 'type[Any]':
197198
return super().register(subclass)
198199

199200

200-
class Sentinel(metaclass=SentinelMeta):
201+
class Sentinel(ReduceToName, metaclass=SentinelMeta):
201202
"""Abstract base of every sentinel type, so each behaves like the type of :obj:`None`.
202203
203204
A subclass sets ``_name`` to the name of the module-level variable that
@@ -210,10 +211,11 @@ class Sentinel(metaclass=SentinelMeta):
210211
:obj:`None`, so a caller who builds one by mistake still gets the
211212
sentinel.
212213
* :meth:`__copy__` and :meth:`__deepcopy__` return ``self``.
213-
* :meth:`__reduce__` returns ``_name``, as ``Ellipsis.__reduce__()``
214-
returns ``'Ellipsis'``, so :mod:`pickle` stores the instance as a
215-
reference to that module-level name and loads it back as the same
216-
object, at every protocol. Guarding :meth:`__new__` alone would not
214+
* :meth:`~pcapkit.utilities.pickle.ReduceToName.__reduce__`, from
215+
:class:`~pcapkit.utilities.pickle.ReduceToName`, returns ``_name``, as
216+
``Ellipsis.__reduce__()`` returns ``'Ellipsis'``, so :mod:`pickle` stores
217+
the instance as a reference to that module-level name and loads it back
218+
as the same object, at every protocol. Guarding :meth:`__new__` alone would not
217219
cover protocols 0 and 1, which rebuild through
218220
:func:`copyreg._reconstructor` and so call :meth:`object.__new__`
219221
directly.
@@ -318,10 +320,6 @@ def __deepcopy__(self, memo: 'dict[int, Any]') -> 'Self':
318320
"""
319321
return self
320322

321-
def __reduce__(self) -> 'str':
322-
"""Reduce to ``_name``, which :mod:`pickle` stores as a global reference."""
323-
return self._name
324-
325323

326324
@final
327325
class NullType(Sentinel):

0 commit comments

Comments
 (0)