Skip to content

Commit f56dc5c

Browse files
committed
fix(foundation): close only an owned input on leaving the with block (#1999)
Extractor.__exit__ closed the input unconditionally, so a with block closed a stream the caller supplied and still owned, against the ownership rule of #610 that _cleanup and the backstops follow. It now closes the input only when _owns_input() holds; the engine is still closed and the finaliser still detached. A caller's non-seekable stream stays open, as the SeekableReader wrapper is built with stream_closing off. - tests: a caller's seekable and non-seekable stream stay open after the block, and a path-opened handle is closed by __exit__ alone - test_extraction: the case pinning the old close now splits owned and caller-supplied input - changelog and close_extractor prose no longer say the block always closes the input
1 parent 33dddb9 commit f56dc5c

6 files changed

Lines changed: 113 additions & 10 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -139,7 +139,7 @@ Preceded by `1.5.0a1` (2026-09-15), `1.5.0b1` to `1.5.0b3` (all 2026-09-19) and
139139

140140
- TCP reassembly mixed absolute sequence numbers with buffer-relative slicing, so on any capture carrying a SYN with a realistic initial sequence number incomplete datagrams were dropped silently, and the `completed=False` branch of the public API was unreachable ([#349](https://github.com/JarryShaw/PyPCAPKit/issues/349), [#376](https://github.com/JarryShaw/PyPCAPKit/pull/376)).
141141
- `format='text'` raised `AttributeError` before writing anything, naming a `dictdumper.Text` that has never existed. It now points at `Tree`, as the `'txt'` alias beside it already did.
142-
- `Extractor` had the ownership of its input stream inverted: a handle it opened itself, from `fin` given as a path, was **never closed**, while a stream the *caller* supplied and still needed **was**. `_cleanup` closed under `not self._flag_s` (the flag that gated the `open()`), and the `SeekableReader` wrapping a non-seekable input asked for `stream_closing=not self._flag_s` in the same wrong direction. The leak was one descriptor per extraction, the root cause of the `ResourceWarning` that reddened [#577](https://github.com/JarryShaw/PyPCAPKit/pull/577), [#596](https://github.com/JarryShaw/PyPCAPKit/pull/596) and [#600](https://github.com/JarryShaw/PyPCAPKit/pull/600) ([#606](https://github.com/JarryShaw/PyPCAPKit/issues/606) had only made the test immune). Both conditions now follow ownership via a single `Extractor._owns_input`, and a finaliser releases the handle of an extraction *abandoned* before end of file (`auto=False`, iterated part way and dropped); the files [#610](https://github.com/JarryShaw/PyPCAPKit/issues/610) names emit no such warning. `__exit__` deliberately still closes unconditionally, since a caller who scopes an `Extractor` with `with` asked for that ([#610](https://github.com/JarryShaw/PyPCAPKit/issues/610)).
142+
- `Extractor` had the ownership of its input stream inverted: a handle it opened itself, from `fin` given as a path, was **never closed**, while a stream the *caller* supplied and still needed **was**. `_cleanup` closed under `not self._flag_s` (the flag that gated the `open()`), and the `SeekableReader` wrapping a non-seekable input asked for `stream_closing=not self._flag_s` in the same wrong direction. The leak was one descriptor per extraction, the root cause of the `ResourceWarning` that reddened [#577](https://github.com/JarryShaw/PyPCAPKit/pull/577), [#596](https://github.com/JarryShaw/PyPCAPKit/pull/596) and [#600](https://github.com/JarryShaw/PyPCAPKit/pull/600) ([#606](https://github.com/JarryShaw/PyPCAPKit/issues/606) had only made the test immune). Both conditions now follow ownership via a single `Extractor._owns_input`, and a finaliser releases the handle of an extraction *abandoned* before end of file (`auto=False`, iterated part way and dropped); the files [#610](https://github.com/JarryShaw/PyPCAPKit/issues/610) names emit no such warning. Leaving a `with` block follows the same rule, so a caller's stream stays open after it ([#1999](https://github.com/JarryShaw/PyPCAPKit/issues/1999)).
143143
- `extract(..., no_eof=True)` never returned. End of stream was detected and `ExtractionWarning: EOF reached` fired, but every loop that handles it (`record_frames`, `__next__`, `__call__`) read `if self._flag_n: continue`, so the flag suppressed the error that *ended* the loop without supplying another ending and an exhausted input was retried forever. The flag is not pointless -- `__main__` sets `no_eof` for `fin='-'`, and the end of what has arrived on a live capture is not the end of the capture -- so this is a termination condition rather than a deletion. The input's own position tells a paused capture from a finished one: `prepare` raises end of stream when zero bytes remain and restores the position first, so two consecutive ends of stream at the same position mean nothing arrived between them (`Extractor._note_eof_progress`). A pipe is unaffected, since a read there *blocks* while the writer is open but idle. This also fixes `pcapkit -`, which hung on **any** finished stdin. **One deliberate narrowing**: a *seekable* input does not block, so a file still being appended to ends at the data present when the extraction reached it; a timed grace period would make the cut-off intermittent rather than absent, so following a growing file is left to a policy of its own. Pre-existing and untouched: an append landing *mid-record* raises `ValueError: read length must be non-negative or -1`. The end-to-end tests are bounded by a **child process** rather than `tests._support.time_limit`, because the in-process deadline proved intermittent on this loop, and an intermittent guard against a hang leaves a wedged suite rather than a red test. Guesses at the cause are ruled out: the parse path's `except Exception` handlers are never entered during the spin, and the memory is the retry loop's `EOF reached` records, which the runner retains ([#620](https://github.com/JarryShaw/PyPCAPKit/issues/620)).
144144
- `register_protocol` displaced one protocol class with another and said nothing. The registry behind `pcapkit.protocols.__proto__` is keyed on `cls.__name__.upper()`, and several dispatchable classes are named `HTTP` (the base `pcapkit.protocols.application.http.HTTP` and the `httpv1` and `httpv2` implementations), so a bare assignment decided the winner: `register_protocol(httpv2.HTTP)` moved `__proto__['HTTP']` to the `httpv2` class with no exception, log line or `RegistryWarning`, and `ProtocolBase.expand_comp('HTTP')` then resolved to whichever registered last. The overwrite now raises a `RegistryWarning` naming the displaced and the replacing class. The sharpest path is not a call to the registrar: `Protocol.__init_subclass__()` registers unconditionally, so merely *defining* a subclass of the public `Protocol` under a name a built-in holds displaced that built-in. A plain `import pcapkit` emits zero new warnings: the built-in `HTTP` classes derive from `ProtocolBase`, so `__init_subclass__` never fires for them, and import-time seeding keys off the distinct names `HTTP`, `HTTPv1` and `HTTPv2`. The guard deliberately reads "key present and the incumbent is a different class", not mere presence ([#726](https://github.com/JarryShaw/PyPCAPKit/pull/726) later gave its sibling registrars the same identity guard): this function is the funnel every wrapper registrar ends in, so registering one class under two codes (`register_tcp` then `register_udp`, a documented path) reaches it twice with nothing displaced. The collision is reported, not resolved: re-keying is a registry-format change that none of the bare-name readers can absorb (each degrades *silently* to a string or `Raw` on a miss), so it belongs to the registry redesign in [#514](https://github.com/JarryShaw/PyPCAPKit/issues/514). The issue called this the one registrar with no warning; in fact none warned by itself, the siblings only by delegating to a guarded classmethod ([#675](https://github.com/JarryShaw/PyPCAPKit/issues/675)).
145145
- the `#:` autodoc comments in `pcapkit/foundation/traceflow/traceflow.py` that named a bare `Type`, which Sphinx resolves against every class named `Type` in the project rather than `typing.Type`, and silently linked to `pcapkit.const.l2tp.type.Type`, an L2TP field-type enum; once [#684](https://github.com/JarryShaw/PyPCAPKit/issues/684) rendered `TraceFlow._foutio`, the built docs pointed at the wrong class with no warning. These sites now spell it `~typing.Type[Dumper]`, among the sites [#709](https://github.com/JarryShaw/PyPCAPKit/issues/709) lists. The other bare `Type` sites, in hand-written `:type:` fields under `docs/source/pcapkit/foundation/`, are fixed separately by [#714](https://github.com/JarryShaw/PyPCAPKit/pull/714) ([#709](https://github.com/JarryShaw/PyPCAPKit/issues/709)).

‎docs/source/changelog/1.5.0.rst‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -976,9 +976,8 @@ Fixed
976976
Both conditions now follow ownership via a single ``Extractor._owns_input``, and a
977977
finaliser releases the handle of an extraction *abandoned* before end of file
978978
(``auto=False``, iterated part way and dropped); the files :issue:`610` names
979-
emit no such warning. ``__exit__`` deliberately still
980-
closes unconditionally, since a caller who scopes an :class:`.Extractor` with ``with``
981-
asked for that (:issue:`610`).
979+
emit no such warning. Leaving a ``with`` block follows the same rule, so a caller's
980+
stream stays open after it (:issue:`1999`).
982981
* ``extract(..., no_eof=True)`` never returned. End of stream was detected and
983982
``ExtractionWarning: EOF reached`` fired, but every loop that handles it
984983
(``record_frames``, ``__next__``, ``__call__``) read ``if self._flag_n: continue``,

‎pcapkit/foundation/extraction.py‎

Lines changed: 21 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1590,9 +1590,26 @@ def __enter__(self) -> 'Extractor':
15901590

15911591
def __exit__(self, exc_type: 'Type[BaseException] | None', exc_value: 'BaseException | None',
15921592
traceback: 'TracebackType | None') -> 'None': # pylint: disable=unused-argument
1593-
"""Close the input file when exits."""
1593+
"""Release what this class holds when the ``with`` block exits.
1594+
1595+
The engine is always closed. The input stream is closed only *if this
1596+
class owns it* -- the rule of :issue:`610`, as :meth:`_owns_input` writes
1597+
it down and :meth:`_cleanup` and the backstops already follow. A stream
1598+
the caller supplied stays open for the caller to close, so leaving the
1599+
block no longer closes it behind the caller's back (:issue:`1999`).
1600+
1601+
For a caller's non-seekable stream, the stream this class owns is the
1602+
:class:`~pcapkit.corekit.io.SeekableReader` wrapped around it, and that
1603+
wrapper is closed; it was built with ``stream_closing`` off, so closing
1604+
it leaves the caller's stream underneath open.
1605+
1606+
"""
15941607
logger.debug('closing %s on context exit after %d frame(s)', self._ifnm, self._frnum)
1595-
self._ifile.close()
1608+
# NOTE: ownership, not the ``with`` statement, decides who closes the
1609+
# input (#610, #1999): entering the block does not hand the caller's
1610+
# stream over to this class.
1611+
if self._owns_input():
1612+
self._ifile.close()
15961613
self._detach_finalizer()
15971614
self._exeng.close()
15981615

@@ -1643,8 +1660,8 @@ def _owns_input(self) -> 'bool':
16431660
"""Whether the input stream is this class's to close.
16441661
16451662
The one place the ownership rule of :issue:`610` is written down, so that
1646-
:meth:`_cleanup`, :meth:`__del__` and the registration of
1647-
:attr:`_finalizer` cannot drift apart on it.
1663+
:meth:`_cleanup`, :meth:`__exit__`, :meth:`__del__` and the registration
1664+
of :attr:`_finalizer` cannot drift apart on it.
16481665
16491666
Returns:
16501667
:data:`True` when the input stream was opened by this class and is

‎tests/_support.py‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1135,8 +1135,9 @@ def close_extractor(extractor: object) -> None:
11351135
:meth:`Extractor._cleanup <pcapkit.foundation.extraction.Extractor._cleanup>`
11361136
or :meth:`Extractor.__exit__ <pcapkit.foundation.extraction.Extractor.__exit__>`,
11371137
so nothing in the library closes up after them. Both of those close the
1138-
input file *and* the engine, and this helper has to do the same: the input
1139-
file is not the only resource. The ``pcap_ct`` and ``pypcap`` engines hold a
1138+
engine as well as an input file the extractor opened itself, and this helper
1139+
closes both, whoever opened the input: the input file is not the only
1140+
resource. The ``pcap_ct`` and ``pypcap`` engines hold a
11401141
live :class:`pcap.pcap` handle, and ``pyshark`` holds a temporary file, so
11411142
dropping the extractor without closing the engine leaks an OS-level handle
11421143
per test. Under a suite that builds hundreds of extractors that accumulates

‎tests/foundation/test_extraction.py‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -949,11 +949,21 @@ def test_record_header_record_frames_iteration_call_and_cleanup(self) -> None:
949949
given_stream._cleanup()
950950
self.assertFalse(given_stream._ifile.closed_by_test)
951951

952+
# Leaving the ``with`` block follows the same ownership rule: it closes
953+
# a handle ``Extractor`` opened, and leaves a caller's stream open. This
954+
# assertion closed the caller's stream until #1999.
955+
manual._flag_s = True
952956
with manual as ctx:
953957
self.assertIs(ctx, manual)
954958
self.assertTrue(manual._ifile.closed_by_test)
955959
self.assertGreaterEqual(manual._exeng.close_calls, 1)
956960

961+
scoped_stream = self._bare_extractor()
962+
with scoped_stream:
963+
pass
964+
self.assertFalse(scoped_stream._ifile.closed_by_test)
965+
self.assertGreaterEqual(scoped_stream._exeng.close_calls, 1)
966+
957967
def test_constructor_initialises_without_running_when_patched(self) -> None:
958968
from pcapkit.foundation.extraction import Extractor
959969

‎tests/foundation/test_extraction_ownership.py‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -337,6 +337,82 @@ def test_non_seekable_stream_survives_its_seekable_wrapper(self) -> None:
337337
stream.close()
338338

339339

340+
@unittest.skipUnless(HAS_RUNTIME, 'runtime dependencies not installed')
341+
class ContextExitFollowsOwnership(unittest.TestCase):
342+
"""#1999: leaving a ``with`` block closes only what ``Extractor`` owns.
343+
344+
``__exit__`` closed the input unconditionally, so scoping an extraction with
345+
``with`` closed a stream the caller had passed in and still held, which the
346+
rule of #610 says is the caller's to close.
347+
348+
"""
349+
350+
def test_callers_seekable_stream_stays_open(self) -> None:
351+
"""The issue's own repro: an ``auto=True`` extraction inside ``with``."""
352+
import pcapkit
353+
354+
with open(sample_path('in.pcap'), 'rb') as handle:
355+
with pcapkit.extract(fin=handle, nofile=True, store=False):
356+
pass
357+
358+
with self.subTest(check='handle not closed'):
359+
self.assertFalse(handle.closed,
360+
"leaving the with block closed the caller's stream")
361+
362+
with self.subTest(check='handle still readable'):
363+
handle.seek(0)
364+
self.assertEqual(handle.read(4), b'\xd4\xc3\xb2\xa1')
365+
366+
def test_callers_non_seekable_stream_stays_open(self) -> None:
367+
"""The wrapper ``Extractor`` built is closed; the stream under it is not."""
368+
import pcapkit
369+
from pcapkit.corekit.io import SeekableReader
370+
371+
with open(sample_path('in.pcap'), 'rb') as source:
372+
payload = source.read()
373+
374+
# The ``sys.stdin.buffer`` shape: see
375+
# ``test_non_seekable_stream_survives_its_seekable_wrapper`` for why a
376+
# bare non-seekable ``BytesIO`` will not do.
377+
stream = io.BufferedReader(_NonSeekableRaw(payload))
378+
self.addCleanup(stream.close)
379+
with pcapkit.extract(fin=stream, nofile=True, store=False,
380+
auto=False) as extractor:
381+
next(extractor)
382+
383+
self.assertIsInstance(extractor._ifile, SeekableReader,
384+
'a non-seekable input should have been wrapped, '
385+
'without which this test proves nothing')
386+
with self.subTest(check='wrapper closed'):
387+
self.assertTrue(extractor._ifile.closed,
388+
'leaving the with block should close the wrapper '
389+
'Extractor built')
390+
with self.subTest(check="caller's stream still open"):
391+
self.assertFalse(stream.closed,
392+
"leaving the with block closed the caller's stream "
393+
'through the wrapper')
394+
395+
def test_path_opened_handle_is_closed(self) -> None:
396+
"""A handle ``Extractor`` opened from a path is closed on leaving the block.
397+
398+
``auto=False`` and one frame read, so the extraction is still short of end
399+
of file and ``_cleanup`` has not run: the close is ``__exit__``'s alone.
400+
401+
"""
402+
import pcapkit
403+
404+
with pcapkit.extract(fin=sample_path('in.pcap'), nofile=True, store=False,
405+
auto=False) as extractor:
406+
next(extractor)
407+
self.assertFalse(extractor._flag_e,
408+
'_cleanup ran inside the block, so this test no '
409+
'longer measures __exit__')
410+
self.assertFalse(extractor._ifile.closed)
411+
412+
self.assertTrue(extractor._ifile.closed,
413+
'leaving the with block left the handle Extractor opened')
414+
415+
340416
@unittest.skipUnless(HAS_RUNTIME, 'runtime dependencies not installed')
341417
class FinaliserBackstopWithoutDel(unittest.TestCase):
342418
"""#1997: the backstop is a :class:`weakref.finalize`, not ``__del__`` alone.

0 commit comments

Comments
 (0)