Skip to content

Commit 5e73e7f

Browse files
committed
fix: run() must own its child from creation, not after logging
`LoggingSubprocess.run()` logged "Command started as pid" and "Output:", and did the POSIX process-group lookup, *before* entering the `try` whose `finally` reaps the child. Anything raising in that window escaped `run()` having never waited on the child: a live process, no exit code recorded, and the runner publishing a terminal state for it anyway. The reachable trigger is a `logging.Filter` that raises. Filters propagate out of `Logger.handle`, unlike handlers, whose exceptions are routed to `handleError` -- and installing a filter is a supported use of this library; `Session` installs its own. Reproduced before the fix: the child stayed alive with `exit_code is None`, while a filter raising from inside the pump (already covered by the `finally`) reaped correctly. The ownership `try` now begins immediately after successful process creation, so every exit path from that point goes through `_reap`. The body binds `proc = self._process` once instead of re-reading the attribute, matching the bind-once convention used by `notify()`/`terminate()`. `_reap`'s own logging is now best-effort. It runs inside `run()`'s `finally`, so anything escaping it replaces the exception already propagating and hides the real cause. That containment was in place for the terminate and the wait but not for the three `logger.error` calls between them -- and a raising logging stack is exactly what routes execution into `_reap`, which makes those calls the likeliest thing to fail there, not the least. Reported by mwiebe on #333: #333 (comment) Two honest notes on the scope of the fix: - Reordering `_reap` to terminate before logging was tried and dropped. With best-effort logging in place the swallow already lets the terminate run, so the ordering changed nothing an `Exception` could observe; it only guarded a `BaseException` from the logging stack, which cannot reach the pool worker thread `run()` executes on. Mutation testing proved it inert, so it is not shipped rather than shipped behind a contrived test. - When the process-group lookup is *itself* what fails, the boundary buys the reap attempt and the handle release but not the kill: with no group recorded, `_posix_signal_subprocess` declines to signal rather than target a possibly recycled pid (the R5-9 convention). That is a pre-existing limitation of the signalling path, not something this change introduces, and the test for that path asserts only what actually happens. Recorded as a follow-up. Tests: 6 added. Every one mutation-checked -- 5 mutants, 0 survivors: the startup logging moved back outside the boundary, the process-group lookup moved back outside it, and each of the three reap log sites bypassing best-effort. The pre-existing 13 tests in the file pass unmodified. Full suite 884 passed; ruff, black, and mypy clean on native and --platform win32. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com> (cherry picked from commit acee655)
1 parent 26655b4 commit 5e73e7f

2 files changed

Lines changed: 382 additions & 64 deletions

File tree

‎src/openjd/sessions/_subprocess.py‎

Lines changed: 99 additions & 62 deletions
Original file line numberDiff line numberDiff line change
@@ -187,56 +187,71 @@ def run(self) -> None:
187187
self._callback()
188188
return
189189

190-
self._pid = self._process.pid
191-
192-
# Would use is_posix(), but it doesn't short-circuit mypy which then complains
193-
# about os.getpgid not being a valid attribute.
194-
if not sys.platform == "win32":
195-
# A trivial command can exit before we get here. Looking up the
196-
# process group of an already-reaped child raises ProcessLookupError
197-
# (ESRCH), which must not fail the action: the child ran, and its
198-
# exit code is still collected below.
199-
try:
200-
if not self._user or self._user.is_process_user():
201-
self._sudo_child_process_group_id = os.getpgid(self._process.pid)
202-
else:
203-
self._sudo_child_process_group_id = find_sudo_child_process_group_id(
204-
logger=self._logger,
205-
sudo_process=self._process,
190+
# Bind the Popen once, and enter the ownership `try` immediately: from here
191+
# on this method owns a live child, so every exit path -- including an
192+
# exceptional one -- has to go through the `finally` that reaps it.
193+
#
194+
# The process-group lookup and the two startup log lines used to sit
195+
# *before* the `try`, outside that guarantee. A `logging.Filter` that
196+
# raises (filters propagate out of `Logger.handle`, unlike handlers) then
197+
# escaped `run()` having never waited on the child: a live process with no
198+
# exit code recorded, while the runner published a terminal state for it.
199+
# Installing such a filter is a supported use of this library -- `Session`
200+
# installs its own. Reproduced before this change; see
201+
# TestStartupLoggingDoesNotAbandonTheChild.
202+
proc = self._process
203+
try:
204+
self._pid = proc.pid
205+
206+
# Would use is_posix(), but it doesn't short-circuit mypy which then complains
207+
# about os.getpgid not being a valid attribute.
208+
if not sys.platform == "win32":
209+
# A trivial command can exit before we get here. Looking up the
210+
# process group of an already-reaped child raises ProcessLookupError
211+
# (ESRCH), which must not fail the action: the child ran, and its
212+
# exit code is still collected below.
213+
try:
214+
if not self._user or self._user.is_process_user():
215+
self._sudo_child_process_group_id = os.getpgid(proc.pid)
216+
else:
217+
self._sudo_child_process_group_id = find_sudo_child_process_group_id(
218+
logger=self._logger,
219+
sudo_process=proc,
220+
)
221+
except ProcessLookupError:
222+
# R5-9 fix: leave this as None -- "no process group known" --
223+
# rather than substituting the pid. The lookup failed *because*
224+
# the process is gone, so its pid is dead: signalling it is at
225+
# best a no-op, and after pid recycling on a busy host `killpg`
226+
# would target an unrelated group. In the sudo branch the pid is
227+
# not even the right identifier, since it belongs to `sudo` and
228+
# not to the workload's process group.
229+
#
230+
# None is the established "unknown" value on this field --
231+
# find_sudo_child_process_group_id returns it for the same reason
232+
# (_linux/_sudo.py), and _posix_signal_subprocess already retries
233+
# the lookup and then declines to signal when it is still unset.
234+
self._sudo_child_process_group_id = None
235+
self._logger.debug(
236+
f"Process {proc.pid} exited before its process group could be "
237+
"determined; no process group will be signalled for it.",
238+
extra=LogExtraInfo(openjd_log_content=LogContent.PROCESS_CONTROL),
206239
)
207-
except ProcessLookupError:
208-
# R5-9 fix: leave this as None -- "no process group known" --
209-
# rather than substituting the pid. The lookup failed *because*
210-
# the process is gone, so its pid is dead: signalling it is at
211-
# best a no-op, and after pid recycling on a busy host `killpg`
212-
# would target an unrelated group. In the sudo branch the pid is
213-
# not even the right identifier, since it belongs to `sudo` and
214-
# not to the workload's process group.
215-
#
216-
# None is the established "unknown" value on this field --
217-
# find_sudo_child_process_group_id returns it for the same reason
218-
# (_linux/_sudo.py), and _posix_signal_subprocess already retries
219-
# the lookup and then declines to signal when it is still unset.
220-
self._sudo_child_process_group_id = None
221-
self._logger.debug(
222-
f"Process {self._process.pid} exited before its process group could be "
223-
"determined; no process group will be signalled for it.",
224-
extra=LogExtraInfo(openjd_log_content=LogContent.PROCESS_CONTROL),
225-
)
226240

227-
self._logger.info(
228-
f"Command started as pid: {self._process.pid}",
229-
extra=LogExtraInfo(openjd_log_content=LogContent.PROCESS_CONTROL),
230-
)
231-
self._logger.info(
232-
"Output:",
233-
extra=LogExtraInfo(openjd_log_content=LogContent.BANNER | LogContent.COMMAND_OUTPUT),
234-
)
241+
self._logger.info(
242+
f"Command started as pid: {proc.pid}",
243+
extra=LogExtraInfo(openjd_log_content=LogContent.PROCESS_CONTROL),
244+
)
245+
self._logger.info(
246+
"Output:",
247+
extra=LogExtraInfo(
248+
openjd_log_content=LogContent.BANNER | LogContent.COMMAND_OUTPUT
249+
),
250+
)
235251

236-
try:
237252
self._log_subproc_stdout() # Blocking
238-
self._process.wait()
239-
self._returncode = self._process.returncode
253+
proc.wait()
254+
self._returncode = proc.returncode
240255
self._log_returncode()
241256
if self._callback:
242257
self._callback()
@@ -245,7 +260,6 @@ def run(self) -> None:
245260
# PopenWindowsAsUser and there's stuff to deallocate. `proc` itself is
246261
# a function local and dies with the frame; a `del proc` here would be
247262
# a no-op, which is what CodeQL's "unnecessary delete" reports.
248-
proc = self._process
249263
self._process = None
250264
# This method owns the subprocess for its whole life, and clearing
251265
# `_process` is the point after which nothing else can reach it:
@@ -276,35 +290,31 @@ def _reap(self, proc: Optional[Popen]) -> None:
276290
# evil -- the alternative is an orphan holding the session directory
277291
# and, for a cross-user action, running as the job user with no
278292
# owner.
279-
self._logger.error(
293+
#
294+
# Every log call in this method goes through
295+
# `_log_error_best_effort`: the failure that routes us here is a
296+
# raising `logging.Filter`, so the log call in front of the kill was
297+
# itself the thing that skipped the kill.
298+
self._log_error_best_effort(
280299
f"Abandoning the subprocess {proc.pid} before it exited; terminating it so it "
281-
"is not orphaned.",
282-
extra=LogExtraInfo(
283-
openjd_log_content=LogContent.PROCESS_CONTROL | LogContent.EXCEPTION_INFO
284-
),
300+
"is not orphaned."
285301
)
286302
try:
287303
self._terminate_process(proc)
288304
except Exception as e: # noqa: BLE001
289-
self._logger.error(
290-
f"Could not terminate the abandoned subprocess {proc.pid}: {e}",
291-
extra=LogExtraInfo(
292-
openjd_log_content=LogContent.PROCESS_CONTROL | LogContent.EXCEPTION_INFO
293-
),
305+
self._log_error_best_effort(
306+
f"Could not terminate the abandoned subprocess {proc.pid}: {e}"
294307
)
295308
try:
296309
# Bounded: a SIGKILLed child is reaped promptly, and this runs on
297310
# the pool worker, so an unbounded wait here would hang the
298311
# runner's future rather than surface anything useful.
299312
proc.wait(timeout=ABANDONED_PROCESS_REAP_TIMEOUT_SECONDS)
300313
except TimeoutExpired:
301-
self._logger.error(
314+
self._log_error_best_effort(
302315
f"Subprocess {proc.pid} did not exit within "
303316
f"{ABANDONED_PROCESS_REAP_TIMEOUT_SECONDS}s of being terminated; it is left "
304-
"unreaped.",
305-
extra=LogExtraInfo(
306-
openjd_log_content=LogContent.PROCESS_CONTROL | LogContent.EXCEPTION_INFO
307-
),
317+
"unreaped."
308318
)
309319
# Record the exit status. Unconditional on purpose: `Popen.returncode` is
310320
# populated by the poll()/wait() above and is the authority either way --
@@ -317,6 +327,33 @@ def _reap(self, proc: Optional[Popen]) -> None:
317327
# it left every test passing. An unfalsifiable check is worse than none.
318328
self._returncode = proc.returncode
319329

330+
def _log_error_best_effort(self, message: str) -> None:
331+
"""Log ``message`` at ERROR, swallowing any failure of the logging stack
332+
itself.
333+
334+
:meth:`_reap` runs inside :meth:`run`'s ``finally``, so anything raised
335+
from it replaces the exception already propagating and hides the real
336+
cause. That containment was already in place for the terminate and the
337+
wait, but not for the log calls between them -- and the failure mode that
338+
routes us into ``_reap`` in the first place is a ``logging.Filter`` that
339+
raises, which makes those calls the *likeliest* thing to raise here, not
340+
the least.
341+
342+
Silently swallowing is normally the wrong answer. It is right here
343+
because the alternative is strictly worse on both counts: the caller
344+
loses the genuine exception, and the child is left unreaped.
345+
"""
346+
try:
347+
self._logger.error(
348+
message,
349+
extra=LogExtraInfo(
350+
openjd_log_content=LogContent.PROCESS_CONTROL | LogContent.EXCEPTION_INFO
351+
),
352+
)
353+
except Exception: # noqa: BLE001
354+
# Nowhere left to report this: the logger is what failed.
355+
pass
356+
320357
def notify(self) -> None:
321358
"""The 'Notify' part of Open Job Description's subprocess cancelation method.
322359
On Linux/macOS:

0 commit comments

Comments
 (0)