Skip to content

killing orphaned child processes - #2463

Merged
bghira merged 1 commit into
mainfrom
bugfix/orphan-pid-cleanup
Jan 21, 2026
Merged

killing orphaned child processes#2463
bghira merged 1 commit into
mainfrom
bugfix/orphan-pid-cleanup

Conversation

@bghira

@bghira bghira commented Jan 21, 2026

Copy link
Copy Markdown
Owner

This pull request improves the reliability and correctness of process termination and child process cleanup in the ProcessKeeper, especially for cases where subprocesses may outlive their parent or ignore termination signals. It also updates and unskips several tests to reflect and validate the improved behavior.

Process termination and child process handling:

  • Added a Linux /proc-based fallback method _collect_child_pids_from_proc to recursively collect all descendant PIDs when psutil is unavailable, ensuring that orphaned child processes are reliably found and terminated.
  • Refactored _force_kill_process_tree to collect all child PIDs before sending any signals, preventing orphaned GPU-hogging processes if the parent exits first. The method now uses either psutil or the new /proc fallback to find children, and then sends signals to all collected PIDs.
  • Updated the final cleanup step to use the collected list of child PIDs for sending SIGKILL, ensuring all descendants are forcefully terminated even if they have been reparented.

Test improvements and re-enabling:

  • Unskipped and improved several tests in tests/test_process_keeper.py to validate process termination, crash detection, and graceful shutdown. Tests now use polling loops to robustly detect process state changes, and timeouts are adjusted for more accurate measurement of graceful termination. [1] [2] [3]

@bghira
bghira merged commit b6c21c8 into main Jan 21, 2026
2 checks passed
@bghira
bghira deleted the bugfix/orphan-pid-cleanup branch January 21, 2026 03:28
samzhang996-hue pushed a commit to samzhang996-hue/SimpleTuner that referenced this pull request Apr 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant