killing orphaned child processes - #2463
Merged
Merged
Conversation
samzhang996-hue
pushed a commit
to samzhang996-hue/SimpleTuner
that referenced
this pull request
Apr 27, 2026
killing orphaned child processes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request improves the reliability and correctness of process termination and child process cleanup in the
ProcessKeeper, especially for cases where subprocesses may outlive their parent or ignore termination signals. It also updates and unskips several tests to reflect and validate the improved behavior.Process termination and child process handling:
/proc-based fallback method_collect_child_pids_from_procto recursively collect all descendant PIDs whenpsutilis unavailable, ensuring that orphaned child processes are reliably found and terminated._force_kill_process_treeto collect all child PIDs before sending any signals, preventing orphaned GPU-hogging processes if the parent exits first. The method now uses eitherpsutilor the new/procfallback to find children, and then sends signals to all collected PIDs.SIGKILL, ensuring all descendants are forcefully terminated even if they have been reparented.Test improvements and re-enabling:
tests/test_process_keeper.pyto validate process termination, crash detection, and graceful shutdown. Tests now use polling loops to robustly detect process state changes, and timeouts are adjusted for more accurate measurement of graceful termination. [1] [2] [3]