Repository navigation
fix(mlops): use spawn for feature ingestion - #6464
lucasjia-aws wants to merge 1 commit into
Conversation
Create the pandas ingestion pool from an explicit spawn context and move its signal initializer to module scope so it is picklable. Add focused coverage for pool configuration and async cleanup. Fixes aws#5312
There was a problem hiding this comment.
🤖 AI code review
This PR switches Feature Store pandas multi-process ingestion from the platform-default multiprocessing start method to an explicit spawn context, and moves the SIGINT-ignoring worker initializer from a nested closure to module scope so it is picklable under spawn. This directly addresses the fork-based deadlock reported in #5312. The change is correct: the staticmethod target and module-level initializer are picklable, the pre-existing from multiprocessing import Pool import is still required (used as the type annotation at line 71), and wait()'s close/join behavior is preserved. Tests cover spawn-context selection, process count, initializer identity/behavior, and pool cleanup, and are consistent with the implementation. No issues found.
Reviewed commit 3a6c142. Automated review; verify before acting.
Summary
Feature Store pandas ingestion currently creates a multiprocessing pool with the platform default start method and defines its initializer inside the calling method. Fork mode can deadlock on recent macOS releases, while the nested initializer is not safe under spawn.
This change creates the pool from an explicit spawn context and moves the signal initializer to module scope so it can be pickled. Existing asynchronous result, close, and join behavior is preserved.
Testing
git diff --checkpassed.sagemaker-mlopswheel and source distribution successfully.Related change
The companion V2 fix is published on #6465.
Fixes #5312