Python has a known limitation - it doesn't always return the memory to the OS.
In an environment that have memory constraints like kubernetes, it causes OOM unnecessarily.
To mitigate this, most frameworks restart the process after certain criteria is met: Celery long supports both worker_max_tasks_per_child and worker_max_memory_per_child, uvicorn has --limit-max-requests.
The current suggested implementation has a major flaw - it restarts all of the sub-processes instead of the specific process that reached the limit.
In multi-process configuration, this causes an unnecessary drainage to the other sub-processes, creating possible latencies until the drain complete. If I were to implement a memory limit middleware, that would be even worse because one sub-process reaching the limit doesn't correlate other sub-processes usage.
Unfortunately, the core of dramatiq doesn't allow this to be implemented: the only support is to restart all of the processes using the SIGHUP signal. A previous attempt #236 was rejected, where it didn't address the actual issues.
I would like to fix this by providing an API that will signal a worker restart.
Would appreciate the go-ahead before doing so.
I do have 2 possible implementation in mind: Creating a dedicate middleware exception or using a specific signal (SIGUSR2).
Python has a known limitation - it doesn't always return the memory to the OS.
In an environment that have memory constraints like kubernetes, it causes OOM unnecessarily.
To mitigate this, most frameworks restart the process after certain criteria is met: Celery long supports both
worker_max_tasks_per_childandworker_max_memory_per_child, uvicorn has--limit-max-requests.The current suggested implementation has a major flaw - it restarts all of the sub-processes instead of the specific process that reached the limit.
In multi-process configuration, this causes an unnecessary drainage to the other sub-processes, creating possible latencies until the drain complete. If I were to implement a memory limit middleware, that would be even worse because one sub-process reaching the limit doesn't correlate other sub-processes usage.
Unfortunately, the core of dramatiq doesn't allow this to be implemented: the only support is to restart all of the processes using the
SIGHUPsignal. A previous attempt #236 was rejected, where it didn't address the actual issues.I would like to fix this by providing an API that will signal a worker restart.
Would appreciate the go-ahead before doing so.
I do have 2 possible implementation in mind: Creating a dedicate middleware exception or using a specific signal (
SIGUSR2).