Repository navigation
perf: preload the app and fork production granian workers - #7079
Conversation
On Linux the production backend now forces the "fork" start method, imports the app in the supervisor, and freezes the GC before granian spawns workers, so every worker shares the framework and app pages copy-on-write instead of re-importing ~1,800 modules privately. A forked child inherits the telemetry ThreadPoolExecutor without its worker thread, so events submitted in a worker would never be sent; an at-fork hook drops the inherited pool so the child creates its own. REFLEX_BACKEND_START_METHOD overrides the start method for apps that are not fork-safe. Blank app, GRANIAN_WORKERS=4, median of 3 runs (PSS over the process tree from /proc/<pid>/smaps_rollup after serving requests): before: 721 MB PSS, 692 MB private dirty, port up in 1.82 s after: 223 MB PSS, 70 MB private dirty, port up in 1.07 s Claude-Session: https://claude.ai/code/session_01CEb3ocfFLeHkjkAKKH8YCy
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
There was a problem hiding this comment.
All reported issues were addressed across 7 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Merging this PR will improve performance by 4.3%
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ⚡ |
test_get_state[memory] |
1.2 ms | 1.1 ms | +4.3% |
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing FarhanAliRaza:farhan/prod-fork-preload (d6cc137) with main (f615739)
masenf
left a comment
There was a problem hiding this comment.
nice improvement.
the telemetry stuff does slightly make me nervous though. i think it's probably worth it to defer sending the "run" telemetry until after the fork and attempt to drain the executor (which theoretically should be empty on the run path if we defer the cli telemetry call). forking with threads is more than just a warning, it can lead to weird and hard-to-debug issues; if we can avoid it, then we should
tests/units/test_telemetry.py: kept the PR's after-fork executor test and main's urllib _send_event tests. tests/units/utils/test_exec.py: took main's file, added the PR's prod fork/preload tests and gc/prerequisites imports; main's prod JSON-logs test now pins _backend_start_method to None so it does not fork/preload. Claude-Session: https://claude.ai/code/session_01VDzz2un9mtjmCY24zSGmca
Address the review asking not to fork granian workers while a telemetry thread is alive: - run_backend_prod, run_frontend_prod, run_granian_backend_prod and run_uvicorn_backend_prod take an on_started callback; `reflex run --env prod` sends the "run-prod" event from it instead of before the server starts. Granian calls it from Server.startup() once the workers are spawned; uvicorn/gunicorn runs in a separate interpreter, so it is called right before that process is launched. - Before forking, the supervisor finishes Plotly serializer setup (_prepare_serializers_for_fork) and drains and shuts down the telemetry executor, so no telemetry thread exists at the fork. - The at-fork child hook also recreates the executor creation lock, so a fork taken while another thread held it cannot block the child's first telemetry send forever. Claude-Session: https://claude.ai/code/session_01VDzz2un9mtjmCY24zSGmca
_shutdown_executor() waited on the telemetry executor without a limit, so a send stalled on DNS (urlopen's timeout does not cover getaddrinfo) or a slow event collector could block production startup forever. It now waits up to two seconds through _flush(), then cancels what is still queued and stops the pool without waiting for a stalled send; the forked children reset the executor and its lock through the at-fork hook. Claude-Session: https://claude.ai/code/session_01VDzz2un9mtjmCY24zSGmca
There was a problem hiding this comment.
All reported issues were addressed across 6 files (changes from recent commits).
You’re at about 92% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
_shutdown_executor() cleared _executor before the old pool had stopped, so a send() racing the pre-fork shutdown could start a replacement pool whose thread was then alive at the fork. The pool now stays published until it has stopped (a racing submit fails and is suppressed) and is only then cleared. The shutdown test now checks the threads of the pool it stops instead of every "reflex-telemetry" thread in the process, and the fork-reset tests stop the pools they orphan, so the check no longer depends on test order. Claude-Session: https://claude.ai/code/session_01VDzz2un9mtjmCY24zSGmca
After the bounded drain, a stalled telemetry send (e.g. blocked in getaddrinfo) keeps its thread alive, and that thread can hold process-wide locks such as the resolver's across a fork. Forking then trades a startup hang for a worker hang. _shutdown_executor() now reports whether the pool's thread actually stopped, _preload_for_fork() passes that on, and run_granian_backend_prod() starts its workers with "spawn" instead of "fork" when it did not, so the supervisor never forks with a live telemetry thread and startup waits at most the drain timeout. Claude-Session: https://claude.ai/code/session_01VDzz2un9mtjmCY24zSGmca
Brings in reflex-dev#7079's merge with main and its telemetry/fork fix commit. reflex/utils/exec.py: kept main's ParentBoundGranian dev supervisor followed by this PR's fork-only _freeze_for_fork(); _preload_for_fork keeps reflex-dev#7079's serializer prep and telemetry drain and freezes via _freeze_for_fork(); run_backend now calls set_dev_start_method() before main's fork-preload check so backend-only runs follow the same rule. tests/units/test_reflex.py: kept both sides' imports and tests. tests/units/utils/test_exec.py: took the prod-fork-preload file and added this PR's dev start-method/freeze tests; main's run_backend preload tests now stub set_start_method and clear REFLEX_BACKEND_START_METHOD. Claude-Session: https://claude.ai/code/session_01VDzz2un9mtjmCY24zSGmca
Brings in reflex-dev#7079's bounded, race-free pre-fork telemetry shutdown (F5-F7); no conflicts. The dev fork path's _freeze_for_fork() uses the same bounded _shutdown_executor(). Claude-Session: https://claude.ai/code/session_01VDzz2un9mtjmCY24zSGmca
Brings in reflex-dev#7079's spawn fallback when the pre-fork telemetry drain leaves a live thread. reflex/utils/exec.py: _freeze_for_fork() (this PR's shared drain+freeze helper) now returns whether the telemetry thread stopped and skips the freeze when it did not; _preload_for_fork() returns its result, matching reflex-dev#7079's bool contract. tests/units/utils/test_exec.py (auto-merged): the dev freeze test's _shutdown_executor stub now reports a stopped thread. Claude-Session: https://claude.ai/code/session_01VDzz2un9mtjmCY24zSGmca
There was a problem hiding this comment.
All reported issues were addressed across 4 files (changes from recent commits).
You’re at about 93% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
…test run_granian_backend_prod() imported the app before selecting the start method, so multiprocessing objects the app creates at import time used the interpreter default context (forkserver on Python 3.14 Linux) rather than the requested one. The requested method is now set first, and only switched to "spawn" when the telemetry drain leaves a live thread. The stalled-drain test now waits until the worker is running the stalled task before shutting down, and joins the worker instead of calling result() on a future that shutdown may have cancelled. Claude-Session: https://claude.ai/code/session_01VDzz2un9mtjmCY24zSGmca
Once _shutdown_executor() had stopped the pool, a send() racing the pending fork could create a fresh pool and thread that granian then forked. The shutdown now pauses pool creation (checked under the executor lock; such sends are dropped with a debug log) until the next fork, which lifts it in the parent through an after_in_parent hook and in the child through the existing after_in_child reset. A caller that keeps forking can pass resume_after_fork=False to stay paused. When the workers are forked, the supervisor stops the pool again after on_started has sent "run-prod", so granian respawns do not fork while a send is in flight. Claude-Session: https://claude.ai/code/session_01VDzz2un9mtjmCY24zSGmca
Brings in reflex-dev#7079's F9-F11 fixes (start method set before the preload, deflaked stalled-drain test, pause on telemetry pool creation until the fork); no conflicts. Claude-Session: https://claude.ai/code/session_01VDzz2un9mtjmCY24zSGmca
The supervisor that forks granian workers also forks again to respawn them, so it must never hold a telemetry thread: a "run-prod" send that stalled (DNS, the lscpu collector) after on_started kept one alive across those respawns, and the earlier bounded re-shutdown could not help. Once the supervisor drains and stops its pool before the first fork, it now stays paused for good (in-process sends are dropped with a debug log; forked children reset it). on_started hands "run-prod" to telemetry._send_detached(), which runs the event collection and delivery in a fresh interpreter: Popen with close_fds (so the server sockets stay in the supervisor) runs a snippet that forks once and exits, so the supervisor reaps its direct child at once and the sender is reparented to init. No process starts when telemetry is disabled. The after-fork parent hook and the post-callback shutdown are gone. Claude-Session: https://claude.ai/code/session_01VDzz2un9mtjmCY24zSGmca
Windows has no os.fork, so the detaching snippet cannot run there; the supervisor never forks on that platform either, so send in-process. Claude-Session: https://claude.ai/code/session_01VDzz2un9mtjmCY24zSGmca
tests/units/test_reflex.py: kept main's MockerFixture import and the PR's reflex_base.constants import; both sides' tests kept. Claude-Session: https://claude.ai/code/session_01VDzz2un9mtjmCY24zSGmca
Brings in reflex-dev#7079's merge of main (reflex-dev#7350 JSON output supervisor). Claude-Session: https://claude.ai/code/session_01VDzz2un9mtjmCY24zSGmca
Comments Outside DiffThese findings could not be posted inline.
|
`reflex run --json` re-runs itself below an output supervisor; as a container's PID 1 (e.g. `CMD exec reflex run --json`) that supervisor inherits the double-forked sender but only waits for its own child, so the sender stayed a zombie for the server's lifetime. Orphans always go to PID 1 (absent a subreaper), so whenever PID 1 is our own parent the sender may linger: _reaps_orphans() now also returns True for os.getppid() == 1, which makes the server send attached and reap the child itself. This covers reflex's output supervisor and any other non-reaping PID 1 parent; a false positive (e.g. a service under systemd) only costs a bounded wait in the supervisor. The ordinary-process test no longer depends on the runner's PIDs or subreaper flag: it fakes getpid/getppid and prctl. Claude-Session: https://claude.ai/code/session_01VDzz2un9mtjmCY24zSGmca
|
CodSpeed on 2562af4, the merge of main, flags
Update for 1e45ee1: Generated by Claude Code |
There was a problem hiding this comment.
All reported issues were addressed across 2 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
aebe6ec sent attached under any PID 1 parent, which includes a working init such as systemd, where the detached sender is reaped fine and the attached send only made the supervisor wait. The PID 1 parent case now requires the REFLEX_OUTPUT_SUPERVISED marker, i.e. the parent is the `reflex run --json` output supervisor, which only waits for its own child. PID 1 itself and child subreapers are unchanged. Claude-Session: https://claude.ai/code/session_01VDzz2un9mtjmCY24zSGmca
On Windows _send_detached sends in-process and never calls Popen, so the test's popen.call_args lookup raised. It now carries the same os.fork guard as the other detached-sender tests; running the module with os.fork deleted shows every test passing or skipping. Claude-Session: https://claude.ai/code/session_01VDzz2un9mtjmCY24zSGmca
|
Fixed: Greptile's outside-diff finding "Detached sender becomes a zombie" (
Covered by:
Generated by Claude Code |
Brings in reflex-dev#7079's F17-F20 fixes to the detached run-event sender and its tests; no conflicts. Claude-Session: https://claude.ai/code/session_01VDzz2un9mtjmCY24zSGmca
|
Proposed test-only fix, for main (the test only checks the warning, not expiry): - state_manager_redis.lock_expiration = LOCK_EXPIRATION
+ # Only the warning is under test: leave the lock enough headroom that a
+ # stalled runner cannot expire it while the substates are written.
+ state_manager_redis.lock_expiration = LOCK_EXPIRATION * 4With it, the test passes with and without the injected pause, still with 7 warnings. #6853, which is open, checks the lock once per write, which would also remove this window. I can't re-run jobs from here (403), so this job needs a maintainer re-run: https://github.com/reflex-dev/reflex/actions/runs/36825635724/job/110250781652 Generated by Claude Code |
tests/units/utils/test_exec.py: kept both sides' imports (gc and logging; prerequisites/telemetry and PrecompressedStaticFiles) and both test blocks (the PR's granian prod fork tests and main's routes-manifest tests). Claude-Session: https://claude.ai/code/session_01VDzz2un9mtjmCY24zSGmca
Brings in reflex-dev#7079's merge of main (reflex-dev#6996 routes-manifest SPA fallback) and its shortened news fragment; no conflicts. Claude-Session: https://claude.ai/code/session_01VDzz2un9mtjmCY24zSGmca
|
On 048718e, This is a transient GitHub Releases error, not this PR. The same leg passed on #7113, which contains this head. The Windows lock-test flake from my earlier comment didn't recur here; every Generated by Claude Code |
masenf
left a comment
There was a problem hiding this comment.
love the fork, hate the on_startup
…rted Address masenf's review: the supervisor sends the run-prod event on its own thread before granian starts, so no telemetry thread exists at the fork and nothing needs to run after the workers start. This removes the on_started callback threaded through run_backend_prod, run_frontend_prod, run_uvicorn_backend_prod and run_granian_backend_prod, the NotifyingGranian subclass, telemetry._send_detached with its detached sender script, _reaps_orphans and the _paused flag. The pre-fork drain (_shutdown_executor) and the at-fork executor reset stay: the compile event is still queued on the thread pool, and a forked child must not inherit a pool whose worker thread it does not own. Claude-Session: https://claude.ai/code/session_01GbqrTagh776xkQvsopdytF
There was a problem hiding this comment.
All reported issues were addressed across 6 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
View guided diff | Re-trigger cubic
…eflex-base A backend-only production run skips the compile event, so its run-prod event is the only one that can carry the one-time $create_alias migration. Send it through _process_event, and deliver the alias on the calling thread with _send so a process about to fork still starts no telemetry thread. From the worker thread this changes nothing but the order: the alias now precedes the event it rides on instead of being queued behind it. REFLEX_BACKEND_START_METHOD is new in reflex-base, so reflex's floor on it moves to 0.10.1.dev0, the next version after the reflex-base-v0.10.0 tag. Claude-Session: https://claude.ai/code/session_01GbqrTagh776xkQvsopdytF
There was a problem hiding this comment.
All reported issues were addressed across 5 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
View guided diff | Re-trigger cubic
|
On d6cc137,
I can't re-run jobs from here (403), so this leg needs a maintainer re-run: https://github.com/reflex-dev/reflex/actions/runs/38471814293/job/115552873697 Generated by Claude Code |
…ender Resolve the conflicts with reflex-dev#7079 as merged: keep main's synchronous run-prod send through telemetry._process_event, and drop this branch's copy of the earlier design (on_started plumbing, NotifyingGranian, telemetry._send_detached, _reaps_orphans and the paused flag) together with its tests. The dev fork changes stay on top of main: set_dev_start_method() before the compile pool and in run_granian_backend, _freeze_for_fork() shared by the production preload and the dev reload supervisor, and the fallback that spawns the dev reload worker when the telemetry drain leaves a live thread.
Summary
run_granian_backend_prodnow forces theforkstart method, imports the app in the supervisor, and runsgc.collect(); gc.freeze()before granian starts workers. Workers share the framework and app pages copy-on-write instead of each re-importing them.REFLEX_BACKEND_START_METHOD(fork/spawn/forkserver) overrides the choice for apps that are not fork-safe. Unset keeps the interpreter default on macOS and Windows (spawn), so nothing changes there.reflex run --env prodsends itsrun-prodevent synchronously on the calling thread (telemetry._send) before the server starts, so it never touches the thread pool. Right before forking, the supervisor drains whatever is still queued (the compile event, queued before the frontend build) and joins the pool's thread (telemetry._shutdown_executor()). A send that stalls past the 2 s drain, e.g. in a DNS lookup, leaves a live thread, and the supervisor then starts its workers withspawninstead of forking. Nothing in the supervisor sends telemetry after that point, so a SIGHUP respawn forks from a thread-free process too._prepare_serializers_for_fork()before forking.ThreadPoolExecutorbut not its worker thread, so any event submitted from a worker would queue forever. Anos.register_at_forkhook drops the inherited pool in the child and recreates its creation lock, which a thread creating the pool at fork time would otherwise leave held. This also affected dev mode on Python <= 3.13, whereforkwas already the Linux default.Measurement
One-page app (a counter
rx.State),GRANIAN_WORKERS=4,reflex run --env prod --backend-only, telemetry off, Python 3.14, granian 2.8.1. PSS summed over the whole process tree from/proc/<pid>/smaps_rollup, after 32/pingrequests, median of 3 runs, measured against current main (which includes #7049)./pingOK afterPer worker: 66 MB PSS → 27 MB PSS. The supervisor goes from 48 MB to 34 MB PSS, and the tree drops the multiprocessing forkserver (16 MB) and resource tracker (7 MB) that main starts under Python 3.14's default
forkservermethod.Functional check: 8 socket.io sessions against the forked workers, each hydrating and sending two
incrementevents, all receivingcount1 then 2. A SIGHUP reload respawned all four workers and/pinganswered 32/32 before and after it.Notes for review
run-prodsend is bounded byurlopen's 5 s timeout, the same as every other CLI telemetry event before the thread pool existed.waitpidand hold no locks.https://claude.ai/code/session_01GbqrTagh776xkQvsopdytF