You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 4199643
Browse filesBrowse the repository at this point in the historyBrowse files
MLX: release a temp slot once its chain is done with it (pytorch#23109)
`ExecutionState::tensors` is never pruned, so a chain ends holding every
intermediate it produced. Whether that costs anything depends on the AOT
memory plan: whisper's reuses a handful of slots, but a gemma4-e2b
prefill plan assigns roughly one slot per instruction, so its 3243
instructions finish with 1918 live tensors whose sizes scale with
sequence length.
Measured on an iPhone 16, gemma4-e2b int4, 2392-character prompt:
| | live slots | liveMB | peak phys_footprint | outcome |
| --- | --- | --- | --- | --- |
| stock | 1918 | 3102.9 MB | 3401 MB | killed by jetsam in prefill |
| this PR | 3 | 1.2 MB | 1379 MB | completes normally |
A 52-character prompt completes on both. Reported downstream as
software-mansion/react-native-executorch#1489.
Only temps are eligible: ids run `Constant -> Input -> Output ->
MutableBuffer -> Temp`, so `>= mutable_buffer_end` is neither a method
output nor a caller-visible buffer. A chain containing SCAN or IF opts
out, since `for_each_tid` does not descend into nested chains.
Independent of pytorch#22932's eval threshold: with the threshold off the
intermediates stay lazy and the growth reappears in the final
`async_eval`; with it on they are materialized and retained. This
release is what makes the bounded path bounded.
cc @nil-is-all@metascroy
---------
Co-authored-by: Scott Roy <161522778+metascroy@users.noreply.github.com>
0 commit comments