Repository navigation
Playground: ask before loading models too big for phones and tablets - #24030
Conversation
Summary: On iPhones, loading a chat model in the ExecuTorch WebGPU Playground closes the Safari tab during download or GPU setup. iOS kills the tab for memory, so the page gets no error it could show. The chat models need far more memory than a phone gives one tab: the downloaded bytes, the runtime's copy of the program, the GPU weights, and the fp32 KV cache each program allocates for its full 8,960-token context (Qwen 3 0.6B: 2.06 GB; Gemma 3 1B: 0.48 GB). Before a demo starts loading, `memory-guard.js` estimates the peak: twice the download, plus the KV cache for chat models. On a phone or tablet (iPadOS included, which reports a Mac user agent), or on a device reporting 4 GB of memory or less, it shows a dialog when the estimate is over 400 MB. The dialog states the estimate, says phones often close the tab, and offers Cancel (focused) or Load anyway, once per model per page load. Today that covers the chat models (Qwen ~2.9 GB, Gemma ~1.8 GB) and Whisper (~0.5 GB); the vision demos stay under the limit and load as before. Desktops are unchanged. This is a guardrail, not a fix. Re-exporting the chat models with a shorter context would cut the KV cache itself. Differential Revision: D126470374
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24030
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 07b439a with merge base d46a886 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
|
@psiddh has exported this pull request. If you are a Meta employee, you can view the originating Diff in D126470374. |
This PR needs a
|
Summary:
On iPhones, loading a chat model in the ExecuTorch WebGPU Playground closes the Safari tab during download or GPU setup. iOS kills the tab for memory, so the page gets no error it could show. The chat models need far more memory than a phone gives one tab: the downloaded bytes, the runtime's copy of the program, the GPU weights, and the fp32 KV cache each program allocates for its full 8,960-token context (Qwen 3 0.6B: 2.06 GB; Gemma 3 1B: 0.48 GB).
Before a demo starts loading,
memory-guard.jsestimates the peak: twice the download, plus the KV cache for chat models. On a phone or tablet (iPadOS included, which reports a Mac user agent), or on a device reporting 4 GB of memory or less, it shows a dialog when the estimate is over 400 MB. The dialog states the estimate, says phones often close the tab, and offers Cancel (focused) or Load anyway, once per model per page load. Today that covers the chat models (Qwen ~2.9 GB, Gemma ~1.8 GB) and Whisper (~0.5 GB); the vision demos stay under the limit and load as before. Desktops are unchanged.This is a guardrail, not a fix. Re-exporting the chat models with a shorter context would cut the KV cache itself.
Differential Revision: D126470374