You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 3a2e6fe
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: docs/docs/04-benchmarks/01-benchmarks.mdx
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -53,7 +53,7 @@ Measurements were collected on physical iOS and Android devices spanning flagshi
53
53
Memory values report peak process footprint during a separate inference pass, including the app baseline: total PSS on Android and `phys_footprint` on iOS. Some supplemental iPhone LLM measurements use a fresh process for each model.
54
54
55
55
:::caution GPU memory accounting
56
-
These sampled process counters do not measure total GPU memory or the peak during model loading. GPU backends can report unexpectedly low values: for example, MLX LFM2.5-VL-1.6B (int8) reports 216 MB despite a roughly 2.8 GB download, and Gemma4 E2B on Vulkan reports 327 MB on S20+ versus 3667 MB on S24. These discrepancies suggest incomplete or device-dependent memory accounting; the exact cause has not been established. Treat the figures as observed process-counter readings, not total model memory requirements or a reliable comparison of GPU memory across devices and backends. Sampling can also miss brief peaks.
56
+
Memory figures reflect sampled process usage during inference. They may under-report GPU memory usage and miss brief peaks, and they exclude peaks during model loading. Memory accounting also varies across devices and backends, so use these values as a guide rather than as minimum RAM requirements or a direct comparison of GPU memory usage.
57
57
:::
58
58
59
59
## Results
@@ -187,7 +187,7 @@ These benchmarks measure execution latency, memory utilization, and initializati
187
187
-**Variants:** Quantized LLMs only.
188
188
-**Collection:** The S26 Ultra LLM results retain the September measurements. Supplemental S20+, S24, and iPhone 17 LLM runs were collected in October using the fixed-prompt harness described below.
189
189
-**Input:** Prompt (`List three uses for a paperclip.`), 64 tokens generated with EOS ignored, temperature 0 (greedy decoding), and the KV cache reset before each independent prompt.
190
-
-**Prefill vs. Decode & Memory:** The short prompt isolates autoregressive token generation throughput (tokens/second) and time to first token (TTFT). Autoregressive decoding executes one forward pass per generated token. Memory requirements include model weights, allocated key-value (KV) cache buffers, and intermediate tensors. Observed process memory reaches 4122 MB for Qwen3 4B on S20+; GPU readings have the accounting limitations described above.
190
+
-**Prefill vs. Decode & Memory:** The short prompt isolates autoregressive token generation throughput (tokens/second) and time to first token (TTFT). Autoregressive decoding executes one forward pass per generated token. Memory requirements include model weights, allocated key-value (KV) cache buffers, and intermediate tensors.
191
191
:::
192
192
193
193
On iPhone 17, Phi-4 Mini was omitted. Qwen3 4B (XNNPACK) terminated after critical memory-pressure warnings, including in a fresh-process retry. On Galaxy S24, Android's low-memory killer terminated Gemma4 Omni E2B (Vulkan) during loading in a fresh-process retry. Only successful measurements are charted.
0 commit comments