Skip to content

Commit 3a2e6fe

Browse files
committed
Simplify benchmark memory caveat
1 parent efd3a09 commit 3a2e6fe

1 file changed

Lines changed: 2 additions & 2 deletions

File tree

‎docs/docs/04-benchmarks/01-benchmarks.mdx‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -53,7 +53,7 @@ Measurements were collected on physical iOS and Android devices spanning flagshi
5353
Memory values report peak process footprint during a separate inference pass, including the app baseline: total PSS on Android and `phys_footprint` on iOS. Some supplemental iPhone LLM measurements use a fresh process for each model.
5454

5555
:::caution GPU memory accounting
56-
These sampled process counters do not measure total GPU memory or the peak during model loading. GPU backends can report unexpectedly low values: for example, MLX LFM2.5-VL-1.6B (int8) reports 216 MB despite a roughly 2.8 GB download, and Gemma4 E2B on Vulkan reports 327 MB on S20+ versus 3667 MB on S24. These discrepancies suggest incomplete or device-dependent memory accounting; the exact cause has not been established. Treat the figures as observed process-counter readings, not total model memory requirements or a reliable comparison of GPU memory across devices and backends. Sampling can also miss brief peaks.
56+
Memory figures reflect sampled process usage during inference. They may under-report GPU memory usage and miss brief peaks, and they exclude peaks during model loading. Memory accounting also varies across devices and backends, so use these values as a guide rather than as minimum RAM requirements or a direct comparison of GPU memory usage.
5757
:::
5858

5959
## Results
@@ -187,7 +187,7 @@ These benchmarks measure execution latency, memory utilization, and initializati
187187
- **Variants:** Quantized LLMs only.
188188
- **Collection:** The S26 Ultra LLM results retain the September measurements. Supplemental S20+, S24, and iPhone 17 LLM runs were collected in October using the fixed-prompt harness described below.
189189
- **Input:** Prompt (`List three uses for a paperclip.`), 64 tokens generated with EOS ignored, temperature 0 (greedy decoding), and the KV cache reset before each independent prompt.
190-
- **Prefill vs. Decode & Memory:** The short prompt isolates autoregressive token generation throughput (tokens/second) and time to first token (TTFT). Autoregressive decoding executes one forward pass per generated token. Memory requirements include model weights, allocated key-value (KV) cache buffers, and intermediate tensors. Observed process memory reaches 4122 MB for Qwen3 4B on S20+; GPU readings have the accounting limitations described above.
190+
- **Prefill vs. Decode & Memory:** The short prompt isolates autoregressive token generation throughput (tokens/second) and time to first token (TTFT). Autoregressive decoding executes one forward pass per generated token. Memory requirements include model weights, allocated key-value (KV) cache buffers, and intermediate tensors.
191191
:::
192192

193193
On iPhone 17, Phi-4 Mini was omitted. Qwen3 4B (XNNPACK) terminated after critical memory-pressure warnings, including in a fresh-process retry. On Galaxy S24, Android's low-memory killer terminated Gemma4 Omni E2B (Vulkan) during loading in a fresh-process retry. Only successful measurements are charted.

0 commit comments

Comments
 (0)