Clarify benchmark memory readings and preserve source runs

Author: barhanc

apps/benchmarks/README.md

@@ -173,8 +173,9 @@ post-processing in it.
 
 ## Publishing to the docs
 
-The docs benchmarks page reads `measured.json`; raw runs are only needed to
-regenerate it. Temporarily collect the runs under the page directory:
+The docs benchmarks page reads `measured.json`. Its source runs are committed
+under `docs/docs/04-benchmarks/runs/`, grouped by device. Add new runs there
+and regenerate the published data:
 
 ```bash
 mkdir -p ../../docs/docs/04-benchmarks/runs/s26ultra
@@ -184,14 +185,13 @@ yarn bench:docs   # regenerates docs/docs/04-benchmarks/measured.json
 
 The directory name is the device id the docs list it under (`DEVICES` in
 `docs/src/data/benchmarks.ts`). Generation replaces `measured.json`, so include
-all runs whose measurements it should retain. Archive the raw runs outside the
-repository and remove the temporary `runs/` directory before committing; the
-docs build only needs the generated JSON. Measured variants replace matching
+all runs whose measurements it should retain. Commit the raw runs alongside
+the generated JSON so others can reproduce it. Measured variants replace matching
 rows in `imported.json`, preserving other published results. To check a build
 against a release, compare against its archived runs:
 
 ```bash
-yarn bench:compare /path/to/archived/<run>.jsonl results/<new>.jsonl
+yarn bench:compare ../../docs/docs/04-benchmarks/runs/s26ultra/<run>.jsonl results/<new>.jsonl
 ```
 
 ## What gets measured

apps/benchmarks/scripts/docs-data.mjs

@@ -4,8 +4,8 @@
  *
  * To generate data, collect raw runs beside the docs page, one
  * directory per device: `runs/<deviceId>/*.jsonl`, where `<deviceId>` is the
- * id the docs list the device under. Only the generated JSON is needed by the
- * docs build; archive raw runs outside the repository after generation.
+ * id the docs list the device under. Commit the source runs alongside the
+ * generated JSON so the published data can be regenerated.
  *
  * This writes `measured.json` beside them. The page merges it with
  * `imported.json`, the published rows that have no raw runs behind them: a

docs/docs/04-benchmarks/01-benchmarks.mdx

@@ -52,6 +52,10 @@ Measurements were collected on physical iOS and Android devices spanning flagshi
 
 Memory values report peak process footprint during a separate inference pass, including the app baseline: total PSS on Android and `phys_footprint` on iOS. Some supplemental iPhone LLM measurements use a fresh process for each model.
 
+:::caution GPU memory accounting
+These sampled process counters do not measure total GPU memory or the peak during model loading. GPU backends can report unexpectedly low values: for example, MLX LFM2.5-VL-1.6B (int8) reports 216 MB despite a roughly 2.8 GB download, and Gemma4 E2B on Vulkan reports 327 MB on S20+ versus 3667 MB on S24. These discrepancies suggest incomplete or device-dependent memory accounting; the exact cause has not been established. Treat the figures as observed process-counter readings, not total model memory requirements or a reliable comparison of GPU memory across devices and backends. Sampling can also miss brief peaks.
+:::
+
 ## Results
 
 :::note Performance and Precision Trade-offs
@@ -183,7 +187,7 @@ These benchmarks measure execution latency, memory utilization, and initializati
 - **Variants:** Quantized LLMs only.
 - **Collection:** The S26 Ultra LLM results retain the September measurements. Supplemental S20+, S24, and iPhone 17 LLM runs were collected in October using the fixed-prompt harness described below.
 - **Input:** Prompt (`List three uses for a paperclip.`), 64 tokens generated with EOS ignored, temperature 0 (greedy decoding), and the KV cache reset before each independent prompt.
-- **Prefill vs. Decode & Memory:** The short prompt isolates autoregressive token generation throughput (tokens/second) and time to first token (TTFT). Autoregressive decoding executes one forward pass per generated token. Peak memory is driven by model weights and allocated key-value (KV) cache buffers, with models larger than 2 GB peaking between 2.4 and 3.5 GB of resident RAM.
+- **Prefill vs. Decode & Memory:** The short prompt isolates autoregressive token generation throughput (tokens/second) and time to first token (TTFT). Autoregressive decoding executes one forward pass per generated token. Memory requirements include model weights, allocated key-value (KV) cache buffers, and intermediate tensors. Observed process memory reaches 4122 MB for Qwen3 4B on S20+; GPU readings have the accounting limitations described above.
 :::
 
 On iPhone 17, Phi-4 Mini was omitted. Qwen3 4B (XNNPACK) terminated after critical memory-pressure warnings, including in a fresh-process retry. On Galaxy S24, Android's low-memory killer terminated Gemma4 Omni E2B (Vulkan) during loading in a fresh-process retry. Only successful measurements are charted.

docs/docs/04-benchmarks/runs/iphone17/v0.11.0-missing-20261026-ios-iPhone18-3.jsonl

@@ -0,0 +1,30 @@
+{
+  "events": [
+    {
+      "caseId": "llm/llama3-2-3-b-xnnpack-spinquant",
+      "event": "process-terminated",
+      "signal": 9,
+      "precedingConsoleWarning": "TRIM_MEMORY_RUNNING_CRITICAL",
+      "consoleWarningTime": "2026-10-26 18:53:21",
+      "assessment": "Memory pressure suspected; no matching Jetsam report was available through devicectl. A memory leak was not established.",
+      "recovery": "Relaunched the same release app in a fresh process; resumed from saved measurements. The retried case completed successfully."
+    },
+    {
+      "caseId": "llm/qwen3-4-b-xnnpack-8-da4-w",
+      "event": "process-terminated",
+      "signal": 9,
+      "precedingConsoleWarning": "TRIM_MEMORY_RUNNING_CRITICAL",
+      "consoleWarningTime": "2026-10-26 19:16:17",
+      "assessment": "Memory pressure suspected. A repeated-load leak has not been established.",
+      "recovery": "Fresh-process retry also terminated with signal 9 immediately after a critical memory warning. Recorded as a failed attempt; no timing published."
+    }
+  ],
+  "remainingCases": "Fresh process per model after each saved measurement.",
+  "skippedByUserRequest": ["llm/phi4-mini-xnnpack-8-da4-w", "llm/phi4-mini-mlx-int4"],
+  "completion": {
+    "successfulVariants": 53,
+    "failedVariants": ["llm/qwen3-4-b-xnnpack-8-da4-w"],
+    "skippedByUserRequest": ["llm/phi4-mini-xnnpack-8-da4-w", "llm/phi4-mini-mlx-int4"],
+    "allRequestedCasesAccountedFor": true
+  }
+}

docs/docs/04-benchmarks/runs/iphone17/v0.11.0-missing-20261026-ios-iPhone18-3.notes.json

@@ -0,0 +1,4 @@
+{"type":"run","label":"v0.11.0-missing-20261026-isolated","platform":"ios","device":{"model":"iPhone18,3","totalMemoryBytes":8044216320,"isEmulator":false,"soc":"iPhone18,3","cpuCores":6,"osVersion":"26.6"},"inputSpecVersion":1,"buildType":"release","clocksPinned":false,"settings":{"suite":"llm/gemma4-omni-e2-b-mlx-int4,llm/phi4-mini-xnnpack-8-da4-w,llm/qwen3-4-b-mlx-int4,llm/lfm2-5-vl-1-6-b-mlx-int8,llm/gemma4-e2-b-xnnpack-8-da4-w,llm/phi4-mini-mlx-int4,llm/gemma4-omni-e2-b-xnnpack-8-da4-w","iterations":20,"warmup":3,"memoryIterations":5,"loadIterations":3,"repeats":1,"maxTempC":37,"maxBytes":6000000000,"urlMap":{}},"startedAt":"2026-10-26T18:28:38.113Z"}
+{"id":"llm/gemma4-omni-e2-b-mlx-int4","task":"llm","model":"llm.GEMMA4_OMNI_E2B.MLX_INT4","backend":"mlx","precision":"int4","bytes":2760555316,"note":"independent single-user chat prompt; KV cache reset each iteration; 64 generated tokens with EOS ignored; multimodal models are driven text-only","mode":"async","progress":{"caseIndex":1,"caseCount":7,"repeat":1,"repeats":1},"status":"ok","downloadMs":40170.667,"taskLoadMs":780.654,"taskLoad":{"count":3,"min":755.83,"max":2001.963,"mean":1179.482,"median":780.654,"p90":2001.963,"iqr":1246.132,"stdev":581.67},"pipeline":{"count":20,"min":1818.928,"max":2115.368,"mean":1927.589,"median":1891.813,"p90":2055.131,"iqr":149.211,"stdev":95.293},"execution":{"perIteration":{"token_embedding":{"count":65,"ms":18.33401405},"text_decoder":{"count":65,"ms":1882.6945878499998}},"totalMs":1901.0286019},"units":64,"detail":{"timeToFirstTokenMs":109.192,"prefillMs":109,"promptTokens":17},"gate":{"kind":"device","waitedS":0,"temperatureC":null,"thermalStatus":0,"timedOut":false},"thermal":{"batteryTemperatureC":-1,"statusName":"none","status":0},"thermalPeak":{"status":0,"statusName":"none","batteryTemperatureC":null,"samples":423},"thermalValid":true,"holdsTimedOut":0,"coolingMs":508,"memory":{"baselineMb":41.31,"loadedMb":749.39,"peakMb":612.31,"disposedMb":471.3}}
+{"id":"llm/phi4-mini-xnnpack-8-da4-w","task":"llm","model":"llm.PHI4_MINI.XNNPACK_8DA4W","backend":"xnnpack","precision":"8da4w","bytes":2833963001,"status":"skipped","mode":"async","error":"Skipped on iPhone at user request due to RAM concerns.","progress":{"caseIndex":2,"caseCount":7,"repeat":1,"repeats":1},"downloadMs":0,"taskLoadMs":0}
+{"id":"llm/phi4-mini-mlx-int4","task":"llm","model":"llm.PHI4_MINI.MLX_INT4","backend":"mlx","precision":"int4","bytes":3283172473,"status":"skipped","mode":"async","error":"Skipped on iPhone at user request due to RAM concerns.","progress":{"caseIndex":6,"caseCount":7,"repeat":1,"repeats":1},"downloadMs":0,"taskLoadMs":0}

docs/docs/04-benchmarks/runs/iphone17/v0.11.0-missing-20261026-isolated-ios-iPhone18-3.jsonl

@@ -0,0 +1,15 @@
+{
+  "procedure": "Each saved measurement is followed by app termination and relaunch. Completed cases are skipped by the collector. Timing and memory iteration counts are unchanged.",
+  "launches": [
+    {
+      "caseId": "llm/gemma4-omni-e2-b-mlx-int4",
+      "procedure": "fresh process before this case",
+      "launchedAtUtc": "2026-10-26T18:28:38.181524+00:00"
+    },
+    {
+      "caseId": "llm/phi4-mini-xnnpack-8-da4-w",
+      "procedure": "fresh process before this case",
+      "launchedAtUtc": "2026-10-26T18:30:20.602376+00:00"
+    }
+  ]
+}

docs/docs/04-benchmarks/runs/iphone17/v0.11.0-missing-20261026-isolated-ios-iPhone18-3.notes.json

@@ -0,0 +1,5 @@
+{"type":"run","label":"v0.11.0-missing-20261026-isolated-no-phi4","platform":"ios","device":{"model":"iPhone18,3","osVersion":"26.6","totalMemoryBytes":8044216320,"soc":"iPhone18,3","isEmulator":false,"cpuCores":6},"inputSpecVersion":1,"buildType":"release","clocksPinned":false,"settings":{"suite":"llm/qwen3-4-b-mlx-int4,llm/lfm2-5-vl-1-6-b-mlx-int8,llm/gemma4-e2-b-xnnpack-8-da4-w,llm/gemma4-omni-e2-b-xnnpack-8-da4-w","iterations":20,"warmup":3,"memoryIterations":5,"loadIterations":3,"repeats":1,"maxTempC":37,"maxBytes":6000000000,"urlMap":{}},"startedAt":"2026-10-26T18:34:24.444Z"}
+{"id":"llm/qwen3-4-b-mlx-int4","task":"llm","model":"llm.QWEN3_4B.MLX_INT4","backend":"mlx","precision":"int4","bytes":2926890664,"note":"independent single-user chat prompt; KV cache reset each iteration; 64 generated tokens with EOS ignored; multimodal models are driven text-only","mode":"async","progress":{"caseIndex":1,"caseCount":4,"repeat":1,"repeats":1},"status":"ok","downloadMs":55305.088,"taskLoadMs":236.253,"taskLoad":{"count":3,"min":227.815,"max":494.229,"mean":319.433,"median":236.253,"p90":494.229,"iqr":266.414,"stdev":123.648},"pipeline":{"count":20,"min":2677.706,"max":3321.982,"mean":2871.853,"median":2828.863,"p90":3144.445,"iqr":301.764,"stdev":192.734},"execution":{"perIteration":{"forward":{"count":65,"ms":2847.97120225}},"totalMs":2847.97120225},"units":64,"detail":{"timeToFirstTokenMs":180.916,"prefillMs":180,"promptTokens":16},"gate":{"kind":"device","waitedS":0,"temperatureC":null,"thermalStatus":0,"timedOut":false},"thermal":{"statusName":"none","batteryTemperatureC":-1,"status":0},"thermalPeak":{"status":1,"statusName":"light","batteryTemperatureC":null,"samples":1452},"thermalValid":true,"holdsTimedOut":0,"coolingMs":90485,"memory":{"baselineMb":43.8,"loadedMb":839.99,"peakMb":815.55,"disposedMb":342.81}}
+{"id":"llm/lfm2-5-vl-1-6-b-mlx-int8","task":"llm","model":"llm.LFM2_5_VL_1_6B.MLX_INT8","backend":"mlx","precision":"int8","bytes":2960510731,"note":"independent single-user chat prompt; KV cache reset each iteration; 64 generated tokens with EOS ignored; multimodal models are driven text-only","mode":"async","progress":{"caseIndex":2,"caseCount":4,"repeat":1,"repeats":1},"status":"ok","downloadMs":62401.724,"taskLoadMs":92.751,"taskLoad":{"count":3,"min":89.299,"max":125.618,"mean":102.556,"median":92.751,"p90":125.618,"iqr":36.32,"stdev":16.368},"pipeline":{"count":20,"min":1487.359,"max":1622.929,"mean":1514.596,"median":1504.482,"p90":1536.406,"iqr":33.035,"stdev":32.024},"execution":{"perIteration":{"token_embedding":{"count":65,"ms":11.83152035},"text_decoder":{"count":65,"ms":1492.30833605}},"totalMs":1504.1398564},"units":64,"detail":{"timeToFirstTokenMs":69.798,"prefillMs":70,"promptTokens":17},"gate":{"kind":"device","waitedS":0,"temperatureC":null,"thermalStatus":0,"timedOut":false},"thermal":{"statusName":"light","batteryTemperatureC":-1,"status":1},"thermalPeak":{"status":1,"statusName":"light","batteryTemperatureC":null,"samples":335},"thermalValid":true,"holdsTimedOut":0,"coolingMs":390,"memory":{"baselineMb":40.13,"loadedMb":200.05,"peakMb":216.44,"disposedMb":216.42}}
+{"id":"llm/gemma4-e2-b-xnnpack-8-da4-w","task":"llm","model":"llm.GEMMA4_E2B.XNNPACK_8DA4W","backend":"xnnpack","precision":"8da4w","bytes":3247469492,"note":"independent single-user chat prompt; KV cache reset each iteration; 64 generated tokens with EOS ignored; multimodal models are driven text-only","mode":"async","progress":{"caseIndex":3,"caseCount":4,"repeat":1,"repeats":1},"status":"ok","downloadMs":62591.546,"taskLoadMs":2150.484,"taskLoad":{"count":3,"min":2085.863,"max":2258.045,"mean":2164.797,"median":2150.484,"p90":2258.045,"iqr":172.182,"stdev":71.018},"pipeline":{"count":20,"min":5547.365,"max":6848.167,"mean":6280.152,"median":6377.982,"p90":6771.84,"iqr":567.203,"stdev":396.557},"execution":{"perIteration":{"forward":{"count":65,"ms":6251.0382644500005}},"totalMs":6251.0382644500005},"units":64,"detail":{"timeToFirstTokenMs":255.627,"prefillMs":255,"promptTokens":17},"gate":{"kind":"device","waitedS":70,"temperatureC":null,"thermalStatus":0,"timedOut":false},"thermal":{"status":0,"batteryTemperatureC":-1,"statusName":"none"},"thermalPeak":{"status":1,"statusName":"light","batteryTemperatureC":null,"samples":4165},"thermalValid":true,"holdsTimedOut":0,"coolingMs":310702,"memory":{"baselineMb":42.44,"loadedMb":2073.32,"peakMb":1933.71,"disposedMb":602.72}}
+{"id":"llm/gemma4-omni-e2-b-xnnpack-8-da4-w","task":"llm","model":"llm.GEMMA4_OMNI_E2B.XNNPACK_8DA4W","backend":"xnnpack","precision":"8da4w","bytes":3651367476,"note":"independent single-user chat prompt; KV cache reset each iteration; 64 generated tokens with EOS ignored; multimodal models are driven text-only","mode":"async","progress":{"caseIndex":4,"caseCount":4,"repeat":1,"repeats":1},"status":"ok","downloadMs":55922.123,"taskLoadMs":2636.022,"taskLoad":{"count":3,"min":2623.778,"max":3718.426,"mean":2992.742,"median":2636.022,"p90":3718.426,"iqr":1094.648,"stdev":513.16},"pipeline":{"count":20,"min":5548.318,"max":6812.044,"mean":6195.531,"median":6248.999,"p90":6657.515,"iqr":625.424,"stdev":377.182},"execution":{"perIteration":{"token_embedding":{"count":65,"ms":6.79186505},"text_decoder":{"count":65,"ms":6158.18550375}},"totalMs":6164.9773688000005},"units":64,"detail":{"timeToFirstTokenMs":281.066,"prefillMs":280,"promptTokens":17},"gate":{"kind":"device","waitedS":70,"temperatureC":null,"thermalStatus":0,"timedOut":false},"thermal":{"batteryTemperatureC":-1,"status":1,"statusName":"light"},"thermalPeak":{"status":1,"statusName":"light","batteryTemperatureC":null,"samples":4268},"thermalValid":true,"holdsTimedOut":0,"coolingMs":324604,"memory":{"baselineMb":42.08,"loadedMb":2528.11,"peakMb":2393.77,"disposedMb":711.27}}

docs/docs/04-benchmarks/runs/iphone17/v0.11.0-missing-20261026-isolated-no-phi4-ios-iPhone18-3.jsonl

@@ -0,0 +1,25 @@
+{
+  "procedure": "Each saved measurement is followed by app termination and relaunch. Completed cases are skipped by the collector. Timing and memory iteration counts are unchanged.",
+  "launches": [
+    {
+      "caseId": "llm/qwen3-4-b-mlx-int4",
+      "procedure": "fresh process before this case",
+      "launchedAtUtc": "2026-10-26T18:34:25.131877+00:00"
+    },
+    {
+      "caseId": "llm/lfm2-5-vl-1-6-b-mlx-int8",
+      "procedure": "fresh process before this case",
+      "launchedAtUtc": "2026-10-26T18:38:15.791811+00:00"
+    },
+    {
+      "caseId": "llm/gemma4-e2-b-xnnpack-8-da4-w",
+      "procedure": "fresh process before this case",
+      "launchedAtUtc": "2026-10-26T18:40:05.264007+00:00"
+    },
+    {
+      "caseId": "llm/gemma4-omni-e2-b-xnnpack-8-da4-w",
+      "procedure": "fresh process before this case",
+      "launchedAtUtc": "2026-10-26T18:50:33.720441+00:00"
+    }
+  ]
+}