docs: Complete quantized LLM, depth and ColBERT benchmarks

apps/benchmarks/BENCHMARK_SPEC.md

@@ -39,9 +39,9 @@ Per variant, per repeat:
 | `memory.disposed`        | Footprint after `dispose` — a leak is a case that never returns to baseline |
 
 `memory.*` is process footprint, not native heap: `phys_footprint` on iOS (what
-jetsam measures an app against) and total PSS on Android. Both count the
-resident pages of a memory-mapped `.pte`, which the heap counters miss entirely
-and which is most of a model's cost.
+jetsam measures an app against) and total PSS on Android. Raw results record the
+process baseline separately, so process memory can be distinguished from a
+model's incremental footprint.
 
 Memory is sampled in a separate pass from the timings. Reading total PSS on
 Android walks `/proc/self/smaps` and costs milliseconds; polling that during a
@@ -148,14 +148,16 @@ would move for reasons unrelated to the device.
 `src/inputs.ts` is the definition; `INPUT_SPEC_VERSION` is recorded in every
 report and the comparator refuses to diff across a change to it.
 
-| Task                                                                                               | Input                                                                                                                                                 |
-| -------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
-| Vision (classification, detection, segmentation, style transfer, OCR, keypoints, image embeddings) | A fixed 512x512 or 640x640 scene: a vertical gradient, three solid ellipses, and a seeded dither                                                      |
-| Text embeddings, privacy filter                                                                    | Fixed prose, with the PII text seeded with the entity types a detector should find                                                                    |
-| VAD, speech-to-text                                                                                | 10 s of a voice-shaped waveform: a 130 Hz pulse train through three formant resonators, 0.7 s bursts alternating with 0.3 s of near-silence           |
-| Text-to-speech                                                                                     | Fixed text; the voice is the alphabetically first the config publishes, so the choice is a property of the voice set rather than of declaration order |
-| Text-to-image                                                                                      | Fixed prompt, fixed seed                                                                                                                              |
-| LLM                                                                                                | Fixed short prompt, **64 tokens decoded with EOS ignored and temperature 0**                                                                          |
+| Task                                                                                                                 | Input                                                                                                                                                 |
+| -------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Vision (classification, depth estimation, detection, segmentation, style transfer, OCR, keypoints, image embeddings) | A fixed 512x512 or 640x640 scene: a vertical gradient, three solid ellipses, and a seeded dither                                                      |
+| ColBERT embeddings                                                                                                   | The fixed text-embedding paragraph, encoded as a document; query encoding and MaxSim scoring are not timed                                            |
+| Image inpainting                                                                                                     | The fixed 512x512 RGBA scene with a central 256x256 hole (25% of the image), marked by a gray mask; includes resizing, inference, and blending        |
+| Text embeddings, privacy filter                                                                                      | Fixed prose, with the PII text seeded with the entity types a detector should find                                                                    |
+| VAD, speech-to-text                                                                                                  | 10 s of a voice-shaped waveform: a 130 Hz pulse train through three formant resonators, 0.7 s bursts alternating with 0.3 s of near-silence           |
+| Text-to-speech                                                                                                       | Fixed text; the voice is the alphabetically first the config publishes, so the choice is a property of the voice set rather than of declaration order |
+| Text-to-image                                                                                                        | Fixed prompt, fixed seed                                                                                                                              |
+| LLM                                                                                                                  | Fixed short prompt, **64 tokens decoded with EOS ignored and temperature 0**                                                                          |
 
 Two caveats worth knowing before reading the output:
 
@@ -166,7 +168,7 @@ Two caveats worth knowing before reading the output:
 - **LLMs.** Decode is pinned to a fixed token count because generation length is
   otherwise a property of the model and its quantisation, and a wall-clock
   figure over a varying token count compares nothing. Divide `pipeline.median`
-  by 64 for milliseconds per token; `detail.timeToFirstTokenMs` and
+  by the recorded generated-token count (`units`) for milliseconds per token; `detail.timeToFirstTokenMs` and
   `detail.prefillMs` are reported separately. Multimodal models are driven
   text-only.
 

apps/benchmarks/README.md

@@ -173,8 +173,9 @@ post-processing in it.
 
 ## Publishing to the docs
 
-The docs benchmarks page is built from raw runs stored beside it, so each docs
-version keeps the runs its numbers came from:
+The docs benchmarks page reads `measured.json`. Its source runs are committed
+under `docs/docs/04-benchmarks/runs/`, grouped by device. Add new runs there
+and regenerate the published data:
 
 ```bash
 mkdir -p ../../docs/docs/04-benchmarks/runs/s26ultra
@@ -183,12 +184,14 @@ yarn bench:docs   # regenerates docs/docs/04-benchmarks/measured.json
 ```
 
 The directory name is the device id the docs list it under (`DEVICES` in
-`docs/src/data/benchmarks.ts`). A device with runs replaces its rows in
-`imported.json`, which holds published numbers that have no raw runs behind
-them. To check a build against a release, compare against its archived runs:
+`docs/src/data/benchmarks.ts`). Generation replaces `measured.json`, so include
+all runs whose measurements it should retain. Commit the raw runs alongside
+the generated JSON so others can reproduce it. Measured variants replace matching
+rows in `imported.json`, preserving other published results. To check a build
+against a release, compare against its archived runs:
 
 ```bash
-yarn bench:compare ../../docs/versioned_docs/version-0.11.0/04-benchmarks/runs/s26ultra/<run>.jsonl results/<new>.jsonl
+yarn bench:compare ../../docs/docs/04-benchmarks/runs/s26ultra/<run>.jsonl results/<new>.jsonl
 ```
 
 ## What gets measured
@@ -322,6 +325,6 @@ a **model or variant** means nothing here beyond `yarn bench:sizes`.
 
 `modules/bench-probe` is a local Expo module reading the process footprint:
 `task_vm_info.phys_footprint` on iOS (what jetsam measures an app against) and
-total PSS on Android. Both count the resident pages of a memory-mapped `.pte`,
-which the native-heap counters miss entirely — and that is most of a model's
-footprint.
+total PSS on Android. These are process-footprint counters, rather than
+native-heap totals. The raw results also record the baseline before loading and
+the footprint after disposal.

apps/benchmarks/app.json

@@ -29,8 +29,7 @@
         "foregroundImage": "./assets/icons/adaptive-icon.png",
         "backgroundColor": "#ffffff"
       },
-      "package": "com.anonymous.benchmarks",
-      "usesCleartextTraffic": true
+      "package": "com.anonymous.benchmarks"
     },
     "web": {
       "favicon": "./assets/icons/favicon.png"
@@ -40,13 +39,15 @@
         "expo-build-properties",
         {
           "android": {
-            "minSdkVersion": 26
+            "minSdkVersion": 26,
+            "usesCleartextTraffic": true
           },
           "ios": {
             "deploymentTarget": "17.0"
           }
         }
-      ]
+      ],
+      "./plugins/withBenchmarkBundleInputs"
     ]
   }
 }

apps/benchmarks/package.json

@@ -24,25 +24,27 @@
     "bench:compare": "node scripts/compare.mjs",
     "bench:summary": "node scripts/summarize.mjs",
     "bench:sizes": "node scripts/fetch-variant-sizes.mjs",
-    "bench:docs": "node scripts/docs-data.mjs"
+    "bench:docs": "node scripts/docs-data.mjs",
+    "test": "node --test scripts/*.test.mjs"
   },
   "dependencies": {
-    "expo": "~56.0.9",
-    "expo-build-properties": "~56.0.17",
-    "expo-keep-awake": "~56.0.3",
+    "expo": "~57.0.28",
+    "expo-build-properties": "~57.0.22",
+    "expo-keep-awake": "~57.0.2",
     "react": "19.2.3",
-    "react-native": "0.85.3",
+    "react-native": "0.86.3",
     "react-native-blob-util": "^0.25.0",
     "react-native-executorch": "workspace:*",
     "react-native-safe-area-context": "~5.7.0",
-    "react-native-worklets": "0.10.3"
+    "react-native-worklets": "0.13.0"
   },
   "devDependencies": {
-    "@babel/core": "^7.29.0",
+    "@babel/core": "^7.29.7",
     "@react-native/metro-config": "^0.86.0",
     "@types/react": "~19.2.0",
-    "babel-preset-expo": "~56.0.14",
-    "react-refresh": "^0.14.0"
+    "babel-preset-expo": "~57.0.15",
+    "react-refresh": "^0.14.0",
+    "typescript": "~5.9.2"
   },
   "private": true
 }

apps/benchmarks/plugins/withBenchmarkBundleInputs.js

@@ -0,0 +1,18 @@
+const { withAppBuildGradle } = require('expo/config-plugins');
+
+// React Native's bundle task tracks source files, but Expo's inlined benchmark
+// settings come from the environment. Give Gradle those inputs as well so a
+// release APK cannot reuse another device's collector or validation settings.
+module.exports = (config) =>
+  withAppBuildGradle(config, (modConfig) => {
+    const marker = '// Benchmark environment inputs';
+    if (!modConfig.modResults.contents.includes(marker)) {
+      modConfig.modResults.contents += `
+${marker}
+tasks.withType(com.facebook.react.tasks.BundleHermesCTask).configureEach {
+    inputs.property("benchmarkEnvironment", providers.environmentVariablesPrefixedBy("EXPO_PUBLIC_BENCH_"))
+}
+`;
+    }
+    return modConfig;
+  });

apps/benchmarks/scripts/docs-data.mjs

@@ -2,15 +2,14 @@
 /**
  * Builds the docs benchmark data from raw runs.
  *
- * The raw runs are the source of truth and live beside the docs page, one
+ * To generate data, collect raw runs beside the docs page, one
  * directory per device: `runs/<deviceId>/*.jsonl`, where `<deviceId>` is the
- * id the docs list the device under. A docs version cut copies them along with
- * the page, so every release keeps the runs its numbers came from, and
- * `bench:compare` can diff a new run against any of them.
+ * id the docs list the device under. Commit the source runs alongside the
+ * generated JSON so the published data can be regenerated.
  *
  * This writes `measured.json` beside them. The page merges it with
  * `imported.json`, the published rows that have no raw runs behind them: a
- * device with runs here replaces its imported rows entirely.
+ * measured variant replaces its matching imported row.
  *
  * Usage:
  *   yarn bench:docs                       # ../../docs/docs/04-benchmarks
@@ -67,8 +66,19 @@ function toRow(deviceId, entries) {
   };
 
   if (task === 'llm') {
-    // `units` is the runner's generated-token count over a pinned-length
-    // generation: 63 for 64 new tokens, the first coming from prefill.
+    if (
+      entries.some(
+        (entry) =>
+          !Number.isFinite(entry.units) ||
+          entry.units <= 0 ||
+          !Number.isFinite(entry.detail?.timeToFirstTokenMs) ||
+          entry.detail.timeToFirstTokenMs < 0
+      )
+    ) {
+      throw new Error(`LLM measurement ${first.id} is missing valid token or TTFT stats`);
+    }
+    // Use the runner's actual generated-token count for the fixed-length
+    // generation; older runtime exports counted the first token separately.
     row.tokensPerSecond = round(first.units / (medianMs / 1000), 1);
     row.ttftMs = round(pick((entry) => entry.detail?.timeToFirstTokenMs));
   } else {

apps/benchmarks/scripts/docs-data.test.mjs

@@ -0,0 +1,86 @@
+import assert from 'node:assert/strict';
+import { spawnSync } from 'node:child_process';
+import { mkdtempSync, mkdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { test } from 'node:test';
+import { fileURLToPath } from 'node:url';
+import { runInNewContext } from 'node:vm';
+import { ModuleKind, ScriptTarget, transpileModule } from 'typescript';
+
+test('partial device runs replace matching variants and retain other published results', () => {
+  const exports = {};
+  const source = readFileSync(
+    new URL('../../../docs/src/data/benchmarks.ts', import.meta.url),
+    'utf8'
+  );
+  const code = transpileModule(source, {
+    compilerOptions: { module: ModuleKind.CommonJS, target: ScriptTarget.ES2022 },
+  }).outputText;
+  runInNewContext(code, { exports });
+  const old = {
+    device: 's20plus',
+    task: 'llm',
+    model: 'test',
+    backend: 'xnnpack',
+    precision: '8da4w',
+    medianMs: 10,
+  };
+  const fresh = { ...old, medianMs: 5 };
+  const otherDevice = { ...old, device: 's26ultra' };
+  const otherTask = { ...old, task: 'classification' };
+  const otherBackend = { ...old, backend: 'vulkan' };
+  const otherPrecision = { ...old, precision: '8da8w' };
+  const rows = exports.mergeBenchmarkRows(
+    [fresh],
+    [old, otherDevice, otherTask, otherBackend, otherPrecision]
+  );
+  assert.equal(rows.length, 5);
+  assert.equal(rows[0], fresh);
+  assert.equal(rows[1], otherDevice);
+  assert.equal(rows[2], otherTask);
+  assert.equal(rows[3], otherBackend);
+  assert.equal(rows[4], otherPrecision);
+});
+
+test('docs generation publishes valid LLM throughput and rejects missing token stats', () => {
+  const page = mkdtempSync(join(tmpdir(), 'rne-docs-test-'));
+  try {
+    const dir = join(page, 'runs', 'iphone17');
+    mkdirSync(dir, { recursive: true });
+    const file = join(dir, 'test.jsonl');
+    const header = { type: 'run', label: 'test', device: { model: 'test' } };
+    const entry = {
+      id: 'llm/test-xnnpack-8-da4-w',
+      status: 'ok',
+      backend: 'xnnpack',
+      precision: '8da4w',
+      pipeline: { mean: 100, median: 100 },
+      units: 64,
+      detail: { timeToFirstTokenMs: 5 },
+      taskLoadMs: 10,
+      bytes: 1024 ** 2,
+      memory: { peakMb: 20 },
+    };
+    const convert = () =>
+      spawnSync(
+        process.execPath,
+        [fileURLToPath(new URL('./docs-data.mjs', import.meta.url)), page],
+        { encoding: 'utf8' }
+      );
+    const write = (measurement) =>
+      writeFileSync(file, `${JSON.stringify(header)}\n${JSON.stringify(measurement)}\n`);
+    write(entry);
+    const valid = convert();
+    assert.equal(valid.status, 0, valid.stderr);
+    const [row] = JSON.parse(readFileSync(join(page, 'measured.json'), 'utf8'));
+    assert.equal(row.tokensPerSecond, 640);
+    assert.equal(row.ttftMs, 5);
+    write({ ...entry, units: undefined });
+    const invalid = convert();
+    assert.equal(invalid.status, 2);
+    assert.match(invalid.stderr, /missing valid token or TTFT stats/);
+  } finally {
+    rmSync(page, { recursive: true, force: true });
+  }
+});

apps/benchmarks/scripts/fetch-variant-sizes.mjs

@@ -51,6 +51,10 @@ function loadRegistry() {
       )
       .replace("} from './extensions/speech/tasks/whisperSpeechToText';", "} from './_stub.ts';")
       .replace("} from './constants';", "} from './constants.ts';")
+      .replace(
+        "import { registerDeprecation } from './deprecation';",
+        "import { registerDeprecation } from './_stub.ts';"
+      )
       // Type-only imports name files that are not copied into the scratch dir.
       // Node strips the annotations but still resolves the specifier, so they
       // have to go.
@@ -70,6 +74,7 @@ function loadRegistry() {
         "export const getRegisteredBackends = () => ['XnnpackBackend', 'CoreMLBackend', 'MLXBackend', 'VulkanBackend'];",
         "export const WHISPER_LANGUAGES = ['en'];",
         'export const WhisperSttModel = undefined;',
+        'export const registerDeprecation = () => {};',
         '',
       ].join('\n')
     );
@@ -119,12 +124,19 @@ async function measure(urls, cached) {
     for (let url = queue.pop(); url !== undefined; url = queue.pop()) {
       try {
         const response = await fetch(url, { method: 'HEAD', redirect: 'follow' });
+        if (!response.ok) throw new Error(`HTTP ${response.status} for ${url}`);
         // Hugging Face serves LFS objects through a redirect and reports the
         // real object size in this header; Content-Length on the redirect hop
         // describes the pointer, not the file.
         const linked = response.headers.get('x-linked-size');
         const length = response.headers.get('content-length');
         sizes[url] = Number(linked ?? length ?? 0) || 0;
+        // Small JSON resources can be served without Content-Length.
+        if (sizes[url] === 0 && new URL(url).pathname.endsWith('.json')) {
+          const body = await fetch(url);
+          if (!body.ok) throw new Error(`HTTP ${body.status} for ${url}`);
+          sizes[url] = (await body.arrayBuffer()).byteLength;
+        }
       } catch {
         sizes[url] = 0;
       }