Repository navigation
MLX backend: large, deterministic logit divergence between Apple Silicon Mac and iPhone on identical inputs (~0.5x logit compression, ~18% answer flips) #1379
Description
Description
TL;DR. We run yes/no visual questions against LFM2.5-VL-1.6B (MLX backend, int4 .pte) and read p("yes") from first-answer-token logprobs. The same model file, same image bytes, same prompt, greedy decoding produce systematically different logits on an iPhone 17 Pro than on an M3 Pro Mac — both platforms perfectly deterministic individually, but the iPhone's logits come out compressed to roughly half of the Mac's, which flips ~18% of near-threshold answers. We've ruled out our image pipeline (details below) and believe this happens inside the MLX-delegated forward pass. We'd like to know if this magnitude is known/expected and whether there's a precision knob we're missing.
A single concrete example (one photo, prompt Is this a photo of a retail purchase receipt? Answer yes or no., greedy, identical everything):
| first-token logprobs | No↔Yes margin | p_yes | |
|---|---|---|---|
| Mac (M3 Pro) | No: −0.003, Yes: −5.941 |
5.9 nats | 0.003 |
| iPhone 17 Pro | No: −0.915, Yes: −1.415 |
0.5 nats | 0.361 |
Same answer here, but the margin collapsed ~12×. On rows where the Mac is less confident, this collapse crosses 0.5 and the answer flips (e.g. another row: Mac p_yes 0.234 → iPhone 0.642). Across our 268-row benchmark the effect is a clean linear map, not noise:
logit_device ≈ 0.52 + 0.50 · logit_mac (R² = 0.85, n = 129 single-call rows;
the full 268-row fit including multi-call recipes: a = 0.22, b = 0.54, R² = 0.73)
Expected: cross-device numeric differences on the order of fp16 rounding, no systematic answer flips.
Actual: a directional ~0.5× logit compression, always toward uncertainty on the phone.
Setup (exact pins)
- iPhone side: react-native-executorch 0.9.3 in-app. The MLX delegate executing the model is the unmodified prebuilt ExecutorchLib binary that 0.9.3 ships — we do not touch it.
- Mac side: a small CLI that compiles RNE 0.9.3's unmodified
common/runner sources (samevision_encoder.cpp, same OpenCV preprocessing) againstsoftware-mansion-labs/executorch@24f0f69— the fork commit RNE 0.9.3's prebuilt binary was built from (the newest fork commit preceding RNEbca2a54, which last updated the binary; the later "compiled-execution" MLX update10f535e4/28d8af6is not in 0.9.3 and not in our build). - Model:
lfm_2_5_vl_1_6b_mlx_int4.ptefromsoftware-mansion/react-native-executorch-lfm-2.5@v0.9.0, same file both sides; same tokenizer files. - Decoding: greedy on both sides (temperature 0, topP 1, minP 0, repetitionPenalty 1).
- Disclosure: both sides carry a small patch to the runner sources (not the delegate): (a) a backport of RNE main's fp16-input dtype fix in
vision_encoder— stable 0.9.3 fails prefill with InvalidArgument on this MLX VL export without it; (b) a read-only tap that reports top-k logprobs of the first answer token. Determinism results below show the tap doesn't perturb decisions.
We're aware the Mac CLI is not a supported configuration — but the claim here is not "the Mac is right." It's that the same delegated graph, built from the same commit, executed on two Apple GPUs, disagrees with itself far beyond fp16-rounding levels, deterministically. The Mac just makes the second data point easy to instrument.
What we observed
- Answer agreement Mac vs iPhone: 81.7% (219/268), mean |Δp_yes| = 0.135; the same 49 rows flip on every run.
- Both platforms individually deterministic: two device runs 10 days apart — across an iOS update (26.5.2 → 26.6) and an app rebuild — gave 268/268 identical answers, max |Δp_yes| = 0.0023. The Mac CLI is exactly reproducible run-to-run.
What we ruled out before blaming the backend
- Pixels (the decisive experiment). We captured the exact JPEGs the device feeds its encoder and fed those same bytes to the Mac CLI. The cross-feed reproduced the Mac's probabilities (mean |Δp| = 0.037, divergence-vs-Mac slope ≈ 0 flat), not the device's (mean |Δp| = 0.124, slope −0.5). Of 20 disagreeing rows in that subset, the cross-feed sided with the Mac 17 times. The gap does not travel with the image bytes.
- Image decode. Identical JPEGs decode near-identically on both sides (max per-pixel diff 3/255). Much larger upstream pixel perturbations showed zero correlation with answer flips.
- Build drift. Both binaries rebuilt from one clean commit of our repo; the gap reproduced to the row.
- Sampling. Greedy both sides; determinism above.
Questions
- Is a systematic ~0.5× logit compression between M-series and A-series GPUs a known behavior of the MLX backend (fp16 accumulation, different kernel paths, tile sizes…)? It is much larger than fp16 rounding folklore suggests, and it is directional.
- Is there a knob we're missing — higher-precision accumulation, fp32 fallback for sensitive ops, or a vision-encoder precision option?
- If this belongs with the ExecuTorch MLX delegate or MLX kernels rather than RNE, we're happy to file upstream — a pointer to the right repo/owners would help.
We can share the full minimal repro (the JPEG above + prompt + complete top-64 logprob dumps from both platforms, and the 268-row paired dataset) — happy to attach or run further diagnostics on our hardware.
Steps to reproduce
- Export LFM2.5-VL-1.6B to
.ptewith the MLX backend (int4), or use the publishedsoftware-mansion/react-native-executorch-lfm-2.5@v0.9.0artifact. - Run a yes/no visual prompt with greedy decoding on an iPhone 17 Pro via react-native-executorch 0.9.3 and capture first-answer-token logprobs.
- Run the identical image bytes + prompt through the same runner sources built against
software-mansion-labs/executorch@24f0f69on an Apple Silicon Mac (or any second Apple GPU). - Compare first-token logprobs: device logits come out compressed ~0.5× relative to the Mac; near-threshold answers flip. Repeating either side reproduces its own outputs exactly.
Snack or a link to a repository
No response (private repo; minimal repro artifacts — JPEG, prompt, both full logprob dumps — available on request)
React Native Executorch version
0.9.3
React Native version
0.81.5
Platforms
iOS
JavaScript runtime
Hermes
Workflow
Expo Dev Client
Architecture
Fabric (New Architecture)
Build type
Release mode
Device
Real device
Device model
iPhone 17 Pro (iOS 26.6); compared against MacBook Pro M3 Pro (macOS)
AI model
LFM2.5-VL-1.6B (lfm_2_5_vl_1_6b_mlx_int4.pte)
Performance logs
No response
Acknowledgements
Yes
Activity
vinceecws commentedon Aug 24, 2026
Follow-up — cross-backend control run. We repeated the same 268-row benchmark on the same two machines with the XNNPACK export (
lfm_2_5_vl_1_6b_xnnpack_8da4w.ptefrom the published registry; runner sources and prompts identical, greedy).Result: the directional compression does not appear on CPU. Regressing
(logit_iphone − logit_mac)onlogit_mac:So the ~0.5× logit shrink between M3 Pro and A19 Pro is specific to the MLX delegate's execution, not a general property of running this model on two different Apple chips.
For completeness/honesty: XNNPACK is not bit-identical across the two machines either — 0/267 rows have exactly equal p_yes, with mean |Δlogit| ≈ 0.73 of non-directional per-row noise (which still flips knife-edge answers). One caveat on that noise figure: our macOS XNNPACK build enables KleidiAI and different threading than the iOS prebuilt, so kernel selection may account for part of it. The MLX slope, by contrast, is directional, large, and survives every control we've run.
Both XNNPACK reports (full per-row logprobs) are available alongside the MLX ones if useful.
vinceecws commentedon Aug 24, 2026
Update — Mac reference rebuilt at the prebuilt's exact source state; results bit-identical.
To close a validity hole in our own comparison, we fingerprinted the ExecutorchLib binary shipped in 0.9.3 (symbol inventory + metallib kernel set + build-path strings) to determine which
software-mansion-labs/executorchcommit it was actually built from. Findings, in case they're useful to you:5133162("Free MLX temp tensors at last use") — it containscompute_temp_lifetimesbut none of the compiled-execution strings from10f535e. The xcframework blob is unchanged from RNE140bf84(feat: ExecuTorch 1.3 with MLX (iOS) and Vulkan (Android) backends + Gemma 4 E2B #1223) through thev0.9.3tag.ce45c52) are identical between that commit and its neighbors.Our Mac CLI had been built from
24f0f69, one C++ commit earlier. We rebuilt it at5133162— the phone's exact MLX C++ state — and re-ran the full 268-row benchmark: bit-identical output to the previous Mac build (0/268 rows changed by any amount). So the temp-lifetime/floor_divide delta is numerically neutral on macOS, and the ~0.5× logit compression reported above is now measured between two builds of the same MLX delegate source, differing only in target (macOS/M3 Pro vs iOS/A19 Pro).Combined with the XNNPACK control (flat slope, previous comment), we're fairly confident the effect lives in MLX GPU execution differences between the two chips (kernel selection/precision), not in source-version skew, our pipeline, or CPU-side code.
msluszniak commentedon Aug 24, 2026
Hi @vinceecws,
Thank you for letting us know. I will investigate this one.
vinceecws commentedon Aug 25, 2026
Additional data point: the logit compression slope is prompt-dependent.
Fitting
logit_device = a + b·logit_macper prompt revealsbis not a constant of the hardware pair:b= 0.32, 0.44, 0.35 for the three prompts respectively. Same pixels, same run, only the prompt text differs.bfrom 0.32 to 0.57 depending on prompt, while any single (prompt, dataset) pair reproduces its own slope deterministically.Implications for triage:
The remaining hypotheses we can't distinguish from our side are (a) reduced-precision attention softmax on the A-series kernel path vs (b) int4 dequant scale math at reduced precision. A non-int4 MLX export run on both machines would separate them (slope flattens → dequant path; persists → general kernel precision), we're happy to run that on our hardware if an 8-bit MLX artifact of LFM2.5-VL exists or can be exported. Per-prompt paired logprob sets available on request.
msluszniak commentedon Aug 25, 2026
@vinceecws I can export INT8 variant for you to check if it works. I'll ping you once it will be published on HF. Otherwise, I will research this case thoroughly.
vinceecws commentedon Aug 27, 2026
@msluszniak
That would be perfect. Thank you. As soon as it's up we'll run the identical paired protocol on both machines (M3 Pro CLI + iPhone 17 Pro, greedy, same runner sources) and report the per-prompt slope fits alongside the int4 numbers.
For clarity on how we'll read the result: if the slope flattens toward 1.0 on INT8, that points at the int4 dequant scale math on the A-series kernel path; if the ~0.4–0.5 compression persists, it's upstream kernel precision (attention/softmax path) independent of weight dtype.
One note on where this lands for us in production: we ship the .pte to end users as an in-app download, so INT8 (~2× the 1.4 GB int4 artifact, plus ~2× weight memory and bandwidth-bound decode cost) is a diagnostic rather than a deployment path on our side. If the ablation does implicate the dequant path, higher-precision dequant math (or a precision knob) within the int4 artifact would be the fix that helps us most.
msluszniak commentedon Aug 27, 2026
@vinceecws the model is available here: https://huggingface.co/software-mansion/react-native-executorch-lfm-2.5/tree/v0.10.0/vl_1_6b/mlx.
msluszniak commentedon Sep 8, 2026
Hi @vinceecws, Have you tried the linked model to compare it with int4 results?
msluszniak commentedon Sep 25, 2026
@vinceecws on iPhone 17 Pro (iOS ≥ 26.2), MLX runs prefill attention and quantized matmuls on neural-accelerator (NAX) kernels. M3 never takes that path. Could you rerun the paired set with
setenv("MLX_METAL_GPU_ARCH", "applegpu_g17p", 1)at app start, before loading the model? It only disables NAX. If the slope returns to ~1, that's the cause. That's more decisive than the int8 run, which also goes through NAX.