[barhanc]

Description

Adds 124 measurements across Galaxy S20+, S24 and iPhone 17 for LLMs, Depth Estimation and ColBERT models. Remove non-quantized LLM rows and documents memory-related failures and skips.

Unfortunately I didn't have access to the exact same devices as other benchmarks that's why I didn't add the newly added backend/precision variants for existing models.

Introduces a breaking change?

• Yes
• No

Type of change

• Bug fix (change which fixes an issue)
• New feature (change which adds functionality)
• Documentation update (improves or adds clarity to existing documentation)
• Other (chores, tests, code style improvements etc.)

Tested on

• iOS
• Android

Testing instructions

N/A

Screenshots

Related issues

Closes #1570

Checklist

• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have updated the documentation accordingly
• My changes generate no new warnings

Additional notes

[msluszniak]

1. Peak memory looks under-counted on GPU backends: MLX LFM2.5-VL-1.6B shows 216 MB for a 2.8 GB model, and Gemma4 E2B Vulkan shows 327 MB on S20+ vs 3667 MB on S24. Please add a caveat. The "2.4 to 3.5 GB" sentence is also outdated now (S20+ Qwen3 4B peaks at 4122 MB).

2. Could the raw runs be committed or attached to Add missing benchmarks to docs before 0.11 release #1570? Regenerating measured.json needs all of them, and right now they only exist locally.

[msluszniak]

Also I've spotted now, that depth estimation lacks demo that we have for other CV tasks inside the docs. Same thing for Colbert retrival.

[msluszniak]

I'm thinking if we should change the navbar below. Some important sections like benchmarks, API references etc. are not visible rn. Maybe they should be collapsed by default and there should be an arrow for expanding them, WDYT?

[barhanc]

Also I've spotted now, that depth estimation lacks demo that we have for other CV tasks inside the docs. Same thing for Colbert retrival.

Do you mean the gif? I want to add them for v1.0 release after new gallery app is ready.

[msluszniak]

Also I've spotted now, that depth estimation lacks demo that we have for other CV tasks inside the docs. Same thing for Colbert retrival.

Do you mean the gif? I want to add them for v1.0 release after new gallery app is ready.

Ok, settled then

[barhanc]

I'm thinking if we should change the navbar below. Some important sections like benchmarks, API references etc. are not visible rn. Maybe they should be collapsed by default and there should be an arrow for expanding them, WDYT?

Not sure this is a problem. I'd rather show the list of tasks we support than things like api ref or the advanced part with core api. I agree, we could move benchmarks above extensions so they are more exposed.