Skip to content

community-benchmarks.md

Latest commit

 

History

History
272 lines (230 loc) · 13.5 KB

community-benchmarks.md

File metadata and controls

272 lines (230 loc) · 13.5 KB
Raw

Community benchmarks: the Apple Silicon matrix

Every number laya-apple ships was measured on one machine, an Apple M4 Max (compatibility.md). This page collects results from other Macs, each one produced by the same script and submitted as a pull request, so the matrix fills in from measurements rather than expectations.

Priority contribution slots

These Macs are the current priority, and each has a good first issue with copy-and-paste steps from git clone to the pull request. You can add a result in 10–25 minutes without changing any code. The matrix below records what has been measured; any Mac without a result is wanted, with or without an issue.

Your Mac Contribute What you run
M1 or M2, any variant #1: M1/M2 MLX-only result MLX only, about 10 minutes
M3 Max #2: M3 Max result MLX, plus ANE if you have 15 more minutes
Any other Apple silicon Mac, including M5 and newer No issue needed: follow Add your Mac Same steps

Matrix

SoC MLX ANE Auto uses ANE Heterogeneous Evidence
M1 – – – – Wanted — submit a benchmark
M1 Pro – – – – Wanted — submit a benchmark
M1 Max – – – – Wanted — submit a benchmark
M1 Ultra – – – – Wanted — submit a benchmark
M2 – – – – Wanted — submit a benchmark
M2 Pro (16 GB, macOS 26.6.2) ✓ untested no untested bundle (#47; --quick, laya-typed-decisions only; MLX-only run without coremltools, so auto stays on MLX (ane_runtime_unavailable) and the ANE and heterogeneous checks do not run)
M2 Max – – – – Wanted — submit a benchmark
M2 Ultra – – – – Wanted — submit a benchmark
M3 – – – – Wanted — submit a benchmark
M3 Pro – – – – Wanted — submit a benchmark
M3 Max – – – – Wanted — submit a benchmark
M3 Ultra – – – – Wanted — submit a benchmark
M4 (32 GB, macOS 26.2) ✓ ✓ no untested bundle (#41; --quick, laya-typed-decisions only; not calibrated, so auto stays on MLX and the heterogeneous check does not run)
M4 Pro (48 GB, macOS 27.0) ✓ ✓ yes, local calibrated profile ✓ bundle (#32; --quick, laya-typed-decisions only)
M4 Max (macOS 26.6.2) ✓ ✓ yes ✓ benchmarks/v1.0.md, bundle
M5 – – – – Wanted — submit a benchmark
M5 Pro – – – – Wanted — submit a benchmark
M5 Max – – – – Wanted — submit a benchmark
M5 Ultra – – – – Wanted — submit a benchmark
Any other or newer Apple silicon – – – – Wanted — submit a benchmark

What each column means:

  • MLX: MLX FP16 passes the parity gate against the shipped PyTorch FP32 goldens for every model measured.
  • ANE: the ANE artifacts built on that machine pass the same gate. untested means no artifacts were built there.
  • Auto uses ANE: device="auto" sends at least one request to the ANE. On a profile that is not shipped and not calibrated, the answer is no with routing_reason platform_not_validated; in an MLX-only run without coremltools it is ane_runtime_unavailable. That is the runtime working as designed, not a failure. yes, local calibrated profile means the submitter ran laya-apple calibrate on that Mac and auto used the resulting local profile. The shipped routing table does not include that profile, so on another Mac with the same SoC auto stays on MLX until that Mac builds its own ANE artifacts and runs laya-apple calibrate (compatibility.md, "What happens on an untested profile").
  • Heterogeneous: a short closed-loop mix of short and long requests has zero answer mismatches and more aggregate throughput with GPU + ANE than GPU alone.

– with Wanted means nobody has submitted a result yet. It says nothing about whether that Mac works.

A filled row records one submitted run on one machine, with the memory, macOS version and scope shown in the row. It is not a claim about every configuration of that SoC.

Add your Mac

You need a Mac with Apple silicon, git, uv (brew install uv) and about 2 GB of free disk, or 5 GB with the optional ANE step. An MLX-only result takes about 10 minutes.

  1. Clone the repository, install it and download the pinned checkpoint (800 MB, hash verified):

    git clone https://github.com/tc3oliver/laya-apple
cd laya-apple
uv sync
uv run laya-apple download laya-typed-decisions

  2. Optional: build the ANE artifacts on your machine.

    uv sync --extra ane --extra convert
uv run laya-apple artifacts build laya-typed-decisions

    This converts the model for each ANE bucket and registers an artifact only if it passes the placement and parity gates on your Mac. It takes about 5–10 minutes. You can skip it: results for MLX alone are useful too, and the report then records the ANE column as untested together with this command.

  3. Run the report:

    uv run python scripts/hardware_report.py --quick

    The script records your SoC, memory, macOS version and build, the package versions and the laya-apple revision on its own. Then it runs MLX parity, ANE parity when artifacts exist, warm latency, the routing decisions of device="auto", and a short heterogeneous mix when auto routes to the ANE. --quick measures laya-typed-decisions with fewer iterations. Without --quick it measures all three models with more iterations, so download them all first (uv run laya-apple download) or pass --models laya-typed-decisions.

  4. Check the new directory before you share it:

    NEW=$(git ls-files --others --exclude-standard hardware-results/ | xargs -n1 dirname | sort -u)
echo "$NEW"                    # exactly one new directory
head -20 "$NEW/summary.md"     # the matrix row and your environment
grep -rn -e "$USER" -e "$(hostname -s)" "$NEW" || echo "OK: no user name or host name in the bundle"

    The script records no host name or serial number, and it replaces your home directory with ~ and the checkout path with ..

  5. Open a pull request that adds the directory the script prints, hardware-results/<soc>-macos<major>/, unchanged. With the GitHub CLI:

    gh repo fork --remote              # your fork becomes origin, this repo becomes upstream
git switch -c bench/add-my-mac
git add hardware-results/
git commit -m "bench(hardware): add <SoC> benchmark result"
git push -u origin HEAD
gh pr create --fill

    Replace <SoC> with your chip, for example M1 Pro or M2 Max. gh pr create --fill uses the commit subject as the pull request title, for example bench(hardware): add M3 Max benchmark result. A different title is fine; the maintainer can adjust it when merging.

    Paste the matrix row the script printed into the description, and name the issue it closes if there is one (#1, #2). Any other Mac needs no issue.

What reviewers check

  • The bundle is complete. It has bundle.json and summary.md, every requested model has a section, and any step that did not run says why (unavailable with the build command, or skipped with a reason). Errors stay in the bundle; they are results too.
  • Parity. The MLX and ANE parity summaries are the verdicts of the same gate laya-apple parity applies, with the per-row maxima recorded. A ✓ needs passed: true for every model measured.
  • No manual edits. summary.md must render from bundle.json, the latency records must keep their raw samples, and the environment must match the directory name. Reviewers regenerate it with uv run python scripts/hardware_report.py --render hardware-results/<dir>/bundle.json and compare. Do not edit either file. If something looks wrong, say so in the pull request instead.
  • Provenance. The laya-apple revision is a published commit, and the dirty flag is false or explained.

How a result changes the matrix

A merged bundle fills in that SoC's row from its summary.md. If two bundles for the same SoC disagree, the row shows both until the reason is understood.

A result does not change what device="auto" does on anyone's machine. The shipped routing table (laya_apple/data/routing.json) applies only to the profiles it was derived on, and a validated ANE result on another profile does not add that profile to it. On your own machine, laya-apple calibrate writes a local profile that lets auto use the ANE there (compatibility.md, "What happens on an untested profile"). Adding a new shipped profile is a separate change that commits its own calibration measurements (support-matrix.md). It is not done through this matrix.

The bundle format is described in hardware-results/README.md.

Summary JSON

--json PATH also writes a compact summary of the run, for a dashboard or a comparison across hardware-results/*/ that should not have to parse bundle.json. It is derived from the bundle and every number in it is the bundle's own. It is optional and not part of the directory you submit: reviewers check bundle.json and summary.md.

uv run python scripts/hardware_report.py --quick --json summary.json

# or, for a bundle you already have, without measuring anything (summary.md is still printed):
uv run python scripts/hardware_report.py --render hardware-results/<dir>/bundle.json --json summary.json

PATH must be a writable file in an existing directory, and with --render it must not be the bundle being read. This is checked before the run starts, and an existing file is overwritten. This is the summary of hardware-results/apple-m2-pro-macos26/bundle.json:

{
 "format": "laya-apple-hardware-summary",
 "schema_version": 1,
 "laya_apple": "1.0.2",
 "started_at": "2026-09-24T05:55:15.663454+00:00",
 "quick": true,
 "platform": {
  "soc": "Apple M2 Pro",
  "macos": "26.6.2",
  "macos_build": "25G83",
  "coremltools": null
 },
 "hardware": {
  "hw_model": "Mac14,12",
  "memory_gb": 16
 },
 "models": {
  "laya-typed-decisions": {
   "revision": "f9ab0b228f0fc0f14d873dbc99038f135c2da1b2",
   "parity": {
    "mlx": true,
    "ane": null
   },
   "forward_p50_ms": [
    {
     "device": "gpu",
     "length": 64,
     "questions": 1,
     "p50_ms": 16.046146163716912
    },
    {
     "device": "gpu",
     "length": 128,
     "questions": 1,
     "p50_ms": 28.06797926314175
    },
    {
     "device": "gpu",
     "length": 512,
     "questions": 1,
     "p50_ms": 96.29687503911555
    },
    {
     "device": "gpu",
     "length": 128,
     "questions": 4,
     "p50_ms": 90.46510397456586
    }
   ]
  }
 }
}

Schema, version 1:

Key Contents
format, schema_version laya-apple-hardware-summary and the integer 1. Any change to a key or to what it means bumps schema_version; a test pins the exact output
laya_apple the laya-apple version that ran
started_at UTC start time of the run, as in the bundle
quick true for a --quick run (fewer iterations), so compare like with like
platform soc, macos, macos_build and coremltools: exactly the keys and values of laya_apple.artifacts.platform_profile(), the profile routing uses. coremltools is null when it is not installed
hardware hw_model (for example Mac14,12) and memory_gb
models.<name>.revision the pinned checkpoint revision that was measured
models.<name>.parity.mlx, .ane the parity gate's verdict: true passed, false failed or errored, null did not run (ANE unavailable on this machine, or the weights step failed; see the bundle)
models.<name>.forward_p50_ms one entry per measured configuration, in measurement order: device (gpu or ane), length (tokens), questions and p50_ms, the median warm forward time in milliseconds

forward_p50_ms is a list, not a map by length, because the same length is measured with different question counts (128 tokens with 1 and with 4 questions above). The forward time is the backend call alone, as in laya-apple benchmark; the end-to-end predict times and the raw samples stay in the bundle. A configuration that was skipped or failed has no entry there, and the reason is in the bundle.