Skip to content

compatibility.md

Latest commit

 

History

History
144 lines (122 loc) · 13.6 KB

compatibility.md

File metadata and controls

144 lines (122 loc) · 13.6 KB
Raw

Compatibility statement

This page states, for each dimension of the runtime, what is tested (the release-validation profile below), what is expected (a hypothesis, not measured), and what is unknown. Where it disagrees with support-matrix.md, that page wins; this page restates its conclusions for someone deciding whether to run laya-apple on a given machine.

Tested profile

The shipped routing thresholds, the release benchmarks and the parity records behind the release were all measured on one profile:
SoC Apple M4 Max
macOS 26.6.2 (25G83)
MLX 0.32.2
coremltools 9.0
NumPy 2.1.3 (the [ane] extra pins numpy>=1.26,<2.2)
Python 3.12.14, but 3.11–3.13 are all in the install matrix (release_gate.py builds and smoke-tests base and ane extras on 3.11, 3.12, 3.13)

The same machine, upgraded to macOS 27.0 (26A428) with the same coremltools (9.0), is the second validated routing profile. All three models' ANE artifacts were built and parity-validated there (research/macos27-validation/), and laya-apple calibrate gave the same auto ANE buckets as on 26.6.2 in two passes. That calibration ships as a routing profile, so device="auto" uses the ANE on an M4 Max with macOS 27 and coremltools 9.0 once its artifacts are built (benchmarks/routing-macos27/). Since 1.6.2 they can also be fetched prebuilt. Known limitation: the release benchmarks, serve and adaptive execution (ane_handoff) were not measured on macOS 27; their results are from macOS 26.6.2 only.

Community hardware results are recorded separately, in community-benchmarks.md and hardware-results/. They do not change the shipped routing table or the status in this page's Tested column. There are three so far, all --quick runs of laya-typed-decisions:

Per-dimension status

Dimension Tested Expected Unknown
SoC Apple M4 Max Other Apple M-series SoCs run MLX correctly (hypothesis: MLX itself is validated across Apple Silicon upstream). One community data point outside the M4 family: on an M2 Pro (macOS 26.6.2), laya-typed-decisions passed MLX FP16 parity; ANE was not tested there ANE placement, correctness and routing thresholds on any other SoC, except one community data point: on an M4 Pro (macOS 27.0, coremltools 9.0), laya-typed-decisions passed MLX and ANE parity with 0 hard mismatches, a locally calibrated profile routed short requests to the ANE, and the heterogeneous check passed; on an M4 (macOS 26.2, coremltools 9.0), laya-typed-decisions passed MLX and ANE parity with 0 hard mismatches, and without calibration auto stayed on MLX (community matrix). Each covers one machine and one model, not every M4 or M4 Pro and not the shipped routing
macOS 26.6.2; 27.0 on the M4 Max (build, parity and a shipped routing profile; no release benchmarks) macOS 15–26 run MLX correctly ANE behaviour on any macOS other than 26.6.2 and, on the M4 Max, 27.0; macOS 27.x specifically — prior third-party work (laya-coreml, M3 Max, macOS 27.2) saw an enumerated-shape package run on the GPU. macOS 27.0 evidence so far:
  • the release M4 Max, after its upgrade to macOS 27.0 (coremltools 9.0): a build and parity validation of all three models, in which every explicit ANE bucket recorded 100% Neural Engine placement with 0 transitions and passed parity (research/macos27-validation/);
  • the M4 Pro community result above, whose ANE placement is inferred from its artifacts registering.

The M4 Max run was then calibrated and ships as a routing profile (benchmarks/routing-macos27/); the M4 Pro result is not one. Neither tested enumerated shapes or macOS 27.2. Other SoCs and macOS 27 profiles are unmeasured. The M4 community result is the only ANE evidence on another macOS 26 release (26.2): artifacts built there passed placement and parity for laya-typed-decisions | | Python | 3.12.14 (benchmarks); 3.11–3.13 (install matrix) | — | Any Python outside 3.11–3.13 (unsupported, not merely untested) | | MLX | 0.32.2 | Other MLX versions within the package's declared constraint are expected to work for the GPU backend | Numerical or performance drift on a materially different MLX version | | coremltools | 9.0 (pinned exactly by the ane/convert extras) | — | Any other coremltools version — the placement and parity gates have not been run against one, and coremltools 9.0 constrains NumPy to <2.2 for a reason: it breaks on NumPy ≥ 2.5 | | NumPy | 2.1.3, and >=1.26,<2.2 when the ane extra is installed | — | Behaviour outside that pinned range with the ANE backend | | laya × MLX | validated | — | — | | laya × Core ML/ANE | validated (buckets 64/96/128, CPU_AND_NE) | — | Batched (B>1), other compute units, other profiles | | laya-multilingual × MLX | validated | — | — | | laya-multilingual × Core ML/ANE | validated (buckets 64/96/128/256, CPU_AND_NE) | — | Batched (B>1), other compute units, other profiles | | laya-typed-decisions × MLX | validated | — | — | | laya-typed-decisions × Core ML/ANE | validated (buckets 64/96/128, CPU_AND_NE) | — | Batched (B>1), other compute units, other profiles | | Compute units | CPU_AND_NE (ANE, shipped), Metal GPU (MLX, shipped) | CPU_AND_GPU is correct but unused (slower than MLX) | — | | Compute units | — | — | CPU_ONLY is invalid (FP16 precision on Core ML's CPU path); ALL is never used because placement is not stable across runs | | Execution: inline | validated (single request at a time, thread-safe) | — | — | | Execution: workers | validated: GPU always in its own worker process; ANE placement (thread/process) chosen per model from measurement on the tested profile | The same GPU-process / ANE-thread-or-process split should hold on other Apple Silicon, since the mechanism (GIL contention, IPC cost) is not M4-Max-specific | Whether the per-model thread vs process choice in laya_apple/data/placement.json is the right one on a different SoC — it was derived from measurements on this machine only | | ane_placement="thread" | validated for laya, laya-typed-decisions | — | — | | ane_placement="process" | validated for laya-multilingual | — | — | | Adaptive ANE execution (ane_handoff, 1.5) | validated on the tested profile for laya and laya-typed-decisions (workers, auto, ANE on a thread): release validation, and the async path's parity against the goldens (research/coreml-adaptive-breaker/) | — | Other Macs and macOS versions, including macOS 27 on the M4 Max, where auto now routes to the ANE but adaptive execution was not measured; serve beside a local LLM; pyobjc-framework-CoreML versions other than 12.2.2 | | Prebuilt ANE artifacts (artifacts fetch, 1.6) | published for Apple M4 Max with coremltools 9.0, on macOS 26 and (since 1.6.2) macOS 27, all 10 model/bucket pairs each; on each profile a clean-cache download on the build machine passed fetch, verify, parity and placement (prebuilt-artifacts-1.6.0.md, prebuilt-artifacts-1.6.2-macos27.md) | The macOS 27 archives on 27.x releases after 27.0 (the profile matches them; only 27.0 was measured) | An independent fetch on a second machine of either profile (recommended, not required for publishing); every other profile has no prebuilt artifacts (ArtifactMissingError, build locally) | | Offline operation | validated (local_files_only=True, HF_HUB_OFFLINE=1, --offline) — no network access once checkpoints/artifacts are cached | — | — | | laya-apple serve against upstream laya.serve | validated since 1.3.0 against unmodified laya.serve 0.3.20 on the same pinned weights: 792 of 792 requests within the FP16 parity gate, 365 of them answered on the ANE (serve-compat) | — | Other upstream laya.serve versions; upstream's LAYA_AUTO_TASK routing and caller language hints (not implemented) | | laya-apple serve with Jev clients | 7 clients at their released versions, unmodified, on 2026-09-25 (list): wire compatibility only | Other clients that speak the same POST /v1/systemone wire format | Other versions of the tested clients; decision quality against Jev (Laya is a different model) | | laya-apple serve beside a local LLM | measured once (published in 1.4.0; measured on 1.3.0) on the release profile, one LLM (Qwen3.8-27B-oQ4e-mtp on oMLX), --model laya, 8 req/s offered (serve.md) | — | Other LLM servers and models, prefill-heavy LLM loads, other request rates, other Macs, and macOS 27 on the M4 Max (not measured there); serve's maximum decision throughput (run 2 used a fixed 8 req/s offered load); the other checkpoints (laya-typed-decisions, --model laya-multilingual) and --model auto |

What happens on an untested profile

laya-apple never guesses. On a platform profile (SoC, macOS major version, coremltools version) that does not match a shipped, validated profile:

  • device="auto" uses MLX only. Every result records routing_reason == "platform_not_validated".

  • device="ane" still works, but only after you validate it yourself on that machine:

    laya-apple artifacts build laya-typed-decisions   # builds locally; each bucket must
                                                   # pass its own placement and parity
                                                   # gate on this machine before it is
                                                   # registered
laya-apple calibrate laya-typed-decisions          # measures MLX and ANE latency here

    calibrate applies the same rule that produced the shipped routing table (docs/support-matrix.md, "How the auto-ANE buckets were derived") to local measurements, and writes <cache>/profiles/<profile>.json. A local profile is used only when no shipped profile matches the running machine — it never overrides a validated shipped profile. Laya.info()["routing_profile"] reports which table is in effect: "shipped", "local:<path>", or None (MLX only, no profile matched). A local profile is bound to the platform profile, the pinned model revision and the artifact hashes it measured. After an upgrade that re-pins a model, or after rebuilding an artifact, it is ignored with a warning until calibrate is run again.

  • Artifacts are tied to the profile that built them. An artifact built on one SoC, macOS major version, or coremltools version is refused when loaded or imported on a different one (ArtifactRevisionError) — it is never silently accepted and re-checked loosely.

  • artifacts export / artifacts import do not trust the exporting machine's checks. The importing machine re-validates everything itself: the manifest against the pinned checkpoint, the build platform profile, the file hash, the compute plan (100% ANE, 0 device transitions), and the full parity gate against the shipped goldens (which requires the checkpoint to be downloaded or already cached there).

This is the same "no silent fallback" policy the runtime applies everywhere else (see no-silent-fallback.md): an unvalidated configuration is never quietly used as if it were validated.

Not measured

laya-apple makes no claim about the following, because none of them has been measured, or measured beyond the community data points above:

  • Energy use. No power or efficiency figure exists for MLX, the ANE or GPU + ANE serving, and no routing decision uses one.
  • Quantized artifacts. Every shipped ANE artifact is FP16 and every MLX path FP16 or FP32. W8-palettized ANE artifacts have been run through the parity gate as research only (research/ane-w8/README.md): every laya and laya-multilingual cell failed it, and only the laya-typed-decisions L64 w8-pt latency result is reproduced. None is shipped, registered or routed to.
  • Cross-SoC validation beyond the community data points above: routing thresholds and ANE placement are not assumed to hold on another SoC.

Where to look for more detail

  • Per-model, per-compute-unit validation status, exact pinned revisions and weight hashes: support-matrix.md.
  • How the auto-ANE bucket list and routing thresholds were derived from evidence: support-matrix.md, "How the auto-ANE buckets were derived".
  • Platform scope, including per-(SoC, macOS major, coremltools) capability profiles: support-matrix.md, "Platform scope", and laya-apple calibrate above.
  • The stable public API and what compatibility promises apply to it (routing reasons, CLI, file formats): api.md.