User guide

The README covers what laya-apple is and why. This page covers how to use it:

the question schema, with score and noul examples;
devices and routing;
language routing between checkpoints, and shortlisting large choice questions;
the MLX fast path;
concurrent serving;
building, managing and moving ANE artifacts;
calibrating another machine;
the failure policy, the CLI, offline use and runtime diagnostics.

Question schema

questions is a dict of {name: {"type": ..., "instructions": ..., "criteria": ...}}, matching upstream Laya's schema:

`type`	`criteria`	Meaning
`choice`	non-empty dict of `{label: description}`, or a list of unique label strings	pick one label
`score`	non-empty list of level descriptions	pick a level, 0-indexed
`noul`	optional dict `{"false": ..., "true": ...}`	yes/no

The rules follow upstream Laya 0.3.20:

noul criteria keys are case-insensitive ("True" reads as "true"). Any key other than true/false raises InvalidRequestError.
noul labels. An optional "labels": {"false": "B", "true": "A"} changes only the option text the model reads (by default false: and true:). The labels must be distinct, non-empty strings. noul is still P(true). labels on a choice or score question raises InvalidRequestError.
Non-string instructions (a dict or list) are sent to the model as JSON, with non-ASCII text kept as it is.
A conversation context (a list of turns) that exceeds the checkpoint's max_len keeps its newest turns: it is truncated from the start. Any other context keeps its start.
questions={} returns empty answers with zero usage. Nothing is tokenized, routed or run, so result.runtime records backend="none", device="none" and the routing reason no_questions, and workers mode emits no trace. This holds for device="ane" too: no device was used, so nothing fell back.

A list of plain strings is also accepted as a convenience; each string becomes a noul question named after itself:

model.predict(context="...", questions=["Does the customer request a refund?"])

Every answer has upstream's fields in upstream's order:

choice: type, choice, probabilities, confidence, answer_confidence, action;
score: type, score, legend, probabilities, confidence, answer_confidence, action;
noul: type, noul, confidence, answer_confidence, action.

answer_confidence is max(p), the calibrated probability of the reported answer. It is the quantity temperature scaling fits, so it is the one to gate on across question types. On choice and score, confidence is the normalised entropy 1 − H(p)/log(k), a different quantity. On noul the two are equal. All values are rounded to 4 decimals.

Known limitation: option order. For choice questions, the decision can depend on the order of the criteria dict. This is a property of the upstream Laya model, not a laya-apple defect: laya-apple's MLX FP32 backend reproduces upstream's decision for every permutation exactly (see research/option-order/README.md). Keep option order fixed across calls for a given question if you need repeatable decisions.

Worked examples: `score` and `noul`

Each example below follows the question schema above and runs as written on the pinned laya-typed-decisions checkpoint, on the MLX GPU (device="gpu", see Devices). The first run downloads the checkpoint, and later runs work offline. Each prints the whole answer for one question. The output is from one Apple M4 Max with float16 weights; another machine may differ in the last digits.

Loading this checkpoint warns that the choice:11+ temperature is replaced. That entry is for choice questions with 11 or more options, so it does not affect these two examples.

score. criteria is a list of level descriptions. Level i is the i-th entry, counting from 0.

import json

from laya_apple import Laya

with Laya.from_pretrained("laya-typed-decisions", device="gpu") as model:
    result = model.predict(
        context="The customer was charged twice for the same invoice and wants the second charge refunded.",
        questions={
            "severity": {
                "type": "score",
                "instructions": "How severe is this problem for the customer?",
                "criteria": [
                    "cosmetic, no real impact",
                    "a minor inconvenience",
                    "blocks the customer until it is fixed",
                    "a financial loss for the customer",
                ],
            }
        },
    )

print(json.dumps(result.answers["severity"], indent=1))

{
 "type": "score",
 "score": 2.1732,
 "legend": {
  "0": "cosmetic, no real impact",
  "1": "a minor inconvenience",
  "2": "blocks the customer until it is fixed",
  "3": "a financial loss for the customer"
 },
 "probabilities": {
  "0": 0.0154,
  "1": 0.2438,
  "2": 0.2931,
  "3": 0.4477
 },
 "confidence": 0.1865,
 "answer_confidence": 0.4477,
 "action": {
  "act_probability": 1.0
 }
}

score is the expected level, the sum of each level times its probability. It is not the most likely level: here level 3 is the most likely (0.4477), and score is 2.1732.
legend maps each level, as a string, back to its description, and probabilities is keyed the same way.
answer_confidence is the probability of the most likely level (3 here), not a confidence in score.
confidence is the normalised entropy 1 − H(p)/log(k), a different quantity (see the answer fields above). It is low, 0.1865, because the probability is spread over levels 1 to 3 rather than concentrated on one.
score and confidence are computed from the unrounded probabilities, and every value is rounded to 4 decimals. Recomputing them from the printed probabilities can therefore differ in the last digit: it gives 2.1731 and 0.1864 here.

noul. criteria is optional. When given, it describes each side as {"false": ..., "true": ...}.

import json

from laya_apple import Laya

with Laya.from_pretrained("laya-typed-decisions", device="gpu") as model:
    result = model.predict(
        context="The customer was charged twice for the same invoice and wants the second charge refunded.",
        questions={
            "refund": {
                "type": "noul",
                "instructions": "Does the customer ask for a refund?",
                "criteria": {
                    "false": "the customer does not ask for money back",
                    "true": "the customer asks for money back",
                },
            }
        },
    )

print(json.dumps(result.answers["refund"], indent=1))

{
 "type": "noul",
 "noul": 0.7457,
 "confidence": 0.7457,
 "answer_confidence": 0.7457,
 "action": {
  "act_probability": 1.0
 }
}

noul is P(true), the probability that the answer is yes: 0.7457 leans yes, and a value below 0.5 would lean no.
confidence and answer_confidence are both max(noul, 1 − noul), the probability of whichever side is more likely. To gate on how sure the model is, use answer_confidence, not noul.

Devices

Laya.from_pretrained(model_id, device="auto" | "gpu" | "ane", ...).

device="gpu": MLX only. Every request runs on the GPU.
device="ane": Core ML / ANE only. A request that does not fit a validated artifact fails loudly (UnsupportedShapeError, ArtifactMissingError, etc.) — it never runs on MLX instead.

device="auto" (default): MLX for everything, except a request routes to the ANE when all of the following hold:

the request has exactly one question;
the longest prompt row fits one of the model's auto-ANE buckets;
a validated artifact for that bucket is present in the cache and passes its placement/parity checks at load;
the platform profile matches a validated profile (currently Apple M4 Max with coremltools 9.0, on macOS 26 or macOS 27).

Otherwise the request runs on MLX. Every result records exactly one routing reason in result.runtime.routing_reason:

Reason	Meaning
`gpu_requested`	`device="gpu"` was requested
`ane_requested`	`device="ane"` was requested
`validated_short_single_question_path`	auto chose the ANE
`multiple_questions`	auto chose MLX: more than one question
`sequence_exceeds_ane_auto_range`	auto chose MLX: the longest row is above the auto buckets
`ane_artifact_unavailable`	auto chose MLX: no validated artifact for the bucket
`ane_runtime_unavailable`	auto chose MLX: coremltools is not installed
`platform_not_validated`	auto chose MLX: unknown hardware/OS profile
`no_questions`	`questions={}`: nothing ran, on any `device`

Auto-ANE buckets, generated from measured evidence (laya_apple/data/routing.json):

Model	Buckets for explicit `device="ane"`	Buckets used by `device="auto"`
`laya`	64, 96, 128	64, 96, 128
`laya-multilingual`	64, 96, 128, 256	64, 96, 128
`laya-typed-decisions`	64, 96, 128	64, 96, 128

Longer validated lengths lose to MLX on latency and are not offered by either explicit device="ane" or auto. Multi-question requests are never auto-routed to the ANE: MLX batching wins at every measured length for 4 and 8 questions, and 2–3 questions were not measured, so they route to MLX conservatively.

Language routing: `from_pretrained("auto")` (since 1.6)

Laya.from_pretrained("auto", ...) returns a LayaRouter. It loads laya and laya-multilingual with the same arguments and picks one per request by the language of the state, with the function laya-apple serve --model auto and upstream's Router use:

from laya_apple import Laya

with Laya.from_pretrained("auto") as model:
    result = model.predict(context="Der Kunde wurde zweimal belastet.", questions=questions)
    print(result.runtime.model, result.runtime.model_routing)   # laya-multilingual language_not_english

LayaRouter has predict, predict_shortlist, submit, apredict, close, wait_for_ane, info and the context manager, with the same contracts as Laya.
The chosen checkpoint is RuntimeInfo.model, and why is RuntimeInfo.model_routing: language_english, language_non_latin_script, language_not_english, language_undecided_default or no_text_default (the last two choose laya, upstream's default). Upstream's routing block is Result.extra["routing"], and the routing key of to_dict().
Device routing inside each checkpoint is unchanged: each loaded Laya routes between the GPU and the ANE as described above.
Both checkpoints stay loaded, so the router uses the memory of both.
Only English against multilingual is decided. Upstream's LAYA_AUTO_TASK routing and caller language hints are not implemented.

Shortlisting large `choice` questions (since 1.6)

predict always scores every label of a choice question. For questions with many labels, predict_shortlist is opt-in and never used by predict:

from laya_apple import Laya, embed_fn_from_laya

with Laya.from_pretrained("laya", device="gpu") as model:
    embed_fn = embed_fn_from_laya(model)     # or any callable: list[str] -> (n, d) array
    result = model.predict_shortlist(context=context, questions=questions, embed_fn=embed_fn, k=20)
    print(result.extra["shortlist"])

Each choice question with more than k labels is cut to the k labels whose embed_fn vectors are most cosine-similar to the state's, then one predict runs on the reduced questions. Other questions pass through unchanged.
Probabilities on a shortlisted question are over the kept labels only.
result.extra["shortlist"][name] (and the shortlist key of to_dict()) holds labels, scores, k, n and passthrough.
embed_fn_from_laya mean-pools the checkpoint's own MLX encoder. It needs execution="inline" and device gpu or auto. A dedicated embedding model passed as embed_fn will usually shortlist better.
On a LayaRouter, predict_shortlist runs on the checkpoint the language router chooses.
It is adapted from upstream laya v0.3.20. No accuracy claim is made: upstream's reported figures are not remeasured here.

The MLX fast path (since 1.6)

Three changes to the MLX forward, for all three checkpoints. Only the first is on by default.

Item	Default	Switch
Token-id cache for prompt texts	on	`model.tokenizer.cache = None` turns it off
Compiled forward (`mx.compile`)	off	`LAYA_APPLE_MLX_COMPILE=1`
Length-bucketed batching	off	`LAYA_APPLE_MLX_LENGTH_BUCKETS=1`

Token-id cache. The tokenizer keeps exact text → token ids, per tokenizer, thread-safe and least-recently-used, bounded to 4096 entries and about 16 MB. The key is the full text, so a hit returns the ids the tokenizer computes: it is exact by construction.
Compiled forward. The same model runs through mx.compile, with batch length padded to a multiple of 32 and the option count to a power of two (masked like the eager padding) so the number of traced graphs stays bounded. RuntimeInfo.artifact_revision gains :compiled when it runs. It was bitwise identical to the eager forward on every golden row of all three models, in FP16 and FP32.
Length-bucketed batching. A request with more rows than batch_size (16) is sorted by length before batching and restored to request order afterwards; smaller requests run as before. Outputs were bitwise identical to in-order batching except laya-multilingual FP32 (max |Δlogit| 9.5e-7, same decisions).
The environment variables also reach the execution="workers" GPU worker, which inherits the parent's environment.
Speed: about 1–4% on laya-typed-decisions, in a screen only, not a full benchmark (benchmarks/mlx-fast-path-screen/README.md, which also records the correctness checks above). The switches are not part of the stable API (api.md).

Concurrent GPU + ANE execution

By default (execution="inline") a Laya instance runs one request at a time in the calling process. For serving many requests, use execution="workers":

from laya_apple import Laya

with Laya.from_pretrained("convaiinnovations/laya-typed-decisions", execution="workers") as laya:
    future = laya.submit(context="...", questions={...})        # concurrent.futures.Future
    result = laya.predict(context="...", questions={...})       # thread-safe, blocking
    # in asyncio code: result = await laya.apredict(context="...", questions={...})

MLX (GPU) runs in its own worker process.
Core ML (ANE) runs either on a dedicated thread in your process or in its own worker process. The choice is made per model from measurements (ane_placement="auto"), and you can override it with "thread" or "process".
Your process builds prompts, routes, and formats answers.
The two devices serve requests at the same time. Each device runs its own queue in arrival order.
close() (or leaving the with block) finishes queued work and stops the worker.

Why this placement, and its limits. Every alternative was measured (see research/v0.2-concurrency/):

Both backends in one interpreter cost the GPU 9–11% of its throughput.
With both devices busy, a device whose requests arrive from another process runs its host-side work 4–6× slower.
Core ML's Python predict holds the GIL for much of an ANE call, so running the ANE in your process costs more at high short-request rates.
Results on the Phase -1 mix with the chosen placement (benchmarks/v0.2.md):

Model Short-stream P99 vs solo Long stream throughput vs solo Aggregate vs GPU-only

laya-typed-decisions +8% −11% 4.6×

laya +5% −12% 2.9×

laya-multilingual +99% −13% 3.6×
Under open-loop load, short requests see 2.5–66× lower P99 than with GPU-only serving.
Complete isolation (each stream within 10% of its solo P99) was not reached on this platform.

Model	Short-stream P99 vs solo	Long stream throughput vs solo	Aggregate vs GPU-only
laya-typed-decisions	+8%	−11%	4.6×
laya	+5%	−12%	2.9×
laya-multilingual	+99%	−13%	3.6×

Routing with queues. On an idle machine, auto behaves exactly like inline. When a device is busy, the router compares expected completion times: the device's backlog plus the request's measured service time. Two additional reasons can then appear:

Reason	Meaning
`ane_backlog_shorter_on_gpu`	a short single-question request went to MLX because the ANE queue was longer
`gpu_backlog_shorter_on_ane`	a request in the tie band (an explicit-only bucket, e.g. multilingual L256) went to the ANE because the GPU queue was longer

Long requests and multi-question requests never go to the ANE under auto, however busy the GPU is. Every RuntimeInfo records execution, queue_wait_ms and the backlog estimates the router saw (gpu_backlog_ms, ane_backlog_ms).

If the ANE worker process dies:

the request running on it raises BackendUnavailableError;
later auto requests run on MLX with reason ane_runtime_unavailable, and a warning is issued once;
explicit device="ane" requests keep raising.

Adaptive ANE execution (in-process ANE, the default since 1.5)

With the ANE on a thread in your process (laya, laya-typed-decisions), coremltools' predict holds the GIL for much of each ANE call. A GPU request whose work has already finished then waits for that hold to end before its result reaches your code (research/coreml-gil-completion-path/). Asynchronous Core ML execution with prebound buffers (backends/coreml_async.py) avoids the synchronous path's long GIL hold. On a contended Mac, though, that path can fall into a slow state, where the median short latency rises from about 10.0 to 12.3 ms. In earlier research, the slow transient was strongly correlated with E-core residency of its threads (research/coreml-async-transient/); that research did not establish why the macOS scheduler made that placement. Adaptive execution (laya_apple/handoff.py) uses the fast path while it is healthy and falls back to the 1.4 path when it is not.

How it works:

Outside a GPU+ANE overlap, every ANE forward runs coremltools' predict, as in 1.4.
An episode starts when an ANE forward arrives while the GPU worker has work queued or running, or finished it at most 1.0 s earlier.
- Its first 64 ANE forwards run coremltools' predict, which steps over the transient at the start of an overlap.
- Later forwards run the asynchronous path.
The breaker: every completed asynchronous request's prepare time (tokenising and layout, RequestTrace.prepare_ms) is observed.
- Three consecutive requests above 0.3 ms open the breaker.
- From then on, every ANE forward of the episode runs coremltools' predict: no retry within the episode.
- A request already running is never cancelled or re-sent.
The episode ends, and the next overlap starts again with 64 synchronous forwards, when the GPU has been idle for more than 1.0 s or no ANE forward came for more than 1.0 s.
Only real requests move the state: no background thread, no timer, no sleep, no CPU counters, no QoS call and no private API. The breaker needs no trace= callback.
"GPU active" means this instance's own GPU worker. GPU work from another process, such as a local LLM, does not start an episode. Without laya-apple's own GPU work, the ANE runs the 1.4 path anyway.
Start-up cost: it loads a second Core ML model per ANE bucket, plus its load checks. That adds to start-up time and memory; neither was measured.

Measured on one Apple M4 Max (research/coreml-adaptive-breaker/):

Healthy async: in 154 production episodes of laya and laya-typed-decisions (product mix, bursts, repeated transitions and a 66-episode typed soak), every episode stayed on the fast path.
- GPU return was 0.035–0.043 ms, against 4.3 ms (laya) and 8.6 ms (typed-decisions) on the 1.4 path.
- Throughput was 1.038–1.042 × the 1.4 path, and the median episode P99 was 0.18–0.53 ms lower.
The slow state: none occurred naturally in those runs. The recovery experiment covered 12 episodes where the asynchronous path was already slow. In all 12 the breaker tripped within 40 ms of watching it, and latency was back to 1.4's within 164–414 ms.

Control. ane_handoff=None (the default) uses adaptive execution wherever it is eligible and available:

It is eligible with execution="workers", device="auto" and a model whose measured ANE placement is thread, not overridden to process.
- laya-multilingual, whose ANE runs in a worker process, never uses it, even with ane_placement="thread".
It is available when:
- the ANE path starts;
- pyobjc-framework-CoreML is importable (it is part of the ane extra);
- Core ML has the asynchronous prediction API;
- the asynchronous models load and pass their load checks.
When an eligible instance cannot use it, the instance runs the 1.4 path. The reason is warned once per process and recorded in info()["ane_handoff"]["disabled_reason"].

The other values:

ane_handoff=False always runs the 1.4 path.
ane_handoff=True requires adaptive execution:
- an ineligible configuration raises ValueError before anything is downloaded;
- ane_startup="background" raises ValueError, because a background start-up cannot raise to the caller;
- an unavailable path raises BackendUnavailableError at start-up.

Correctness. Both paths run the same verified artifact on CPU_AND_NE in FP16; only the Python call into Core ML differs.

At load, each asynchronous bucket's outputs must be identical to coremltools' (names, shapes, dtypes and values), and it must pass the placement probe's ratio.
The parity suite runs the asynchronous path against the goldens and requires every output to equal coremltools' exactly.
Every asynchronous predict reads its outputs from the array Core ML returned for that predict.

Failures. The coremltools path is the production path, not a fallback device: a request never changes device, compute units, artifact or precision because of adaptive execution.

If an asynchronous predict fails, or does not complete within 5 s, that forward runs again on coremltools' predict, still on the Neural Engine. Adaptive execution is then disabled with one warning, and every later ANE forward runs the 1.4 path.
If its own state is ever found inconsistent, it disables itself the same way.
info()["ane_handoff"] reports:
- the state, the guard, the episode count and the forwards per path;
- the breaker's count, whether it is open and its trips;
- whether it is disabled and why, and whether its state is consistent.

The ANE path: building artifacts

device="ane" and the auto-ANE path need a Core ML artifact for the exact (model, revision, bucket) tuple. Artifacts are never downloaded or committed — build them locally:

laya-apple artifacts build laya-typed-decisions

This builds every offered bucket for the model (or pass --length 64 to build one). Each bucket goes through:

layout — the channel-first BC1S PyTorch body is checked against the upstream FP32 DecisionModel on an exact-length row;
conversion — traced and converted to Core ML with FP16 compute and FP16 I/O, fixed shape (B=1 × L=<bucket>), macOS 15 deployment target;
compile — to a model.mlmodelc (the intermediate .mlpackage is not kept, since loading it recompiles for the ANE on every process — 37.8s vs. 0.21s for a compiled artifact);
placement — the Core ML compute plan on CPU_AND_NE must show 100% of operations on the Neural Engine and 0 device transitions;
parity — every shipped golden row that fits the bucket, checked against the upstream PyTorch FP32 reference, with the unchanged Phase -1 gate (see Correctness);
atomic registration — the artifact is built in a temporary directory and renamed into place only after every step above passes. A partial or failed build never appears as usable; a failure leaves the manifest under artifacts/rejected/ as evidence.

Building one bucket takes 1.5–3.5 minutes on the tested hardware, including the pre-warm load at the registered path.

Artifacts are cached under $LAYA_APPLE_CACHE if set, else $XDG_CACHE_HOME/laya-apple, else ~/.cache/laya-apple. They live outside the source tree and are never committed to Git or redistributed with the package.

Re-check everything already registered:

laya-apple artifacts verify

This reloads every cached artifact and re-checks its manifest schema, revision and weight hash, graph variant, bucket, compute units, parity status, build platform profile, file hashes and Core ML placement.

At runtime the manifest, revision, profile and compute-unit checks run on every load. The file hash and the placement check (about 1.1 s per bucket together) run on first load and again whenever the artifact's files, the macOS build or the coremltools version change; a passing result is stamped under <cache>/verified/. artifacts verify ignores the stamp.

Cold start. Loading a compiled artifact triggers Core ML's on-device ANE compile the first time a given artifact location is loaded (25–86 s per bucket on the tested machine); the system caches the result. artifacts build pays this cost once at the registered path, so later loads take 0.14–0.31 s per bucket. If the system evicts its cache, the next load pays it again.

Artifact lifecycle

Concurrent builds are safe. One build per (model, revision, bucket) runs at a time across processes, under a file lock in <cache>/artifacts/.locks/. A second builder waits, then finds the artifact already registered.
Corruption is quarantined. An artifact whose files no longer match their manifest hash, or whose manifest cannot be read, is moved to <cache>/artifacts/quarantine/ when it is detected. The error names the rebuild command. Nothing corrupt stays where the runtime looks.
Cleanup is explicit. laya-apple artifacts prune lists what it would delete and why:
- other revisions or weights, unregistered models, buckets no longer offered;
- builds from another platform profile, unvalidated or rejected builds;
- quarantined entries, abandoned staging directories, orphaned verification stamps;
- unpublished installs a killed import left at the registered path, once that import has exited and they are older than an hour;
- a .pruning-<pid>-<rand> copy a killed prune left (prune renames a staging directory or an unpublished install to such a copy and checks it again before deleting it). It is removed once that prune has exited, no running build or import marks it and it is older than an hour. One that holds a published artifact (manifest.json) is never deleted: prune reports it with the mv command that puts it back. Add --yes to delete. It only ever deletes inside the cache.
Warm after eviction. laya-apple artifacts warm MODEL pays Core ML's on-device ANE compile ahead of the first request.

Moving artifacts between machines. Artifacts are never downloaded implicitly: loading a model never fetches one. You can build them once and carry them to another machine yourself:

laya-apple artifacts export laya-typed-decisions --out exports/   # one .tar.gz per offered bucket
# on the other machine:
laya-apple artifacts import exports/laya-typed-decisions-L64.tar.gz

An import is registered only after the receiving machine has checked, itself:

the manifest against the pinned checkpoint;
the build platform profile;
the file hash;
the compute plan (100% ANE, 0 transitions);
the full parity gate against the shipped goldens (the checkpoint must be downloaded).

The local results are recorded in the manifest under imported. The archive, manifest, profile and file hash are checked in a staging directory. Core ML's on-device compile is tied to the model's path, so on a first install the checks that load the model (compute plan, parity gate) run at the registered path, with the manifest withheld until they pass. The import pays the per-path compile at one path instead of two. On one M4 Max (macOS 27.0, laya-typed-decisions L128, an empty cache) that took an artifacts import from 149–154 s to 76–91 s, and an artifacts fetch-style import from about 146 s to about 75 s, with half the new Core ML compile bundles (research/import-compile-once/, raw data in raw-2026-10-03-phaseb/ and raw-2026-10-03-full/, 3 imports each). Until the manifest is published, the runtime treats that path as having no artifact. If a check fails or the import is killed, nothing is registered: the next import of that bucket removes the unpublished directory, and artifacts prune lists it once its import has exited and it is older than an hour. --force (replacing a registered artifact) runs the checks on the staged copy, then moves it into place and loads it there.

Prebuilt artifacts (since 1.6). laya-apple artifacts fetch MODEL [--length L ...] [--repo OWNER/NAME] [--revision REV] downloads artifacts export archives from a Hugging Face repository and registers each one through the same import, so every check above runs on this machine. In addition:

the archive's SHA-256 must match the repository's index.json;
only archives built on this machine's platform profile (same SoC, macOS major and coremltools) are selected. Any other profile gets ArtifactMissingError, which names the build command;
each fetched bucket must pass the runtime placement probe at the registered path. On a first install it runs there once, after the parity gate and before the manifest is published. With --force it runs on the staged copy before the move into place, and again at the registered path after a full load there. The fetch result reports the probe at the registered path. Registration is all-or-nothing: a bucket that fails any check is not kept, and with --force the artifact it would replace is kept or put back;
a bucket is skipped as already registered only if its artifact passes the runtime's load checks (manifest, platform profile, file hash, compute plan). One that does not is fetched again; a corrupt one is quarantined first.

Without --revision, the default repository is read at the commit this release was validated against (44a54765f89d434e9d85f401126cfee7d2059e7d since 1.6.2; 1.6.0 and 1.6.1 read 93181067cfee9c6117a7919321eb303ec36fcbd4), never at its mutable main, so one laya-apple version always gets the same index and archives. --revision overrides it. A repository given with --repo or LAYA_APPLE_PREBUILT_REPO defaults to main.

A download is trusted no more than a local build. The repository is --repo, or LAYA_APPLE_PREBUILT_REPO, and otherwise the default tc3oliver/laya-apple-artifacts. At the 1.6.2 pin it holds two platform profiles: Apple M4 Max with coremltools 9.0, on macOS 26 and on macOS 27. Any other profile gets ArtifactMissingError and builds locally. Fetching removes the build and its PyTorch dependency. It does not remove Core ML's on-device ANE compile, which still runs once when the artifact is first loaded at its registered location.

How the published artifacts were checked. On each profile they were built for, they were downloaded into an empty cache and passed fetch, verify, parity and placement for all 10 model/bucket pairs: macOS 26.6.2 (benchmarks/prebuilt-artifacts-1.6.0.md) and macOS 27.0 (benchmarks/prebuilt-artifacts-1.6.2-macos27.md). Both checks ran on the build machine. An independent check on a second machine of either profile has not been done yet; it is recommended, not required (publishing.md). The macOS 27 profile matches every 27.x, but only 27.0 was measured.

Fetching does not give a fast cold start. On one M4 Max (macOS 26.6.2), the first start of laya-typed-decisions (buckets 64, 96, 128) at a new location took 273.5 s with ane_startup="wait", against 2.7 s warm (research/coreml-compile-cache/screen.md, one run). On macOS 27.0 it took 210.2 s, the median of 3 (research/coreml-compile-cache/results.md). That time is Core ML's on-device compile, so prebuilt artifacts do not bring a cold start under 30 s. Use ane_startup="background" to serve on MLX while it runs.

Limitation: Core ML's compile cache is never evicted. Core ML stores the on-device compile under ~/Library/Caches/<process name>/com.apple.e5rt.e5bundlecache/<macOS build>/. The cache grows with every new artifact location (a fetch, an import, a moved or copied cache), by 0.7–1.4 GB per bucket in that screen. laya-apple never evicts it, and artifacts prune does not touch it.

Artifact provenance

laya-apple artifacts list --capabilities prints one JSON record per registered artifact, whatever its manifest carries — a field the manifest does not have comes back null, it never raises. Each record:

Field	From the manifest
`model`, `repo`, `revision`	`source.model`, `source.repo`, `source.revision`
`source_weights_sha256`	`source.weights_sha256`
`conversion_revision.laya_apple_version` / `.git_revision` / `.code_sha256`	`conversion.laya_apple_version` / `.git_revision` / `.code_sha256`
`graph`	`artifact.graph` (the graph variant, e.g. `bc1s-masked`)
`bucket`, `batch`	`artifact.length`, `artifact.batch` (the fixed shape)
`compute_target.compute_units` / `.ops` / `.transitions`	`placement.compute_units` / `.ops` / `.transitions`
`precision`	`artifact.precision`
`platform`	`platform` (SoC, macOS, coremltools — the build profile)
`parity.passed` / `.tolerance` / `.prob_max_abs` / `.hard_mismatches`	the matching `parity.*` fields
`parity.near_tie_flips`	the count of `parity.near_tie_flips`, not the rows themselves
`artifact_sha256`	`integrity.artifact_sha256`
`offered_by_auto`	whether `bucket` is in the model's `auto_ane_buckets` (§ Devices)
`offered_explicit`	whether `bucket` is in the model's `ane_buckets`

To read them rather than parse them, add --format table (to list or list --capabilities; both print the same table). It has one row per registered artifact, sorted by model, revision and bucket (the JSON keeps the registry order):

MODEL                 BUCKET  STATUS     PARITY PASSED  COMPUTE PLAN
laya-typed-decisions     128  validated  yes            CPU_AND_NE (ane=<n> gpu=0 cpu=0 transitions=0)

PARITY PASSED is yes or no, - stands for any field the manifest does not have, and ? replaces a control character in one. The compute plan is compute_target.compute_units with the op counts per device and the transitions. With nothing registered it says so and names artifacts fetch and artifacts build. The default format, --format json, is unchanged; scripts should keep reading that.

An unsupported request shape (too long, wrong graph, a dynamic length) is never padded or reshaped to fit a bucket — it raises UnsupportedShapeError instead. Only buckets that passed the parity gate on this machine (its SoC, macOS major version and coremltools version — see Calibrating another machine) are ever loaded; an artifact built elsewhere fails verify_profile rather than running unverified.

Cold start. With execution="workers", ane_startup="background" makes from_pretrained return as soon as MLX is ready. The ANE finishes loading behind it, including any on-device compile. Until it is ready, auto routes to MLX with reason ane_starting. laya.wait_for_ane() blocks until it is ready, and info()["ane_ready"] reports it. Measured start times are in benchmarks/v0.3.md.

Calibrating another machine

The shipped routing profiles apply only to the profiles they were measured on (SoC, macOS major version, coremltools version): Apple M4 Max with coremltools 9.0, on macOS 26 or 27. On any other profile, auto uses MLX only (platform_not_validated). To enable the ANE on your machine:

laya-apple artifacts build laya-typed-decisions   # builds and parity-validates here
laya-apple calibrate laya-typed-decisions          # measures MLX and ANE latency here

calibrate applies the same rule that produced the shipped table to local measurements. It writes <cache>/profiles/<profile>.json. Such a local profile is used only when no shipped profile matches the machine. Laya.info()["routing_profile"] reports which table is in effect: shipped, local:<path>, or None (MLX only).

No silent fallback

An explicit device="ane" request runs on the exact validated artifact with the exact validated compute units, or it raises. It never silently runs on MLX, CPU, or a different Core ML placement.
Under device="auto", a decision to use MLX instead of the ANE is made before the request runs, and the reason is recorded in result.runtime.routing_reason. It is never a reaction to Core ML changing placement mid-flight.
The Core ML compute plan is checked against the declared placement (100% ANE, 0 transitions) at build time, at first load, and again after any change to the artifact files, the macOS build or coremltools. A mismatch raises ComputeUnitMismatchError instead of silently running on CPU.

Failure policy

Every failure the runtime can detect is a specific exception from laya_apple.errors:

Condition	Error
model not registered	`UnsupportedModelError`
malformed questions	`InvalidRequestError`
ANE requested, request exceeds the largest offered bucket, or the model/shape pair is not validated	`UnsupportedShapeError`
ANE requested, no artifact in the cache	`ArtifactMissingError`
artifact built from another revision or other weights	`ArtifactRevisionError`
artifact files do not match their manifest hash	`ArtifactIntegrityError`
artifact has no passing parity record	`ArtifactParityError`
requested compute units differ from the validated ones, or the loaded compute plan is not 100% ANE / 0 transitions	`ComputeUnitMismatchError`
MLX or coremltools unavailable for the requested device	`BackendUnavailableError`
offline and not cached	`BackendUnavailableError` (with download instructions)

CLI reference

The --offline flag is global and must come before the subcommand:

laya-apple --offline predict laya-typed-decisions --context "..." --questions '{...}'

Commands:

laya-apple predict MODEL --context TEXT --questions JSON [--device auto|gpu|ane] [--dtype float16|float32]
laya-apple info [MODEL]
laya-apple download MODEL...
laya-apple artifacts build MODEL [--length L ...] [--force] [--skip-existing]
laya-apple artifacts list [--capabilities] [--format json|table]
laya-apple artifacts verify [MODEL] [--length L ...]
laya-apple artifacts warm [MODEL] [--length L ...]
laya-apple artifacts prune [--yes]
laya-apple artifacts export MODEL [--length L ...] [--out DIR]
laya-apple artifacts import ARCHIVE.tar.gz [--force]
laya-apple artifacts fetch MODEL [--length L ...] [--repo OWNER/NAME] [--revision REV] [--force]
laya-apple calibrate [MODEL ...] [--warmup N] [--iters N]
laya-apple parity MODEL [--device gpu|ane] [--dtype float16|float32]
laya-apple benchmark MODEL [--device auto|gpu|ane] [--lengths L ...] [--questions N] [--warmup N] [--iters N] [--output FILE]
laya-apple switchyard [--seed N] [--duration S] [--out DIR] [--no-open] [--setup-ane] [--replay DIR]
laya-apple serve [--model auto|MODEL] [--preload MODEL ...] [--host H] [--port P] [--device auto|gpu|ane] [--allow-remote] [--log-level L]

serve runs the local Jev-compatible decision server; see serve.md.

--context and --questions each accept inline text/JSON, @file to read from a file, or - to read from stdin.

Example:

laya-apple --offline predict laya-typed-decisions \
  --context "The customer was charged twice." \
  --questions '{"refund": {"type": "noul", "instructions": "Does the customer request a refund?"}}'

laya-apple --offline info laya-typed-decisions

Switchyard

The easiest way to see heterogeneous serving is to run it. laya-apple switchyard runs the frozen switchyard-v1 workload (bursty seeded arrivals, single-question routing decisions as "trains", background long-context requests loading the GPU), then replays the recorded run as a rail yard in your browser. All measurement happens headless, before anything opens; the browser only replays the recorded requests.

uvx laya-apple switchyard

The first run downloads the pinned laya-typed-decisions checkpoint (about 800 MB) and runs offline after that. A standard run (60 s timed window, plus a 5 s warmup, for one or two rounds) takes about 2–3 minutes once the checkpoint is cached.

What you get depends on Neural Engine state:

No built ANE artifact: switchyard measures a gpu_only round and prints a setup command:
```
uvx --from "laya-apple[convert]" laya-apple switchyard --setup-ane
```
This builds and parity-validates the ANE artifact, calibrates this machine if needed, then runs switchyard again. Missing ANE never fails the command — it runs gpu_only and shows why.
ANE ready: switchyard measures gpu_only and hybrid (GPU + ANE) rounds on the identical seeded timetable, and the result card compares them: late trains, P99 decision latency, P99 queue wait.

Useful flags: --seed, --duration, --out DIR (default: ./switchyard-results/<timestamp>-<soc>/), --no-open (skip opening the browser), --replay DIR (rebuild replay.html from a previous run's result.json and trace.jsonl without re-measuring).

The workload, measurement boundaries, round order and result schema are frozen and described in docs/switchyard.md. The official measured campaign is in benchmarks/switchyard/README.md.

Offline use

Once checkpoints (and, if used, ANE artifacts) are cached, no network access is needed. Three equivalent ways to force this:

Laya.from_pretrained(model_id, local_files_only=True)
HF_HUB_OFFLINE=1 in the environment
laya-apple --offline <subcommand> ... on the CLI

Running offline against an uncached checkpoint raises BackendUnavailableError with download instructions instead of hanging or silently going online.

This covers artifacts fetch too: offline, it reads the repository's index.json and the archives from the Hugging Face cache only (for example after hf download OWNER/NAME FILE --revision REV), runs the same validating import, and raises BackendUnavailableError when a file is not cached. An online fetch does not keep its downloads, so it does not fill that cache.

Runtime diagnostics

Every Result.runtime (a RuntimeInfo) records:

backend (mlx / coreml) and device (gpu / ane);
model and model_revision;
sequence_length (longest prompt row, in tokens) and question_count;
routing_reason;
artifact_revision (the ANE artifact hash, or mlx:<weights sha256 prefix>:<dtype>, with :compiled appended when the compiled MLX forward ran);
model_routing (since 1.6): why a LayaRouter chose the checkpoint, None otherwise;
compute_units and buckets (Core ML only);
dtype;
latency_ms, from the call to the result (queueing included);
execution (inline / workers); with workers, queue_wait_ms and the gpu_backlog_ms / ane_backlog_ms estimates the router used.

`laya-apple info`

laya-apple info [MODEL] prints one JSON object, the first thing to paste into a bug report: laya_apple (the version), platform (SoC, macOS version and build, coremltools version), platform_validated_for_auto_ane, coremltools_available and models (each checkpoint's pin, dtypes, ANE buckets and the status of its artifacts). environment adds what platform does not carry:

"environment": {
 "python": "3.12.14",
 "mlx": {"available": true, "version": "0.32.2"},
 "offline": {"local_files_only": false, "hf_hub_offline": false, "effective": false}
}

python: the interpreter's version.
mlx.available: whether import mlx.core works here. mlx.version is null when it does not.
offline: whether resolving a pinned checkpoint stays off the network (see Offline use). laya-apple itself reads no environment variable for this. It passes local_files_only to huggingface_hub, False unless the caller gives local_files_only=True or the CLI gets --offline, and huggingface_hub has its own offline mode, on when HF_HUB_OFFLINE is 1, ON, YES or TRUE when it is imported. The legacy TRANSFORMERS_OFFLINE is consulted only when HF_HUB_OFFLINE is unset or empty.
- local_files_only: what laya-apple passes for this invocation, that is, whether --offline was given.
- hf_hub_offline: huggingface_hub's offline mode. null if huggingface_hub cannot be imported.
- effective: local_files_only or hf_hub_offline. When true, an uncached checkpoint raises BackendUnavailableError instead of downloading.

Navigation Menu

Repository navigation

guide.md

guide.md

User guide

Question schema

Worked examples: `score` and `noul`

Devices

Language routing: `from_pretrained("auto")` (since 1.6)

Shortlisting large `choice` questions (since 1.6)

The MLX fast path (since 1.6)

Concurrent GPU + ANE execution

Adaptive ANE execution (in-process ANE, the default since 1.5)

The ANE path: building artifacts

Artifact lifecycle

Artifact provenance

Calibrating another machine

No silent fallback

Failure policy

CLI reference

Switchyard

Offline use

Runtime diagnostics

`laya-apple info`

FilesExpand file tree

guide.md

Latest commit

History

guide.md

File metadata and controls

User guide

Question schema

Worked examples: score and noul

Devices

Language routing: from_pretrained("auto") (since 1.6)

Shortlisting large choice questions (since 1.6)

The MLX fast path (since 1.6)

Concurrent GPU + ANE execution

Adaptive ANE execution (in-process ANE, the default since 1.5)

The ANE path: building artifacts

Artifact lifecycle

Artifact provenance

Calibrating another machine

No silent fallback

Failure policy

CLI reference

Switchyard

Offline use

Runtime diagnostics

laya-apple info

Worked examples: `score` and `noul`

Language routing: `from_pretrained("auto")` (since 1.6)

Shortlisting large `choice` questions (since 1.6)

`laya-apple info`