Skip to content

v1.6.0.md

Latest commit

 

History

History
129 lines (108 loc) · 8.29 KB

v1.6.0.md

File metadata and controls

129 lines (108 loc) · 8.29 KB
Raw

laya-apple 1.6.0: language routing, prebuilt ANE artifacts, opt-in shortlisting

The 1.6 flow: laya-apple artifacts fetch → from_pretrained("auto") → predict / predict_shortlist.

  • Laya.from_pretrained("auto") returns a LayaRouter that loads laya and laya-multilingual and picks one per request by the language of the context, with the same function laya-apple serve --model auto uses.
  • laya-apple artifacts fetch MODEL downloads a prebuilt Neural Engine artifact from tc3oliver/laya-apple-artifacts and registers it through the validating artifacts import. It replaces the local build and its PyTorch dependency: the convert extra is not needed.
  • predict works as before. predict_shortlist is opt-in, for choice questions with many labels.
pip install -U 'laya-apple[ane]'
laya-apple artifacts fetch laya
laya-apple artifacts fetch laya-multilingual
from laya_apple import Laya

with Laya.from_pretrained("auto") as model:
    result = model.predict(context=context, questions=questions)
    print(result.runtime.model, result.runtime.model_routing, result.runtime.device)

Prebuilt artifacts exist for exactly one platform profile: Apple M4 Max, macOS 26, coremltools 9.0. On every other Mac, fetch raises ArtifactMissingError naming the build command, and you build locally as before (laya-apple artifacts build MODEL, with the convert extra).

A fetched artifact is trusted no more than a local build

Every archive goes through the same checks on the receiving machine before it is registered:

  • its SHA-256 against the repository's index.json;
  • the manifest against the pinned checkpoint, and the build platform profile;
  • the compute plan: 100% ANE, 0 transitions;
  • the full FP16 parity gate against the shipped goldens;
  • the runtime placement probe, once per fetched bucket.

The default repository is read at the commit this release was validated against, 93181067cfee9c6117a7919321eb303ec36fcbd4, not at its mutable main. A later upload reaches users only through a release that moves the pin. --revision overrides it; --repo (or LAYA_APPLE_PREBUILT_REPO) selects another repository, which defaults to main.

Measured results

Result Source
The published artifacts, downloaded into an empty cache on the profile they were built for: fetch, verify, parity and placement passed for all 10 model/bucket pairs (laya 64/96/128, laya-multilingual 64/96/128/256, laya-typed-decisions 64/96/128). This ran on the build machine with an empty cache; no independent second-machine check yet benchmarks/prebuilt-artifacts-1.6.0.md
Cold start: the first start of laya-typed-decisions (buckets 64/96/128) at a new artifact location took 273.5 s, against 2.7 s warm. One run on one M4 Max, macOS 26.6.2 research/coreml-compile-cache/screen.md
MLX fast path: about 1–4% on laya-typed-decisions, in a screen only (no full release_bench) benchmarks/mlx-fast-path-screen/README.md

What else is in it

  • Language routing in Python. The choice is in RuntimeInfo.model_routing (stable codes: language_english, language_non_latin_script, language_not_english, language_undecided_default, no_text_default) and in Result.extra["routing"], which to_dict() returns as a top-level routing key, as upstream's Router does.
  • predict_shortlist(..., embed_fn, k=20), adapted from upstream laya v0.3.20. For each choice question with more than k labels it keeps the k labels whose embed_fn vectors are most cosine-similar to the request's, then runs one predict. predict itself is unchanged and still scores every label. embed_fn_from_laya builds an embed_fn from the checkpoint's own MLX encoder. No accuracy claim is made: upstream's reported figures are not remeasured here.
  • The MLX fast path, for all three checkpoints:
    • a token-id cache, on by default. It is exact by construction and bounded to about 16 MB; laya.tokenizer.cache = None turns it off (on a LayaRouter from from_pretrained("auto"), set it on each of router.instances.values());
    • a compiled forward (mx.compile), opt-in: LAYA_APPLE_MLX_COMPILE=1. artifact_revision gains :compiled when it runs. Bitwise identical to the eager forward on every golden row;
    • length-bucketed batching, opt-in: LAYA_APPLE_MLX_LENGTH_BUCKETS=1, for requests with more than 16 questions. Bitwise identical to in-order batching except laya-multilingual FP32 (max |Δlogit| 9.5e-7, same decisions).
  • laya-apple serve reports adaptive ANE execution in /health (ane.<model>.handoff).
  • CI runs the MLX-only integration tests on a hosted Apple silicon runner.
  • A new example, examples/auto_fetch_shortlist.py.

Research recorded, not shipped

Limitations

  • One platform profile has prebuilt artifacts: Apple M4 Max, macOS 26, coremltools 9.0. Every other Mac builds locally.
  • The prebuilt artifacts have no independent check on a second machine yet. The clean-cache fetch, verify, parity and placement check ran on the machine that built them. A second machine of the same profile is recommended, not required, for publishing (docs/publishing.md); every receiving machine re-validates each artifact before registering it.
  • Fetching does not make a cold start fast. The first Core ML load still compiles on the device, about 4.5 minutes for laya-typed-decisions in the one run measured (273.5 s against 2.7 s warm). Fetching removes the build, not this compile. ane_startup="background" serves on MLX in the meantime.
  • Core ML's compile cache grows without eviction: 0.7–1.4 GB per bucket for every new artifact location, under ~/Library/Caches/<process name>/com.apple.e5rt.e5bundlecache/. laya-apple never evicts it.
  • from_pretrained("auto") loads two checkpoints, laya and laya-multilingual, with the same arguments. It covers English against multilingual only; upstream's LAYA_AUTO_TASK routing and caller language hints are not implemented.
  • The MLX speedups come from a screen, not a full benchmark, and are not a headline.
  • Every measurement is from one Apple M4 Max.

Upgrade notes

  • No code change is needed. predict, routing thresholds, parity tolerances and the default execution paths are unchanged.
  • The token-id cache is the only new default. Set laya.tokenizer.cache = None (on a LayaRouter from from_pretrained("auto"), set it on each of router.instances.values()) to tokenize every text every time, as 1.5 did.
  • Result.to_dict() now adds each Result.extra entry as a top-level key, for example routing and shortlist. It never replaces model, answers, usage or runtime.
  • To use prebuilt artifacts, install the ane extra and run laya-apple artifacts fetch MODEL. --repo or LAYA_APPLE_PREBUILT_REPO selects another repository.

The release gate is in benchmarks/release-gate-1.6.0.md.

Details: docs/guide.md and CHANGELOG.md.