Studio: refuse a hand-set Metal context only past the GPU wired limit

[Lyxot]

Summary

On Apple Silicon, Studio refuses a hand-set GGUF context whenever its estimated footprint exceeds the free-memory budget. That refusal is stricter than the failure it guards against: it rejects configurations that load and run fine while macOS compresses or swaps other apps. This PR moves the hard refusal to the limit that actually takes the machine down, the GPU wired-memory limit. A context between the free-memory budget and that limit now loads with a memory warning instead of HTTP 500.

Refs #9942 (the false refusal; the persistent "Load anyway" setting requested there is not part of this PR). Follow-up to #9172, which added the refusal.

Problem

• fix(studio): refuse a hand-set context unified memory cannot hold #9172 refuses an explicit context whose footprint exceeds min(Metal recommended working set, psutil available) * 0.85.
• In [Bug] Metal Memory Guard Blocks Working Qwen3.8-27B Config, Add Persistent 'Load anyway' Setting #9942, a Mac Studio M1 Ultra 64 GB running Qwen3.8-27B-UD-Q6_K_XL at 131,072 tokens (a 38 GiB estimate, about 32 GB free) is refused. The same configuration previously ran reliably. It still loads and generates with UNSLOTH_ALLOW_METAL_CTX_OVERCOMMIT=1.
• The desktop app offers no way to set that variable when launched from Finder or the Dock.

Root cause

Free memory is not the safety boundary on unified memory. When psutil-available memory runs out, macOS compresses or swaps other apps: the load slows but survives. The unrecoverable case, the kernel panics behind #9172, is over-committing wired GPU memory. Wired pages cannot be reclaimed, so Jetsam cannot step in. Refusing at free memory therefore blocks working loads without adding safety beyond the wired limit.

What changed

Backend (studio/backend/core/inference/llama_cpp.py, utils/hardware/hardware.py)

• New _apple_metal_wired_ceiling_bytes() measures the wired headroom:

  • the limit is iogpu.wired_limit_mb, or Metal's recommended working set when the sysctl reads 0 (the kernel default);
  • minus system-wide GPU memory in use, from ioreg In use system memory;
  • times 0.95.

• It returns 0 (unknown) when:

  • the host is not Apple Silicon;
  • the sysctl cannot be read or parsed;
  • MLX is unavailable;
  • the ioreg counter is missing;
  • the headroom is already used up.

• _read_apple_gpu_stats now omits vram_used_bytes when ioreg lacks the counter, instead of reporting 0.

• The explicit-context verdict in the Metal fit arm now works as follows:

  Situation	Result
  Headroom unknown, or the fitted weights alone exceed it	Previous free-memory refusal, unchanged
  Requested footprint above the wired headroom	Refused, naming the largest context under the headroom (or "No context fits")
  Between the free-memory budget and the wired headroom	Loads; a warning is logged and returned as memory_warning
  Within the free-memory budget	Loads as before

• The requested footprint is priced the way the context fit prices it, SWA context checkpoints (--ctx-checkpoints) included, so an admitted request never exceeds the ceiling a refusal names.

• The wired-limit refusal does not suggest leaving the context on Auto, because Auto sizes from free memory, not the wired headroom. The free-memory refusal text is unchanged.

• Unchanged: Auto context sizing, the published max_context_length, placement on discrete GPUs, the paravirtual and manual-layer exemptions, and the UNSLOTH_ALLOW_METAL_CTX_OVERCOMMIT=1 opt-out.

Frontend

On unified memory, the two context-length warnings (chat settings sheet and model config page) no longer describe the published max context as what fits with nothing to offload to. They now say the context is above Studio's free-memory estimate, may still load, and macOS may have to compress or swap other apps. Discrete-GPU wording is unchanged.

Risks and trade-offs

• Headroom measurement. The ioreg in-use counter was checked on an M3 Max with an MLX allocation: it tracked an 8 GiB allocation within 0.1 GiB and released it within 0.01 s of the process exiting. Whether it also counts another llama-server's no-copy mmap buffers has not been verified on hardware. The 5% margin absorbs estimate error, not a missing counter.
• New refusal. If other processes hold a lot of GPU memory, the wired headroom can fall below the free-memory budget. A context that fits free memory can then be refused where it previously loaded. Auto still sizes from free memory in that state; capping Auto at the wired headroom is left for a follow-up.
• Fallback states match previous behavior. When the headroom is unknown or the weights alone exceed it, the free-memory refusal is kept.
• Warning visibility. memory_warning is returned by the load API, but no UI consumer shows it yet; the reworded slider warnings are the visible signal.
• Probe cost. The probe runs sysctl and ioreg once per explicit-context load on Apple Silicon.

Validation

Targeted test files, each run in its own process:

cd studio/backend
python -m pytest -q -p no:cacheprovider tests/test_metal_explicit_context_guard.py         # 100 passed
python -m pytest -q -p no:cacheprovider tests/test_llama_cpp_context_fit.py                # 73 passed
python -m pytest -q -p no:cacheprovider tests/test_metal_paravirtual_guard.py              # 119 passed
python -m pytest -q -p no:cacheprovider tests/test_metal_never_starts_at_native_context.py # 71 passed

• New tests cover:
  • the warning and refusal boundaries;
  • naming a context below the fit floor;
  • weights that exceed the headroom;
  • SWA checkpoint pricing (in the general case and at the fit floor);
  • warning rounding;
  • refusal wording on both paths;
  • the wired probe's unreadable and off-Apple states.
• Every new test was mutation-checked: weakening the production line it covers fails that test and no other.
• A conftest fixture keeps the suite from reading the host's live GPU memory.

End-to-end load_model run on a 128 GB M3 Max, with llama-server launch and health check stubbed, using the real Qwen3.8-Flash-Next UD-IQ4_XS GGUF header (87 GiB weights). The free-memory budget read about 74 GiB and the wired headroom about 100 GiB:

n_ctx=       0  launched -c 8192, no warning
n_ctx=  131072  launched -c 131072, warning: needs about 91 GB, more than the 74 GB Studio budgets from the memory free right now
n_ctx=  262144  launched -c 262144, warning: needs about 94 GB
n_ctx= 4194304  refused: "Lower the context to 524,032 or less, or use a more quantized GGUF."

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-11T17:03:28.295778Z	bf778c4	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bf778c4c4b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Lyxot]

@codex review

[danielhanchen]

Confirmed the hand-set path in studio/backend/core/inference/llama_cpp.py still refuses at the free-memory budget, so a config that loads fine gets a 500. The new wired-headroom ceiling can also fall below that budget when other processes hold GPU memory, refusing a context that loads today, and it admits loads the old guard blocked against kernel panic; could you confirm the ioreg counter includes another llama-server's mmap buffers before I review?