README.md

Core ML predict and the GPU completion path

Issue: #45. Machine: Mac Studio M4 Max, macOS 26, Python 3.12.14, coremltools 9.0, PyObjC 12.2.2. Model: laya-typed-decisions. Every number below comes from results.json / tables.md, generated by scripts/analyze.py from raw/.

Question

With the ANE on a thread in the caller's process, a GPU request whose forward has already finished stays unresolved until a concurrent Core ML predict finishes (research/gpu-ane-interference/ledger/: GPU L128 return leg 0.05 → 7.41 ms, 91.1% of its occupancy increase). Does that delay come from the GIL, and can it be removed without costing the ANE throughput?

Answer

1. The GPU reply stops exactly at take_gil. The GPU dispatcher's read of the worker's reply returns from the read syscall, then waits to re-acquire the GIL until predict returns. Every later step (unpickling, set_result, the runtime's done-callback) takes ≤0.03 ms. Nothing waits in the pipe, the Future or the callback.
2. The GIL is the cause, established by intervention, not only by timing:
   • holding the GIL without Core ML reproduces the delay;
   • running the same Core ML predict without holding the GIL removes it.
3. Releasing the GIL around predict meets the GPU criterion. GPU return leg: 7.67 → 0.14 ms (P50), 7.74 → 0.16 ms (mean). GPU occupancy P50 drops from 20.24 to 13.79 ms, and GPU throughput rises from 48.1 to 69.5 req/s.
4. It fails the ANE criterion as written. With the GPU busy, ANE throughput is −13.5% against the product path (95.9 → 83.0 req/s) and ANE e2e P99 ×1.12. ANE alone: −0.6%. Process placement, measured the same way, gives ANE −9.0% and total throughput 159.7 req/s against 152.4 for the GIL release.

Configurations

Fixed workload: GPU L128 closed loop and ANE L128 closed loop, one client each, through the runtime (Laya(execution="workers"), the gpu-ane-interference harness, device plan). Cells: each device alone, then both at once. 3 cycles × 20 s windows.

	ANE placement	Core ML call	GIL during predict
A	thread (product default for this model)	coremltools CompiledMLModel.predict	held
B	process (the control)	coremltools, in the worker process	not the parent's
C	thread	PyObjC predictionFromFeatures:error: (scripts/nogil_predict.py)	released

C loads the same verified model.mlmodelc with the same CPU_AND_NE compute units and feeds the same FP16 features; only the binding differs. Two checks were made before any run. They are interactive checks, not recorded in raw/; the runs' answer matches are recorded:

• its outputs were checked bit-identical to coremltools on 20 inputs;
• another thread's 1 ms sleep woke 0.515 ms late during a PyObjC predict loop, against 8.47 ms with coremltools (idle: ~0.51 ms).

The success criteria were fixed in #45 before any run and are applied here unchanged.

1. Where the reply stops

Parent-side stamps (scripts/boundary.py) replace the GPU worker's connection and repeat Connection.recv with a clock read between its steps. The table uses the matrix cell, times after service_end in ms:

	service_end → header read (syscall returned, GIL re-held) P50 / P95	header → body	body → unpickled	unpickled → received (set_result + callback)
A thread	7.638 / 8.393	0.005	0.013	0.003
B process	0.060 / 0.147	0.006	0.020	0.007
C thread, GIL released	0.091 / 0.338	0.009	0.030	0.008

Alignment with the ANE's predict. In A, 2,871 of 2,885 GPU forwards ended while an ANE predict was running. The header read completed a median 0.012 ms before that predict's recorded end (P5..P95 −0.395..+0.305 ms). The span end is stamped after predict returns to Python, so a hand-off can land just before it. The wait correlates with the predict time left when the forward ended (0.70). In B and C the reply does not wait for the predict: the header comes a median 5.1–5.4 ms before it ends, and the correlation is −0.03 / −0.13.

Native stacks. sample at 1 ms took the GPU dispatcher's stacks during 3 s of the probe below. Counts are of samples under the dispatcher's os_read / os_write:

load thread	in the read syscall	read returned, in take_gil	write returned, in take_gil
coremltools predict	629	872	216
PyObjC predict	2,114	0	0

So for the questions in #45:

• The reply is available: the worker has sent it, and the read syscall has returned.
• The dispatcher thread is runnable but cannot acquire the GIL. The job's send also waits for the GIL after write, which is smaller.
• The Future, its callback and result parsing are not delayed; they are ≤0.03 ms together.
• The Python operation that resumes when predict ends is the dispatcher's os.read inside Connection._recv.

2. The 2×2: Core ML or the GIL?

scripts/completion_probe.py. One runtime GPU worker serves GPU L128 closed loop from the main thread, while a load thread in the same interpreter loops one call. There are 3 repeats of 10 s, interleaved. usleep lasts 9,477 µs, the median coremltools predict.

load	Core ML	GIL	GPU return P50 per repeat (ms)	GPU req/s
idle	–	–	0.026 / 0.025 / 0.094	80.3–80.5
coremltools predict	yes	held	15.57 / 15.67 / 15.64	26.6–29.1
PyObjC predict	yes	released	0.030 / 0.031 / 0.090	79.6–79.8
usleep via ctypes.PyDLL	no	held	15.85 / 13.45 / 12.14	13.4–16.3
usleep via ctypes.CDLL	no	released	0.027 / 0.084 / 0.094	79.8–80.3

• Holding the GIL is enough to cause the delay: a plain C sleep that holds it reproduces it with no Core ML involved.
• Holding the GIL is also necessary for it: the same Core ML predict without the GIL shows none.
• The probe's delay (~15 ms) exceeds the ledger's (~7.7 ms) because the load thread calls predict back to back. The dispatcher often misses the short gap between calls and waits through a second one. In the product the ANE thread does Python work between calls.
• No answer changed in any window.

3. The runtime, A / B / C (matrix cell unless noted)

GPU L128:

	return P50 / P95 / P99	occupancy P50 / P95 / P99	service mean	mx.eval mean	e2e P50 / P95 / P99	req/s
A solo	0.041 / 0.105 / 0.127	12.10 / 12.32 / 12.52	12.03	11.55	12.28 / 12.64 / 12.88	80.8
A	7.672 / 8.577 / 9.322	20.24 / 21.38 / 22.58	12.60	12.04	20.47 / 21.76 / 23.24	48.1
B	0.094 / 0.209 / 0.508	13.34 / 14.63 / 15.04	13.07	11.91	13.80 / 15.37 / 15.97	72.5
C	0.143 / 0.396 / 0.686	13.79 / 14.43 / 14.90	13.52	12.11	14.36 / 15.15 / 15.66	69.5

ANE L128:

	req/s	predict mean / P99	host mean	dispatch + return mean	e2e P50 / P95 / P99	CPU per request
A solo	99.0	9.62 / 9.90	0.25	0.004	10.01 / 10.42 / 11.06	0.64
C solo	98.4	9.71 / 9.97	0.24	0.004	10.14 / 10.36 / 10.56	0.69
A	95.9	9.75 / 10.47	0.35	0.005	10.25 / 10.93 / 12.08	0.74
B	87.2	10.16 / 10.71	0.58	0.183	11.43 / 12.40 / 12.78	1.38
C	83.0	10.72 / 12.00	0.72	0.011	11.97 / 13.07 / 13.55	1.96

Both at once: total throughput A 144.0, B 159.7, C 152.4 req/s. Every configuration had 0 answer mismatches against the inline references. Routing is explicit per device in this plan (gpu_requested / ane_requested only), so no routing decision can differ.

Against the success criteria (#45, as written)

criterion	result	met?
GPU return leg toward the process control, ideally ≤0.5 ms	7.74 → 0.16 ms mean (P50 0.14); B 0.13 ms	yes
GPU device execution not worse	mx.eval mean 12.04 (A) → 12.11 ms (C)	yes
ANE throughput regression < 5%	−0.6% alone; −13.5% with the GPU busy	no
no clear ANE tail regression	e2e P99 ×0.96 alone; ×1.12 with the GPU busy	no
answers and routing unchanged	0 mismatches; routing explicit	yes

What the ANE cost is, and is not, known to be

• Measured: in C the ANE's predict span grows by +1.0 ms with the GPU busy (A: +0.13). Its host work doubles (0.35 → 0.72 ms), and CPU per request rises 0.74 → 1.96 ms.
• Confounded: the closed-loop GPU client runs 44% more requests once it is no longer blocked (48.1 → 69.5 req/s), so the ANE shares the machine with a busier GPU. Process placement, with a similar GPU rate (72.5), also costs the ANE (−9.0%). How much of C's −13.5% is the GPU rate and how much is the GIL's new owner was not separated. That would need the GPU held at A's rate.
• Hypothesis, not measured: once the GIL is released, the ANE thread has to re-acquire it when predict returns, behind the GPU dispatcher and the client threads. That would appear inside the predict span. The stacks above were taken only for the GPU dispatcher.
• The PyObjC path's own copies cost ~0.09 ms per call (predict alone: 9.62 → 9.71 ms).

Process placement's cost (Experiment 3), from the same runs

With the GPU busy, against the product thread placement:

• ANE: −9.0% throughput; e2e P50 10.25 → 11.43 ms and P99 12.08 → 12.78 ms.
• ANE IPC: 0.18 ms per request (dispatch + return).
• CPU per ANE request: 0.74 → 1.38 ms.
• GPU gains: e2e P99 23.24 → 15.97 ms, and +51% throughput.

Start-up was 4.9 s against 3.0 s for both instances, from one sample each. Memory was not measured.

Verdict

• GIL hypothesis: supported by intervention. The delay needs the GIL held and is caused by the GIL held even without Core ML. The dispatcher's stack is in take_gil after its read has returned.
• The in-interpreter fix works for the GPU and moves cost to the ANE. It does not meet the ANE criteria written in #45, so it is not a production change as it stands.
• Against process placement, under this workload, C gives:
  • the same GPU completion path (0.14 vs 0.09 ms P50);
  • less ANE throughput (83.0 vs 87.2 req/s);
  • less total throughput (152.4 vs 159.7 req/s);
  • no extra process: start-up and memory are not compared here.

Threats to validity and changes of method

• A discarded first run of C. The first version of nogil_predict.py ran PyObjC calls without an autorelease pool. A Python thread has none, so every call leaked Core ML's IOSurface-backed outputs. The ANE slowed down within the run (predict 10.2 → 11.1 ms), IOSurface allocation then failed, and the run stopped after one matrix cycle.
  • Fix: every predict now runs in objc.autorelease_pool(). Over 20,000 calls on a worker thread, predict time held at 9.85–9.98 ms and RSS stayed flat.
  • C and the probe (whose PyObjC windows had the same leak) were re-run. Only the re-runs are in raw/; A and B never load PyObjC and were kept.
• One run per configuration, 3 cycles each. B's matrix cycle 0 differed from cycles 1–2 (GPU occupancy P50 12.25 vs 13.9 ms); all three cycles are included.
• Stack sampling perturbs timing. The sampled windows are separate from the latency windows and are used only for the counts.
• This is not the ledger campaign. Windows are 20 s, not 25 s, and there are no sweep or control cells. The A numbers reproduce the ledger's (GPU return mean 7.74 vs 7.41 ms), not exactly.
• PyObjC is not a laya-apple dependency. It is pulled in by uv run --with for these scripts only.

Reproduce

# ~20 min, idle machine on AC power, other GPU/ANE services stopped
LAYA_APPLE_CACHE=... HF_HUB_OFFLINE=1 sh research/coreml-gil-completion-path/scripts/run.sh
uv run python research/coreml-gil-completion-path/scripts/analyze.py      # --check to verify

file	what
scripts/nogil_predict.py	Core ML predict through PyObjC, GIL released (research only)
scripts/boundary.py	parent-side stamps on the GPU worker's reply
scripts/run_cell.py	the gpu-ane-interference harness + --ane-predict + the stamps
scripts/completion_probe.py	the 2×2 probe and the dispatcher stack samples
scripts/analyze.py	results.json, tables.md
raw/	A/B/C in the harness's raw format (+ completion_path stamps), the probe