perf: speed up cold start (~34s) — it is all aroma model loading

[rvnminers-A-and-N]

Server cold start is ~34s, and it is almost entirely loading the 164 aroma RandomForest .joblib files (628 MB) at import. The substitute index is now fast (1.6s, cached). Options:

• Parallel model loading — load the 164 joblibs across a thread pool (deserialization + I/O); likely ~3x faster. Quick win.
• ONNX — convert aroma RFs to ONNX like the taste heads (export_onnx.py); faster load + smaller on disk.
• Lazy load — load aroma heads on first aroma request instead of at import (moves the cost, does not remove it).
  Not urgent (the box stays up), but a real quality-of-life + demo-startup win.

[rvnminers-A-and-N]

Measured update — the 'parallel model loading' option in this issue is a dead end; don't retry it.

I implemented it and benchmarked: loading the heads across a thread pool made cold start worse, ~34 s serial vs ~86 s across 14 threads. joblib.load is dominated by GIL-bound Python unpickling, so threads only add contention. Loading is now deliberately serial and commented as such in predict._load_all_models.

What did land instead: heads load on a background thread, so uvicorn binds its port within a second and serves a branded warming-up page (live loaded/total count + ETA, /api/status, /healthz) rather than the old ~90 s startup 502. The cold start itself is unchanged — the experience is fixed, the cost isn't.

Remaining real options if this is ever worth revisiting: ONNX conversion (smaller/faster to load, as already done for taste), or process-based loading with a shared-memory handoff. Note the process-per-unit pattern does work elsewhere — it's what made build_profile_index ~3 min → ~35 s and the aroma retrain ~30 min → minutes — it just can't help a single process's own import.

[rvnminers-A-and-N]

Current numbers after the retrain in #258, measured on the box at startup:

{"loaded":191,"total":191,"phase":"ready","ready":true,"elapsed":47.5}

47.5s for 191 artifacts (190 heads + the sweetness-intensity regressor), up from ~34s at 164 aroma heads. The per-head cost is essentially flat — we added 3 aroma heads and the extra time tracks the larger corpus in each forest, not a regression in the loader.

Worth restating so it isn't retried: threading the load makes it worse, not better (34s serial → 86s across 14 threads). joblib unpickling is GIL-bound, so threads only add contention. Real parallelism here needs ProcessPoolExecutor with the models handed back over shared memory, which is the open idea on this issue.

The user-facing sting is already largely handled — the branded warming page with live loaded/total and elapsed means a cold start is visible progress rather than a 502. So this is now a polish issue, not an availability one.

[rvnminers-A-and-N]

Measured both parallelisation options properly. Neither can ship as things stand, and the reason is more interesting than the timing.

Threads — worse than serial. Already known and re-confirmed: ~34s serial vs ~86s across 14 threads. joblib.load is dominated by GIL-bound Python unpickling, so threads only add contention.

Processes — 3.1× faster in isolation, and they deadlock in practice.

Benchmarked on 40 aroma models:

	time
serial	11.49s
processes, deserialise only (discard in worker)	1.16s
processes, returned to parent	3.74s

So the transfer cost is real (2.58s for 40 files) but small next to the unpickling it replaces. Extrapolated to the full roster that is roughly 49s → 16s.

Then it hung. _load_all_models() runs during module import, and forking while the interpreter holds the import lock deadlocks — the children inherit locked import machinery they can never acquire. Parent and every worker sat indefinitely; I had to SIGKILL them.

So the real blocker is not parallelism, it is when loading happens. The process pool becomes safe the moment loading is deferred out of import time — a FastAPI startup hook, or an explicit warm() the server calls after import completes. That is the actual fix this issue needs, and it is a bigger change than a pool: import predict currently having models loaded as a side effect is relied on by the CLI, the batch scripts and the test suite.

Reverting to serial with all of the above written into the docstring so nobody re-attempts either approach blind. Keeping this open — the 3× is worth having, it just needs the load moved first.