status.md

Project status

An honest snapshot of what works, what does not, and what is unknown. Read this before investing time, and update it when you learn something new.

The one-paragraph summary

The full compile pipeline works end to end: a Hugging Face checkpoint goes through ONNX → HAR → surgery → quantization → HEF, and the resulting self-contained HEF loads in genai and hailo-ollama. Prefill inference on hardware is numerically faithful (cosine ≈ 1.0 against float32). Multi-token generation through the KV-cache path now works: hailo-ollama serves long, coherent text from a checkpoint compiled by this pipeline (TinyStories-25M, 128-token context). The historic blocker was a KV-cache memory-layout mismatch between the __prefill and __tbt groups, fixed at compile time (findings/tbt-cache-read.md), plus two server-side tokenizer quirks fixed in step 3 (findings/tokenizer-bos-mismatch.md). Validated end to end on TinyStories-25M only so far; larger checkpoints still carry the separate hidden-size fidelity gap described below.

A separate SDK (not hardware) bug was found and closed this session (findings/sdk-native-cosine-drift.md): DFC's SDK_NATIVE/SDK_BIT_EXACT emulation normalizes softmax across all attention heads combined instead of per-head — confirmed bit-exact by reproducing it from raw QK^T scores, and confirmed present on every checkpoint tested including the original TinyStories default (masked there by a final-logits cosine that happens to still land near 1.0). It looked at first like it might explain the (since fixed) KV-cache incoherence, but that connection is ruled out: TinyStories' real-hardware base-scope generation is coherent despite carrying this exact SDK-emulation defect, so real silicon does not share it. Net effect: don't trust SDK_NATIVE/SDK_BIT_EXACT cosine as an attention-fidelity signal on multi-head checkpoints.

Generalization in progress. The pipeline no longer hardcodes one checkpoint: pipeline/s1_export_onnx.py --model <hf-id> derives every architecture constant from transformers.AutoConfig and steps 2-6 pick it up automatically. Validated end to end (step 1 + step 2's SDK_NATIVE fidelity gate) on 4 additional checkpoints spanning GQA and MHA, 4-22 layers, 64-768 hidden — all pass step 1; step 2's cosine degrades with model scale on larger checkpoints, a second open issue (findings/sdk-native-cosine-drift.md). Tied embeddings, QK-Norm (Qwen3-style), explicit head_dim, q/k/v projection biases (present on Qwen2-style architectures, silently dropped by the exporter until this session — a real bug, found and fixed while validating on a real, non-distilled Qwen2.5-0.5B-Instruct checkpoint chosen specifically to test on), and sliding-window attention (Mistral-style — needed zero exporter changes at all, since this project's mask is entirely host-computed and content-agnostic to the graph; verified end to end including hailo-ollama serving on Locutusque/TinyMistral-248M — findings/sliding-window-attention.md) are now supported and verified through quantization on real checkpoints (nickypro/tinyllama-15M, Qwen/Qwen3-0.6B, tabularisai/Qwen3-0.3B-distil, Qwen/Qwen2.5-0.5B-Instruct, Locutusque/TinyMistral-248M).

A third real-hardware fidelity gap was found, this time on base-scope (no-cache) generation — previously this project's control test for "is the compiled model itself sound", always coherent before now. Reproduced on three independent checkpoints (TinyMistral-248M, SmolLM2-135M — the latter also confirming the historical "30-layer memory wall" no longer applies with this project's current recipe/memory discipline, since it compiled cleanly — and Felladrin/Llama-160M-Chat-v1). The identical prompt through float32 HF is coherent in every case, ruling out weak checkpoints as the explanation. Three hypotheses were directly tested and cleared: the lm_head-splitting surgery (forcing it onto the always-coherent TinyStories baseline produced fully coherent output, unaffected), non-power-of-2 GQA ratio (Felladrin/Llama-160M-Chat-v1 has NREP=1, pure MHA, and degrades identically), and checkpoint depth — merging this finding's checkpoints with older hardware data from findings/sdk-native-cosine-drift.md (never previously cross-referenced) turned up a 6-layer checkpoint that already degrades, at the same depth as a coherent one, ruling out NLAYERS too. hidden size is the one variable that separates every coherent checkpoint from every degraded one, across all eight data points now on record: coherent only at hidden≤288, degraded starting at hidden≥576 (findings/tinymistral-base-scope-degenerate.md).

Large-vocabulary lm_head is now fixed. Every checkpoint sharing Qwen's ~152K-token vocabulary used to fail lm_head placement as one monolithic matmul. Three sharding mechanisms (ONNX multi-output export, the native defuse() model-script command, a naive pre-quantization HN surgery) all hit the same DFC-internal post-fuser bug once shard fan-out reached 3; a fourth — duplicating the shared ancestor normalization/slice chain per shard before quantization, so the fuser never sees that fan-out — works (findings/large-vocab-lm-head-sharding.md). Verified end to end on a real, non-distilled Qwen/Qwen2.5-0.5B-Instruct checkpoint (24 layers, VOCAB=151936): compiles, registers with hailo-ollama, and serves a live genai generation response — the first real Qwen-family checkpoint to clear every wall in this pipeline. A separate, downstream numeric-fidelity issue was found once this checkpoint could actually run: real-hardware prefill cosine 0.858 with a wrong argmax (previously-tested smaller checkpoints all showed ≈0.998-1.0 exact) — isolated to be neither a tokenizer/BOS desync nor the __tbt cache-read bug, root cause not yet found (findings/large-checkpoint-prefill-drift.md). A separate, now-moot DFC limitation was also found and documented while chasing the abandoned defuse() path: a deterministic __tbt context-partition topology error specific to large (24-layer) bodies, never reproduced with the fix that actually landed (findings/large-body-multicontext-topology.md).

Stage-by-stage

Stage	State	Notes
HF → ONNX export	✅ solid	cosine 1.000000 vs float32; matmul-trick graph; lm_head + last-position slice
ONNX → HAR parse (hailo10h)	✅ solid on hardware; ⚠️ SDK_NATIVE cosine numbers alone are not trustworthy	see the softmax finding — DFC's SDK emulation (SDK_NATIVE/SDK_BIT_EXACT) has a confirmed defect that makes attention-layer cosine look worse than it is; confirmed NOT present on real hardware (TinyStories base-scope generation is coherent despite carrying the defect in emulation)
Graph surgery + resources	✅ solid on hardware, same emulation caveat	mask_surgery()'s input_layer2 rewiring is correct — it was the prime suspect until hardware evidence ruled it out
Quantization (KV-cache)	✅ runs (~30 s GPU)	recipe validated by comparison with official .alls; no emulator check possible (see below)
Conv repair pass	✅ kept as safety net	finds 0 issues with the final recipe — its historical cause was ew_add_fusing
HAR → HEF compile	✅ works (~5–10 min)	both network groups emitted, with explicit token-major layout conversions on every KV-cache edge (findings/tbt-cache-read.md); large vocabularies handled by lm_head sharding (Finding 12)
Notebooks (../notebooks/)	⚠️ stale	walkthrough.ipynb still uses SEQ=24 and predates the KV-cache layout fix — its HEF reproduces the old degraded tbt behavior; needs re-running against the current pipeline
genai.LLM load	✅ works	HEF passes the full runtime contract (six inputs, embedded resources, config keys)
hailo-ollama serving	✅ coherent generation	long, coherent text on TinyStories-25M with a 128-token context; BOS and spacing handled in step 3
Prefill numerics	✅ exact	per-position cosines ≈ 1.0 vs float32 reference on hardware
Base-scope greedy generation (no cache)	✅ coherent	real English text; proves weights/RoPE/GQA/lm_head are sound
tbt generation via KV-cache	✅ fixed	greedy KV-cache generation matches float32 HF (see runtime/diagnostics/kv_greedy.py)

Known SDK/toolchain behaviors worth knowing

Documented in detail in findings/sdk-behavior-notes.md; short version:

• the quantized emulator (SDK_QUANTIZED) is structurally broken on KV-cache graphs — hardware is the only judge after step 4;
• optimization_level > 0 silently re-enables adaround/bias_correction;
• Keras deserialization of optimized HARs needs the acceleras layer registration preamble (../pipeline/config.py);
• a host-side EINTR/ioctl failure can interrupt long-lived LLM sessions — genai_generate.py isolates attempts in subprocesses to absorb it.
• SDK_NATIVE/SDK_BIT_EXACT softmax emulation ignores the HN's own groups metadata and normalizes across all attention heads combined — confirmed not present on real hardware (see findings/sdk-native-cosine-drift.md), but don't trust either context's cosine as an attention-fidelity signal.

What would move the needle next

Ranked guesses, informed by everything eliminated so far (full list inside the open findings):

1. Run larger checkpoints (SmolLM2-135M, Qwen2.5-0.5B) through the fixed pipeline and serve them through hailo-ollama — the KV-cache path is no longer the bottleneck, so what remains is the fidelity gap below.
2. Integrate the optional un-expanded KV-cache (llm_modifications, see the last section of findings/tbt-cache-read.md) without having to disable the float32 fidelity gates — it matched float32 HF 8/8 tokens on TinyStories-25M.
3. Bring the notebooks back in line with the pipeline (SEQ, cache layout conversions, tokenizer resources).
4. Continue the hidden-size threshold investigation (Finding 16) — now confirmed across 8 checkpoints (coherent only at hidden≤288, degraded starting at hidden≥576; NLAYERS and GQA ratio both directly ruled out). INT8 measurably better than INT4, calibset_size increase made it worse not better; root cause of why hidden size specifically degrades hardware fidelity still open (see findings/tinymistral-base-scope-degenerate.md and the "Downstream symptom" section of findings/sdk-native-cosine-drift.md, whose data resolved the NLAYERS question). Sharpest next step: find or construct a checkpoint at hidden roughly 300-575 to bisect the threshold further.
5. Run DFC's Layer Noise Analysis checker (hailo analyze-noise <har> --data-path <data>, confirmed from the official user guide). Blocked by the Cache/SDK_QUANTIZED bug on this project's standard KV-cache-duplicated quantized.har — but confirmed working (reaches real per-layer noise computation) on a HAR quantized without set_kv_cache_global_params, before hitting a separate, real shape-mismatch error (conv11, 256 vs 272 — plausibly GQA/repeat_kv related, not yet investigated). See findings/quantization-recipe.md's "Layer Noise Analysis" section for the full evidence and the exact no-KV-cache recipe used. Next step: chase the conv11 shape mismatch specifically — it's the one remaining blocker on the no-KV-cache path, and understanding it might also inform the KV-cache Cache bug itself.
6. Shrink the quantized HAR. On Qwen2.5-0.5B-Instruct (~1 GB of bf16 weights) quantized.har is ~13.7 GB (about ×14): three graph copies (base, __prefill, __tbt), each carrying float weights, integer weights and calibration statistics. Writing and re-reading it costs ~5 min per step (s4 save, s5 rewrite). ClientRunner.save_har() takes two options the pipeline never passes: compilation_only=True ("a reduced size har, containing only compilation related data" — likely the SDK's QUANTIZED_SLIM_MODEL state, quantized params only) and compressed=True. Untested. Check on TinyStories-25M that s5/s6 accept the reduced HAR and that the HEF is unchanged; expect it to rule out float emulation and re-quantization from that file, so keep a full HAR alongside when those are needed.
7. Apply Hailo's own LLM quantization recipe to the checkpoints that stay incoherent. Step 4 now loads recipes from .alls files (--recipe, see recipes/README.md); the repository ships the default recipe, Hailo's Qwen2 recipe unmodified, and hailo-llm.alls, the same recipe with role placeholders so it applies to any checkpoint. Why it matters: on NanoLM-25M-Instruct (hidden 312, so Finding 16's hidden threshold does not explain it) the chip faithfully runs a quantized model that is itself wrong — quantized emulation of the prefill scores Pearson 0.42 against HF, the chip 0.52, float emulation 0.98. Layer by layer the residual stream holds up to layer 6 and collapses after it; the HF residual stream carries massive activations (one channel grows to ~600, about 100× the median), and the operations that lose the most precision are the 8-bit SwiGLU products (values up to ~760 for a median of 0.1), the softmax reduce_max and the RMSNorm square. Those are exactly the operations Hailo's recipe keeps in 16 bits or treats with quarot, smart_softmax_stats and layer_norm_decomposition (token_equalization); this pipeline's default recipe treats none of them. On TinyStories-25M (no massive activations), measured on the chip against HF (Pearson, HF top-1 agreement over prefill + 15 decode steps): default recipe 0.982, 13/16; Hailo's recipe without the fused softmax mask, quarot, the norm decomposition and smart_softmax_stats 0.985, 15/16; the full hailo-llm.alls 0.936, 7/16 (embeddings fed rotated — quarot rotates the residual stream from the input on, and the DFC rotates the embedding table stored in the HEF accordingly, so the runtime needs no change). Adding the four features back one at a time to the reduced form isolates the loss: quarot (0.983, 14/16), the norm decomposition (0.985, 15/16) and smart_softmax_stats (0.985, 15/16) are neutral; only set_input_mask_to_softmax() degrades (0.945, 9/16), and only when attention spans several cached tokens — why it does here and not in Hailo's own model is open. Next: NanoLM-25M with Hailo's recipe minus the fused mask. Two SDK limits surfaced on it: weight_group_size=128 has no conv to apply to (hidden 312, MLP 1092), and a16_w4_a16 on its first down projection fails while creating hardware parameters — the 16-bit conv is split into high and low sub-convs, and the shift search for the low one takes log2 of a ratio that is negative when its input statistics do not straddle zero (a16_w16 passes). hailo-llm.alls needs a single lm_head conv (llm_modifications splits it into 4 itself) and refuses to resolve on a run whose step 1 sharded the lm_head, which step 1 does for every vocabulary wider than one shard (NanoLM's 32064 included) — a step-1 option to keep the lm_head whole is needed first.

Provenance note

Every claim above was verified on real hardware with reproducible tests; the diagnostics under ../runtime/diagnostics/ re-derive the key ones.