Chinese fine-tune: the #364 benchmark caught a generalization collapse; fixed via multi-label-space mixing

[ZLHAOOO]

Following up on #479: we released v0.2.0 of our Chinese fine-tune, and the #364 benchmark caught a failure mode worth sharing.

What happened. Our v0.1 fine-tune (heads-only, fixed 13-label routing) scored well on its own eval but collapsed on the #364 set: choice accuracy 0.44 (criteria) / 0.28 (scenario), wrong with high confidence. The base model scores 0.72/0.78 there. The cause: training on a single fixed criteria set — the head stopped reading options and pattern-matched label strings instead.

The fix. Mix task families with different criteria sets (7 new heads: code / marketing / finance / life-admin / robot-commands / generic-intent / emotion), and paraphrase the criteria descriptions across samples (27% of rows). After retraining: 0.72 / 0.83 / 0.67 on #364 — at or above the base model. On a 9-head held-out eval (1,116 decisions, never seen in training): 0.750 vs 0.681 for the base checkpoint, ahead on 8/9 heads.

Trade-off. Specialist accuracy on the original routing task dropped (0.854 → 0.756 on our hand-written set); head capacity is competed for across tasks. Documented, not hidden.

Shipped (Apache-2.0): weights ZLHAOOO/laya-mlx-zh (MLX, Apple Silicon), full synthetic dataset, recipes, eval sets, and an MLX-native runner for #364 (no torch needed). Also a small spec for domain packs (.card.md) — external files read at inference, so a new scenario needs no retraining.

Repo: https://github.com/ZLHAOOO/laya-mlx-zh

Thanks for the frozen benchmark — it did exactly its job.