Skip to content

mimo.md

Latest commit

 

History

History
28 lines (19 loc) · 1.74 KB

mimo.md

File metadata and controls

28 lines (19 loc) · 1.74 KB
Raw

Xiaomi-MiMo

Xiaomi-MiMo models use a Qwen2-style causal language backbone with Multi-Token Prediction support. Megatron Bridge supports MiMo causal language models through MimoBridge, which extends the Qwen2 bridge and adds MTP weight mappings.

Supported Variants

Megatron Bridge supports Hugging Face checkpoints using the MiMoForCausalLM architecture and mimo model type.

Architecture Notes

  • Qwen2-style attention behavior with QKV bias enabled.
  • Optional MTP layers are enabled from num_nextn_predict_layers in the Hugging Face config.
  • The bridge maps MTP token/hidden layernorms, projection layers, attention weights, and gated MLP weights.
  • Input projection halves are swapped during import/export to match Megatron and Hugging Face layouts.

Examples

Checkpoint conversion and text-generation examples for MiMo causal language models live under examples/models/mimo. For MiMo-V2-Flash conversion and inference examples, see the MiMo-V2-Flash examples README. Heterogeneous multimodal training examples live separately under examples/megatron_mimo.

Related Implementation