Skip to content

Commit 6229ed2

Browse files
committed
Moved under tutorials/
1 parent ee5e445 commit 6229ed2

7 files changed

Lines changed: 2077 additions & 0 deletions

File tree

‎tutorials/moe/README.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
# MoE Playbook (Megatron-Core & Megatron-Bridge)
2+
3+
Practical playbook for **Mixture of Experts**: concepts, single-GPU training walkthrough, and optional multi-GPU / production material in an appendix.
4+
5+
- **`moe.ipynb`** — Main notebook, structured in three layers:
6+
- **Path 1 (~30 min):** §1 when to use MoE → §2 concepts (routing, aux loss, total vs active compute) → §3 micro-train on one GPU → §6 short checklist
7+
- **Path 2:** Path 1 + **Appendix** (dispatch, EP/TP/PP, Mixtral-style configs, performance tuning, extended tips)
8+
- **`moe_permute_impl.py`** — Naive (PyTorch) token permutation helpers plus a small benchmark runner (optional; appendix / advanced).
9+
10+
Open `moe.ipynb` and follow **“How to read this notebook”** at the top. Only the environment check and MoE micro-train cells execute by default; Megatron-Bridge blocks are reference-only unless that stack is installed.

‎tutorials/moe/moe.ipynb‎

Lines changed: 1497 additions & 0 deletions
Large diffs are not rendered by default.
138 KB
Loading

0 commit comments

Comments
 (0)