Repository navigation
Commit 81dea65
authored
Add fused MoE Triton kernel for CUDA backend (pytorch#18384)
INT4 W4A16 fused MoE kernel that performs grouped GEMM for
Mixture-of-Experts, loading only active experts' weights from HBM.
Each Triton program handles one (token-expert pair, N-block).
The INT4 dequantization logic is adapted from vLLM's
fused_moe_kernel_gptq_awq (Apache-2.0). Key difference: no token
sorting (moe_align_block_size) — uses per-pair programs with
vector-matrix multiply instead of tl.dot. This is optimal for decode
(M=1, memory-bandwidth-bound) but suboptimal for prefill.
The triton_op wrapper orchestrates the full MoE: GEMM1 (gate+up) ->
SiLU activation -> GEMM2 (down) -> weighted sum across top-k experts.
Tests: eager correctness, shapes, single-expert, export, e2e C++ runner.
```
Qwen 3.5 MoE (35B-A3B): hidden=2048, inter=512, E=256, top_k=8
Naive: 16 grouped nn.Linear (compute all 256 experts, bf16)
Fused: Triton kernel (compute only 8 experts, INT4)
M Naive (ms) Fused (ms) Speedup
--- ---------- ---------- -------
1 1.849 0.391 4.7x
2 1.809 0.368 4.9x
4 1.720 0.371 4.6x
8 1.709 0.556 3.1x
```
repro script:
https://gist.github.com/mergennachin/1f344db1ee6faf150426fc7a98116a211 parent 07b7c7e commit 81dea65
3 files changed
Lines changed: 716 additions & 0 deletions
0 commit comments