Skip to content

Commit 81dea65

Browse files
authored
Add fused MoE Triton kernel for CUDA backend (pytorch#18384)
INT4 W4A16 fused MoE kernel that performs grouped GEMM for Mixture-of-Experts, loading only active experts' weights from HBM. Each Triton program handles one (token-expert pair, N-block). The INT4 dequantization logic is adapted from vLLM's fused_moe_kernel_gptq_awq (Apache-2.0). Key difference: no token sorting (moe_align_block_size) — uses per-pair programs with vector-matrix multiply instead of tl.dot. This is optimal for decode (M=1, memory-bandwidth-bound) but suboptimal for prefill. The triton_op wrapper orchestrates the full MoE: GEMM1 (gate+up) -> SiLU activation -> GEMM2 (down) -> weighted sum across top-k experts. Tests: eager correctness, shapes, single-expert, export, e2e C++ runner. ``` Qwen 3.5 MoE (35B-A3B): hidden=2048, inter=512, E=256, top_k=8 Naive: 16 grouped nn.Linear (compute all 256 experts, bf16) Fused: Triton kernel (compute only 8 experts, INT4) M Naive (ms) Fused (ms) Speedup --- ---------- ---------- ------- 1 1.849 0.391 4.7x 2 1.809 0.368 4.9x 4 1.720 0.371 4.6x 8 1.709 0.556 3.1x ``` repro script: https://gist.github.com/mergennachin/1f344db1ee6faf150426fc7a98116a21
1 parent 07b7c7e commit 81dea65

3 files changed

Lines changed: 716 additions & 0 deletions

File tree

0 commit comments

Comments
 (0)