|
| 1 | +# Qwen 3.5 MoE |
| 2 | + |
| 3 | +Self-contained ExecuTorch implementation of |
| 4 | +[Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B), |
| 5 | +a ~35B total / ~3B active parameter Mixture-of-Experts language model. |
| 6 | +Weights are loaded directly from the HuggingFace safetensors checkpoint. |
| 7 | +CUDA backend only. See [model.md](model.md) for architecture and |
| 8 | +implementation details. |
| 9 | + |
| 10 | +## Overview |
| 11 | + |
| 12 | +The pipeline has two stages: **export** (Python, once) and **inference** |
| 13 | +(C++ runner, repeated). Export converts the HuggingFace checkpoint into a |
| 14 | +`model.pte` file with int4 quantization. At inference time, the C++ runner |
| 15 | +loads the `.pte`, `.ptd`, and a HuggingFace tokenizer, then generates text. |
| 16 | + |
| 17 | +## Prerequisites |
| 18 | + |
| 19 | +- ExecuTorch installed from source (see [building from source](../../../docs/source/using-executorch-building-from-source.md)) |
| 20 | +- [safetensors](https://pypi.org/project/safetensors/) (`pip install safetensors`) |
| 21 | +- NVIDIA GPU with CUDA toolkit (tested on A100 80GB) |
| 22 | +- Python dependencies: `pip install -r requirements.txt` (installs FLA / Flash Linear Attention) |
| 23 | +- Model weights downloaded from HuggingFace. The directory should contain |
| 24 | + `config.json`, `model.safetensors.index.json`, safetensors shards, and |
| 25 | + `tokenizer.json`. |
| 26 | + |
| 27 | +## Export |
| 28 | + |
| 29 | +Export produces a `model.pte` and `aoti_cuda_blob.ptd` containing the |
| 30 | +compiled CUDA kernels and quantized weights. Int4 quantization is |
| 31 | +recommended — the model is too large to fit in VRAM at bf16. |
| 32 | + |
| 33 | +```bash |
| 34 | +python export.py \ |
| 35 | + --model-dir ~/models/Qwen3.5-MoE-A3B \ |
| 36 | + --output-dir ./qwen35_moe_exports \ |
| 37 | + --qlinear 4w \ |
| 38 | + --qembedding 8w |
| 39 | +``` |
| 40 | + |
| 41 | +### Options |
| 42 | + |
| 43 | +| Flag | Default | Description | |
| 44 | +|------|---------|-------------| |
| 45 | +| `--model-dir` | (required) | HuggingFace model directory with `config.json` + safetensors | |
| 46 | +| `--output-dir` | `./qwen35_moe_exports` | Output directory | |
| 47 | +| `--max-seq-len` | `4096` | KV cache length | |
| 48 | +| `--qlinear` | (none) | Linear layer quantization: `4w`, `8w`, `8da4w`, `8da8w` | |
| 49 | +| `--qlinear-group-size` | `32` | Group size for linear quantization | |
| 50 | +| `--qembedding` | (none) | Embedding quantization: `8w` | |
| 51 | + |
| 52 | +## Build |
| 53 | + |
| 54 | +ExecuTorch must be installed from source first (see |
| 55 | +[Prerequisites](#prerequisites)). The `make` target handles building |
| 56 | +core libraries and the runner binary. |
| 57 | + |
| 58 | +```bash |
| 59 | +make qwen3_5_moe-cuda |
| 60 | +``` |
| 61 | + |
| 62 | +This builds ExecuTorch with CUDA backend support, then the runner binary |
| 63 | +at `cmake-out/examples/models/qwen3_5_moe/qwen3_5_moe_runner`. |
| 64 | + |
| 65 | +## Run |
| 66 | + |
| 67 | +The runner requires: |
| 68 | +- `model.pte` — exported model (see [Export](#export)) |
| 69 | +- `aoti_cuda_blob.ptd` — CUDA delegate data file (produced during export) |
| 70 | +- `tokenizer.json` — HuggingFace tokenizer from the model weights directory |
| 71 | + |
| 72 | +```bash |
| 73 | +cmake-out/examples/models/qwen3_5_moe/qwen3_5_moe_runner \ |
| 74 | + --model_path qwen35_moe_exports/model.pte \ |
| 75 | + --data_path qwen35_moe_exports/aoti_cuda_blob.ptd \ |
| 76 | + --tokenizer_path ~/models/Qwen3.5-MoE-A3B/tokenizer.json \ |
| 77 | + --prompt "The meaning of life is" \ |
| 78 | + --max_new_tokens 128 |
| 79 | +``` |
| 80 | + |
| 81 | +### Options |
| 82 | + |
| 83 | +| Flag | Default | Description | |
| 84 | +|------|---------|-------------| |
| 85 | +| `--model_path` | (required) | Path to exported `.pte` model | |
| 86 | +| `--data_path` | (none) | Path to `.ptd` delegate data file (required for CUDA) | |
| 87 | +| `--tokenizer_path` | (required) | Path to HuggingFace `tokenizer.json` | |
| 88 | +| `--prompt` | `"Hello"` | Input prompt text | |
| 89 | +| `--temperature` | `0.8` | Sampling temperature (0 = greedy) | |
| 90 | +| `--max_new_tokens` | `128` | Maximum tokens to generate | |
| 91 | + |
| 92 | +## Troubleshooting |
| 93 | + |
| 94 | +- **OOM during export**: The model requires significant GPU memory even |
| 95 | + with int4 quantization. Try reducing `--max-seq-len` or using a GPU |
| 96 | + with more VRAM. |
| 97 | +- **OOM during loading**: The 35B parameter model requires ~70 GB RAM to |
| 98 | + load from safetensors before quantization. Ensure sufficient system |
| 99 | + memory. |
| 100 | +- **Missing `aoti_cuda_blob.ptd`**: This file is produced during export |
| 101 | + alongside the `.pte`. Both files are required for inference. |
0 commit comments