|
1 | | -.. _quantization-aware-training: |
| 1 | +:orphan: |
2 | 2 |
|
3 | | -=============================================== |
4 | 3 | Quantization-Aware Training and Distillation |
5 | | -=============================================== |
| 4 | +============================================ |
6 | 5 |
|
7 | | -Quantization-aware training (QAT) and quantization-aware distillation (QAD) recover |
8 | | -quality lost when a model is quantized. Both train with simulated quantization |
9 | | -enabled, so the resulting checkpoint retains its ModelOpt quantization state for |
10 | | -deployment. |
11 | | - |
12 | | -QAT versus QAD |
13 | | -============== |
14 | | - |
15 | | -QAT is standard supervised fine-tuning of a quantized model. It uses the usual |
16 | | -cross-entropy (CE) loss against labeled data while the quantized forward pass lets |
17 | | -the weights adapt to quantization error. Use QAT to adapt a quantized model to a |
18 | | -task or dataset. |
19 | | - |
20 | | -QAD uses knowledge distillation: a frozen BF16 teacher guides the quantized student |
21 | | -with a logit-level KL-divergence loss. Use QAD after PTQ to recover accuracy lost |
22 | | -specifically to quantization. It requires the teacher during training and therefore |
23 | | -uses more memory and compute than QAT. |
24 | | - |
25 | | -In both cases, start from a PTQ checkpoint and retain its quantization configuration |
26 | | -during training. Choose QAT for task adaptation; choose QAD for quantization-accuracy |
27 | | -recovery. |
28 | | - |
29 | | -QAD rationale |
30 | | -============= |
31 | | - |
32 | | -`Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery |
33 | | -<https://arxiv.org/abs/2601.20088>`_ recommends QAD for recovery after aggressive |
34 | | -quantization, especially for models that have passed through multi-stage |
35 | | -post-training such as SFT, RL, or model merging. The teacher signal makes recovery |
36 | | -more robust when training-data quality or coverage is limited. |
37 | | - |
38 | | -For broader background, see `How Quantization-Aware Training Enables Low-Precision |
39 | | -Accuracy Recovery |
40 | | -<https://developer.nvidia.com/blog/how-quantization-aware-training-enables-low-precision-accuracy-recovery/>`_. The |
41 | | -`Nemotron 3.5 Lightning QAD blog |
42 | | -<https://developer.nvidia.com/blog/developing-nemotron-3-5-lightning-nvfp4-with-qad-using-nvidia-model-optimizer/>`_ |
43 | | -discusses the PTQ-to-QAD-to-export workflow and its scale-handling considerations. |
44 | | - |
45 | | -Choose a framework |
46 | | -================== |
47 | | - |
48 | | -ModelOpt supports QAT with Hugging Face, Megatron-Bridge, and Megatron-LM. |
49 | | - |
50 | | -.. list-table:: |
51 | | - :header-rows: 1 |
52 | | - :widths: 20 40 40 |
53 | | - |
54 | | - * - Framework |
55 | | - - Advantages |
56 | | - - Trade-offs |
57 | | - * - Hugging Face |
58 | | - - Starts directly from Hugging Face checkpoints and is the simplest path for |
59 | | - small to medium models. It supports FSDP2, DDP, and DeepSpeed through |
60 | | - Accelerate, with no conversion to Megatron-Core. |
61 | | - - Its parallelism is less efficient for large-scale training, so it is better |
62 | | - suited to smaller models than the Megatron-based options. |
63 | | - * - Megatron-Bridge |
64 | | - - Automatically converts Hugging Face models to Megatron-Core and uses |
65 | | - Megatron-LM's distributed training stack. It is a convenient scalable |
66 | | - workflow without a separate conversion step. |
67 | | - - The high-level workflow exposes fewer customization points than working |
68 | | - directly in Megatron-LM. |
69 | | - * - Megatron-LM |
70 | | - - Provides the most control over model configuration, data, parallelism, and |
71 | | - training behavior, making it the most customizable option for large-model |
72 | | - training. |
73 | | - - Requires manually converting the model to Megatron-Core and managing that |
74 | | - checkpoint workflow. |
75 | | - |
76 | | -Implementation guides |
77 | | -===================== |
78 | | - |
79 | | -Use the framework README as the executable source of truth. Each guide owns its |
80 | | -prerequisites, commands, data preparation, distributed topology, and export options. |
81 | | - |
82 | | -* `Hugging Face QAT/QAD Quick Start |
83 | | - <https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/llm_qat#quick-start>`_ |
84 | | -* `Megatron-Bridge README |
85 | | - <https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/megatron_bridge>`_ |
86 | | -* `Megatron-LM ModelOpt post-training documentation |
87 | | - <https://github.com/NVIDIA/Megatron-LM/tree/main/examples/post_training/modelopt>`_ |
88 | | - |
89 | | -QAD launcher examples |
90 | | -=================================== |
91 | | - |
92 | | -Model Optimizer includes a `launcher |
93 | | -<https://github.com/NVIDIA/Model-Optimizer/tree/main/tools/launcher>`_ for |
94 | | -running supported QAD pipelines as one-click commands. Follow the launcher's |
95 | | -`Quick Start <https://github.com/NVIDIA/Model-Optimizer/tree/main/tools/launcher#quick-start>`_ |
96 | | -to set it up. |
97 | | - |
98 | | -The `NVIDIA Nemotron 3.5 Lightning QAD launcher examples |
99 | | -<https://github.com/NVIDIA/Model-Optimizer/tree/main/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16>`_ |
100 | | -provide Slurm pipelines for Megatron-Bridge and Megatron-LM. Both create an NVFP4 |
101 | | -student with PTQ, distill it from the BF16 teacher, and export a deployable Hugging |
102 | | -Face checkpoint. |
103 | | - |
104 | | -Review the example directory's README and the selected YAML before running. Adapt |
105 | | -the model and data locations, output paths, and Slurm topology for your environment. |
106 | | - |
107 | | -Run the Megatron-Bridge pipeline: |
108 | | - |
109 | | -.. code-block:: bash |
110 | | -
|
111 | | - cd tools/launcher |
112 | | - source .env-slurm |
113 | | - uv run launch.py \ |
114 | | - --yaml examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/mbridge_qad.yaml \ |
115 | | - --yes |
116 | | -
|
117 | | -Run the Megatron-LM pipeline: |
118 | | - |
119 | | -.. code-block:: bash |
120 | | -
|
121 | | - cd tools/launcher |
122 | | - source .env-slurm |
123 | | - uv run launch.py \ |
124 | | - --yaml examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/megatron_lm_qad.yaml \ |
125 | | - --yes |
| 6 | +This guide has moved. See |
| 7 | +:doc:`quantization_aware_training_and_distillation`. |
0 commit comments