Repository navigation
Commit c461421
authored
Qualcomm AI Engine Direct - [LLM QAT] LLM Quant-Aware Distillation (QAD) (pytorch#21036)
### Summary
Training:
- Add BaseTrainer, Trainer (CE), KDTrainer (knowledge distillation)
- Add CrossEntropyLoss, KLDivergenceLoss, linear warm-up cosine LR
scheduler
- Add TrainingArgs dataclass (epochs, lr, alpha, temperature,
grad_accum_steps, warmup_ratio, lr_config)
Data pipeline:
- Add DecoderDatasetBuilder.from_hf_source for HuggingFace SFT chat
datasets
- Add build_qat_dataloaders: explicit calib/train split
- Add LLMTrainingCollator, make_causal_labels, make_conversation_labels
- Add DataConfig train fields: train_tasks, train_hf_dataset,
train_hf_limit
Quantization Strategy:
- Add QATStrategy: PTQ calibration pass followed by KDTrainer/Trainer
fine-tuning
- Branch TextDecoder.quantize on --qat: prepare_qat_pt2e +
move_exported_model_to_train vs prepare_pt2e
- Select qat_recipe over quant_recipe when --qat is active
Quant recipe:
- Add StaticLLMQATRecipe base class as a example
- Add Smollm2QATQuantRecipe; add qat_recipe field to LLMModelConfig
- Add 16a8w QAT qconfig
Fix:
- SeqMSE: unwrap FakeQuantize wrapper before extracting observer
CI Testing:
- Add test_static_llm_qat: assert QAT PPL < PTQ PPL on smollm2_135m
README:
- `examples/qualcomm/oss_scripts/llama/quantization_guidance.md`
#### E2E script:
``` bash
python examples/qualcomm/oss_scripts/llama/llama.py --build_folder build-android --device ${SERIAL_NUM} --soc_model ${SOC_MODEL} --decoder_model smollm2_135m --model_mode hybrid --max_seq_len 1024 --prompt "What dose it mean to edit written content?" --qat --train_hf_dataset "HuggingFaceTB/smol-smoltalk" --train_hf_limit 1000 --calib_hf_dataset "HuggingFaceTB/smol-smoltalk" --calib_hf_limit 1000
```
#### Result
``` bash
I 00:00:00.284922 executorch:runner.cpp:228] Reading metadata from model
I 00:00:00.285219 executorch:runner.cpp:360] creating io_memory
I 00:00:00.285856 executorch:prompt_processor.cpp:273] Prompt Processor: total 18 prompt tokens (AR-32 * 1 iters)
I 00:00:00.295011 executorch:runner.cpp:473] RSS after prompt prefill: 181.175781 MiB (0 if unsupported)
I 00:00:01.262212 executorch:token_generator.cpp:347]
Reached to the end of generation
I 00:00:01.262249 executorch:runner.cpp:490] RSS after finishing text generation: 181.175781 MiB (0 if unsupported)
I 00:00:01.262275 executorch:stats.h:161] Prompt Tokens: 18 Generated Tokens: 355
I 00:00:01.262284 executorch:stats.h:167] Model Load Time: 0.284000 (seconds)
I 00:00:01.262291 executorch:stats.h:176] Total inference time: 0.977000 (seconds) Rate: 363.357216 (tokens/second)
I 00:00:01.262298 executorch:stats.h:185] Prompt evaluation: 0.010000 (seconds) Rate: 1800.000000 (tokens/second)
I 00:00:01.262305 executorch:stats.h:197] Generated 355 tokens: 0.967000 (seconds) Rate: 367.114788 (tokens/second)
I 00:00:01.262311 executorch:stats.h:205] Time to first generated token: 0.010000 (seconds)
I 00:00:01.262318 executorch:stats.h:212] Sampling time over 373 tokens: 0.112000 (seconds)
[INFO] [Qnn ExecuTorch]: Destroy Qnn context
[INFO] [Qnn ExecuTorch]: Destroy Qnn device
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend
PyTorchObserver {"prefill_token_per_sec":1800,"decode_token_per_sec":367.115,"prompt_tokens":18,"generated_tokens":355,"model_load_start_ms":1753685358963,"model_load_end_ms":1753685359247,"inference_start_ms":1753685359247,"inference_end_ms":1753685360224,"prompt_eval_end_ms":1753685359257,"first_token_ms":1753685359257,"aggregate_sampling_time_ms":112,"SCALING_FACTOR_UNITS_PER_SECOND":1000}
/data/local/tmp/yuyazhua/executorch/static_llm/outputs/outputs.txt: 1 file pulled, 0 skipped. 1.8 MB/s (1714 bytes in 0.001s)
/data/local/tmp/yuyazhua/executorch/static_llm/outputs/inference_speed.txt: 1 file pulled, 0 skipped. 0.0 MB/s (7 bytes in 0.001s)
[INFO 2026-07-20 12:10:52,191 llama.py:290] Device Inference Results[0]:
<|im_start|>user
What dose it mean to edit written content?<|im_end|>
<|im_start|>assistant
1. **Editing for clarity**: When you edit your writing, you aim to make it clear and concise, ensuring that your message is easy for readers to understand and retain. This involves cutting unnecessary words, phrases, or sentences, and rephrasing or reorganizing your content to make it more readable.
2. **Editing for grammar**: Grammar is another crucial aspect of writing. When you edit your writing, you check for errors in grammar, syntax, and punctuation, which can make your writing more polished and error-free.
3. **Editing for style**: The final stage of editing is where you refine your writing to make it more engaging, clear, and engaging. This involves using language that is engaging, yet clear, and engaging in the way you want it to be.
4. **Editing for impact**: The final stage of editing is where you aim to make your writing more impactful. This means that you strive to make your message clear, concise, and impactful, and to convey it in a way that resonates with your readers.
5. **Editing for consistency**: When you edit your writing, you ensure that it is consistent in terms of tone, style, and voice. This involves using the same language, vocabulary, and structure to create a consistent tone, which is essential for effective communication.
To get started, take a step back and look at your writing. Read it aloud, if possible, to get a sense of the tone, style, and voice you're using. Then, read your writing aloud to get a sense of the rhythm, cadence, and flow.
When you're on the right track, you can begin to refine your writing and make it more engaging and effective.<|im_end|>
```
### Test plan
``` bash
python backends/qualcomm/tests/test_qnn_delegate.py TestExampleLLMScript.test_static_llm_qat --device ${SERIAL_NUM} --soc_model ${SOC_MODEL} -r . -a . --build_folder build-android
```
cc: @shewu-quic @haowhsu-quic @winskuo-quic @psiddh @abhinaykukkadapu1 parent 4753d75 commit c461421
35 files changed
Lines changed: 1870 additions & 80 deletions
File tree
- backends/qualcomm
- _passes
- quantizer
- tests
- examples/qualcomm/oss_scripts/llama
- assets/quantization
- dataset
- inference
- quantize
- train
- config
- wrappers
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
160 | 160 | | |
161 | 161 | | |
162 | 162 | | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
163 | 166 | | |
164 | 167 | | |
165 | 168 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
998 | 998 | | |
999 | 999 | | |
1000 | 1000 | | |
1001 | | - | |
| 1001 | + | |
1002 | 1002 | | |
1003 | 1003 | | |
1004 | 1004 | | |
1005 | 1005 | | |
1006 | 1006 | | |
1007 | 1007 | | |
1008 | 1008 | | |
1009 | | - | |
| 1009 | + | |
1010 | 1010 | | |
1011 | 1011 | | |
1012 | 1012 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
207 | 207 | | |
208 | 208 | | |
209 | 209 | | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
210 | 219 | | |
211 | 220 | | |
212 | 221 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9149 | 9149 | | |
9150 | 9150 | | |
9151 | 9151 | | |
| 9152 | + | |
| 9153 | + | |
| 9154 | + | |
| 9155 | + | |
| 9156 | + | |
| 9157 | + | |
| 9158 | + | |
| 9159 | + | |
| 9160 | + | |
| 9161 | + | |
| 9162 | + | |
| 9163 | + | |
| 9164 | + | |
| 9165 | + | |
| 9166 | + | |
| 9167 | + | |
| 9168 | + | |
| 9169 | + | |
| 9170 | + | |
| 9171 | + | |
| 9172 | + | |
| 9173 | + | |
| 9174 | + | |
| 9175 | + | |
| 9176 | + | |
| 9177 | + | |
| 9178 | + | |
| 9179 | + | |
| 9180 | + | |
| 9181 | + | |
| 9182 | + | |
| 9183 | + | |
| 9184 | + | |
| 9185 | + | |
| 9186 | + | |
| 9187 | + | |
| 9188 | + | |
| 9189 | + | |
| 9190 | + | |
| 9191 | + | |
| 9192 | + | |
| 9193 | + | |
| 9194 | + | |
| 9195 | + | |
| 9196 | + | |
| 9197 | + | |
| 9198 | + | |
| 9199 | + | |
| 9200 | + | |
| 9201 | + | |
| 9202 | + | |
| 9203 | + | |
| 9204 | + | |
| 9205 | + | |
| 9206 | + | |
| 9207 | + | |
| 9208 | + | |
| 9209 | + | |
| 9210 | + | |
| 9211 | + | |
| 9212 | + | |
| 9213 | + | |
| 9214 | + | |
| 9215 | + | |
| 9216 | + | |
| 9217 | + | |
| 9218 | + | |
| 9219 | + | |
| 9220 | + | |
| 9221 | + | |
| 9222 | + | |
| 9223 | + | |
9152 | 9224 | | |
9153 | 9225 | | |
9154 | 9226 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
584 | 584 | | |
585 | 585 | | |
586 | 586 | | |
587 | | - | |
588 | | - | |
589 | 587 | | |
590 | 588 | | |
591 | 589 | | |
| |||
617 | 615 | | |
618 | 616 | | |
619 | 617 | | |
620 | | - | |
621 | | - | |
622 | | - | |
623 | | - | |
624 | | - | |
625 | | - | |
626 | | - | |
627 | | - | |
628 | | - | |
629 | | - | |
630 | | - | |
631 | | - | |
632 | | - | |
| 618 | + | |
633 | 619 | | |
634 | 620 | | |
635 | 621 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
87 | 87 | | |
88 | 88 | | |
89 | 89 | | |
| 90 | + | |
90 | 91 | | |
91 | 92 | | |
92 | 93 | | |
| |||
129 | 130 | | |
130 | 131 | | |
131 | 132 | | |
| 133 | + | |
| 134 | + | |
132 | 135 | | |
133 | 136 | | |
134 | 137 | | |
| |||
145 | 148 | | |
146 | 149 | | |
147 | 150 | | |
| 151 | + | |
148 | 152 | | |
149 | 153 | | |
150 | 154 | | |
| |||
545 | 549 | | |
546 | 550 | | |
547 | 551 | | |
| 552 | + | |
548 | 553 | | |
549 | 554 | | |
550 | 555 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
11 | 11 | | |
12 | 12 | | |
13 | 13 | | |
| 14 | + | |
14 | 15 | | |
15 | 16 | | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
16 | 20 | | |
17 | 21 | | |
18 | 22 | | |
| |||
21 | 25 | | |
22 | 26 | | |
23 | 27 | | |
| 28 | + | |
24 | 29 | | |
25 | 30 | | |
26 | 31 | | |
27 | 32 | | |
28 | 33 | | |
29 | 34 | | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
30 | 39 | | |
31 | 40 | | |
32 | 41 | | |
33 | 42 | | |
34 | 43 | | |
35 | 44 | | |
| 45 | + | |
| 46 | + | |
36 | 47 | | |
37 | 48 | | |
38 | 49 | | |
39 | 50 | | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
40 | 55 | | |
41 | 56 | | |
42 | 57 | | |
43 | 58 | | |
44 | 59 | | |
45 | 60 | | |
| 61 | + | |
46 | 62 | | |
47 | 63 | | |
48 | 64 | | |
| |||
0 commit comments