You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit d19320d
Browse filesBrowse the repository at this point in the historyBrowse files
Qualcomm AI Engine Direct - update llm document (pytorch#20449)
### Summary
- Add --calib_num_fewshot option to control few-shot count for lm_eval
calibration tasks
- Update all README example commands from short flags (-b/-s/-m) to
flags (--build_folder/--device/--soc_model)
### Test plan
LLM CI
python examples/qualcomm/oss_scripts/llama/llama.py -b build-android -s${SERIAL_NUM} -m${SOC_MODEL} --checkpoint consolidated.00.pth --params params.json --tokenizer_model tokenizer.model --decoder_model llama3_2-1b_instruct --model_mode hybrid --prefill_ar_len 128 --max_seq_len 1024 --prompt "I would like to learn python, could you teach me with a simple example?" --calib_tasks wikitext --calib_limit 1
126
+
python examples/qualcomm/oss_scripts/llama/llama.py --build_folder build-android --device${SERIAL_NUM} --soc_model${SOC_MODEL} --checkpoint consolidated.00.pth --params params.json --tokenizer_model tokenizer.model --decoder_model llama3_2-1b_instruct --model_mode hybrid --prefill_ar_len 128 --max_seq_len 1024 --prompt "I would like to learn python, could you teach me with a simple example?" --calib_tasks wikitext --calib_limit 1
127
127
```
128
128
129
129
#### LLAMA3.2 3B Instruct
130
130
Default example using hybrid mode.
131
131
```bash
132
-
python examples/qualcomm/oss_scripts/llama/llama.py -b build-android -s ${SERIAL_NUM} -m ${SOC_MODEL} --checkpoint consolidated.00.pth --params params.json --tokenizer_model tokenizer.model --decoder_model llama3_2-3b_instruct --model_mode hybrid --prefill_ar_len 128 --max_seq_len 1024 --prompt "I would like to learn python, could you teach me with a simple example?" --calib_tasks wikitext --calib_limit 1
132
+
python examples/qualcomm/oss_scripts/llama/llama.py --build_folder build-android --device ${SERIAL_NUM} --soc_model ${SOC_MODEL} --checkpoint consolidated.00.pth --params params.json --tokenizer_model tokenizer.model --decoder_model llama3_2-3b_instruct --model_mode hybrid --prefill_ar_len 128 --max_seq_len 1024 --prompt "I would like to learn python, could you teach me with a simple example?" --calib_tasks wikitext --calib_limit 1
python examples/qualcomm/oss_scripts/llama/llama.py -b build-android -s${SERIAL_NUM} -m${SOC_MODEL} --temperature 0 --model_mode hybrid --max_seq_len 1024 --prefill_ar_len 128 --decoder_model gemma-2b --prompt "I would like to learn python, could you teach me with a simple example?" --calib_tasks wikitext --calib_limit 1
145
+
python examples/qualcomm/oss_scripts/llama/llama.py --build_folder build-android --device${SERIAL_NUM} --soc_model${SOC_MODEL} --temperature 0 --model_mode hybrid --max_seq_len 1024 --prefill_ar_len 128 --decoder_model gemma-2b --prompt "I would like to learn python, could you teach me with a simple example?" --calib_tasks wikitext --calib_limit 1
145
146
```
146
147
147
148
#### Gemma2 2B
148
149
Default example using hybrid mode
149
150
```bash
150
-
python examples/qualcomm/oss_scripts/llama/llama.py -b build-android -s${SERIAL_NUM} -m${SOC_MODEL} --temperature 0 --model_mode hybrid --max_seq_len 1024 --prefill_ar_len 128 --decoder_model gemma2-2b --prompt "I would like to learn python, could you teach me with a simple example?" --calib_tasks wikitext --calib_limit 1
151
+
python examples/qualcomm/oss_scripts/llama/llama.py --build_folder build-android --device${SERIAL_NUM} --soc_model${SOC_MODEL} --temperature 0 --model_mode hybrid --max_seq_len 1024 --prefill_ar_len 128 --decoder_model gemma2-2b --prompt "I would like to learn python, could you teach me with a simple example?" --calib_tasks wikitext --calib_limit 1
151
152
```
152
153
153
154
#### Gemma3 1B
154
155
Default example using hybrid mode
155
156
```bash
156
-
python examples/qualcomm/oss_scripts/llama/llama.py -b build-android -s${SERIAL_NUM} -m${SOC_MODEL} --temperature 0 --model_mode hybrid --max_seq_len 1024 --prefill_ar_len 128 --decoder_model gemma3-1b --prompt "I would like to learn python, could you teach me with a simple example?" --calib_tasks wikitext --calib_limit 1
157
+
python examples/qualcomm/oss_scripts/llama/llama.py --build_folder build-android --device${SERIAL_NUM} --soc_model${SOC_MODEL} --temperature 0 --model_mode hybrid --max_seq_len 1024 --prefill_ar_len 128 --decoder_model gemma3-1b --prompt "I would like to learn python, could you teach me with a simple example?" --calib_tasks wikitext --calib_limit 1
157
158
```
158
159
159
160
#### GLM 1.5B
160
161
Default example using hybrid mode
161
162
```bash
162
-
python examples/qualcomm/oss_scripts/llama/llama.py -b build-android -s${SERIAL_NUM} -m${SOC_MODEL} --temperature 0 --model_mode hybrid --max_seq_len 1024 --prefill_ar_len 128 --decoder_model glm-1_5b --prompt "I would like to learn python, could you teach me with a simple example?" --calib_tasks wikitext --calib_limit 1
163
+
python examples/qualcomm/oss_scripts/llama/llama.py --build_folder build-android --device${SERIAL_NUM} --soc_model${SOC_MODEL} --temperature 0 --model_mode hybrid --max_seq_len 1024 --prefill_ar_len 128 --decoder_model glm-1_5b --prompt "I would like to learn python, could you teach me with a simple example?" --calib_tasks wikitext --calib_limit 1
163
164
```
164
165
165
166
#### Granite3.3 2B
166
167
Default example using hybrid mode
167
168
```bash
168
-
python examples/qualcomm/oss_scripts/llama/llama.py -b build-android -s${SERIAL_NUM} -m${SOC_MODEL} --temperature 0 --model_mode hybrid --max_seq_len 1024 --prefill_ar_len 128 --decoder_model granite_3_3-2b_instruct --prompt "I would like to learn python, could you teach me with a simple example?" --eval_methods tasks_eval --eval_tasks hellaswag --eval_limit 10 --calib_tasks hellaswag --calib_limit 10
169
+
python examples/qualcomm/oss_scripts/llama/llama.py --build_folder build-android --device${SERIAL_NUM} --soc_model${SOC_MODEL} --temperature 0 --model_mode hybrid --max_seq_len 1024 --prefill_ar_len 128 --decoder_model granite_3_3-2b_instruct --prompt "I would like to learn python, could you teach me with a simple example?" --eval_methods tasks_eval --eval_tasks hellaswag --eval_limit 10 --calib_tasks hellaswag --calib_limit 10
169
170
```
170
171
171
172
#### Phi4-mini-instruct
172
173
Default example using hybrid mode.
173
174
```bash
174
-
python examples/qualcomm/oss_scripts/llama/llama.py -b build-android -s${SERIAL_NUM} -m${SOC_MODEL} --decoder_model phi_4_mini --model_mode hybrid --prefill_ar_len 128 --max_seq_len 1024 --prompt "I would like to learn python, could you teach me with a simple example?" --calib_tasks wikitext --calib_limit 1
175
+
python examples/qualcomm/oss_scripts/llama/llama.py --build_folder build-android --device${SERIAL_NUM} --soc_model${SOC_MODEL} --decoder_model phi_4_mini --model_mode hybrid --prefill_ar_len 128 --max_seq_len 1024 --prompt "I would like to learn python, could you teach me with a simple example?" --calib_tasks wikitext --calib_limit 1
175
176
```
176
177
177
178
#### QWEN2.5 0.5B
178
179
Default example using hybrid mode
179
180
```bash
180
-
python examples/qualcomm/oss_scripts/llama/llama.py -b build-android -s${SERIAL_NUM} -m${SOC_MODEL} --temperature 0 --model_mode hybrid --max_seq_len 1024 --prefill_ar_len 128 --decoder_model qwen2_5-0_5b --prompt "I would like to learn python, could you teach me with a simple example?" --calib_tasks wikitext --calib_limit 1
181
+
python examples/qualcomm/oss_scripts/llama/llama.py --build_folder build-android --device${SERIAL_NUM} --soc_model${SOC_MODEL} --temperature 0 --model_mode hybrid --max_seq_len 1024 --prefill_ar_len 128 --decoder_model qwen2_5-0_5b --prompt "I would like to learn python, could you teach me with a simple example?" --calib_tasks wikitext --calib_limit 1
181
182
```
182
183
183
184
#### QWEN2.5 1.5B
184
185
Default example using hybrid mode
185
186
```bash
186
-
python examples/qualcomm/oss_scripts/llama/llama.py -b build-android -s${SERIAL_NUM} -m${SOC_MODEL} --temperature 0 --model_mode hybrid --prefill_ar_len 128 --max_seq_len 1024 --decoder_model qwen2_5-1_5b --prompt "I would like to learn python, could you teach me with a simple example?" --calib_tasks wikitext --calib_limit 1
187
+
python examples/qualcomm/oss_scripts/llama/llama.py --build_folder build-android --device${SERIAL_NUM} --soc_model${SOC_MODEL} --temperature 0 --model_mode hybrid --prefill_ar_len 128 --max_seq_len 1024 --decoder_model qwen2_5-1_5b --prompt "I would like to learn python, could you teach me with a simple example?" --calib_tasks wikitext --calib_limit 1
187
188
```
188
189
189
190
#### QWEN3 0.6B
190
191
Default example using hybrid mode
191
192
```bash
192
-
python examples/qualcomm/oss_scripts/llama/llama.py -b build-android -s${SERIAL_NUM} -m${SOC_MODEL} --temperature 0 --model_mode hybrid --max_seq_len 1024 --prefill_ar_len 128 --decoder_model qwen3-0_6b --prompt "I would like to learn python, could you teach me with a simple example?" --calib_tasks wikitext --calib_limit 1
193
+
python examples/qualcomm/oss_scripts/llama/llama.py --build_folder build-android --device${SERIAL_NUM} --soc_model${SOC_MODEL} --temperature 0 --model_mode hybrid --max_seq_len 1024 --prefill_ar_len 128 --decoder_model qwen3-0_6b --prompt "I would like to learn python, could you teach me with a simple example?" --calib_tasks wikitext --calib_limit 1
193
194
```
194
195
195
196
#### QWEN3 1.7B
196
197
Default example using hybrid mode
197
198
```bash
198
-
python examples/qualcomm/oss_scripts/llama/llama.py -b build-android -s${SERIAL_NUM} -m${SOC_MODEL} --temperature 0 --model_mode hybrid --prefill_ar_len 128 --max_seq_len 1024 --decoder_model qwen3-1_7b --prompt "I would like to learn python, could you teach me with a simple example?" --calib_tasks wikitext --calib_limit 1
199
+
python examples/qualcomm/oss_scripts/llama/llama.py --build_folder build-android --device${SERIAL_NUM} --soc_model${SOC_MODEL} --temperature 0 --model_mode hybrid --prefill_ar_len 128 --max_seq_len 1024 --decoder_model qwen3-1_7b --prompt "I would like to learn python, could you teach me with a simple example?" --calib_tasks wikitext --calib_limit 1
199
200
```
200
201
201
202
#### SmolLM2
202
203
Default example using hybrid mode.
203
204
```bash
204
-
python examples/qualcomm/oss_scripts/llama/llama.py -b build-android -s${SERIAL_NUM} -m${SOC_MODEL} --decoder_model smollm2_135m --model_mode hybrid --prefill_ar_len 128 --max_seq_len 1024 --prompt "I would like to learn python, could you teach me with a simple example?" --calib_tasks wikitext --calib_limit 1
205
+
python examples/qualcomm/oss_scripts/llama/llama.py --build_folder build-android --device${SERIAL_NUM} --soc_model${SOC_MODEL} --decoder_model smollm2_135m --model_mode hybrid --prefill_ar_len 128 --max_seq_len 1024 --prompt "I would like to learn python, could you teach me with a simple example?" --calib_tasks wikitext --calib_limit 1
205
206
```
206
207
207
208
#### SmolLM3
208
209
Default example using hybrid mode.
209
210
```bash
210
-
python examples/qualcomm/oss_scripts/llama/llama.py -b build-android -s${SERIAL_NUM} -m${SOC_MODEL} --decoder_model smollm3-3b --model_mode hybrid --prefill_ar_len 128 --max_seq_len 1024 --prompt "I would like to learn python, could you teach me with a simple example?" --calib_tasks wikitext --calib_limit 1
211
+
python examples/qualcomm/oss_scripts/llama/llama.py --build_folder build-android --device${SERIAL_NUM} --soc_model${SOC_MODEL} --decoder_model smollm3-3b --model_mode hybrid --prefill_ar_len 128 --max_seq_len 1024 --prompt "I would like to learn python, could you teach me with a simple example?" --calib_tasks wikitext --calib_limit 1
211
212
```
212
213
213
214
#### Using custom calibration samples for LLMs
214
215
215
216
Instead of `--calib_tasks`, you can supply your own conversation JSON files via `--calib_samples`. The samples are fed into the quantization calibration pass to collect activation observer statistics — they do not affect the inference prompt. This is useful when you want to calibrate on domain-specific or instruct-format data rather than a generic lm_eval task.
216
217
217
218
```bash
218
-
python examples/qualcomm/oss_scripts/llama/llama.py -b build-android -s${SERIAL_NUM} -m${SOC_MODEL} --decoder_model smollm2_135m --model_mode hybrid --prefill_ar_len 128 --max_seq_len 1024 --prompt "I would like to learn python, could you teach me with a simple example?" --calib_samples examples/qualcomm/oss_scripts/llama/assets/samples/text.json
219
+
python examples/qualcomm/oss_scripts/llama/llama.py --build_folder build-android --device${SERIAL_NUM} --soc_model${SOC_MODEL} --decoder_model smollm2_135m --model_mode hybrid --prefill_ar_len 128 --max_seq_len 1024 --prompt "I would like to learn python, could you teach me with a simple example?" --calib_samples examples/qualcomm/oss_scripts/llama/assets/samples/text.json
219
220
```
220
221
221
222
You can also provide both `--calib_tasks` and `--calib_samples` at the same time; the pipeline concatenates both data sources for calibration.
222
223
223
224
225
+
## Multimodal Support
224
226
225
227
### Overview
226
228
@@ -455,19 +457,16 @@ The VLM inference pipeline consists of:
455
457
- Special tokens (e.g., `<image>`, `<|fake_token_around_image|>`, `<fake_token_around_image>`) mark modality boundaries (see [tokenizer.py](tokenizer.py))
456
458
457
459
```python
458
-
# Token fields on each encoder config subclass (encoder/encoder_config.py)
459
-
@dataclass(init=False, frozen=True)
460
-
class SmolVLMEncoder(VisionModalityConfig):
461
-
img_token = "<image>"
462
-
fake_wrap_start = "<fake_token_around_image>"
463
-
fake_wrap_end = "<fake_token_around_image>"
464
-
global_img_token = "<global-img>"
465
-
466
-
@dataclass(init=False, frozen=True)
467
-
class InternVL3Encoder(VisionModalityConfig):
468
-
img_token = "<IMG_CONTEXT>"
469
-
fake_wrap_start = "<img>"
470
-
fake_wrap_end = "</img>"
460
+
# Special tokens for Vision-Language Model
461
+
VLM_SPECIAL_TOKENS= {
462
+
"smolvlm_500m_instruct": {
463
+
"image_token": "<image>",
464
+
"global_img": "<global-img>",
465
+
"fake_wrap_start": "<fake_token_around_image>",
466
+
"fake_wrap_end": "<fake_token_around_image>",
467
+
},
468
+
...
469
+
}
471
470
```
472
471
- Final fused sequence: `[batch, img_seq_len + text_seq_len, hidden_dim]`
473
472
@@ -543,18 +542,18 @@ To evaluate the perplexity across all 3 phases, users should provide the `--eval
543
542
544
543
For example, using the Qwen model and 1 wikitext sample as the evaluation task, users can assess all 3 phases perplexity score in a single run by including the appropriate configuration:
545
544
```bash
546
-
python examples/qualcomm/oss_scripts/llama/llama.py -b build-android -s${SERIAL_NUM} -m${SOC_MODEL} --prompt "I would like to learn python, could you teach me with a simple example?" --temperature 0 --model_mode kv --max_seq_len 1024 --decoder_model qwen2_5-0_5b --eval_methods tasks_eval --calib_tasks wikitext --calib_limit 1 --eval_tasks wikitext --eval_limit 1 --verbose
545
+
python examples/qualcomm/oss_scripts/llama/llama.py --build_folder build-android --device${SERIAL_NUM} --soc_model${SOC_MODEL} --prompt "I would like to learn python, could you teach me with a simple example?" --temperature 0 --model_mode kv --max_seq_len 1024 --decoder_model qwen2_5-0_5b --eval_methods tasks_eval --calib_tasks wikitext --calib_limit 1 --eval_tasks wikitext --eval_limit 1 --verbose
547
546
```
548
547
549
548
From the example script above, 1 wikitext sample is used to evaluate all 3 phases. However, there are cases where a user may want to use one sample for quantization calibration and multiple samples for perplexity evaluation. In this case, the process should be split into two runs. In the 1st run, the model is compiled using one sample. In the 2nd run, the user can provide a different configuration for QNN device execution.
550
549
Example:
551
550
```bash
552
551
# 1st run to compile with --calib_limit 1
553
-
python examples/qualcomm/oss_scripts/llama/llama.py -b build-android -s${SERIAL_NUM} -m${SOC_MODEL} --prompt "I would like to learn python, could you teach me with a simple example?" --temperature 0 --model_mode kv --max_seq_len 1024 --decoder_model qwen2_5-0_5b --eval_methods tasks_eval --calib_tasks wikitext --calib_limit 1 -a ${FOLDER_TO_PRE_GEN_PTE} --compile_only
552
+
python examples/qualcomm/oss_scripts/llama/llama.py --build_folder build-android --device${SERIAL_NUM} --soc_model${SOC_MODEL} --prompt "I would like to learn python, could you teach me with a simple example?" --temperature 0 --model_mode kv --max_seq_len 1024 --decoder_model qwen2_5-0_5b --eval_methods tasks_eval --calib_tasks wikitext --calib_limit 1 --compile_only
554
553
```
555
554
```bash
556
555
# 2nd run to perform QNN device execution with --eval_limit 3
557
-
python examples/qualcomm/oss_scripts/llama/llama.py -b build-android -s${SERIAL_NUM} -m${SOC_MODEL} --prompt "I would like to learn python, could you teach me with a simple example?" --temperature 0 --model_mode kv --max_seq_len 1024 --decoder_model qwen2_5-0_5b --eval_methods tasks_eval --eval_tasks wikitext --eval_limit 3 --pre_gen_pte ${FOLDER_TO_PRE_GEN_PTE}
556
+
python examples/qualcomm/oss_scripts/llama/llama.py --build_folder build-android --device${SERIAL_NUM} --soc_model${SOC_MODEL} --prompt "I would like to learn python, could you teach me with a simple example?" --temperature 0 --model_mode kv --max_seq_len 1024 --decoder_model qwen2_5-0_5b --eval_methods tasks_eval --eval_tasks wikitext --eval_limit 3 --pre_gen_pte ${FOLDER_TO_PRE_GEN_PTE}
558
557
```
559
558
560
559
@@ -620,7 +619,7 @@ To automatically identify sensitive layers and generate a mixed-precision recipe
620
619
621
620
Example:
622
621
```bash
623
-
python examples/qualcomm/oss_scripts/llama/llama.py -b build-android -s${SERIAL_NUM} -m${SOC_MODEL} --prompt "I would like to learn python, could you teach me with a simple example?" --temperature 0 --model_mode kv --max_seq_len 1024 --decoder_model qwen3-1_7b --calib_tasks wikitext --calib_limit 1 --quant_recipe_suggestion --compile_only
622
+
python examples/qualcomm/oss_scripts/llama/llama.py --build_folder build-android --device${SERIAL_NUM} --soc_model${SOC_MODEL} --prompt "I would like to learn python, could you teach me with a simple example?" --temperature 0 --model_mode kv --max_seq_len 1024 --decoder_model qwen3-1_7b --calib_tasks wikitext --calib_limit 1 --quant_recipe_suggestion --compile_only
624
623
```
625
624
626
625
After the run, pick one of the generated classes from `qwen3-1_7b_suggest_recipe.py` as your new recipe. For a full walkthrough, see [quantization_guidance.md](quantization_guidance.md).
@@ -649,7 +648,7 @@ This feature supports fluent multi-turn conversations and manages long-context s
649
648
Example:
650
649
```bash
651
650
# Compile llama pte file and attention sink evictor pte file with sink_size = 4 and batch_eviction_size = 64
652
-
python examples/qualcomm/oss_scripts/llama/llama.py -b build-android -s ${SERIAL_NUM} -m ${SOC_MODEL} --checkpoint consolidated.00.pth --params params.json --tokenizer_model tokenizer.model --decoder_model llama3_2-1b_instruct --model_mode hybrid --prefill_ar_len 128 --max_seq_len 4096 --max_context_len 1024 --prompt "I would like to learn python, could you teach me with a simple example?" --calib_tasks wikitext --calib_limit 1 --use_attention_sink 4,64 --compile_only
651
+
python examples/qualcomm/oss_scripts/llama/llama.py --build_folder build-android --device${SERIAL_NUM} --soc_model${SOC_MODEL} --checkpoint consolidated.00.pth --params params.json --tokenizer_model tokenizer.model --decoder_model llama3_2-1b_instruct --model_mode hybrid --prefill_ar_len 128 --max_seq_len 4096 --max_context_len 1024 --prompt "I would like to learn python, could you teach me with a simple example?" --calib_tasks wikitext --calib_limit 1 --use_attention_sink 4,64 --compile_only
653
652
```
654
653
655
654
After running this, the `attention_sink_evictor.pte` file will be generated in the artifacts directory. This file is necessary for using the attention sink feature, as it handles removing the `eviction_batch_size` tokens from the kv cache, retaining the first `sink_size` tokens, and re-rotating the remaining tokens in the kv cache.
0 commit comments