Repository navigation
Commit 92a8ba9
authored
Qualcomm AI Engine Direct - Update value cache for all heads in lookahead mode (pytorch#16362)
### Summary
- As mentioned in the title, it currently only updates once, but it
should update for each num_heads.
- Fixed soc_model of SM8350
<img width="730" height="241" alt="image"
src="https://github.com/user-attachments/assets/372a6ff1-4f6b-44f8-91a3-0852758a6e36"
/>
### Test Plan
```
python examples/qualcomm/oss_scripts/llama/llama.py -b build-android -H <host> -s <device id> -m SM8750 --temperature 0 --checkpoint consolidated.00.pth --params params.json --tokenizer_model tokenizer.model --decoder_model llama3_2-1b_instruct --model_mode lookahead --prefill_ar_len 64 --ngram 9 --window 4 --gcap 4 --max_seq_len 1024 --prompt "I would like to learn python, could you teach me with a simple example?" --artifact test_llama321b_lookahead944/
```1 parent 6e8b206 commit 92a8ba9
3 files changed
Lines changed: 15 additions & 11 deletions
File tree
- backends/qualcomm/serialization
- examples/qualcomm/oss_scripts/llama/runner
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
37 | | - | |
| 37 | + | |
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
43 | | - | |
| 43 | + | |
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
348 | 348 | | |
349 | 349 | | |
350 | 350 | | |
351 | | - | |
352 | | - | |
353 | | - | |
354 | | - | |
355 | | - | |
356 | | - | |
357 | | - | |
358 | | - | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
359 | 362 | | |
360 | | - | |
| 363 | + | |
| 364 | + | |
361 | 365 | | |
362 | 366 | | |
363 | 367 | | |
| |||
0 commit comments