Repository navigation
Commit c55c31b
authored
Qualcomm AI Engine Direct - Add HTP context graph splitting option (pytorch#22595)
## Summary
- Exposes QNN's HTP context graph-splitting config
(`QNN_HTP_CONTEXT_CONFIG_OPTION_GRAPH_SPLITTING_CONFIGS` /
`QnnHtpContext_GraphSplit_t`) via a new `use_graph_splitting` option on
`generate_htp_compiler_spec()`, see
https://docs.qualcomm.com/doc/80-63442-10/topic/htp_backend.html. This
option significantly reduces finalization time at the cost of a slight
increase in execution time.
### Misc
- Wired up for the offline-prepare (host/x86 export) path only, since
this feature is used in finalize.
- Gated behind `QNN_HTP_API_VERSION_MAJOR >= 5 &&
QNN_HTP_API_VERSION_MINOR >= 49` since this option doesn't exist in
older QNN_HTP_API_VERSION
## Test plan
```
python backends/qualcomm/tests/test_qnn_delegate.py -k "test_qnn_backend_graph_splitting" --device ef5e4029 --host localhost --soc_model SM8850 --build_folder build-android --executorch_root .
```
LLAMA3.2 3B performance:
| Graph | Graph Preparation Initializing (us) | Graph Optimizations (us)
| Post Graph Optimization (us) | Graph Sequencing for Target (us) | VTCM
Allocation (us) | Parallelization Optimization (us) | Finalizing Graph
Sequence (us) | Completion (us) | QNN (execute) time (us) |
|---|---|---|---|---|---|---|---|---|---|
| kv_forward1_without_gpe | 1146 | 26843138 | 433549 | 6828982 | 449030
| 5929118 | 266695 | 15171 | 9552 |
| kv_forward1_with_gpe | 496 | 2014382 | 26630 | 427743 | 27504 | 246911
| 22066 | 1268 | 9880 |
| kv_forward2_without_gpe | 373 | 27618966 | 1049136 | 10700233 | 313808
| 7947260 | 291464 | 14841 | 9452 |
| kv_forward2_with_gpe | 599 | 2511473 | 34099 | 577162 | 36742 | 353654
| 31248 | 1754 | 9787 |
| kv_forward3_without_gpe | 405 | 30262309 | 541066 | 8523872 | 407082 |
8058194 | 314898 | 16692 | 11136 |
| kv_forward3_with_gpe | 512 | 5575498 | 75421 | 1322133 | 111640 |
1009982 | 51157 | 3371 | 11437 |
| kv_forward4_without_gpe | 1966 | 10304161 | 36241 | 1151113 | 87287 |
749097 | 34706 | 1944 | 6703 |
| kv_forward4_with_gpe | 342 | 8387236 | 41997 | 1104533 | 96270 |
870457 | 31896 | 2245 | 6686 |
| prefill_forward1_without_gpe | 403 | 23425582 | 521963 | 20288202 |
28141820 | 13415019 | 534725 | 19491 | 23283 |
| prefill_forward1_with_gpe | 635 | 2404414 | 42852 | 928676 | 607167 |
368336 | 25194 | 1790 | 20824 |
| prefill_forward2_without_gpe | 354 | 22249865 | 496426 | 16837420 |
24738364 | 9634081 | 464298 | 16200 | 20473 |
| prefill_forward2_with_gpe | 557 | 2521494 | 41107 | 929431 | 765368 |
426039 | 28336 | 1955 | 20764 |
| prefill_forward3_without_gpe | 744 | 31356868 | 553758 | 17889195 |
17232990 | 9485483 | 500217 | 15025 | 21125 |
| prefill_forward3_with_gpe | 1933 | 5689220 | 99188 | 2404130 | 2256161
| 1236475 | 79918 | 6054 | 21835 |
| prefill_forward4_without_gpe | 889 | 10117697 | 43571 | 1450046 |
1323893 | 1062400 | 35717 | 2122 | 9431 |
| prefill_forward4_with_gpe | 331 | 8484639 | 50456 | 1647489 | 1798310
| 1306534 | 45152 | 2760 | 9446 |
This table is created by extracting data from finalization log:
<img width="392" height="351" alt="image"
src="https://github.com/user-attachments/assets/76f876f3-5084-4ac3-8be6-54b7971402e8"
/>
This PR was authored with assistance from Claude Code (Anthropic).
cc @cbilgin @psiddh1 parent c9595a6 commit c55c31b
8 files changed
Lines changed: 179 additions & 0 deletions
File tree
- backends/qualcomm
- runtime/backends
- htp/host
- serialization
- tests
- rework
- htp/feature/v68
- src
- utils
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
60 | 60 | | |
61 | 61 | | |
62 | 62 | | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
63 | 66 | | |
64 | 67 | | |
65 | 68 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
29 | 52 | | |
30 | 53 | | |
31 | 54 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
206 | 206 | | |
207 | 207 | | |
208 | 208 | | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
209 | 213 | | |
210 | 214 | | |
211 | 215 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
203 | 203 | | |
204 | 204 | | |
205 | 205 | | |
| 206 | + | |
206 | 207 | | |
207 | 208 | | |
208 | 209 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
15 | 25 | | |
16 | 26 | | |
17 | 27 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
71 | 71 | | |
72 | 72 | | |
73 | 73 | | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
74 | 149 | | |
75 | 150 | | |
76 | 151 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7248 | 7248 | | |
7249 | 7249 | | |
7250 | 7250 | | |
| 7251 | + | |
| 7252 | + | |
| 7253 | + | |
| 7254 | + | |
| 7255 | + | |
| 7256 | + | |
| 7257 | + | |
| 7258 | + | |
| 7259 | + | |
| 7260 | + | |
| 7261 | + | |
| 7262 | + | |
| 7263 | + | |
| 7264 | + | |
| 7265 | + | |
| 7266 | + | |
| 7267 | + | |
| 7268 | + | |
| 7269 | + | |
| 7270 | + | |
| 7271 | + | |
| 7272 | + | |
| 7273 | + | |
| 7274 | + | |
| 7275 | + | |
| 7276 | + | |
| 7277 | + | |
| 7278 | + | |
| 7279 | + | |
| 7280 | + | |
7251 | 7281 | | |
7252 | 7282 | | |
7253 | 7283 | | |
| |||
8326 | 8356 | | |
8327 | 8357 | | |
8328 | 8358 | | |
| 8359 | + | |
| 8360 | + | |
| 8361 | + | |
| 8362 | + | |
| 8363 | + | |
| 8364 | + | |
| 8365 | + | |
| 8366 | + | |
| 8367 | + | |
| 8368 | + | |
| 8369 | + | |
| 8370 | + | |
| 8371 | + | |
| 8372 | + | |
| 8373 | + | |
| 8374 | + | |
| 8375 | + | |
| 8376 | + | |
| 8377 | + | |
| 8378 | + | |
| 8379 | + | |
| 8380 | + | |
| 8381 | + | |
| 8382 | + | |
| 8383 | + | |
| 8384 | + | |
| 8385 | + | |
| 8386 | + | |
| 8387 | + | |
8329 | 8388 | | |
8330 | 8389 | | |
8331 | 8390 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1094 | 1094 | | |
1095 | 1095 | | |
1096 | 1096 | | |
| 1097 | + | |
1097 | 1098 | | |
1098 | 1099 | | |
1099 | 1100 | | |
| |||
1113 | 1114 | | |
1114 | 1115 | | |
1115 | 1116 | | |
| 1117 | + | |
| 1118 | + | |
1116 | 1119 | | |
1117 | 1120 | | |
1118 | 1121 | | |
| |||
1131 | 1134 | | |
1132 | 1135 | | |
1133 | 1136 | | |
| 1137 | + | |
1134 | 1138 | | |
1135 | 1139 | | |
1136 | 1140 | | |
| |||
0 commit comments