Skip to content

Commit 3887eea

Browse files
authored
Qualcomm AI Engine Direct - HF LLM Optimization Stage 1 (pytorch#22170)
### Summary As HF use case is increasing, we aim to align accuracy and performance of HF LLM with Static LLM(static_llama.py). Currently, there are many gaps, and we have a series of PR to fill the gaps. For now, we will first focus on Llama3.2 1B decode mode. #### What is done - Annotate KV 8bit IO. This largely reduces the KV IO size from 16bits to 8 bits, significantly reducing runtimes memory consumption and improve performance. In order to reuse the existing, annotate_kv_8bit that is used by static llama, we have also made corresponding changes to HF-LLM so it has the same pattern as static llama. - HF LLM Quant Recipe: In order to align performance and accuracy with static llama, a hf llm will also have its own quant_recipe under `examples/qualcomm/oss_scripts/llm_utils/hf_llm_quant_recipe.py`. The quantization technique is mostly migrated from `executorch/examples/qualcomm/oss_scripts/llama/static_llm_quant_recipe.py`, with some minor changes, especially the weight naming as HF-LLM and Static-LLM structure naming is different. - Reuse qnn_llama_runner instead of using Llama Main. This is done by aligning the IO shapes of HF llama in AoT stage, meaning we don't need to modify qnn_llama_runner at all. This provides us more flexibility on model modification and reusing a lot of features already enabled in qnn_llama_runner. - Custom Static Cache: In mainline, we can only dump all KV caches, so the output shape is directly affected by `seq_len`. This increases the memory consumption and is also bad for performance. To resolve this issue, we have implemented `QnnCustomStaticCache` so mode now only outputs the new pair of kv-cache. - Precompute Rope: This can greatly improve the accuracy especially for model that has huge `inv_freq` Additionally, since we are precomputing rope, we can also improve performance. - Convert_Linear_to_Conv added to the stage of annotation: This might be counter intuitive because a linear node is decomposed to a set of nodes {conv2d, reshape, squeeze, permute}. All these should be sharing the same quant_attrs with since nodes since conv2d is equivalent to linear here, and reshape, squeeze, permute is just changing the shape but not values, so they should be able to share the same quant_attrs. However, this becomes tricky when this pass is mixed with `annotate_kv_8bit` in the mainline. This issue is happens during v_proj linear node. When these 2 are used together, the `permute` and `view_copy` will be tagged as 16bit, so it becomes conv(16bit) ->permute(16bit) -> view_copy(16bit) -> cast(8bit). This is bad and different from static llama since if we insert cast(8bit) right after conv(16bit), rest of the ops can run in 8bits, which is faster. After this change, the behavior now aligns with static llama, which is conv(16bit) -> cast(8bit) ->permute(8bit) -> view_copy(8bit). - R3 is removed on purpose for this PR. First, without R3, all models are still generating pretty accurate outputs. Second, to make R3 well align with static_llama, the matrix might be spreading around. It would be better to enable this feature in later PRs when supporting advanced features. - FuseConsecutiveReshape: This fold's extra view_copy node and also allow the model shape to match mha_to_sha. #### Optimization Result Compared with Llama3.2 1B_instruct (HF uses non-instruct verison) For now, accuracy is not compared as HF flow has not yet support SQNR evaluation. Official accuracy features will be supported in Upcoming PRs. However, this PR did ensure HF output is reasonable and readable. | llama3.2-1b | Static (Baseline) | HF Before (Mainline) | HF Optimized (This PR)| |--------|--------|--------|--------| | performance (tok/sec) | 64 | 19 | 55 | <img width="1668" height="657" alt="image" src="https://github.com/user-attachments/assets/40fe8738-4438-46fc-9073-075418e97932" /> #### Sample Script `python examples/qualcomm/oss_scripts/hf_causal_lm.py --prompt "Simply put, the theory of relativity states that" --soc_model SM8750 --device $DEVICE --build_folder build-android/ --decoder_model llama3_2-1b --max_seq_len 1024` #### Upcoming Features Performance & Accuracy - As we are now filling the gaps, HF seq=1024 improves from 16 to 55 tokens, and it is now getting close to static llm's performance. For stage 2, we will be focusing on using Qualcomm SDK tools such as qhas to do detailed profiling to compare HF and Static model, understanding the gap, analyze and see if we can fill rest of the gaps. Feature - Reusing Eval Tools under Static Llama - Reorg model specific configs. - Adding R3 and migrating more StaticLlama source model optimization. ### Test plan - Test for pass: 1. `pytest backends/qualcomm/tests/rework/passes/test.py -k "test_fuse_consecutive_reshape"` 2. `pytest backends/qualcomm/tests/rework/passes/test.py -k "test_convert_linear_to_conv2d"` - E2E test updated: `python backends/qualcomm/tests/test_qnn_delegate.py TestExampleLLMScript.test_hf_causal_lm --device $DEVICE--soc_model SM8750 --build_folder build-android --executorch_root . --artifact_dir ./llama3_2_1b --model_name llama3_2-1b`
1 parent 89387f6 commit 3887eea

19 files changed

Lines changed: 1132 additions & 292 deletions

‎backends/qualcomm/_passes/__init__.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,7 @@
5353
from .fixed_linear_keep_dim import FixedLinearKeepDim
5454
from .fold_qdq import FoldQDQ
5555
from .fuse_consecutive_cast import FuseConsecutiveCast
56+
from .fuse_consecutive_reshape import FuseConsecutiveReshape
5657
from .fuse_consecutive_transpose import FuseConsecutiveTranspose
5758
from .i64_to_i32 import I64toI32
5859
from .insert_cast_for_fp_act_quantized_weight import InsertCastForFpActQuantizedWeight
@@ -123,6 +124,7 @@
123124
FixedLinearKeepDim,
124125
FoldQDQ,
125126
FuseConsecutiveCast,
127+
FuseConsecutiveReshape,
126128
FuseConsecutiveTranspose,
127129
I64toI32,
128130
InsertCastForFpActQuantizedWeight,

‎backends/qualcomm/_passes/backends/gpu/qnn_gpu_pass_manager.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ def get_passes_dependency_for_capture_program(cls):
3232
return deps
3333

3434
@classmethod
35-
def get_annotation_passes(cls):
35+
def get_annotation_passes(cls, convert_linear_to_conv2d: bool = False):
3636
# The annotation pipeline is skipped for the GPU backend, as it does not
3737
# support quantized data types. Return an empty list to indicate a no-op.
3838
return []

‎backends/qualcomm/_passes/backends/htp/qnn_htp_pass_manager.py‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -36,9 +36,13 @@ def get_passes_dependency_for_capture_program(cls):
3636
return deps
3737

3838
@classmethod
39-
def get_annotation_passes(cls):
39+
def get_annotation_passes(cls, convert_linear_to_conv2d: bool = False):
4040
passes = [DecomposeReciprocal, RecomposeHadamard]
41-
passes.extend(super().get_annotation_passes())
41+
passes.extend(
42+
super().get_annotation_passes(
43+
convert_linear_to_conv2d=convert_linear_to_conv2d,
44+
)
45+
)
4246
return passes
4347

4448
@classmethod

‎backends/qualcomm/_passes/backends/lpai/qnn_lpai_pass_manager.py‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -65,9 +65,13 @@ def _validate_edge_passes(self) -> None:
6565
), "Please ensure LpaiPartitionFallbackSupport is the last pass."
6666

6767
@classmethod
68-
def get_annotation_passes(cls):
68+
def get_annotation_passes(cls, convert_linear_to_conv2d: bool = False):
6969
passes = [DecomposeHardsigmoid, DecomposeReciprocal]
70-
passes.extend(super().get_annotation_passes())
70+
passes.extend(
71+
super().get_annotation_passes(
72+
convert_linear_to_conv2d=convert_linear_to_conv2d,
73+
)
74+
)
7175
return passes
7276

7377
@classmethod

‎backends/qualcomm/_passes/convert_linear_to_conv2d.py‎

Lines changed: 108 additions & 65 deletions
Original file line numberDiff line numberDiff line change
@@ -4,11 +4,16 @@
44
# This source code is licensed under the BSD-style license found in the
55
# LICENSE file in the root directory of this source tree.
66

7+
from typing import Optional
8+
79
import torch
810

911
from executorch.backends.qualcomm._passes.utils import copy_meta
1012
from executorch.backends.qualcomm.builders.node_visitor import dq_ops
11-
from executorch.backends.qualcomm.builders.utils import get_parameter
13+
from executorch.backends.qualcomm.builders.utils import (
14+
get_attr_from_target,
15+
get_parameter,
16+
)
1217
from executorch.exir.dialects._ops import ops as exir_ops
1318
from executorch.exir.dialects.edge._ops import EdgeOpOverload
1419
from executorch.exir.pass_base import ExportPass, PassResult
@@ -25,12 +30,47 @@ def _pad_list_to_4(lst):
2530

2631
class ConvertLinearToConv2d(ExportPass):
2732
"""
28-
Replace aten.linear.default with equivalent 1x1 conv2d using call_function nodes.
33+
Replace linear with an equivalent 1x1 convolution using call_function nodes.
2934
"""
3035

31-
def __init__(self, edge_program: torch.export.ExportedProgram):
36+
def __init__(
37+
self, edge_program: Optional[torch.export.ExportedProgram] = None
38+
) -> None:
3239
super().__init__()
3340
self.edge_program = edge_program
41+
self.aten_linear_target = torch.ops.aten.linear.default
42+
self.edge_linear_target = exir_ops.edge.aten.linear.default
43+
self._op_table = {
44+
# (is_edge) -> {role: target}
45+
True: {
46+
"view": exir_ops.edge.aten.view_copy.default,
47+
"permute": exir_ops.edge.aten.permute_copy.default,
48+
"conv": exir_ops.edge.aten.convolution.default,
49+
"linear": exir_ops.edge.aten.linear.default,
50+
},
51+
False: {
52+
"view": torch.ops.aten.reshape.default,
53+
"permute": torch.ops.aten.permute.default,
54+
"conv": torch.ops.aten.conv2d.default,
55+
"linear": torch.ops.aten.linear.default,
56+
},
57+
}
58+
59+
def _rewrite_module_stack(self, node, suffix: str):
60+
"""
61+
Append a suffix to nn_module_stack so the quant recipe can differentiate
62+
the converted node. Without this, the conv inherits the linear's
63+
nn_module_stack verbatim, which makes regex targeting in the recipe
64+
ambiguous (e.g. ``model.lm_head`` vs ``model.lm_head.conv``).
65+
"""
66+
stack = node.meta.get("nn_module_stack")
67+
if not stack:
68+
return
69+
stack = dict(stack)
70+
last_key = next(reversed(stack))
71+
path, qualname = stack[last_key]
72+
stack[last_key] = (path + suffix, qualname)
73+
node.meta["nn_module_stack"] = stack
3474

3575
def _register_tensor(
3676
self,
@@ -79,47 +119,26 @@ def _reshape_weight_for_all_users(
79119
self,
80120
graph_module: torch.fx.GraphModule,
81121
weight_placeholder_node: torch.fx.Node,
122+
ops: dict,
82123
) -> torch.fx.Node:
83-
weight_val = get_parameter(weight_placeholder_node, self.edge_program)
124+
weight_val = (
125+
get_attr_from_target(graph_module, weight_placeholder_node.target)
126+
if weight_placeholder_node.op == "get_attr"
127+
else get_parameter(weight_placeholder_node, self.edge_program)
128+
)
84129
assert weight_val is not None, "Cannot get the weight in linear node."
85130

86131
weight_val = weight_val.reshape(*weight_val.shape, 1, 1).contiguous().detach()
87132
get_attr_node = self._register_tensor(
88133
graph_module, weight_placeholder_node, weight_val
89134
)
90-
if list(weight_placeholder_node.users)[0].target in dq_ops:
91-
# Scenarios where multiple linear nodes share the same weights, such as the embedding and lm_head in LLM.
92-
for dq_node in list(weight_placeholder_node.users):
93-
if (
94-
list(dq_node.users)[0].target
95-
is not exir_ops.edge.aten.linear.default
96-
):
97-
# Add a safety check to prevent replacing weights of non-linear nodes with updated weights.
98-
continue
99-
# For shared weights, a dequantize node is inserted per user after quantization.
100-
# Reuse the dequantize node after weight updates by replacing its input with the corresponding `get_attr` node.
101-
dq_node.replace_input_with(weight_placeholder_node, get_attr_node)
102-
103-
fake_mode = detect_fake_mode(get_attr_node.meta["val"])
104-
converter = fake_mode.fake_tensor_converter
105-
dq_node.meta["val"] = converter.from_real_tensor(fake_mode, weight_val)
106-
107-
# Update block size for per-block quant
108-
if dq_node.target is exir_ops.edge.torchao.dequantize_affine.default:
109-
new_args = list(dq_node.args)
110-
# pad block size
111-
new_args[1] = _pad_list_to_4(list(new_args[1]))
112-
dq_node.args = tuple(new_args)
113-
114-
return dq_node
115-
else:
116-
for user in list(weight_placeholder_node.users):
117-
if user.target is not exir_ops.edge.aten.linear.default:
118-
# Add a safety check to prevent replacing weights of non-linear nodes with updated weights.
119-
continue
120-
user.replace_input_with(weight_placeholder_node, get_attr_node)
135+
for user in list(weight_placeholder_node.users):
136+
if user.target is not ops["linear"]:
137+
# Add a safety check to prevent replacing weights of non-linear nodes with updated weights.
138+
continue
139+
user.replace_input_with(weight_placeholder_node, get_attr_node)
121140

122-
return get_attr_node
141+
return get_attr_node
123142

124143
def call(self, graph_module: GraphModule):
125144
graph = graph_module.graph
@@ -128,14 +147,16 @@ def call(self, graph_module: GraphModule):
128147
preprocessed_linear_weights_set = set()
129148

130149
for node in graph.nodes:
131-
if node.target is exir_ops.edge.aten.linear.default:
150+
if node.target in {self.aten_linear_target, self.edge_linear_target}:
151+
is_edge = node.target == self.edge_linear_target
152+
ops = self._op_table[is_edge]
132153
input_node = node.args[0]
133-
weight_placeholder_node = (
134-
# QDQ graph
135-
node.args[1].args[0]
136-
if node.args[1].target in dq_ops
137-
# FP graph
138-
else node.args[1]
154+
weight_placeholder_node = node.args[1]
155+
assert weight_placeholder_node.target not in dq_ops, (
156+
"ConvertLinearToConv2d does not handle quantized weights. "
157+
"A quantized flow must convert during the annotation "
158+
"pipeline, before q/dq nodes exist. Please set "
159+
"quantizer.set_convert_linear_to_conv2d(True)."
139160
)
140161
bias_arg = node.args[2] if len(node.args) > 2 else None
141162

@@ -158,33 +179,27 @@ def call(self, graph_module: GraphModule):
158179
cur_meta_val = cur_meta_val.reshape(shape)
159180
reshape_node = self._create_node(
160181
graph_module,
161-
exir_ops.edge.aten.view_copy.default,
182+
ops["view"],
162183
(input_node, shape),
163184
cur_meta_val,
164185
)
165186

166-
# This pass is scheduled after the `FoldQDQ` pass. After copying the metadata,
167-
# the quantization attributes are also propagated to the target node.
168187
order = (0, 3, 1, 2) if rank == 4 else (0, 2, 3, 1)
169188
cur_meta_val = cur_meta_val.permute(order)
170189
permute_node = self._create_node(
171190
graph_module,
172-
exir_ops.edge.aten.permute_copy.default,
191+
ops["permute"],
173192
(reshape_node, order) if rank <= 3 else (input_node, order),
174193
cur_meta_val,
175194
)
176195

177196
# Step 2: reshape weight
178197
if weight_placeholder_node.name not in preprocessed_linear_weights_set:
179198
weight_arg = self._reshape_weight_for_all_users(
180-
graph_module, weight_placeholder_node
199+
graph_module, weight_placeholder_node, ops
181200
)
182201
# Add the name of the preprocessed weights to the list.
183-
preprocessed_linear_weights_set.add(
184-
weight_arg.args[0].name
185-
if weight_arg.target in dq_ops
186-
else weight_arg.name
187-
)
202+
preprocessed_linear_weights_set.add(weight_arg.name)
188203
else:
189204
# Skip preprocessing as the weights have already been processed due to shared weights.
190205
weight_arg = node.args[1]
@@ -198,30 +213,57 @@ def call(self, graph_module: GraphModule):
198213
output_padding = [0, 0]
199214
groups = 1
200215
"""
216+
The two dialects take different argument lists:
217+
201218
Spec for `aten.convolution` (https://docs.pytorch.org/docs/stable/torch.compiler_ir.html)
202219
convolution(Tensor input, Tensor weight, Tensor? bias, SymInt[] stride, SymInt[] padding,
203220
SymInt[] dilation, bool transposed, SymInt[] output_padding, SymInt groups) -> Tensor
221+
222+
aten::conv2d(Tensor input, Tensor weight, Tensor? bias=None, SymInt[2] stride=[1, 1],
223+
SymInt[2] padding=[0, 0], SymInt[2] dilation=[1, 1], SymInt groups=1) -> Tensor
224+
225+
conv2d has no transposed / output_padding, so passing the 9-arg
226+
convolution list to it raises "expected at most 7 argument(s)".
204227
"""
205228
conv_args = (
206-
permute_node,
207-
weight_arg,
208-
bias_arg,
229+
(
230+
permute_node,
231+
weight_arg,
232+
bias_arg,
233+
stride,
234+
padding,
235+
dilation,
236+
transposed,
237+
output_padding,
238+
groups,
239+
)
240+
if is_edge
241+
else (
242+
permute_node,
243+
weight_arg,
244+
bias_arg,
245+
stride,
246+
padding,
247+
dilation,
248+
groups,
249+
)
250+
)
251+
# Shape inference always goes through the 9-arg convolution;
252+
# conv2d is just the narrower spelling of the same op.
253+
cur_meta_val = exir_ops.edge.aten.convolution.default(
254+
cur_meta_val,
255+
weight_meta_val,
256+
bias_meta_val,
209257
stride,
210258
padding,
211259
dilation,
212260
transposed,
213261
output_padding,
214262
groups,
215263
)
216-
cur_meta_val = exir_ops.edge.aten.convolution.default(
217-
cur_meta_val,
218-
weight_meta_val,
219-
bias_meta_val,
220-
*conv_args[3:],
221-
)
222264
conv_node = self._create_node(
223265
graph_module,
224-
exir_ops.edge.aten.convolution.default,
266+
ops["conv"],
225267
conv_args,
226268
cur_meta_val,
227269
meta_source_node=node,
@@ -235,7 +277,7 @@ def call(self, graph_module: GraphModule):
235277
cur_meta_val = cur_meta_val.permute(order)
236278
permute_node = self._create_node(
237279
graph_module,
238-
exir_ops.edge.aten.permute_copy.default,
280+
ops["permute"],
239281
(conv_node, order),
240282
cur_meta_val,
241283
)
@@ -244,7 +286,7 @@ def call(self, graph_module: GraphModule):
244286
cur_meta_val = cur_meta_val.reshape(target_shape)
245287
reshape_node = self._create_node(
246288
graph_module,
247-
exir_ops.edge.aten.view_copy.default,
289+
ops["view"],
248290
(permute_node, target_shape),
249291
cur_meta_val,
250292
)
@@ -253,6 +295,7 @@ def call(self, graph_module: GraphModule):
253295
node.replace_all_uses_with(permute_node)
254296

255297
graph.erase_node(node)
298+
self._rewrite_module_stack(conv_node, ".conv")
256299

257300
dead_code_elimination_pass(graph_module)
258301
return PassResult(graph_module, True)
Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
# Copyright (c) Qualcomm Innovation Center, Inc.
2+
# All rights reserved
3+
#
4+
# This source code is licensed under the BSD-style license found in the
5+
# LICENSE file in the root directory of this source tree.
6+
7+
8+
import torch
9+
from executorch.backends.qualcomm.utils.constants import QCOM_REQUANTIZE
10+
from executorch.exir.dialects._ops import ops as exir_ops
11+
from executorch.exir.pass_base import ExportPass, PassResult
12+
from executorch.exir.passes import dead_code_elimination_pass
13+
14+
15+
class FuseConsecutiveReshape(ExportPass):
16+
"""
17+
Collapse chains of consecutive view_copy into a single view_copy.
18+
This condition is not added in remove_redundancy.py since
19+
quantized(QDQ) models are hard to handle, and remove_redundancy runs before FoldQDQ.
20+
Some reshape shows up at edge dialect, so annotation phase of remove_redundancy
21+
also won't work.
22+
This pass is designed to be conservative and will not fold if requantize meta is
23+
found during traversal. The only exception is when the requantize meta is on the
24+
last view_copy of the chain, since that node is the one being rewritten rather
25+
than a node being folded away.
26+
"""
27+
28+
def __init__(self):
29+
super().__init__()
30+
self.view = exir_ops.edge.aten.view_copy.default
31+
32+
def _fuse(self, graph_module: torch.fx.GraphModule):
33+
for node in graph_module.graph.nodes:
34+
if node.target != self.view:
35+
continue
36+
# Walk back to the first non-view source node; every intermediate view
37+
# is redundant because this node restates the full target shape.
38+
# Need to ensure this optimization doesn't break requantize logic.
39+
src = original_src = node.args[0]
40+
while (
41+
isinstance(src, torch.fx.Node)
42+
and src.target == self.view
43+
and QCOM_REQUANTIZE not in src.meta
44+
):
45+
src = src.args[0]
46+
47+
if src is not original_src and QCOM_REQUANTIZE not in src.meta:
48+
node.args = (src, *node.args[1:])
49+
50+
def call(self, graph_module: torch.fx.GraphModule):
51+
self._fuse(graph_module)
52+
dead_code_elimination_pass(graph_module)
53+
return PassResult(graph_module, True)

0 commit comments

Comments
 (0)