Skip to content

Commit cfc96cc

Browse files
Add quantize/dequantize_per_channel_group support to QNN backend (pytorch#19629)
## Motivation `quantized_decomposed.quantize_per_channel_group` and `dequantize_per_channel_group` are what group-wise (int4) LLM weight quantization lowers to, but the QNN backend did not recognize them, causing two distinct failures: 1. The ops were decomposed away during `torch.export` instead of being preserved for the backend. 2. `InsertIOQDQ` raised a `KeyError` when a pre-quantized weight annotated with a `dequantize_per_channel_group` encoding fed the graph output, because the op was absent from `q_dq_map`. This change gives them the same treatment `torchao.quantize_affine` / `dequantize_affine` already receive. ## Changes **`backends/qualcomm/builders/node_visitor.py`** - Register both ops in `q_ops` / `dq_ops`. - Add both entries to the module-level `q_dq_map` (this is the dict `InsertIOQDQ` imports and looks up, so this is what fixes the `KeyError`). - Add the `PER_CHANNEL_GROUP_ENCODING` set and route it through `make_qnn_per_block_config`, since QNN models group-wise quantization as a per-block encoding. - `make_qnn_per_block_config` now rejects non-zero `zero_points` (`zero_points` / `zero_point`) with a `ValueError`: QNN blockwise expansion hardcodes a per-channel offset of 0, so an asymmetric weight would otherwise silently lower to a symmetric encoding. **`backends/qualcomm/partition/utils.py`** - Add both ops to `get_skip_decomp_table` so they survive export and reach the backend unchanged. **`backends/qualcomm/partition/qnn_partitioner.py`** - `QnnOperatorSupport` refuses to delegate `per_channel_group` q/dq when `soc_info.htp_info.htp_arch < HtpArch.V69`, mirroring `validate_lpbq_support`. Pre-quantized weights bypass the quantizer, so without this there is no arch check on this path; blockwise expansion hard-fails on V68. The op falls back to CPU with a warning. **`backends/qualcomm/_passes/utils.py`** - Expose the block scale tensor as `QCOM_SCALE` in `get_quant_attrs`, which is what `make_qnn_per_block_config` reads. The remap is gated on the encoding rather than on the presence of a `"scales"` key, because `quantize_per_channel.default` also has a `"scales"` argument but is consumed as a per-channel (not per-block) config. - Validate that `scales.shape[-1] * group_size` matches the weight's input-feature count, so a mismatched checkpoint fails at annotation time instead of producing a silently wrong encoding (`QCOM_NUM_BLOCKS_PER_AXIS` is inferred from the scale shape downstream). ## Testing ``` pytest backends/qualcomm/tests/test_passes.py -k "insert_io_qdq or per_block or q_dq_map" ``` - `test_q_dq_map_pins_per_channel_group_pairs` pins the `q_dq_map` entries through `to_q_op` / `to_dq_op`: `to_q_op(dequantize_per_channel_group) is quantize_per_channel_group`, `to_dq_op(quantize_per_channel_group) is dequantize_per_channel_group`, plus the identity cases. `to_dq_op` short-circuits for targets already in `dq_ops`, so this is the only place the cross-pair mapping is exercised; it fails with a self-referential `dequantize_per_channel_group` entry. - `test_insert_io_qdq_per_channel_group_dequantizes_output` annotates a parameter with a `dequantize_per_channel_group` encoding and wires it to the graph output. That is the dequantize-before-output branch of `InsertIOQDQ` (the insert-quantize-after-input branch is skipped for parameters); it asserts a `dequantize_per_channel_group` node now feeds the output. - `test_make_qnn_per_block_config_rejects_asymmetric` covers the non-zero `zero_points` rejection. The encoding gating in `get_quant_attrs` was checked directly: `quantize_per_channel.default` no longer gets a `QCOM_SCALE` entry written, while `dequantize_per_channel_group.default` does. **Not validated on device**: no HTP device is available in this environment, so end-to-end numerics of the per-block lowering (scale reshaping, symmetric encoding) have not been verified on any HTP arch. The partitioner arch gate (< V69 falls back to CPU) and the asymmetric/`group_size` validations are there so the unverified paths fail loudly or fall back instead of lowering silently wrong. cc @qti-horodnic @cbilgin @psiddh @cccclai @abhinaykukkadapu --------- Co-authored-by: Siddartha Pothapragada <sidart@meta.com>
1 parent 60dce2a commit cfc96cc

5 files changed

Lines changed: 183 additions & 1 deletion

File tree

‎backends/qualcomm/_passes/utils.py‎

Lines changed: 25 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,12 +7,18 @@
77
from typing import Callable, Dict, List
88

99
import torch
10-
from executorch.backends.qualcomm.builders.node_visitor import dq_ops, q_ops
10+
from executorch.backends.qualcomm.builders.node_visitor import (
11+
dq_ops,
12+
PER_CHANNEL_GROUP_ENCODING,
13+
q_ops,
14+
)
1115
from executorch.backends.qualcomm.builders.utils import get_parameter
1216
from executorch.backends.qualcomm.utils.constants import (
1317
QCOM_DTYPE,
1418
QCOM_ENCODING,
1519
QCOM_QUANT_ATTRS,
20+
QCOM_SCALE,
21+
QCOM_SCALES,
1622
)
1723
from executorch.exir.dialects._ops import ops as exir_ops
1824
from torch._subclasses import FakeTensor
@@ -146,6 +152,24 @@ def get_quant_attrs(
146152
if dtype := quant_attrs.get("input_dtype", None):
147153
quant_attrs[QCOM_DTYPE] = dtype
148154

155+
# per-channel-group ops name their block scale tensor "scales"; QNN's
156+
# per-block config reads it as QCOM_SCALE. Gate on the encoding rather than
157+
# on the presence of the key: quantize_per_channel.default also has a
158+
# "scales" argument but is consumed as a per-channel (not per-block) config.
159+
if quant_node.target in PER_CHANNEL_GROUP_ENCODING:
160+
scales = quant_attrs[QCOM_SCALES]
161+
group_size = quant_attrs["group_size"]
162+
num_input_features = quant_node.args[0].meta["val"].shape[-1]
163+
# make_qnn_per_block_config infers blocks-per-axis from the scale
164+
# shape; a mismatched checkpoint would otherwise lower silently wrong
165+
if scales.shape[-1] * group_size != num_input_features:
166+
raise ValueError(
167+
f"group-wise scale shape {tuple(scales.shape)} is inconsistent "
168+
f"with group_size {group_size} for weight with "
169+
f"{num_input_features} input features"
170+
)
171+
quant_attrs[QCOM_SCALE] = scales
172+
149173
quant_attrs[QCOM_ENCODING] = quant_node.target
150174
return quant_attrs
151175

‎backends/qualcomm/builders/node_visitor.py‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,8 +83,16 @@
8383
exir_ops.edge.quantized_decomposed.dequantize_per_tensor.tensor,
8484
}
8585

86+
# Group-wise (a.k.a. per-channel-group / blockwise) weight quantization, as used
87+
# by int4 LLM weights. QNN represents it with a per-block encoding.
88+
PER_CHANNEL_GROUP_ENCODING = {
89+
exir_ops.edge.quantized_decomposed.quantize_per_channel_group.default,
90+
exir_ops.edge.quantized_decomposed.dequantize_per_channel_group.default,
91+
}
92+
8693
q_ops = {
8794
exir_ops.edge.quantized_decomposed.quantize_per_channel.default,
95+
exir_ops.edge.quantized_decomposed.quantize_per_channel_group.default,
8896
exir_ops.edge.quantized_decomposed.quantize_per_tensor.default,
8997
exir_ops.edge.quantized_decomposed.quantize_per_tensor.tensor,
9098
exir_ops.edge.torchao.quantize_affine.default,
@@ -94,16 +102,19 @@
94102
exir_ops.edge.quantized_decomposed.dequantize_per_tensor.default,
95103
exir_ops.edge.quantized_decomposed.dequantize_per_tensor.tensor,
96104
exir_ops.edge.quantized_decomposed.dequantize_per_channel.default,
105+
exir_ops.edge.quantized_decomposed.dequantize_per_channel_group.default,
97106
exir_ops.edge.torchao.dequantize_affine.default,
98107
}
99108

100109
q_dq_map = {
101110
exir_ops.edge.quantized_decomposed.quantize_per_tensor.default: exir_ops.edge.quantized_decomposed.dequantize_per_tensor.default,
102111
exir_ops.edge.quantized_decomposed.quantize_per_tensor.tensor: exir_ops.edge.quantized_decomposed.dequantize_per_tensor.tensor,
103112
exir_ops.edge.quantized_decomposed.quantize_per_channel.default: exir_ops.edge.quantized_decomposed.dequantize_per_channel.default,
113+
exir_ops.edge.quantized_decomposed.quantize_per_channel_group.default: exir_ops.edge.quantized_decomposed.dequantize_per_channel_group.default,
104114
exir_ops.edge.quantized_decomposed.dequantize_per_tensor.default: exir_ops.edge.quantized_decomposed.quantize_per_tensor.default,
105115
exir_ops.edge.quantized_decomposed.dequantize_per_tensor.tensor: exir_ops.edge.quantized_decomposed.quantize_per_tensor.tensor,
106116
exir_ops.edge.quantized_decomposed.dequantize_per_channel.default: exir_ops.edge.quantized_decomposed.quantize_per_channel.default,
117+
exir_ops.edge.quantized_decomposed.dequantize_per_channel_group.default: exir_ops.edge.quantized_decomposed.quantize_per_channel_group.default,
107118
}
108119

109120

@@ -176,6 +187,19 @@ def _get_tensor(node, index):
176187
def make_qnn_per_block_config(self, node: torch.fx.Node, quant_attrs: Dict):
177188
import math
178189

190+
# blockwise expansion hardcodes a per-channel offset of 0, so an
191+
# asymmetric weight would silently lower to a symmetric encoding
192+
if quant_attrs[QCOM_ENCODING] in PER_CHANNEL_GROUP_ENCODING:
193+
zero_points = quant_attrs.get(QCOM_ZERO_POINTS)
194+
if zero_points is None:
195+
zero_points = quant_attrs.get(QCOM_ZERO_POINT)
196+
if zero_points is not None and torch.any(torch.as_tensor(zero_points) != 0):
197+
raise ValueError(
198+
"per-channel-group quantization with non-zero zero_points is "
199+
"not supported; QNN blockwise expansion requires symmetric "
200+
"quantization"
201+
)
202+
179203
quant_config = {
180204
QCOM_DTYPE: quant_attrs[QCOM_DTYPE],
181205
QCOM_QUANT_MIN: quant_attrs[QCOM_QUANT_MIN],
@@ -339,6 +363,7 @@ def get_quant_encoding_conf(
339363
per_block_encoding = {
340364
exir_ops.edge.torchao.quantize_affine.default,
341365
exir_ops.edge.torchao.dequantize_affine.default,
366+
*PER_CHANNEL_GROUP_ENCODING,
342367
}
343368
if quant_attrs[QCOM_ENCODING] in per_block_encoding:
344369
return self.make_qnn_per_block_config(node, quant_attrs)

‎backends/qualcomm/partition/qnn_partitioner.py‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,15 @@
1010

1111
import torch
1212
from executorch.backends.qualcomm.builders import node_visitor_manager
13+
from executorch.backends.qualcomm.builders.node_visitor import (
14+
PER_CHANNEL_GROUP_ENCODING,
15+
)
1316
from executorch.backends.qualcomm.builders.qnn_constants import OpContextLoader
1417
from executorch.backends.qualcomm.qnn_preprocess import QnnBackend
18+
from executorch.backends.qualcomm.serialization.qc_schema import (
19+
HtpArch,
20+
QnnExecuTorchBackendType,
21+
)
1522
from executorch.backends.qualcomm.serialization.qc_schema_serialize import (
1623
flatbuffer_to_option,
1724
)
@@ -73,6 +80,8 @@ def __init__(
7380
# Label prefixed to op-support log lines so callers that reuse this
7481
# checker (e.g. the LPAI fallback pass) are distinguishable in the logs.
7582
self.phase = phase
83+
self.soc_info = python_options.soc_info
84+
self.backend_type = python_options.backend_options.backend_type
7685
self.nodes_to_wrappers = defaultdict(dict)
7786
self.qnn_manager = get_current_qnn_manager(
7887
python_options.backend_options.backend_type, compiler_specs
@@ -82,6 +91,22 @@ def is_node_supported(self, _, node: torch.fx.Node) -> bool:
8291
if node.op != "call_function" or node.target in not_supported_operator:
8392
return False
8493

94+
# Per-channel-group lowers to blockwise expansion (the encoding LPBQ
95+
# uses), which requires HTP >= V69. Pre-quantized weights bypass the
96+
# quantizer's validate_lpbq_support check, so gate the delegation here.
97+
# Scoped to HTP: other backends reusing this checker have no htp_arch.
98+
if (
99+
node.target in PER_CHANNEL_GROUP_ENCODING
100+
and self.backend_type == QnnExecuTorchBackendType.kHtpBackend
101+
and self.soc_info.htp_info.htp_arch < HtpArch.V69
102+
):
103+
logger.warning(
104+
f"[{self.phase}] {node.target.__name__} | Skipped: blockwise "
105+
f"(per-channel-group) encoding requires HTP arch >= V69, got "
106+
f"{self.soc_info.htp_info.htp_arch!r}"
107+
)
108+
return False
109+
85110
if node.target in to_be_implemented_operator:
86111
logger.info(
87112
f"[{self.phase}] {node.target.__name__} | Skipped, this op can be supported, please report an issue in https://github.com/pytorch/executorch/issues"

‎backends/qualcomm/partition/utils.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,5 +77,7 @@ def get_skip_decomp_table() -> List[torch._ops.OperatorBase]:
7777
torch.ops.aten.unbind.int,
7878
torch.ops.torchao.quantize_affine.default,
7979
torch.ops.torchao.dequantize_affine.default,
80+
torch.ops.quantized_decomposed.quantize_per_channel_group.default,
81+
torch.ops.quantized_decomposed.dequantize_per_channel_group.default,
8082
]
8183
return do_not_decompose

‎backends/qualcomm/tests/test_passes.py‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,11 +15,17 @@
1515
from executorch.backends.qualcomm._passes.qnn_pass_manager import (
1616
get_qnn_pass_manager_cls,
1717
)
18+
from executorch.backends.qualcomm.builders.node_visitor import (
19+
NodeVisitor,
20+
to_dq_op,
21+
to_q_op,
22+
)
1823
from executorch.backends.qualcomm.builders.op_custom_op import (
1924
_resolve_qnn_data_type,
2025
CustomOp,
2126
)
2227
from executorch.backends.qualcomm.builders.qnn_constants import OpContextLoader
28+
from executorch.backends.qualcomm.builders.utils import is_parameter
2329
from executorch.backends.qualcomm.partition.qnn_partitioner import QnnOperatorSupport
2430
from executorch.backends.qualcomm.qnn_preprocess import QnnBackend
2531
from executorch.backends.qualcomm.quantizer.quantizer import QnnQuantizer, QuantDtype
@@ -34,6 +40,12 @@
3440
Reciprocal,
3541
TopKandIndex,
3642
)
43+
from executorch.backends.qualcomm.utils.constants import (
44+
QCOM_ENCODING,
45+
QCOM_QUANT_ATTRS,
46+
QCOM_SCALE,
47+
QCOM_ZERO_POINTS,
48+
)
3749
from executorch.backends.qualcomm.utils.utils import (
3850
generate_htp_compiler_spec,
3951
generate_qnn_executorch_compiler_spec,
@@ -212,6 +224,100 @@ def test_insert_io_qdq_no_revisit(self):
212224
# one quantize (input) and one dequantize (output) = +2 nodes.
213225
self.assertEqual(node_count_after, node_count_before + 2)
214226

227+
def test_q_dq_map_pins_per_channel_group_pairs(self):
228+
"""to_q_op / to_dq_op must map per_channel_group q<->dq to each other.
229+
230+
``to_dq_op`` short-circuits for targets already in ``dq_ops``, so the
231+
map is only consulted when converting across the pair; a wrong entry
232+
here would surface later as an assertion in ``insert_quant_node``.
233+
"""
234+
quantize_per_channel_group = (
235+
exir_ops.edge.quantized_decomposed.quantize_per_channel_group.default
236+
)
237+
dequantize_per_channel_group = (
238+
exir_ops.edge.quantized_decomposed.dequantize_per_channel_group.default
239+
)
240+
self.assertIs(to_q_op(dequantize_per_channel_group), quantize_per_channel_group)
241+
self.assertIs(
242+
to_dq_op(quantize_per_channel_group), dequantize_per_channel_group
243+
)
244+
self.assertIs(to_q_op(quantize_per_channel_group), quantize_per_channel_group)
245+
self.assertIs(
246+
to_dq_op(dequantize_per_channel_group), dequantize_per_channel_group
247+
)
248+
249+
def test_insert_io_qdq_per_channel_group_dequantizes_output(self):
250+
"""InsertIOQDQ must insert a dequantize_per_channel_group before output.
251+
252+
A pre-quantized group-wise (e.g. int4 LLM) weight parameter that feeds
253+
the graph output takes the dequantize-before-output branch. The
254+
insert-quantize-after-input branch is skipped for parameters, so the
255+
output branch is the only place the encoding is resolved.
256+
"""
257+
graph_module, exported_program = self._build_quantized_graph()
258+
259+
parameter_node = next(
260+
node
261+
for node in graph_module.graph.nodes
262+
if node.op == "placeholder" and is_parameter(node, exported_program)
263+
)
264+
output_node = next(
265+
node for node in graph_module.graph.nodes if node.op == "output"
266+
)
267+
268+
# Annotate the parameter as a group-wise quantized weight.
269+
scales = torch.ones(4, 1)
270+
parameter_node.meta[QCOM_QUANT_ATTRS] = {
271+
QCOM_ENCODING: exir_ops.edge.quantized_decomposed.dequantize_per_channel_group.default,
272+
QCOM_SCALE: scales,
273+
"scales": scales,
274+
"zero_points": None,
275+
"quant_min": -8,
276+
"quant_max": 7,
277+
"dtype": torch.int8,
278+
"group_size": 1,
279+
"output_dtype": torch.float32,
280+
}
281+
282+
existing_outputs = output_node.args[0]
283+
if not isinstance(existing_outputs, tuple):
284+
existing_outputs = (existing_outputs,)
285+
output_node.args = (existing_outputs + (parameter_node,),)
286+
graph_module.graph.lint()
287+
graph_module.recompile()
288+
289+
InsertIOQDQ(exported_program)._insert(graph_module)
290+
291+
dequantize_target = (
292+
exir_ops.edge.quantized_decomposed.dequantize_per_channel_group.default
293+
)
294+
dequantize_nodes_feeding_output = [
295+
node
296+
for node in graph_module.graph.nodes
297+
if node.op == "call_function"
298+
and node.target == dequantize_target
299+
and any(user.op == "output" for user in node.users.keys())
300+
]
301+
self.assertEqual(len(dequantize_nodes_feeding_output), 1)
302+
303+
def test_make_qnn_per_block_config_rejects_asymmetric(self):
304+
"""QNN blockwise expansion is symmetric-only; non-zero zero_points must
305+
raise instead of silently lowering to a symmetric encoding."""
306+
graph_module, exported_program = self._build_quantized_graph()
307+
parameter_node = next(
308+
node
309+
for node in graph_module.graph.nodes
310+
if node.op == "placeholder" and is_parameter(node, exported_program)
311+
)
312+
313+
visitor = NodeVisitor({}, exported_program, enable_tensor_dump=False)
314+
quant_attrs = {
315+
QCOM_ENCODING: exir_ops.edge.quantized_decomposed.dequantize_per_channel_group.default,
316+
QCOM_ZERO_POINTS: torch.tensor([0, 1, 0, 0]),
317+
}
318+
with self.assertRaisesRegex(ValueError, "symmetric"):
319+
visitor.make_qnn_per_block_config(parameter_node, quant_attrs)
320+
215321
def test_insert_reshape_for_argmax(self):
216322
class ArgmaxModule(torch.nn.Module):
217323
def forward(self, x):

0 commit comments

Comments
 (0)