Skip to content

Commit 12668f8

Browse files
authored
Support per-channel weights end to end (pytorch#22227)
Summary: Per-channel weight quantization went through `prepare_pt2`/`convert_pt2` and then quietly stopped being quantized: fusion bailed on the per-channel dequant, so the convs fell out to float `cadence.fully_connected`. On HiFi it is worse than a fallback, because `op_quantized_linear_out` reads `out_multiplier[0]` even in the tensor-qparam overload and would requantize every channel with channel 0's scale, silently. This makes the whole path work, AoT and kernels. Granularity is carried by the existing overload split, with `.out` (tensor qparams) now being the per-channel form and `.per_tensor_out` the per-tensor one. No new operator names are introduced. Only the generic kernels implement per-channel, so `operator_fallback.bzl` routes every per-channel op there: the HiFi and TIE rows for the four tensor-qparam ops are removed, and `quantized_depthwise_conv1d_{ncl,nlc}.out` gain generic rows they never had. Backends fall back instead of failing, and a follow-up can reclaim performance by adding a row back. NNLib's conv entry points are already per-channel capable, so HiFi conv is the obvious first candidate; its fully-connected and matmul entry points take a single (multiplier, shift) pair and structurally cannot be. The two new depthwise rows also need schemas in `min_runtime/custom_ops.yaml`: that file is not OSS-only, the Jarvis xtensa codegen resolves every selective-build op name through it and fails the build outright on one it cannot find. - The quantizer aligns a pattern's derived bias spec with the granularity of its weight spec. Previously six patterns hardcoded `per_tensor_affine` on the bias, so `bias_scale = act_scale * weight_scale` produced a vector that a per-tensor spec could not hold (`ValueError: only one element tensors can be converted to Python scalars`). `CadenceAtenQuantizer.annotate` is the one place that sees both specs, so the alignment lives there. - Fusion accepts a per-channel weight dequant and emits the tensor-qparam overload, materializing bias_scale / out_multiplier / out_shift / weight_zero_point as lifted constants. Reading the weight scale and adding constants both need the ExportedProgram, which `PassBase.call` does not get, so `QuantFusionPass` takes one and stashes it for the duration of the pass rather than changing `fuse()` on ~20 patterns. - Both conv and linear fuse per-channel. Linear is what fully-connected models are made of, so leaving it out would have made this a no-op for the target model. `fuse_linear` mirrors `fuse_conv`; `quantized_linear` has no bias_scale operand, so the scale reaches the kernel only through out_multiplier/out_shift. `AddmmPattern` still declines: its weight is `[in_features, out_features]`, so an output-channel axis of 0 is meaningless there, and linear is preserved from decomposition anyway. - Depthwise selection now runs before the per-channel overload swap in `fuse_conv`. It compared against the `.per_tensor` target after the swap had already moved to `.default`, so a per-channel depthwise conv silently stayed a dense conv. `quantized_depthwise_conv1d_ncl`/`_nlc` had no `.default` overload at all; the schemas, metas and reference implementations are added here. - The conv lowering exits carry per-channel through. `ReplaceConvWithChannelLastConvPass` picks the layout from the input rank as before but takes the overload from the incoming node, instead of hardcoding `.per_tensor`. `ReplaceConvWithIm2RowAndLinear` normally recomputes out_multiplier/out_shift from `bias_scale / out_scale`, which is scalar arithmetic; for per-channel it reuses the conv's existing multiplier/shift nodes, which were derived from exactly that ratio at fusion time. - The generic conv kernels do per-channel requantization. The conv2d and conv1d cores take weight_zero_point and bias_scale as a pointer plus a stride, resolved once per output channel: stride 0 keeps every channel on element 0 (per-tensor), stride 1 walks the channel axis. Existing callers pass 0, so per-tensor behaviour and the vectorizable inner loops are unchanged. The `.out` entry points previously read element 0 and applied channel 0's scale to every output channel. - The conv reference implementations do real per-channel requantization. The four tensor-qparam variants used to call `.item()` on their qparams and delegate to the per-tensor path, which raised on a vector. `quantized_conv_per_tensor` becomes `quantized_conv_common` and broadcasts weight_zero_point and bias_scale over the output-channel axis. Reviewed By: ethansfng Differential Revision: D117164581 Pull Request resolved: pytorch#22227
1 parent b2896ae commit 12668f8

22 files changed

Lines changed: 3826 additions & 353 deletions

‎backends/cadence/aot/BUCK‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -434,6 +434,7 @@ fbcode_target(_kind = python_unittest,
434434
"//executorch/backends/cadence/aot:compiler",
435435
"//executorch/backends/test:graph_builder",
436436
"//executorch/backends/cadence/aot:pass_utils",
437+
"//executorch/backends/cadence/aot/quantizer:utils",
437438
"//executorch/exir:pass_base",
438439
"//executorch/exir/dialects:lib",
439440
"//executorch/exir/passes:lib",
@@ -645,6 +646,7 @@ fbcode_target(_kind = python_unittest,
645646
deps = [
646647
":typing_stubs",
647648
"//executorch/backends/cadence/aot:ops_registrations",
649+
"//executorch/backends/cadence/aot/quantizer:utils",
648650
"//caffe2:torch",
649651
]
650652
)
@@ -664,6 +666,21 @@ fbcode_target(_kind = python_unittest,
664666
],
665667
)
666668

669+
fbcode_target(_kind = python_unittest,
670+
name = "test_pattern_utils",
671+
srcs = [
672+
"tests/test_pattern_utils.py",
673+
],
674+
deps = [
675+
"//caffe2:torch",
676+
"//executorch/backends/test:graph_builder",
677+
"//executorch/backends/test:program_builder",
678+
"//executorch/backends/cadence/aot:compiler_funcs",
679+
"//executorch/backends/cadence/aot:ops_registrations",
680+
"//executorch/backends/cadence/aot/quantizer:quantizer",
681+
],
682+
)
683+
667684
fbcode_target(_kind = python_unittest,
668685
name = "test_to_out_var_pass",
669686
srcs = [

‎backends/cadence/aot/compiler.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -170,7 +170,7 @@ def apply_pre_edge_transform_passes(
170170
PassManager(
171171
[
172172
FuseQATConvBN(converted_program),
173-
QuantFusionPass(patterns),
173+
QuantFusionPass(patterns, converted_program),
174174
]
175175
)(converted_program.graph_module)
176176

‎backends/cadence/aot/compiler_funcs.py‎

Lines changed: 32 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -13,8 +13,12 @@
1313

1414
import torch
1515

16+
from executorch.backends.cadence.aot.quantizer.pattern_utils import (
17+
EXPORTED_PROGRAM_META_KEY,
18+
)
1619
from executorch.backends.transforms.permute_pass_utils import get_arg
1720
from torch._inductor.decomposition import remove_decompositions
21+
from torch.export.exported_program import ExportedProgram
1822
from torch.fx import GraphModule
1923
from torch.fx.passes.infra.pass_base import PassBase, PassResult
2024
from torchao.quantization.pt2e.quantize_pt2e import prepare_pt2e, prepare_qat_pt2e
@@ -714,26 +718,41 @@ class QuantFusionPass(PassBase):
714718
"""
715719
Iterates patterns, finds anchor ops in the converted graph, and calls
716720
pattern.fuse() to replace dq-op-q subgraphs with fused ops.
721+
722+
``exported_program`` is optional but required for per-channel weights:
723+
fusion has to read the weight scale vector and materialize the derived
724+
qparam tensors, and by this point both live in the program's constants
725+
rather than in the graph.
717726
"""
718727

719-
def __init__(self, patterns: Sequence[object]) -> None:
728+
def __init__(
729+
self,
730+
patterns: Sequence[object],
731+
exported_program: Optional[ExportedProgram] = None,
732+
) -> None:
720733
super().__init__()
721734
self.patterns = patterns
735+
self.exported_program = exported_program
722736

723737
def call(self, graph_module: GraphModule) -> Optional[PassResult]:
724738
changed = False
725-
for pattern in self.patterns:
726-
pattern_changed = False
727-
for target in pattern.anchor_ops(): # pyre-ignore[16]
728-
for node in graph_module.graph.find_nodes(
729-
op="call_function", target=target
730-
):
731-
result = pattern.fuse(graph_module, node) # pyre-ignore[16]
732-
if result is not None:
733-
changed = True
734-
pattern_changed = True
735-
if pattern_changed:
736-
graph_module.graph.eliminate_dead_code()
739+
if self.exported_program is not None:
740+
graph_module.meta[EXPORTED_PROGRAM_META_KEY] = self.exported_program
741+
try:
742+
for pattern in self.patterns:
743+
pattern_changed = False
744+
for target in pattern.anchor_ops(): # pyre-ignore[16]
745+
for node in graph_module.graph.find_nodes(
746+
op="call_function", target=target
747+
):
748+
result = pattern.fuse(graph_module, node) # pyre-ignore[16]
749+
if result is not None:
750+
changed = True
751+
pattern_changed = True
752+
if pattern_changed:
753+
graph_module.graph.eliminate_dead_code()
754+
finally:
755+
graph_module.meta.pop(EXPORTED_PROGRAM_META_KEY, None)
737756
if changed:
738757
graph_module.recompile()
739758
return PassResult(graph_module, changed)

‎backends/cadence/aot/ops_registrations.py‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -268,12 +268,24 @@ def register_fake(
268268
lib.define(
269269
"quantized_conv1d_nlc.per_tensor_out(Tensor input, Tensor weight, Tensor bias, int[] stride, SymInt[] padding, int[] dilation, int groups, int input_zero_point, int weight_zero_point, float bias_scale, float out_scale, int out_zero_point, int out_multiplier, int out_shift, Tensor? offset=None, *, Tensor(a!) out) -> Tensor(a!)"
270270
)
271+
lib.define(
272+
"quantized_depthwise_conv1d_ncl(Tensor input, Tensor weight, Tensor bias, int[] stride, SymInt[] padding, int[] dilation, int groups, int input_zero_point, Tensor weight_zero_point, Tensor bias_scale, float out_scale, int out_zero_point, Tensor out_multiplier, Tensor out_shift) -> (Tensor Z)"
273+
)
274+
lib.define(
275+
"quantized_depthwise_conv1d_ncl.out(Tensor input, Tensor weight, Tensor bias, int[] stride, SymInt[] padding, int[] dilation, int groups, int input_zero_point, Tensor weight_zero_point, Tensor bias_scale, float out_scale, int out_zero_point, Tensor out_multiplier, Tensor out_shift, *, Tensor(a!) out) -> Tensor(a!)"
276+
)
271277
lib.define(
272278
"quantized_depthwise_conv1d_ncl.per_tensor(Tensor input, Tensor weight, Tensor bias, int[] stride, SymInt[] padding, int[] dilation, int groups, int input_zero_point, int weight_zero_point, float bias_scale, float out_scale, int out_zero_point, int out_multiplier, int out_shift) -> (Tensor Z)"
273279
)
274280
lib.define(
275281
"quantized_depthwise_conv1d_ncl.per_tensor_out(Tensor input, Tensor weight, Tensor bias, int[] stride, SymInt[] padding, int[] dilation, int groups, int input_zero_point, int weight_zero_point, float bias_scale, float out_scale, int out_zero_point, int out_multiplier, int out_shift, *, Tensor(a!) out) -> Tensor(a!)"
276282
)
283+
lib.define(
284+
"quantized_depthwise_conv1d_nlc(Tensor input, Tensor weight, Tensor bias, int[] stride, SymInt[] padding, int[] dilation, int groups, int input_zero_point, Tensor weight_zero_point, Tensor bias_scale, float out_scale, int out_zero_point, Tensor out_multiplier, Tensor out_shift) -> (Tensor Z)"
285+
)
286+
lib.define(
287+
"quantized_depthwise_conv1d_nlc.out(Tensor input, Tensor weight, Tensor bias, int[] stride, SymInt[] padding, int[] dilation, int groups, int input_zero_point, Tensor weight_zero_point, Tensor bias_scale, float out_scale, int out_zero_point, Tensor out_multiplier, Tensor out_shift, *, Tensor(a!) out) -> Tensor(a!)"
288+
)
277289
lib.define(
278290
"quantized_depthwise_conv1d_nlc.per_tensor(Tensor input, Tensor weight, Tensor bias, int[] stride, SymInt[] padding, int[] dilation, int groups, int input_zero_point, int weight_zero_point, float bias_scale, float out_scale, int out_zero_point, int out_multiplier, int out_shift) -> (Tensor Z)"
279291
)
@@ -1403,6 +1415,76 @@ def quantized_depthwise_conv1d_nlc_per_tensor_meta(
14031415
return input.new_empty(output_size, dtype=input.dtype)
14041416

14051417

1418+
@register_fake("cadence::quantized_depthwise_conv1d_ncl")
1419+
def quantized_depthwise_conv1d_ncl_meta(
1420+
input: torch.Tensor,
1421+
weight: torch.Tensor,
1422+
bias: torch.Tensor,
1423+
stride: Tuple[int],
1424+
padding: Tuple[int],
1425+
dilation: Tuple[int],
1426+
groups: int,
1427+
in_zero_point: int,
1428+
weight_zero_point: torch.Tensor,
1429+
bias_scale: torch.Tensor,
1430+
output_scale: float,
1431+
output_zero_point: int,
1432+
out_multiplier: torch.Tensor,
1433+
out_shift: torch.Tensor,
1434+
) -> torch.Tensor:
1435+
return quantized_depthwise_conv1d_ncl_per_tensor_meta(
1436+
input,
1437+
weight,
1438+
bias,
1439+
stride,
1440+
padding,
1441+
dilation,
1442+
groups,
1443+
in_zero_point,
1444+
0,
1445+
1.0,
1446+
output_scale,
1447+
output_zero_point,
1448+
1,
1449+
0,
1450+
)
1451+
1452+
1453+
@register_fake("cadence::quantized_depthwise_conv1d_nlc")
1454+
def quantized_depthwise_conv1d_nlc_meta(
1455+
input: torch.Tensor,
1456+
weight: torch.Tensor,
1457+
bias: torch.Tensor,
1458+
stride: Tuple[int],
1459+
padding: Tuple[int],
1460+
dilation: Tuple[int],
1461+
groups: int,
1462+
in_zero_point: int,
1463+
weight_zero_point: torch.Tensor,
1464+
bias_scale: torch.Tensor,
1465+
output_scale: float,
1466+
output_zero_point: int,
1467+
out_multiplier: torch.Tensor,
1468+
out_shift: torch.Tensor,
1469+
) -> torch.Tensor:
1470+
return quantized_depthwise_conv1d_nlc_per_tensor_meta(
1471+
input,
1472+
weight,
1473+
bias,
1474+
stride,
1475+
padding,
1476+
dilation,
1477+
groups,
1478+
in_zero_point,
1479+
0,
1480+
1.0,
1481+
output_scale,
1482+
output_zero_point,
1483+
1,
1484+
0,
1485+
)
1486+
1487+
14061488
@register_fake("cadence::quantized_conv2d_nchw")
14071489
def quantized_conv2d_nchw_meta(
14081490
input: torch.Tensor,

0 commit comments

Comments
 (0)