Skip to content

Commit 5f4b03a

Browse files
authored
Row-packed sub-byte fully-connected weights (pytorch#23403)
Summary: Adds 4- and 6-bit row-packed weight storage for fully-connected layers, from the packing utilities all the way through to a dispatchable operator. Nothing is landed unreachable: the packer, the operator, the registration, the lowering hook and the backend fallback rows all arrive together. The operator is `quantized_fully_connected_packed` rather than a packed `quantized_linear` because wakeword stage 2 lowers to fully-connected, so this is the shape the model will actually hit. Layout: a [out_dim, in_dim] int8 weight becomes [out_dim, ceil(in_dim*bits/8)]. Packing per row is what keeps `out_dim` as `weight.size(0)`, so the existing `out_size[-1] = weight.size(0)` meta computation still holds. `in_dim` and `weight_bits` become explicit operator arguments, since neither can be read off the packed shape any more. 6-bit follows the torchao UINT6 byte order; signed values are stored as offset binary. Packing is storage-only and never changes a value. 6-bit means "quantize with quant_min/quant_max of -32/31, then store it in 6 bits", not "round an 8-bit weight down". A layer whose weights do not fit the narrower range, or whose in_dim is not a multiple of the group size, is skipped and stays 8-bit rather than failing the compile. Selection is an explicit compile-time option: a `weight_bits` field on `EdgePassesConfig`, defaulting to 8 (off). The transform runs in `apply_exir_ops_passes` after `.transform(...)` rather than as a pipeline pass, because it changes a constant's *shape* and so needs the ExportedProgram, not the GraphModule a pass is handed. Scalar qparams on a `.per_tensor` source are lifted to length-1 constants, since the packed operator only has the tensor-qparam form. The reference implementation unpacks and delegates to `quantized_linear_common` so the packed path cannot drift from the unpacked arithmetic. The C++ operator wraps the same inner kernel `op_quantized_fully_connected.cpp` uses. Backend wiring: only the generic row is added to `operator_fallback.bzl`. HiFi and vision resolve to it through the existing fallback chain, which is the point of the chain - a backend gets a row only when it has a specialized implementation. Verified by removing the redundant HiFi row and confirming the wakeword build still resolves the operator. HiFi's CMake build lists generic fallback sources explicitly, so the new source is added there. Jarvis keeps its own operator schema registry in `min_runtime/custom_ops.yaml` which `codegen.py` reads instead of the cadence yamls; a cadence op missing from it fails the runtime build with a bare KeyError, so it is declared there too. The requant expression in the packed kernel is deliberately byte-for-byte identical to `quantized_linear.h`, including the `(1 << 31)`, because the tests assert the packed and dense kernels agree exactly. Differential Revision: D117289220 Pull Request resolved: pytorch#23403
1 parent efc7f2e commit 5f4b03a

14 files changed

Lines changed: 1151 additions & 0 deletions

‎backends/cadence/aot/BUCK‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,8 @@ fbcode_target(_kind = runtime.python_library,
9797
],
9898
deps = [
9999
":fuse_ops",
100+
":pass_utils",
101+
":weight_packing",
100102
":remove_ops",
101103
":reorder_ops",
102104
":replace_ops",
@@ -131,6 +133,7 @@ fbcode_target(_kind = runtime.python_library,
131133
deps = [
132134
"fbcode//caffe2:torch",
133135
"fbcode//executorch/backends/cadence/aot:utils",
136+
"fbcode//executorch/backends/cadence/aot:weight_packing",
134137
"fbcode//executorch/exir:scalar_type",
135138
"fbcode//executorch/kernels/quantized:custom_ops_generated_lib",
136139
],
@@ -157,6 +160,7 @@ fbcode_target(_kind = executorch_generated_lib,
157160
"//executorch/backends/cadence/generic/operators:op_quantized_depthwise_conv1d_ncl",
158161
"//executorch/backends/cadence/generic/operators:op_quantized_depthwise_conv1d_nlc",
159162
"//executorch/backends/cadence/generic/operators:op_quantized_fully_connected",
163+
"//executorch/backends/cadence/generic/operators:op_quantized_fully_connected_packed",
160164
"//executorch/backends/cadence/generic/operators:op_quantized_layer_norm",
161165
"//executorch/backends/cadence/generic/operators:op_quantized_linear",
162166
"//executorch/backends/cadence/generic/operators:op_quantized_matmul",
@@ -637,6 +641,37 @@ fbcode_target(_kind = python_unittest,
637641
],
638642
)
639643

644+
fbcode_target(_kind = runtime.python_library,
645+
name = "weight_packing",
646+
srcs = [
647+
"weight_packing.py",
648+
],
649+
deps = [
650+
"fbcode//caffe2:torch",
651+
"//executorch/backends/cadence/aot/quantizer:pattern_utils",
652+
"//executorch/exir:lib",
653+
],
654+
)
655+
656+
fbcode_target(_kind = python_unittest,
657+
name = "test_weight_packing",
658+
srcs = [
659+
"tests/test_weight_packing.py",
660+
],
661+
supports_static_listing = False,
662+
deps = [
663+
"//caffe2:torch",
664+
"//executorch/backends/cadence/aot:compiler",
665+
"//executorch/backends/cadence/aot:ops_registrations",
666+
"//executorch/backends/cadence/aot:pass_utils",
667+
"//executorch/backends/cadence/aot:weight_packing",
668+
"//executorch/backends/cadence/aot:graph_builder",
669+
"//executorch/backends/cadence/aot/quantizer:fusion_pass",
670+
"//executorch/backends/cadence/aot/quantizer:quantizer",
671+
"//pytorch/ao:torchao",
672+
]
673+
)
674+
640675
fbcode_target(_kind = python_unittest,
641676
name = "test_ref_implementations",
642677
srcs = [
@@ -646,6 +681,7 @@ fbcode_target(_kind = python_unittest,
646681
deps = [
647682
":typing_stubs",
648683
"//executorch/backends/cadence/aot:ops_registrations",
684+
"//executorch/backends/cadence/aot:weight_packing",
649685
"//executorch/backends/cadence/aot/quantizer:utils",
650686
"//caffe2:torch",
651687
]

‎backends/cadence/aot/functions.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -494,6 +494,11 @@
494494
- arg_meta: null
495495
kernel_name: impl::generic::quantized_fully_connected_asym8uxasym8u_asym8u_per_tensor_out
496496

497+
- func: cadence::quantized_fully_connected_packed.out(Tensor src, Tensor weight, Tensor bias, int in_dim, int weight_bits, int src_zero_point, Tensor? weight_zero_point, Tensor out_multiplier, Tensor out_shift, int out_zero_point, Tensor? offset, *, Tensor(a!) out) -> Tensor(a!)
498+
kernels:
499+
- arg_meta: null
500+
kernel_name: impl::generic::quantized_fully_connected_packed_out
501+
497502
- func: cadence::requantize.out(Tensor input, Tensor in_scale, Tensor in_zero_point, Tensor out_scale, Tensor out_zero_point, ScalarType out_dtype, *, Tensor(a!) out) -> Tensor(a!)
498503
kernels:
499504
- arg_meta: null

‎backends/cadence/aot/functions_hifi.yaml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -460,6 +460,13 @@
460460
- arg_meta: null
461461
kernel_name: impl::HiFi::quantized_conv2d_nhwc_depthwise_asym8uxsym8u_asym8u_per_tensor_out
462462

463+
- func: cadence::quantized_fully_connected_packed.out(Tensor src, Tensor weight, Tensor bias, int in_dim, int weight_bits, int src_zero_point, Tensor? weight_zero_point, Tensor out_multiplier, Tensor out_shift, int out_zero_point, Tensor? offset, *, Tensor(a!) out) -> Tensor(a!)
464+
kernels:
465+
- arg_meta: null
466+
# No packed kernel on this backend yet: fall back to the generic one
467+
# rather than failing to resolve the operator.
468+
kernel_name: impl::generic::quantized_fully_connected_packed_out
469+
463470
- func: cadence::quantized_layer_norm.out(Tensor input, Tensor in_scale, Tensor in_zero_point, int[] normalized_shape, Tensor weight, Tensor bias, float eps, float output_scale, int output_zero_point, *, Tensor(a!) out) -> Tensor(a!)
464471
kernels:
465472
- arg_meta: null

‎backends/cadence/aot/ops_registrations.py‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -482,6 +482,13 @@ def register_fake(
482482
"quantized_fully_connected_asym8uxasym8u_asym8u.per_tensor(Tensor src, Tensor weight, Tensor bias, int src_zero_point, "
483483
"int weight_zero_point, int out_multiplier, int out_shift, int out_zero_point, Tensor? offset) -> (Tensor Z)"
484484
)
485+
# Sub-byte weights: `weight` is a [out_dim, packed_row_bytes] int8 blob, so
486+
# in_dim and the bit width can no longer be read off its shape.
487+
lib.define(
488+
"quantized_fully_connected_packed(Tensor src, Tensor weight, Tensor bias, int in_dim, int weight_bits, "
489+
"int src_zero_point, Tensor? weight_zero_point, Tensor out_multiplier, Tensor out_shift, int out_zero_point, "
490+
"Tensor? offset) -> (Tensor Z)"
491+
)
485492
lib.define("where_Scalar(Tensor condition, float self, float other) -> (Tensor Z)")
486493
lib.define(
487494
"where_Scalar.out(Tensor condition, float self, float other, *, Tensor(a!) out) -> Tensor(a!)"
@@ -666,6 +673,11 @@ def register_fake(
666673
"quantized_fully_connected_asym8uxasym8u_asym8u.per_tensor_out(Tensor src, Tensor weight, Tensor bias, int src_zero_point, "
667674
"int weight_zero_point, int out_multiplier, int out_shift, int out_zero_point, Tensor? offset, *, Tensor(a!) out) -> Tensor(a!)"
668675
)
676+
lib.define(
677+
"quantized_fully_connected_packed.out(Tensor src, Tensor weight, Tensor bias, int in_dim, int weight_bits, "
678+
"int src_zero_point, Tensor? weight_zero_point, Tensor out_multiplier, Tensor out_shift, int out_zero_point, "
679+
"Tensor? offset, *, Tensor(a!) out) -> Tensor(a!)"
680+
)
669681
lib.define(
670682
"quantized_embedding_byte.out(Tensor weight, Tensor weight_scales, Tensor? weight_zero_points, "
671683
"Tensor indices, bool pruned_weights=False, *, Tensor(a!) out) -> Tensor(a!)"
@@ -2769,6 +2781,33 @@ def quantized_fully_connected_per_tensor_meta(
27692781
return src.new_empty(out_size, dtype=src.dtype)
27702782

27712783

2784+
@register_fake("cadence::quantized_fully_connected_packed")
2785+
def quantized_fully_connected_packed_meta(
2786+
src: torch.Tensor,
2787+
weight: torch.Tensor,
2788+
bias: torch.Tensor,
2789+
in_dim: int,
2790+
weight_bits: int,
2791+
in_zero_point: int,
2792+
weight_zero_point: Optional[torch.Tensor],
2793+
out_multiplier: torch.Tensor,
2794+
out_shift: torch.Tensor,
2795+
out_zero_point: int,
2796+
offset: Optional[torch.Tensor],
2797+
) -> torch.Tensor:
2798+
torch._check(bias.dtype == torch.int32, lambda: "expected int32")
2799+
torch._check(weight.dim() == 2, lambda: "expected 2D tensor")
2800+
torch._check(src.size(0) == 1, lambda: "expected batch size of 1")
2801+
# src comes in shape [leading_dims, in_dim]
2802+
# weight comes in shape [out_dim, packed_row_bytes], so out_dim is still
2803+
# size(0) - that is what packing per row buys us - but in_dim is an
2804+
# argument rather than size(1).
2805+
torch._check(src.size(-1) == in_dim, lambda: "src last dim must equal in_dim")
2806+
out_size = list(src.size())
2807+
out_size[-1] = weight.size(0)
2808+
return src.new_empty(out_size, dtype=src.dtype)
2809+
2810+
27722811
@register_fake("cadence::quantized_fully_connected_asym8sxasym8s_asym8s.per_tensor")
27732812
def quantized_fully_connected_asym8sxasym8s_asym8s_per_tensor_meta(
27742813
src: torch.Tensor,

‎backends/cadence/aot/pass_utils.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,10 @@ class CompileMode(Enum):
4242
@dataclass(frozen=True)
4343
class EdgePassesConfig:
4444
use_im2row_transform: bool = False
45+
# Storage width for fully-connected weights. 8 leaves them as-is; 4 and 6
46+
# physically pack them, which is only a size win, not an accuracy one - the
47+
# values are expected to already be clamped to the narrower range.
48+
weight_bits: int = 8
4549

4650

4751
# Return the overload packet for the edge or torch op.

‎backends/cadence/aot/passes.py‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,9 @@
6161
SimplifySliceOpPass,
6262
)
6363
from executorch.backends.cadence.aot.type_dispatch import CompileTimeTypeDispatchPass
64+
from executorch.backends.cadence.aot.weight_packing import (
65+
fold_and_pack_fully_connected_weights,
66+
)
6467
from executorch.exir import EdgeProgramManager
6568
from executorch.exir.pass_base import ExportPass, PassResult
6669
from executorch.exir.pass_manager import PassManager, PassType
@@ -198,6 +201,13 @@ def apply_exir_ops_passes(
198201
list[Callable[[torch.fx.GraphModule], Optional[PassResult]]], cadence_passes
199202
)
200203
)
204+
# Weight packing runs last and outside the pass list on purpose: it changes
205+
# the shape of a weight constant, so it needs the ExportedProgram rather
206+
# than the GraphModule that `transform` hands to a pass.
207+
config = edge_passes_config or EdgePassesConfig()
208+
fold_and_pack_fully_connected_weights(
209+
cadence_prog_manager.exported_program(), config.weight_bits
210+
)
201211
return cadence_prog_manager
202212

203213

‎backends/cadence/aot/ref_implementations.py‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313
import torch.nn as nn
1414
import torch.nn.functional as F
1515
from executorch.backends.cadence.aot.utils import is_depthwise_conv
16+
from executorch.backends.cadence.aot.weight_packing import unpack_rows
1617
from executorch.exir.scalar_type import ScalarType
1718
from torch.library import impl, Library
1819

@@ -683,6 +684,37 @@ def quantized_fully_connected_asym8sxasym8s_asym8s_per_tensor() -> torch.Tensor:
683684
def quantized_fully_connected_asym8uxasym8u_asym8u_per_tensor() -> torch.Tensor: ...
684685

685686

687+
@impl_tracked(m, "quantized_fully_connected_packed")
688+
def quantized_fully_connected_packed(
689+
src: torch.Tensor,
690+
weight: torch.Tensor,
691+
bias: torch.Tensor,
692+
in_dim: int,
693+
weight_bits: int,
694+
in_zero_point: int,
695+
weight_zero_point: Optional[torch.Tensor],
696+
out_multiplier: torch.Tensor,
697+
out_shift: torch.Tensor,
698+
out_zero_point: int,
699+
offset: Optional[torch.Tensor] = None,
700+
) -> torch.Tensor:
701+
"""Sub-byte weights: unpack, then reuse the ordinary quantized path.
702+
703+
Keeping the arithmetic in `quantized_linear_common` means packing cannot
704+
drift from the unpacked operator, which is exactly what the tests assert.
705+
"""
706+
return quantized_linear_common(
707+
src,
708+
unpack_rows(weight, in_dim, weight_bits),
709+
bias,
710+
in_zero_point,
711+
weight_zero_point,
712+
out_multiplier,
713+
out_shift,
714+
out_zero_point,
715+
)
716+
717+
686718
@impl_tracked(m, "fully_connected")
687719
def fully_connected(
688720
input_tensor: torch.Tensor,

0 commit comments

Comments
 (0)