You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 5f4b03a
Browse filesBrowse the repository at this point in the historyBrowse files
Summary:
Adds 4- and 6-bit row-packed weight storage for fully-connected layers, from
the packing utilities all the way through to a dispatchable operator. Nothing
is landed unreachable: the packer, the operator, the registration, the
lowering hook and the backend fallback rows all arrive together.
The operator is `quantized_fully_connected_packed` rather than a packed
`quantized_linear` because wakeword stage 2 lowers to fully-connected, so this
is the shape the model will actually hit.
Layout: a [out_dim, in_dim] int8 weight becomes [out_dim, ceil(in_dim*bits/8)].
Packing per row is what keeps `out_dim` as `weight.size(0)`, so the existing
`out_size[-1] = weight.size(0)` meta computation still holds. `in_dim` and
`weight_bits` become explicit operator arguments, since neither can be read off
the packed shape any more. 6-bit follows the torchao UINT6 byte order; signed
values are stored as offset binary.
Packing is storage-only and never changes a value. 6-bit means "quantize with
quant_min/quant_max of -32/31, then store it in 6 bits", not "round an 8-bit
weight down". A layer whose weights do not fit the narrower range, or whose
in_dim is not a multiple of the group size, is skipped and stays 8-bit rather
than failing the compile.
Selection is an explicit compile-time option: a `weight_bits` field on
`EdgePassesConfig`, defaulting to 8 (off). The transform runs in
`apply_exir_ops_passes` after `.transform(...)` rather than as a pipeline pass,
because it changes a constant's *shape* and so needs the ExportedProgram, not
the GraphModule a pass is handed. Scalar qparams on a `.per_tensor` source are
lifted to length-1 constants, since the packed operator only has the
tensor-qparam form.
The reference implementation unpacks and delegates to `quantized_linear_common`
so the packed path cannot drift from the unpacked arithmetic. The C++ operator
wraps the same inner kernel `op_quantized_fully_connected.cpp` uses.
Backend wiring: only the generic row is added to `operator_fallback.bzl`. HiFi
and vision resolve to it through the existing fallback chain, which is the
point of the chain - a backend gets a row only when it has a specialized
implementation. Verified by removing the redundant HiFi row and confirming the
wakeword build still resolves the operator. HiFi's CMake build lists generic
fallback sources explicitly, so the new source is added there. Jarvis keeps its
own operator schema registry in `min_runtime/custom_ops.yaml` which
`codegen.py` reads instead of the cadence yamls; a cadence op missing from it
fails the runtime build with a bare KeyError, so it is declared there too.
The requant expression in the packed kernel is deliberately byte-for-byte
identical to `quantized_linear.h`, including the `(1 << 31)`, because the tests
assert the packed and dense kernels agree exactly.
Differential Revision: D117289220
Pull Request resolved: pytorch#23403
0 commit comments