Repository navigation
Commit 12668f8
authored
Support per-channel weights end to end (pytorch#22227)
Summary:
Per-channel weight quantization went through `prepare_pt2`/`convert_pt2`
and then quietly stopped being quantized: fusion bailed on the
per-channel dequant, so the convs fell out to float `cadence.fully_connected`.
On HiFi it is worse than a fallback, because `op_quantized_linear_out`
reads `out_multiplier[0]` even in the tensor-qparam overload and would
requantize every channel with channel 0's scale, silently.
This makes the whole path work, AoT and kernels. Granularity is carried by
the existing overload split, with `.out` (tensor qparams) now being the
per-channel form and `.per_tensor_out` the per-tensor one. No new operator
names are introduced.
Only the generic kernels implement per-channel, so `operator_fallback.bzl`
routes every per-channel op there: the HiFi and TIE rows for the four
tensor-qparam ops are removed, and `quantized_depthwise_conv1d_{ncl,nlc}.out`
gain generic rows they never had. Backends fall back instead of failing, and
a follow-up can reclaim performance by adding a row back. NNLib's conv entry
points are already per-channel capable, so HiFi conv is the obvious first
candidate; its fully-connected and matmul entry points take a single
(multiplier, shift) pair and structurally cannot be.
The two new depthwise rows also need schemas in
`min_runtime/custom_ops.yaml`: that file is not OSS-only, the Jarvis xtensa
codegen resolves every selective-build op name through it and fails the
build outright on one it cannot find.
- The quantizer aligns a pattern's derived bias spec with the granularity
of its weight spec. Previously six patterns hardcoded `per_tensor_affine`
on the bias, so `bias_scale = act_scale * weight_scale` produced a vector
that a per-tensor spec could not hold (`ValueError: only one element
tensors can be converted to Python scalars`). `CadenceAtenQuantizer.annotate`
is the one place that sees both specs, so the alignment lives there.
- Fusion accepts a per-channel weight dequant and emits the tensor-qparam
overload, materializing bias_scale / out_multiplier / out_shift /
weight_zero_point as lifted constants. Reading the weight scale and
adding constants both need the ExportedProgram, which `PassBase.call`
does not get, so `QuantFusionPass` takes one and stashes it for the
duration of the pass rather than changing `fuse()` on ~20 patterns.
- Both conv and linear fuse per-channel. Linear is what fully-connected
models are made of, so leaving it out would have made this a no-op for
the target model. `fuse_linear` mirrors `fuse_conv`; `quantized_linear`
has no bias_scale operand, so the scale reaches the kernel only through
out_multiplier/out_shift. `AddmmPattern` still declines: its weight is
`[in_features, out_features]`, so an output-channel axis of 0 is
meaningless there, and linear is preserved from decomposition anyway.
- Depthwise selection now runs before the per-channel overload swap in
`fuse_conv`. It compared against the `.per_tensor` target after the swap
had already moved to `.default`, so a per-channel depthwise conv silently
stayed a dense conv. `quantized_depthwise_conv1d_ncl`/`_nlc` had no
`.default` overload at all; the schemas, metas and reference
implementations are added here.
- The conv lowering exits carry per-channel through.
`ReplaceConvWithChannelLastConvPass` picks the layout from the input rank
as before but takes the overload from the incoming node, instead of
hardcoding `.per_tensor`. `ReplaceConvWithIm2RowAndLinear` normally
recomputes out_multiplier/out_shift from `bias_scale / out_scale`, which
is scalar arithmetic; for per-channel it reuses the conv's existing
multiplier/shift nodes, which were derived from exactly that ratio at
fusion time.
- The generic conv kernels do per-channel requantization. The conv2d and
conv1d cores take weight_zero_point and bias_scale as a pointer plus a
stride, resolved once per output channel: stride 0 keeps every channel on
element 0 (per-tensor), stride 1 walks the channel axis. Existing callers
pass 0, so per-tensor behaviour and the vectorizable inner loops are
unchanged. The `.out` entry points previously read element 0 and applied
channel 0's scale to every output channel.
- The conv reference implementations do real per-channel requantization.
The four tensor-qparam variants used to call `.item()` on their qparams
and delegate to the per-tensor path, which raised on a vector.
`quantized_conv_per_tensor` becomes `quantized_conv_common` and broadcasts
weight_zero_point and bias_scale over the output-channel axis.
Reviewed By: ethansfng
Differential Revision: D117164581
Pull Request resolved: pytorch#222271 parent b2896ae commit 12668f8
22 files changed
Lines changed: 3826 additions & 353 deletions
File tree
- backends/cadence
- aot
- quantizer
- tests
- generic/operators
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
434 | 434 | | |
435 | 435 | | |
436 | 436 | | |
| 437 | + | |
437 | 438 | | |
438 | 439 | | |
439 | 440 | | |
| |||
645 | 646 | | |
646 | 647 | | |
647 | 648 | | |
| 649 | + | |
648 | 650 | | |
649 | 651 | | |
650 | 652 | | |
| |||
664 | 666 | | |
665 | 667 | | |
666 | 668 | | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
| 674 | + | |
| 675 | + | |
| 676 | + | |
| 677 | + | |
| 678 | + | |
| 679 | + | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
667 | 684 | | |
668 | 685 | | |
669 | 686 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
170 | 170 | | |
171 | 171 | | |
172 | 172 | | |
173 | | - | |
| 173 | + | |
174 | 174 | | |
175 | 175 | | |
176 | 176 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
16 | 19 | | |
17 | 20 | | |
| 21 | + | |
18 | 22 | | |
19 | 23 | | |
20 | 24 | | |
| |||
714 | 718 | | |
715 | 719 | | |
716 | 720 | | |
| 721 | + | |
| 722 | + | |
| 723 | + | |
| 724 | + | |
| 725 | + | |
717 | 726 | | |
718 | 727 | | |
719 | | - | |
| 728 | + | |
| 729 | + | |
| 730 | + | |
| 731 | + | |
| 732 | + | |
720 | 733 | | |
721 | 734 | | |
| 735 | + | |
722 | 736 | | |
723 | 737 | | |
724 | 738 | | |
725 | | - | |
726 | | - | |
727 | | - | |
728 | | - | |
729 | | - | |
730 | | - | |
731 | | - | |
732 | | - | |
733 | | - | |
734 | | - | |
735 | | - | |
736 | | - | |
| 739 | + | |
| 740 | + | |
| 741 | + | |
| 742 | + | |
| 743 | + | |
| 744 | + | |
| 745 | + | |
| 746 | + | |
| 747 | + | |
| 748 | + | |
| 749 | + | |
| 750 | + | |
| 751 | + | |
| 752 | + | |
| 753 | + | |
| 754 | + | |
| 755 | + | |
737 | 756 | | |
738 | 757 | | |
739 | 758 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
268 | 268 | | |
269 | 269 | | |
270 | 270 | | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
271 | 277 | | |
272 | 278 | | |
273 | 279 | | |
274 | 280 | | |
275 | 281 | | |
276 | 282 | | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
277 | 289 | | |
278 | 290 | | |
279 | 291 | | |
| |||
1403 | 1415 | | |
1404 | 1416 | | |
1405 | 1417 | | |
| 1418 | + | |
| 1419 | + | |
| 1420 | + | |
| 1421 | + | |
| 1422 | + | |
| 1423 | + | |
| 1424 | + | |
| 1425 | + | |
| 1426 | + | |
| 1427 | + | |
| 1428 | + | |
| 1429 | + | |
| 1430 | + | |
| 1431 | + | |
| 1432 | + | |
| 1433 | + | |
| 1434 | + | |
| 1435 | + | |
| 1436 | + | |
| 1437 | + | |
| 1438 | + | |
| 1439 | + | |
| 1440 | + | |
| 1441 | + | |
| 1442 | + | |
| 1443 | + | |
| 1444 | + | |
| 1445 | + | |
| 1446 | + | |
| 1447 | + | |
| 1448 | + | |
| 1449 | + | |
| 1450 | + | |
| 1451 | + | |
| 1452 | + | |
| 1453 | + | |
| 1454 | + | |
| 1455 | + | |
| 1456 | + | |
| 1457 | + | |
| 1458 | + | |
| 1459 | + | |
| 1460 | + | |
| 1461 | + | |
| 1462 | + | |
| 1463 | + | |
| 1464 | + | |
| 1465 | + | |
| 1466 | + | |
| 1467 | + | |
| 1468 | + | |
| 1469 | + | |
| 1470 | + | |
| 1471 | + | |
| 1472 | + | |
| 1473 | + | |
| 1474 | + | |
| 1475 | + | |
| 1476 | + | |
| 1477 | + | |
| 1478 | + | |
| 1479 | + | |
| 1480 | + | |
| 1481 | + | |
| 1482 | + | |
| 1483 | + | |
| 1484 | + | |
| 1485 | + | |
| 1486 | + | |
| 1487 | + | |
1406 | 1488 | | |
1407 | 1489 | | |
1408 | 1490 | | |
| |||
0 commit comments