Skip to content

Commit bf88c64

Browse files
authored
Cortex-M: expose explicit-layout AOT (pytorch#22545)
### Summary Expose the experimental explicit-layout Cortex-M pipeline through `--cortex-m-explicit-layout`. Explicit mode keeps model inputs contiguous and selects the matching quantizer mode and blessed pass sequence. Legacy mode remains the default and continues converting inputs to channels-last before quantization. Explicit mode requires quantization and never falls back to legacy spatial operators. The tests cover mode-specific operator selection, removal of intermediate channels-last dialect nodes, int8 layout-copy boundaries, copy ceilings for MobileNetV2, ResNet8, and Silero VAD, and serialized Conv1d execution on the M55 FVP. The documentation also records how the explicit mode becomes the default and how legacy AOT generation is subsequently removed without removing legacy runtime operators. ### Test plan `source examples/arm/arm-scratch/setup_path.sh && python -m pytest --config-file=backends/arm/test/pytest.ini backends/cortex_m/test/test_explicit_layout.py` `lintrunner -m origin/main` AI-assisted: Codex. --- <sub>Stack created with <a href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
1 parent f624c34 commit bf88c64

5 files changed

Lines changed: 226 additions & 7 deletions

File tree

‎backends/arm/scripts/aot_arm_compiler.py‎

Lines changed: 32 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -626,6 +626,14 @@ def _get_args():
626626
choices=TARGETS,
627627
help=f"Target backend. For delegated models: Ethos-U/VGF/TOSA variants. For non-delegated: cortex-m<variant> (CMSIS-NN portable kernels). Valid targets: {TARGETS}",
628628
)
629+
parser.add_argument(
630+
"--cortex-m-explicit-layout",
631+
action="store_true",
632+
help=(
633+
"Use explicit NCHW/NHWC permutes for Cortex-M instead of dim-order "
634+
"operators. This is an experimental Cortex-M-only option."
635+
),
636+
)
629637
# TODO: Remove --evaluate and --evaluate_config completely after a suitable time.
630638
# They are deprecated and no longer functional in this script.
631639
parser.add_argument(
@@ -923,9 +931,16 @@ def _to_edge_cortex_m(
923931
"""Cortex-M/CMSIS-NN compilation path with no delegation."""
924932
logging.info(
925933
f"Using Cortex-M/CMSIS-NN compilation path for cpu={target_config.cpu.name} "
926-
f"backend={target_config.backend.name}"
934+
f"backend={target_config.backend.name} "
935+
f"layout={'explicit' if args.cortex_m_explicit_layout else 'dim-order'}"
927936
)
928937

938+
if args.cortex_m_explicit_layout and not args.quantize:
939+
raise RuntimeError(
940+
"--cortex-m-explicit-layout requires --quantize; explicit layout "
941+
"does not fall back to portable float spatial operators."
942+
)
943+
929944
def _to_channels_last(x):
930945
if isinstance(x, torch.Tensor):
931946
if x.dim() == 4:
@@ -949,17 +964,26 @@ def _to_channels_last(x):
949964
)
950965
model_quant = None
951966
else:
952-
model = model.to(memory_format=torch.channels_last) # type: ignore[call-overload]
953-
example_inputs = tuple(_to_channels_last(x) for x in example_inputs)
967+
if not args.cortex_m_explicit_layout:
968+
model = model.to(memory_format=torch.channels_last) # type: ignore[call-overload]
969+
example_inputs = tuple(_to_channels_last(x) for x in example_inputs)
970+
# Refresh fake-tensor strides after changing the captured module's
971+
# memory format so the legacy quantizer sees channels-last inputs.
972+
model = torch.export.export(
973+
model, example_inputs, strict=args.strict_export
974+
).module()
975+
976+
quantizer = CortexMQuantizer(use_explicit_layout=args.cortex_m_explicit_layout)
954977

955-
quantizer = CortexMQuantizer()
956978
prepared = prepare_pt2e(model, quantizer)
957979

958980
if calibration_samples is None:
959981
calibration_samples = [example_inputs]
960982

961983
for sample in calibration_samples:
962-
prepared(*tuple(_to_channels_last(x) for x in sample))
984+
if not args.cortex_m_explicit_layout:
985+
sample = tuple(_to_channels_last(x) for x in sample)
986+
prepared(*sample)
963987

964988
model_quant = convert_pt2e(prepared)
965989

@@ -973,7 +997,9 @@ def _to_channels_last(x):
973997
)
974998

975999
pass_manager = CortexMPassManager(
976-
edge.exported_program(), target_config=target_config
1000+
edge.exported_program(),
1001+
target_config=target_config,
1002+
use_explicit_layout=args.cortex_m_explicit_layout,
9771003
)
9781004
edge._edge_programs["forward"] = pass_manager.transform()
9791005

‎backends/cortex_m/README.md‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,17 @@
55
66
## Overview
77

8-
The Cortex-M backend is implemented as an operator dialect/library based on [CMSIS-NN](https://github.com/ARM-software/CMSIS-NN), together with the `CortexMQuantizer` which targets supported ops, and the `CortexMPassManager` which modifies the exported program to use Cortex-M operators where possible. It is intended for use with **channels-last input** since this is what the accelerated kernels are using.
8+
The Cortex-M backend is implemented as an operator dialect/library based on [CMSIS-NN](https://github.com/ARM-software/CMSIS-NN), together with the `CortexMQuantizer` which targets supported ops, and the `CortexMPassManager` which modifies the exported program to use Cortex-M operators where possible.
9+
10+
The default AOT path retains the established channels-last input and dim-order contract. An experimental explicit-layout path accepts ordinary contiguous inputs, inserts graph-visible NHWC copies, and lowers spatial kernels to the `cortex_m::*_nhwc` operator family. Enable it with `--cortex-m-explicit-layout`; the two modes do not fall back to or mix with each other. Explicit-layout compilation fails when a spatial operator is not eligible for NHWC lowering, so models using unsupported configurations must use the legacy mode.
11+
12+
### Explicit-layout migration
13+
14+
The `use_explicit_layout=True` modes on `CortexMQuantizer` and `CortexMPassManager` are temporary staging APIs. They keep the experimental path isolated while the default modes and `CortexMTester` continue to exercise the legacy path.
15+
16+
When explicit layout becomes the default, its support table and pass list will become the defaults in `CortexMQuantizer` and `CortexMPassManager`. The existing legacy support table and pass list will remain temporarily behind an opt-out AOT flag. This keeps the public Python entry points stable and switches `CortexMTester` to explicit layout without changing its callers. The direct NHWC kernel tests and explicit-only model tests will then be folded into the normal operator and model suites.
17+
18+
After the legacy AOT compatibility period, the legacy support table, pass list, input conversion, opt-out flag, and remaining dual-mode tests will be removed. Legacy runtime operators will remain registered so programs serialized by the old AOT path continue to load.
919

1020
For a detailed example of the full lowering flow, see `examples/arm/cortex_m_mv2_example.ipynb`.
1121

‎backends/cortex_m/test/build_test_runner.sh‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,7 @@ ops_list=(
5252
aten::full.out
5353
aten::ge.Tensor_out
5454
aten::unsqueeze_copy.out
55+
aten::squeeze_copy.dim_out
5556
aten::select_copy.int_out
5657
aten::amax.out
5758
cortex_m::quantize_per_tensor.out
Lines changed: 180 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,180 @@
1+
# Copyright (c) Meta Platforms, Inc. and affiliates.
2+
# All rights reserved.
3+
#
4+
# This source code is licensed under the BSD-style license found in the
5+
# LICENSE file in the root directory of this source tree.
6+
7+
from types import SimpleNamespace
8+
9+
import pytest
10+
import torch
11+
from executorch.backends.cortex_m.target_config import CortexM, CortexMTargetConfig
12+
from executorch.backends.cortex_m.test.tester import CortexMSerialize
13+
from executorch.exir.dialects._ops import ops as exir_ops
14+
15+
16+
_LEGACY_SPATIAL_OPS = {
17+
exir_ops.edge.cortex_m.quantized_conv2d.default,
18+
exir_ops.edge.cortex_m.quantized_depthwise_conv2d.default,
19+
exir_ops.edge.cortex_m.quantized_transpose_conv2d.default,
20+
exir_ops.edge.cortex_m.quantized_avg_pool2d.default,
21+
exir_ops.edge.cortex_m.quantized_max_pool2d.default,
22+
}
23+
24+
# Temporary dual-mode acceptance coverage; see "Explicit-layout migration" in
25+
# the Cortex-M README.
26+
27+
28+
class Conv2d(torch.nn.Module):
29+
def __init__(self) -> None:
30+
super().__init__()
31+
self.conv = torch.nn.Conv2d(3, 4, 3, padding=1)
32+
33+
def forward(self, x):
34+
return self.conv(x)
35+
36+
37+
class Conv1d(torch.nn.Module):
38+
def __init__(self) -> None:
39+
super().__init__()
40+
self.conv = torch.nn.Conv1d(2, 4, 3, padding=1)
41+
42+
def forward(self, x):
43+
return self.conv(x)
44+
45+
46+
def _compile(module, inputs, *, explicit_layout: bool, quantize: bool = True):
47+
from executorch.backends.arm.scripts.aot_arm_compiler import _to_edge_cortex_m
48+
49+
exported_program = torch.export.export(module, inputs, strict=True)
50+
return _to_edge_cortex_m(
51+
exported_program,
52+
SimpleNamespace(
53+
cortex_m_explicit_layout=explicit_layout,
54+
quantize=quantize,
55+
strict_export=True,
56+
),
57+
exported_program.module(),
58+
inputs,
59+
None,
60+
CortexMTargetConfig(cpu=CortexM.M55),
61+
)
62+
63+
64+
def _count(exported_program, target) -> int:
65+
return sum(node.target == target for node in exported_program.graph.nodes)
66+
67+
68+
@pytest.mark.parametrize(
69+
"explicit_layout,expected,unexpected",
70+
[
71+
(
72+
False,
73+
exir_ops.edge.cortex_m.quantized_conv2d.default,
74+
exir_ops.edge.cortex_m.quantized_conv2d_nhwc.default,
75+
),
76+
(
77+
True,
78+
exir_ops.edge.cortex_m.quantized_conv2d_nhwc.default,
79+
exir_ops.edge.cortex_m.quantized_conv2d.default,
80+
),
81+
],
82+
)
83+
def test_aot_layout_mode_selects_operator_family(explicit_layout, expected, unexpected):
84+
_, edge, _ = _compile(
85+
Conv2d().eval(),
86+
(torch.randn(1, 3, 8, 8),),
87+
explicit_layout=explicit_layout,
88+
)
89+
program = edge.exported_program()
90+
91+
assert _count(program, expected) == 1
92+
assert _count(program, unexpected) == 0
93+
94+
95+
def test_aot_explicit_layout_requires_quantization():
96+
with pytest.raises(RuntimeError, match="requires --quantize"):
97+
_compile(
98+
Conv2d().eval(),
99+
(torch.randn(1, 3, 8, 8),),
100+
explicit_layout=True,
101+
quantize=False,
102+
)
103+
104+
105+
def _assert_explicit_copy_ceiling(module, inputs, ceiling):
106+
_, edge, _ = _compile(module, inputs, explicit_layout=True)
107+
program = edge.exported_program()
108+
copies = [
109+
node
110+
for node in program.graph.nodes
111+
if node.target
112+
in {
113+
exir_ops.edge.cortex_m.transpose.default,
114+
exir_ops.edge.aten.view_copy.default,
115+
}
116+
]
117+
118+
assert len(copies) <= ceiling
119+
assert all(
120+
node.args[0].meta["val"].dtype == torch.int8
121+
for node in copies
122+
if node.target == exir_ops.edge.cortex_m.transpose.default
123+
)
124+
assert not any(node.target in _LEGACY_SPATIAL_OPS for node in program.graph.nodes)
125+
assert not any(
126+
getattr(node.target, "namespace", None) == "channels_last"
127+
for node in program.graph.nodes
128+
)
129+
130+
131+
def test_mobilenet_v2_explicit_copy_ceiling():
132+
torchvision = pytest.importorskip("torchvision")
133+
_assert_explicit_copy_ceiling(
134+
torchvision.models.mobilenet_v2(weights=None).eval(),
135+
(torch.randn(1, 3, 224, 224),),
136+
ceiling=2,
137+
)
138+
139+
140+
def test_resnet8_explicit_copy_ceiling():
141+
from executorch.examples.models.mlperf_tiny.resnet8 import ResNet8
142+
143+
_assert_explicit_copy_ceiling(
144+
ResNet8().eval(),
145+
(torch.rand(1, 3, 32, 32) * 2 - 1,),
146+
ceiling=2,
147+
)
148+
149+
150+
def test_silero_explicit_copy_ceiling():
151+
from executorch.examples.models.silero_vad.export_silero_vad import (
152+
CONTEXT_SIZE,
153+
HIDDEN_DIM,
154+
SileroVAD16k,
155+
WINDOW_SIZE,
156+
)
157+
158+
_assert_explicit_copy_ceiling(
159+
SileroVAD16k().eval(),
160+
(
161+
torch.randn(1, CONTEXT_SIZE + WINDOW_SIZE),
162+
torch.zeros(2, 1, HIDDEN_DIM),
163+
),
164+
ceiling=12,
165+
)
166+
167+
168+
def test_explicit_conv1d_runs_on_fvp():
169+
inputs = (torch.linspace(-5, 5, steps=16).reshape(1, 2, 8),)
170+
model_quant, edge, runtime_inputs = _compile(
171+
Conv1d().eval(), inputs, explicit_layout=True
172+
)
173+
program = edge.exported_program()
174+
assert _count(program, exir_ops.edge.cortex_m.quantized_conv2d_nhwc.default) == 1
175+
176+
serialized = CortexMSerialize(CortexMTargetConfig(cpu=CortexM.M55))
177+
serialized.run(edge.to_executorch())
178+
[actual] = serialized.run_artifact(runtime_inputs)
179+
expected = model_quant(*runtime_inputs)
180+
torch.testing.assert_close(actual, expected, atol=0.05, rtol=1e-3)

‎docs/source/backends/arm-cortex-m/arm-cortex-m-overview.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@ This backend is in **beta**. It has been validated with a set of small models (e
66

77
The Arm&reg; Cortex&reg;-M backend accelerates quantized model execution on Arm Cortex-M CPUs using [CMSIS-NN](https://arm-software.github.io/CMSIS-NN/latest/) optimized kernels. Unlike delegate-based backends, it operates as an operator library: quantized subgraphs are replaced with CMSIS-NN accelerated kernels during the pass-lowering stage, while unsupported operators fall back to portable fp32 kernels.
88

9+
The default AOT flow uses channels-last inputs and the existing dim-order representation. The experimental explicit-layout flow uses ordinary contiguous inputs, represents NCHW/NHWC conversions as graph operators, and selects the experimental `cortex_m::*_nhwc` kernels. Enable it with `--cortex-m-explicit-layout`. Layout modes are selected independently of the Cortex-M CPU target and never mix operator families.
10+
911
## Target Support
1012

1113
The backend targets Arm Cortex-M CPUs via CMSIS-NN, which provides optimized kernel implementations for three instruction set variants:

0 commit comments

Comments
 (0)