Skip to content

Commit a48ff9f

Browse files
authored
Register a Cortex-M flow in the backend test suite (pytorch#21854)
### Summary The shared suite at backends/test/suite runs 23 torchvision and torchaudio models against every other major backend; Cortex-M had none, so its model coverage was hand-written one file at a time. This adds one, running on the Corstone-300 FVP. Five models pass and 15 are listed in CORTEX_M_SKIPS with a comment recording why each fails; read that list as a to-do. A pass means the model lowered and ran rather than that it was accurate, for the reason the module docstring gives. The suite drives to_edge_transform_and_lower, so CortexMTester grows a stage of that name: to_edge with no partitioner followed by CortexMPassManager, since Cortex-M rewrites operators in place rather than delegating a subgraph. The flow converts 4D inputs to channels_last, without which CortexMConv2DCheck rejects every convolution in every model and raises the FVP timeout, which an ImageNet-sized model exceeds at the 120 second default. ### Test plan CI goes through test_backend.sh like every other backend, path-gated because every case installs the Arm toolchain, builds a runner and drives the FVP. The operator suite is excluded and the exclusion says why. Authored with Claude Code.
1 parent eaa2dc6 commit a48ff9f

6 files changed

Lines changed: 240 additions & 2 deletions

File tree

‎.ci/scripts/test_backend.sh‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,15 @@ if [[ "$FLOW" == *arm* ]]; then
9595
fi
9696
fi
9797

98+
if [[ "$FLOW" == *cortex_m* ]]; then
99+
# Cortex-M runs on the Corstone-300 FVP, using the same Arm toolchain as the
100+
# Ethos-U flows but its own semihosting runner.
101+
.ci/scripts/setup-arm-baremetal-tools.sh
102+
source examples/arm/arm-scratch/setup_path.sh
103+
104+
backends/cortex_m/test/build_test_runner.sh
105+
fi
106+
98107
if [[ "$FLOW" == *openvino* ]]; then
99108
# Setup OpenVINO environment
100109
source .ci/scripts/setup-openvino.sh --nightly
Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
name: Test Cortex-M Backend
2+
3+
on:
4+
schedule:
5+
- cron: 0 2 * * *
6+
push:
7+
branches:
8+
- main
9+
- release/*
10+
tags:
11+
- ciflow/nightly/*
12+
pull_request:
13+
workflow_dispatch:
14+
15+
concurrency:
16+
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref_name }}-${{ github.ref_type == 'branch' && github.sha }}-${{ github.event_name == 'workflow_dispatch' }}-${{ github.event_name == 'schedule' }}
17+
cancel-in-progress: true
18+
19+
jobs:
20+
# Emits PR diff file list; non-PR events emit '*' so the per-job
21+
# `if:` short-circuits via `event_name != 'pull_request'`.
22+
changed-files:
23+
name: Get changed files
24+
uses: ./.github/workflows/_get-changed-files.yml
25+
26+
test-cortex-m:
27+
needs: changed-files
28+
# Every case in this flow serializes a .pte and runs it on the Corstone-300
29+
# FVP, on top of an Arm toolchain install and a runner build. The Arm
30+
# workflow keeps that off unrelated PRs by running only its TOSA and VGF
31+
# flows there; Cortex-M has no non-FVP flow to fall back to, so it is
32+
# path-gated instead and still runs in full on the nightly schedule.
33+
#
34+
# backends/arm is gated whole rather than by subdirectory. The flow borrows
35+
# Arm's test harness, and importing it pulls in around 350 arm modules --
36+
# the passes, the TOSA operators, the quantizer -- so a break almost
37+
# anywhere in that tree reaches this job. It costs little to be broad:
38+
# over the last 300 commits, 66 touched backends/arm and 61 of those
39+
# touched backends/arm/test or backends/arm/scripts anyway.
40+
if: |
41+
github.event_name != 'pull_request' ||
42+
contains(needs.changed-files.outputs.changed-files, 'backends/cortex_m') ||
43+
contains(needs.changed-files.outputs.changed-files, 'backends/arm') ||
44+
contains(needs.changed-files.outputs.changed-files, 'examples/arm') ||
45+
contains(needs.changed-files.outputs.changed-files, 'backends/test/suite') ||
46+
contains(needs.changed-files.outputs.changed-files, 'backends/test/harness') ||
47+
contains(needs.changed-files.outputs.changed-files, '.ci/scripts/test_backend.sh') ||
48+
contains(needs.changed-files.outputs.changed-files, '.ci/scripts/setup-arm-baremetal-tools.sh') ||
49+
contains(needs.changed-files.outputs.changed-files, '.github/workflows/test-backend-cortex-m.yml') ||
50+
contains(needs.changed-files.outputs.changed-files, '.github/workflows/_test_backend.yml')
51+
uses: ./.github/workflows/_test_backend.yml
52+
with:
53+
backend: cortex_m
54+
flows: '["cortex_m"]'
55+
# Most of the operator suite still fails on a kernel the runner does not
56+
# carry. Growing ops_list in backends/cortex_m/test/build_test_runner.sh
57+
# is the wrong fix -- portable kernels bloat the binary, and most of these
58+
# operators should be lowered, folded or decomposed instead of reaching
59+
# one. Excluded until enough of them lower to be worth the runtime.
60+
exclude: '[{"flow": "cortex_m", "suite": "operators"}]'
61+
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
62+
timeout: 120
63+
run-linux: true
64+
docker-image: ci-image:executorch-ubuntu-22.04-arm-sdk

‎backends/cortex_m/test/tester.py‎

Lines changed: 30 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@
2525
RunPasses,
2626
StageType,
2727
ToEdge,
28+
ToEdgeTransformAndLower,
2829
ToExecutorch,
2930
)
3031

@@ -51,15 +52,37 @@ def __init__(self, target_config: Optional[CortexMTargetConfig] = None):
5152
)
5253

5354

54-
class CortexMSerialize(Serialize):
55+
class CortexMToEdgeTransformAndLower(ToEdgeTransformAndLower):
56+
"""to_edge with no partitioner, then CortexMPassManager.
57+
58+
Cortex-M rewrites edge operators in place rather than delegating a subgraph,
59+
so this is its equivalent of to_edge_transform_and_lower, which is the only
60+
lowering entry point the shared backend test suite drives.
61+
"""
62+
5563
def __init__(self, target_config: Optional[CortexMTargetConfig] = None):
64+
super().__init__(edge_compile_config=cortex_m_edge_compile_config())
65+
self._run_passes = CortexMRunPasses(target_config)
66+
67+
def run(self, artifact, inputs=None, generate_etrecord: bool = False) -> None:
68+
super().run(artifact, inputs, generate_etrecord=generate_etrecord)
69+
self._run_passes.run(self.edge_dialect_program, inputs) # type: ignore[arg-type]
70+
71+
72+
class CortexMSerialize(Serialize):
73+
def __init__(
74+
self,
75+
target_config: Optional[CortexMTargetConfig] = None,
76+
timeout: int = 120,
77+
):
5678
target_config = target_config or CortexMTargetConfig(cpu=CortexM.M55)
5779
compile_spec = get_u55_compile_spec()
5880
# Select the runner built for this target (build_test_runner.sh writes
5981
# one runner per target into a target-suffixed directory).
6082
super().__init__(
6183
compile_spec,
6284
None,
85+
timeout=timeout,
6386
build_dir_suffix=f"_{target_config.target_string}",
6487
)
6588

@@ -69,6 +92,7 @@ def __init__(self, target_config: Optional[CortexMTargetConfig] = None):
6992
StageType.QUANTIZE: CortexMQuantize,
7093
StageType.RUN_PASSES: CortexMRunPasses,
7194
StageType.TO_EDGE: CortexMToEdge,
95+
StageType.TO_EDGE_TRANSFORM_AND_LOWER: CortexMToEdgeTransformAndLower,
7296
StageType.TO_EXECUTORCH: ToExecutorch,
7397
StageType.SERIALIZE: CortexMSerialize,
7498
}
@@ -80,6 +104,7 @@ def __init__(
80104
module,
81105
example_inputs,
82106
target_config: Optional[CortexMTargetConfig] = None,
107+
timeout: int = 120,
83108
):
84109
if callable(example_inputs):
85110
resolved_example_inputs = example_inputs()
@@ -92,8 +117,11 @@ def __init__(
92117
stage_classes[StageType.RUN_PASSES] = lambda: CortexMRunPasses(
93118
target_config=target_config
94119
)
120+
stage_classes[StageType.TO_EDGE_TRANSFORM_AND_LOWER] = (
121+
lambda: CortexMToEdgeTransformAndLower(target_config=target_config)
122+
)
95123
stage_classes[StageType.SERIALIZE] = lambda: CortexMSerialize(
96-
target_config=target_config
124+
target_config=target_config, timeout=timeout
97125
)
98126
super().__init__(module, resolved_example_inputs, stage_classes)
99127

‎backends/test/suite/conftest.py‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,13 @@ def pytest_collection_modifyitems(config, items):
3030
should_skip, reason = flow.should_skip_test(test_name, callspec.params)
3131
if should_skip:
3232
item.add_marker(pytest.mark.skip(reason))
33+
elif flow.should_xfail_test(test_name):
34+
item.add_marker(
35+
pytest.mark.xfail(
36+
reason=f"Expected to fail by {flow.name} xfail_patterns",
37+
strict=True,
38+
)
39+
)
3340

3441
item_path = str(getattr(item, "path", ""))
3542
for suite_prefix, timeout_s in FLOW_TEST_CASE_TIMEOUTS.items():

‎backends/test/suite/flow.py‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55

66
import logging
77
import os
8+
import shutil
89

910
from dataclasses import dataclass, field
1011
from typing import Any, Callable
@@ -44,6 +45,11 @@ class TestFlow:
4445
skip_patterns: list[str] = field(default_factory=lambda: [])
4546
""" Tests with names containing any substrings in this list are skipped. """
4647

48+
xfail_patterns: list[str] = field(default_factory=lambda: [])
49+
""" Tests with names containing any substrings in this list are expected to fail.
50+
They still run, so the report keeps recording how they fail; the marker is strict,
51+
so one that starts passing is reported rather than silently ignored. """
52+
4753
param_skip_reasons: dict[str, dict[Any, str]] = field(default_factory=dict)
4854
""" Skip tests with a given reason when a pytest parameter matches a given value."""
4955

@@ -69,6 +75,9 @@ def should_skip_test(
6975

7076
return False, ""
7177

78+
def should_xfail_test(self, test_name: str) -> bool:
79+
return any(pattern in test_name for pattern in self.xfail_patterns)
80+
7281
def __str__(self):
7382
return self.name
7483

@@ -176,6 +185,17 @@ def _load_arm() -> list[TestFlow]:
176185
]
177186

178187

188+
def _load_cortex_m() -> list[TestFlow]:
189+
# Every case runs on the FVP, so without it the whole flow fails the same way.
190+
if not shutil.which("FVP_Corstone_SSE-300_Ethos-U55"):
191+
logger.info("Skipping Cortex-M flow registration: Corstone-300 FVP not on PATH")
192+
return []
193+
194+
from executorch.backends.test.suite.flows.cortex_m import CORTEX_M_TEST_FLOW
195+
196+
return [CORTEX_M_TEST_FLOW]
197+
198+
179199
def all_flows() -> dict[str, TestFlow]:
180200
from executorch.backends.test.suite.flows.portable import PORTABLE_TEST_FLOW
181201

@@ -188,6 +208,7 @@ def all_flows() -> dict[str, TestFlow]:
188208
+ _register_flow(_load_openvino, "OpenVINO")
189209
+ _register_flow(_load_qnn, "QNN")
190210
+ _register_flow(_load_arm, "ARM")
211+
+ _register_flow(_load_cortex_m, "Cortex-M")
191212
)
192213

193214
try:
Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
# Copyright (c) Meta Platforms, Inc. and affiliates.
2+
# All rights reserved.
3+
#
4+
# This source code is licensed under the BSD-style license found in the
5+
# LICENSE file in the root directory of this source tree.
6+
7+
"""Cortex-M flow for the backend test suite.
8+
9+
Every case is serialized and executed on the Corstone-300 FVP with the
10+
semihosting runner from backends/cortex_m/test/build_test_runner.sh. That runner
11+
registers the cortex_m kernels along with the portable fallbacks listed in its
12+
ops_list, so an operator the backend does not lower still runs when the list
13+
happens to carry one.
14+
15+
The kernels are CMSIS-NN compiled for the device and cannot run on a host CPU,
16+
so the simulator is the only place to execute them. The Arm Ethos-U flows reach
17+
the Corstone FVP the same way. flow.py leaves the flow unregistered when the FVP
18+
is not on PATH.
19+
20+
Comparison uses the suite's fixed atol of 1e-1, which is not tuned for int8, so
21+
a mismatch is not by itself evidence of a lowering bug.
22+
"""
23+
24+
import torch
25+
26+
from executorch.backends.cortex_m.test.tester import CortexMQuantize, CortexMTester
27+
from executorch.backends.test.suite.flow import TestFlow
28+
29+
30+
def _create_cortex_m_tester(model, inputs, **kwargs) -> CortexMTester:
31+
# The CMSIS-NN kernels are NHWC, and CortexMConv2DCheck rejects any pattern
32+
# whose tensors are not channels_last. The suite hands over contiguous
33+
# tensors, which silently costs every convolution in the model.
34+
inputs = tuple(
35+
(
36+
t.to(memory_format=torch.channels_last)
37+
if isinstance(t, torch.Tensor) and t.dim() == 4
38+
else t
39+
)
40+
for t in inputs
41+
)
42+
# The Serialize stage is also the stage that invokes the ELF, so its timeout
43+
# is the FVP's --timelimit rather than a serialization budget, and 120s is
44+
# not enough for an ImageNet-sized model on an M55 with no NPU. Kept under
45+
# the suite conftest's 1200s pytest timeout so the FVP reports the overrun
46+
# instead of pytest killing the process.
47+
return CortexMTester(model, inputs, timeout=900, **kwargs)
48+
49+
50+
# Models that cannot run on the FVP runner at all, so there is nothing for the
51+
# report to record. A skipped test produces no row, which is why the sizes are
52+
# written down here.
53+
CORTEX_M_SKIPS = [
54+
# Over the runner's 60 MiB pool even fully quantized: 86.6M and 68.9M
55+
# parameters. Raising ET_ARM_BAREMETAL_SEMIHOSTING_FILE_ALLOCATOR_POOL_SIZE
56+
# is the only thing that would change that.
57+
"test_vit_b_16",
58+
"test_wide_resnet50_2",
59+
# The program is under a megabyte; it is the activation arena that does not
60+
# fit, and it crosses the pool at a sequence length of about 353. The test
61+
# draws lengths from randint(1, 400), so it fits roughly one run in three.
62+
"test_conformer",
63+
]
64+
65+
# Models that run to completion and fail. They stay in the suite so the report
66+
# keeps recording how they fail; the marker only keeps a known state from
67+
# turning the job red, and it is strict, so anything fixed reports XPASS and
68+
# this list has to shrink.
69+
CORTEX_M_XFAILS = [
70+
# Whatever a model leaves unlowered runs on the portable kernels compiled
71+
# into the runner, and ops_list in build_test_runner.sh is a fixed list.
72+
"test_convnext_small",
73+
"test_densenet161",
74+
"test_maxvit_t",
75+
"test_mnasnet1_0",
76+
"test_shufflenet_v2_x1_0",
77+
# 4.00 bytes per parameter: none of the convolutions lower, so the weights
78+
# stay fp32 and the 100 MiB program overruns the pool. Lowering conv1d
79+
# brings it to 26 MiB, which fits.
80+
"test_wav2letter",
81+
# Runs on target and does not match.
82+
"test_inception_v3",
83+
"test_resnet50",
84+
"test_squeezenet1_1",
85+
# AtenToCortexMPass rejects the graph.
86+
"test_swin_v2_t",
87+
]
88+
89+
90+
def _create_cortex_m_flow() -> TestFlow:
91+
return TestFlow(
92+
"cortex_m",
93+
backend="cortex_m",
94+
tester_factory=_create_cortex_m_tester,
95+
quantize=True,
96+
quantize_stage_factory=CortexMQuantize,
97+
is_delegated=False,
98+
param_skip_reasons={
99+
"use_dynamic_shapes": {
100+
True: "Cortex-M lowers for a fixed shape; the CMSIS-NN kernels "
101+
"take their dimensions from the graph."
102+
}
103+
},
104+
skip_patterns=CORTEX_M_SKIPS,
105+
xfail_patterns=CORTEX_M_XFAILS,
106+
)
107+
108+
109+
CORTEX_M_TEST_FLOW = _create_cortex_m_flow()

0 commit comments

Comments
 (0)