Skip to content

Commit 2f4fb0f

Browse files
NXP backend: Add Neutron backend to /backend/test/suite/operators testing. (pytorch#21712)
### Summary Use the testing suite in `/backend/test/suite/operators` to test the Neutron backend. Currently, all tests are passing except for 2 `cat` tests and all `lstm` tests due to known bugs in the NeutronConverter (so these tests are skipped). ### Test plan `pytest -c /dev/nul backends/test/suite/operators/ -m backend_nxp -n auto` cc @robert-kalmar @JakeStevens @digantdesai @rascani
1 parent 09bcc32 commit 2f4fb0f

51 files changed

Lines changed: 1190 additions & 100 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.ci/scripts/test_backend.sh‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,23 @@ if [[ "$FLOW" == *cortex_m* ]]; then
104104
backends/cortex_m/test/build_test_runner.sh
105105
fi
106106

107+
if [[ "$FLOW" == *nxp* ]]; then
108+
# Install the eIQ Toolkit Python packages (NSYS simulator and Neutron converter).
109+
pip install -r backends/nxp/requirements-eiq.txt
110+
111+
# Enable the Neutron delegate, portable kernels, pybindings and extensions
112+
# required by the operator test suite.
113+
EXTRA_BUILD_ARGS+=" -DEXECUTORCH_BUILD_NXP_NEUTRON=ON"
114+
EXTRA_BUILD_ARGS+=" -DEXECUTORCH_BUILD_NXP_NEUTRON_RUNNER=ON"
115+
EXTRA_BUILD_ARGS+=" -DEXECUTORCH_BUILD_KERNELS_PORTABLE=ON"
116+
EXTRA_BUILD_ARGS+=" -DEXECUTORCH_BUILD_PYBIND=ON"
117+
EXTRA_BUILD_ARGS+=" -DEXECUTORCH_BUILD_EXTENSION_MODULE=ON"
118+
EXTRA_BUILD_ARGS+=" -DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON"
119+
EXTRA_BUILD_ARGS+=" -DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON"
120+
EXTRA_BUILD_ARGS+=" -DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON"
121+
EXTRA_BUILD_ARGS+=" -DEXECUTORCH_BUILD_EXTENSION_NAMED_DATA_MAP=ON"
122+
fi
123+
107124
if [[ "$FLOW" == *openvino* ]]; then
108125
# Setup OpenVINO environment
109126
source .ci/scripts/setup-openvino.sh --nightly
@@ -117,6 +134,23 @@ else
117134
fi
118135
CMAKE_ARGS="$EXTRA_BUILD_ARGS" ${CONDA_RUN_CMD} $SETUP_SCRIPT --build-tool cmake --build-mode Release --editable true
119136

137+
if [[ "$FLOW" == *nxp* ]]; then
138+
# Install test-time Python requirements (neutron-test helpers, etc.).
139+
pip install -r backends/nxp/requirements-tests-pypi.txt
140+
PYTHON_EXECUTABLE=python bash examples/nxp/setup.sh
141+
142+
# Build nxp_executor_runner as a standalone binary. The cmake-out subproject
143+
# build may produce a differently-linked binary; the standalone build is known
144+
# to work correctly with the NSYS simulator firmware.
145+
mkdir -p examples/nxp/executor_runner/build
146+
pushd examples/nxp/executor_runner/build
147+
cmake -DCMAKE_BUILD_TYPE=Release ..
148+
make -j"$(nproc)" nxp_executor_runner
149+
popd
150+
151+
export NXP_RUNNER_PATH="$(pwd)/examples/nxp/executor_runner/build/nxp_executor_runner"
152+
fi
153+
120154
GOLDEN_DIR="${ARTIFACT_DIR}/golden-artifacts"
121155
export GOLDEN_ARTIFACTS_DIR="${GOLDEN_DIR}"
122156

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
name: Test NXP Backend
2+
3+
on:
4+
schedule:
5+
- cron: 0 2 * * *
6+
push:
7+
branches:
8+
- main
9+
- release/*
10+
tags:
11+
- ciflow/nightly/*
12+
pull_request:
13+
workflow_dispatch:
14+
15+
concurrency:
16+
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref_name }}-${{ github.ref_type == 'branch' && github.sha }}-${{ github.event_name == 'workflow_dispatch' }}-${{ github.event_name == 'schedule' }}
17+
cancel-in-progress: true
18+
19+
jobs:
20+
# Emits PR diff file list; non-PR events emit '*' so the per-job
21+
# `if:` short-circuits via `event_name != 'pull_request'`.
22+
changed-files:
23+
name: Get changed files
24+
uses: ./.github/workflows/_get-changed-files.yml
25+
26+
test-nxp:
27+
needs: changed-files
28+
# Every case in this flow runs the NSYS Neutron simulator, so the full
29+
# eIQ Toolkit install and nxp_executor_runner build are always required.
30+
# Path-gate on pull_request to avoid running on unrelated changes, while
31+
# still running in full on the nightly schedule and on pushes to main.
32+
if: |
33+
github.event_name != 'pull_request' ||
34+
contains(needs.changed-files.outputs.changed-files, 'backends/nxp') ||
35+
contains(needs.changed-files.outputs.changed-files, 'examples/nxp') ||
36+
contains(needs.changed-files.outputs.changed-files, 'backends/test/suite') ||
37+
contains(needs.changed-files.outputs.changed-files, 'backends/test/harness') ||
38+
contains(needs.changed-files.outputs.changed-files, '.ci/scripts/test_backend.sh') ||
39+
contains(needs.changed-files.outputs.changed-files, '.github/workflows/test-backend-nxp.yml') ||
40+
contains(needs.changed-files.outputs.changed-files, '.github/workflows/_test_backend.yml')
41+
uses: ./.github/workflows/_test_backend.yml
42+
with:
43+
backend: nxp
44+
flows: '["nxp_neutron_imxrt700_int8_ptq"]'
45+
# The models suite (torchvision/torchaudio) is excluded until a curated
46+
# list of models known to pass on the Neutron backend is established.
47+
exclude: '[{"flow": "nxp_neutron_imxrt700_int8_ptq", "suite": "models"}]'
48+
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
49+
timeout: 150
50+
run-linux: true
51+
docker-image: ci-image:executorch-ubuntu-22.04-clang12

‎backends/nxp/backend/edge_helper.py‎

Lines changed: 24 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -413,18 +413,18 @@ def try_get_arg(node: Node, idx: int) -> Argument | None:
413413
return node.args[idx] if idx < len(node.args) else None
414414

415415

416-
def input_quantization_type(
416+
def input_quantization_parameters(
417417
node: Node, input_index: int | tuple[int, int]
418-
) -> torch.dtype | None:
419-
"""Return the quantization input datatype of the QDQ quantized `node`.
418+
) -> tuple[Scale, ZeroPoint, torch.dtype] | None:
419+
"""Return the input quantization parameters of the QDQ quantized `node`.
420420
421421
:param node: The compute node.
422422
:param input_index: The index into the `node.args`. If a tuple of 2 ints is provided,
423423
`args[input_index[0]][input_index[1]]` is used instead.
424-
:return: The input quantization datatype of the QDQ quantized `node`, or `None` if the graph does not follow the
424+
:return: The input quantization parameters of the QDQ quantized `node`, or `None` if the graph does not follow the
425425
QDQ pattern or some metadata is incomplete or an invalid input index is given.
426426
427-
│ <returned type>
427+
│ <returned parameters>
428428
┌─────▼──────┐
429429
│ Dequantize │
430430
└─────┬──────┘
@@ -455,17 +455,24 @@ def input_quantization_type(
455455
if (dequantize_input_val := dequantize_node.args[0].meta.get("val")) is None:
456456
return None # Invalid metadata.
457457

458-
return dequantize_input_val.dtype
458+
params = get_quantization_parameters_for(dequantize_node)
459+
dtype = dequantize_input_val.dtype
460+
if params is None or dtype is None:
461+
return None
462+
463+
return *params, dtype
459464

460465

461-
def output_quantization_type(node: Node, output_index: int) -> torch.dtype | None:
462-
"""Return the quantization output datatype of the QDQ quantized `node`.
466+
def output_quantization_parameters(
467+
node: Node, output_index: int
468+
) -> tuple[Scale, ZeroPoint, torch.dtype] | None:
469+
"""Return the output quantization parameters of the QDQ quantized `node`.
463470
464471
:param node: The compute node.
465472
:param output_index: If the `node` has multiple outputs and therefore multiple `getitem` nodes follow it, the
466473
index selects the output. If no `getitem` nodes follow it, the operator
467474
produces only 1 output (most common case), and the value `0` must be used.
468-
:return: The output quantization datatype of the QDQ quantized `node`, or `None` if the graph does not follow the
475+
:return: The output quantization parameters of the QDQ quantized `node`, or `None` if the graph does not follow the
469476
QDQ pattern or some metadata is incomplete or an invalid input index is given.
470477
471478
┌───▼────┐
@@ -477,10 +484,10 @@ def output_quantization_type(node: Node, output_index: int) -> torch.dtype | Non
477484
┌────▼─────┐ or │ getitem(output_index) │ ...
478485
│ Quantize │ └─────────┬─────────────┘
479486
└────┬─────┘ │ float
480-
│ <returned type> ┌────▼─────┐
487+
│ <returned parameters> ┌────▼─────┐
481488
│ Quantize │
482489
└────┬─────┘
483-
│ <returned type>
490+
│ <returned parameters>
484491
"""
485492
users = list(node.users)
486493
if len(users) == 1 and _is_quantize(quantize_node := users[0]):
@@ -512,4 +519,9 @@ def output_quantization_type(node: Node, output_index: int) -> torch.dtype | Non
512519
if (quantize_val := quantize_node.meta.get("val")) is None:
513520
return None # Invalid metadata.
514521

515-
return quantize_val.dtype
522+
params = get_quantization_parameters_for(quantize_node)
523+
dtype = quantize_val.dtype
524+
if params is None or dtype is None:
525+
return None
526+
527+
return *params, dtype

‎backends/nxp/backend/ir/converter/node_converter.py‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,8 @@
1616
)
1717
from executorch.backends.nxp.backend.data_format import DataFormat, NXP_NODE_FORMAT
1818
from executorch.backends.nxp.backend.edge_helper import (
19-
input_quantization_type,
20-
output_quantization_type,
19+
input_quantization_parameters,
20+
output_quantization_parameters,
2121
)
2222
from executorch.backends.nxp.backend.ir import logger as logger
2323
from executorch.backends.nxp.backend.ir.conversion_context import ConversionContext
@@ -119,8 +119,9 @@ def _is_supported_in_IR(
119119
"""
120120
pass
121121

122-
@staticmethod
122+
@classmethod
123123
def _is_supported_on_target(
124+
cls,
124125
node: Node,
125126
neutron_target_spec: NeutronTargetSpec,
126127
parameters_mapping: dict[str, Parameter],
@@ -390,7 +391,8 @@ def uses_quantization_type_for_inputs(
390391
:return: True, if the `node` is QDQ quantized and has quantization input types in `supported_types`.
391392
"""
392393
return all(
393-
input_quantization_type(node, input_index) in supported_types
394+
(params := input_quantization_parameters(node, input_index)) is not None
395+
and params[2] in supported_types
394396
for input_index in input_indices
395397
)
396398

@@ -412,7 +414,8 @@ def uses_quantization_type_for_outputs(
412414
:return: True, if the `node` is QDQ quantized and has quantization output types in `supported_types`.
413415
"""
414416
return all(
415-
output_quantization_type(node, output_index) in supported_types
417+
(q_params := output_quantization_parameters(node, output_index)) is not None
418+
and q_params[2] in supported_types
416419
for output_index in output_indices
417420
)
418421

‎backends/nxp/backend/ir/converter/node_converters/ops_converters/abs_converter.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,8 +27,9 @@ def _is_supported_in_IR(
2727
) -> bool:
2828
return True
2929

30-
@staticmethod
30+
@classmethod
3131
def _is_supported_on_target(
32+
cls,
3233
node: Node,
3334
neutron_target_spec: NeutronTargetSpec,
3435
parameters_mapping: dict[str, Parameter],

‎backends/nxp/backend/ir/converter/node_converters/ops_converters/adaptive_avg_pool_2d_converter.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -58,8 +58,9 @@ def _is_supported_in_IR(
5858

5959
return True
6060

61-
@staticmethod
61+
@classmethod
6262
def _is_supported_on_target(
63+
cls,
6364
node: Node,
6465
neutron_target_spec: NeutronTargetSpec,
6566
parameters_mapping: dict[str, Parameter],

‎backends/nxp/backend/ir/converter/node_converters/ops_converters/add_tensor_converter.py‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,9 @@
1919

2020

2121
class AddTensorConverter(NodeConverter):
22-
@staticmethod
22+
@classmethod
2323
def _is_supported_on_target(
24+
cls,
2425
node: Node,
2526
neutron_target_spec: NeutronTargetSpec,
2627
parameters_mapping: dict[str, Parameter],
@@ -46,7 +47,7 @@ def _is_supported_in_IR(
4647
if len(node.args) != 2:
4748
return False
4849

49-
if hasattr(node.kwargs, "alpha"):
50+
if node.kwargs.get("alpha", 1) != 1:
5051
return False
5152

5253
return True

‎backends/nxp/backend/ir/converter/node_converters/ops_converters/addmm_converter.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,8 +62,9 @@ def _is_supported_in_IR(
6262

6363
return True
6464

65-
@staticmethod
65+
@classmethod
6666
def _is_supported_on_target(
67+
cls,
6768
node: Node,
6869
neutron_target_spec: NeutronTargetSpec,
6970
parameters_mapping: dict[str, Parameter],

‎backends/nxp/backend/ir/converter/node_converters/ops_converters/amax_converter.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,9 @@
2525

2626
class AmaxConverter(NodeConverter):
2727

28-
@staticmethod
28+
@classmethod
2929
def _is_supported_on_target(
30+
cls,
3031
node: Node,
3132
neutron_target_spec: NeutronTargetSpec,
3233
parameters_mapping: dict[str, Parameter],

‎backends/nxp/backend/ir/converter/node_converters/ops_converters/amin_converter.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,9 @@
2525

2626
class AminConverter(NodeConverter):
2727

28-
@staticmethod
28+
@classmethod
2929
def _is_supported_on_target(
30+
cls,
3031
node: Node,
3132
neutron_target_spec: NeutronTargetSpec,
3233
parameters_mapping: dict[str, Parameter],

0 commit comments

Comments
 (0)