Skip to content

Commit d7138b4

Browse files
authored
CI: test the generated Arduino library on pull requests (pytorch#21582)
The library is generated from this repository, so an ExecuTorch change can break it without touching examples/arduino at all. Nothing checked that. The job builds the library, verifies the shipped models against it, compiles every example for arduino:zephyr:unoq, and runs arduino-lint in Library Manager submission mode. It runs on changes under examples/arduino, backends/cortex_m, kernels/portable, runtime, and schema, which is the set that can break any of the four. verify_models.py is the part worth having. A .pte records how many values each operator call puts on the stack, and the generated kernel wrappers reject anything else; the two only agree when the model and the library came from the same commit. Cortex-M schemas change - scratch was added to the conv operators in pytorch#19636 and pytorch#19825 - and a mismatch is invisible until far too late, because the program loads, every operator resolves, and then execute returns InvalidProgram naming nothing useful. Tracking that down by hand took the better part of a day. The check compares the two directly in about a second, needs no board, and reports: FAIL KeywordSpotting cortex_m::quantized_avg_pool2d.out: model supplies 10, library expects 11 Compiling is deliberately not the whole job. Every failure worth finding in this library compiled cleanly first, so the model check runs before the sketches and arduino-lint guards the thing users actually install.
1 parent 39a3e0c commit d7138b4

4 files changed

Lines changed: 272 additions & 13 deletions

File tree

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
name: Test Arduino library
2+
3+
permissions:
4+
id-token: write
5+
contents: read
6+
7+
on:
8+
workflow_call:
9+
inputs:
10+
timeout:
11+
description: 'Job timeout in minutes'
12+
required: false
13+
type: number
14+
default: 90
15+
16+
jobs:
17+
run:
18+
uses: pytorch/test-infra/.github/workflows/linux_job_v2.yml@main
19+
permissions:
20+
id-token: write
21+
contents: read
22+
with:
23+
job-name: arduino-library
24+
runner: linux.2xlarge
25+
docker-image: ci-image:executorch-ubuntu-22.04-arm-sdk
26+
submodules: 'recursive'
27+
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
28+
timeout: ${{ inputs.timeout }}
29+
script: |
30+
# The generic Linux job chooses to use base env, not the one setup by the image
31+
CONDA_ENV=$(conda env list --json | jq -r ".envs | .[-1]")
32+
conda activate "${CONDA_ENV}"
33+
34+
source .ci/scripts/utils.sh
35+
install_executorch "--use-pt-pinned-commit"
36+
37+
# Schema headers come from a cmake build; the library build script
38+
# refuses to run without them.
39+
cmake -B cmake-out -DCMAKE_BUILD_TYPE=Release
40+
cmake --build cmake-out --target program_schema -j$(nproc)
41+
42+
cd examples/arduino
43+
./build_arduino_library.sh
44+
45+
# A model exported against a different ExecuTorch commit than the one
46+
# that built the library loads fine, resolves every operator, and then
47+
# fails inside Method::execute. Catch that here rather than on a board.
48+
python verify_models.py arduino_lib/ExecuTorch
49+
50+
# arduino-cli, the Uno Q core, and the Serial dependency the core
51+
# hard-errors without.
52+
# Pinned release tarballs rather than piping an installer script from a
53+
# moving branch: reproducible, and one less thing to fail transiently.
54+
ARDUINO_CLI_VERSION=1.5.1
55+
ARDUINO_LINT_VERSION=1.3.0
56+
# RUNNER_TEMP and GITHUB_WORKSPACE both point at host paths this
57+
# container cannot write to, and HOME may too, so keep the toolchain
58+
# and every arduino-cli directory somewhere local to the container.
59+
ARDUINO_CI_DIR=/tmp/.arduino-ci
60+
export ARDUINO_DIRECTORIES_USER="${ARDUINO_CI_DIR}/user"
61+
export ARDUINO_DIRECTORIES_DATA="${ARDUINO_CI_DIR}/data"
62+
export ARDUINO_DIRECTORIES_DOWNLOADS="${ARDUINO_CI_DIR}/downloads"
63+
mkdir -p "${ARDUINO_DIRECTORIES_USER}" "${ARDUINO_DIRECTORIES_DATA}" \
64+
"${ARDUINO_DIRECTORIES_DOWNLOADS}" "${ARDUINO_CI_DIR}/bin"
65+
export PATH="${ARDUINO_CI_DIR}/bin:${PATH}"
66+
curl -fsSL --retry 3 --retry-delay 5 --retry-all-errors \
67+
"https://downloads.arduino.cc/arduino-cli/arduino-cli_${ARDUINO_CLI_VERSION}_Linux_64bit.tar.gz" \
68+
| tar xz -C "${ARDUINO_CI_DIR}/bin" arduino-cli
69+
arduino-cli core update-index
70+
arduino-cli core install arduino:zephyr
71+
arduino-cli lib install Arduino_RouterBridge
72+
73+
# link_mode=static is not optional. The board defaults to Dynamic,
74+
# which builds the sketch as a Zephyr loadable extension; a library
75+
# this size never starts that way and prints nothing at all.
76+
FQBN="arduino:zephyr:unoq:link_mode=static"
77+
for sketch in arduino_lib/ExecuTorch/examples/*/; do
78+
echo "::group::compile $(basename "${sketch}")"
79+
arduino-cli compile --fqbn "${FQBN}" \
80+
--libraries arduino_lib "${sketch}"
81+
echo "::endgroup::"
82+
done
83+
84+
# The library is published through the Arduino Library Manager, which
85+
# applies these rules at submission time.
86+
curl -fsSL --retry 3 --retry-delay 5 --retry-all-errors \
87+
"https://downloads.arduino.cc/arduino-lint/arduino-lint_${ARDUINO_LINT_VERSION}_Linux_64bit.tar.gz" \
88+
| tar xz -C "${ARDUINO_CI_DIR}/bin" arduino-lint
89+
cp ../../LICENSE arduino_lib/ExecuTorch/LICENSE
90+
cp README.md arduino_lib/ExecuTorch/README.md
91+
(cd arduino_lib/ExecuTorch && arduino-lint \
92+
--project-type library --library-manager submit --compliance strict)

‎.github/workflows/pull.yml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,23 @@ jobs:
6161
6262
PYTHON_EXECUTABLE=python bash .ci/scripts/test_wheel_package_qnn.sh "${{ matrix.python-version }}"
6363
64+
test-arduino-library:
65+
needs: changed-files
66+
if: |
67+
github.event_name != 'pull_request' ||
68+
contains(needs.changed-files.outputs.changed-files, '.github/workflows/_test_arduino_library.yml') ||
69+
contains(needs.changed-files.outputs.changed-files, '.github/workflows/pull.yml') ||
70+
contains(needs.changed-files.outputs.changed-files, 'backends/cortex_m/') ||
71+
contains(needs.changed-files.outputs.changed-files, 'examples/arduino/') ||
72+
contains(needs.changed-files.outputs.changed-files, 'kernels/portable/') ||
73+
contains(needs.changed-files.outputs.changed-files, 'runtime/') ||
74+
contains(needs.changed-files.outputs.changed-files, 'schema/')
75+
name: test-arduino-library
76+
uses: ./.github/workflows/_test_arduino_library.yml
77+
permissions:
78+
id-token: write
79+
contents: read
80+
6481
test-minimal-wheel-linux:
6582
needs: changed-files
6683
if: |

‎examples/arduino/build_arduino_library.sh‎

Lines changed: 42 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -189,18 +189,22 @@ for yaml in "$TORCHGEN/packaged/ATen/native/tags.yaml" \
189189
fi
190190
done
191191

192-
# The exporter and the runtime must be the same ExecuTorch. A pip release wheel
193-
# can be months behind this checkout, and a model exported against one schema
194-
# fails at Method::execute against a library built from another.
195-
ET_PY=$("$PYTHON" -c "import executorch; print(next(iter(executorch.__path__), ''))" 2>/dev/null || echo "")
196-
case "$ET_PY" in
197-
"$ET_ROOT"*) ;;
198-
"") echo " NOTE: no executorch Python package found; the library will build but" ;
199-
echo " you cannot export models with this interpreter." ;;
200-
*) echo " WARNING: $PYTHON imports executorch from $ET_PY, not $ET_ROOT." ;
201-
echo " Models exported with it may not match this library. See" ;
202-
echo " 'Keeping this working' in examples/arduino/README.md." ;;
203-
esac
192+
# The exporter and the runtime must be the same ExecuTorch. A model exported
193+
# against one operator schema fails at Method::execute against a library built
194+
# from another, and nothing in that error says so. Compare what the installed
195+
# package was built from against this checkout; an install from here lands in
196+
# site-packages, so the path alone tells us nothing.
197+
ET_PY_SHA=$("$PYTHON" -c "import executorch.version as v; print(getattr(v, 'git_version', ''))" 2>/dev/null || echo "")
198+
ET_GIT_SHA=$(git -C "$ET_ROOT" rev-parse HEAD 2>/dev/null || echo "")
199+
if [ -z "$ET_PY_SHA" ]; then
200+
echo " NOTE: no executorch Python package found. The library will build, but"
201+
echo " you cannot export models with $PYTHON."
202+
elif [ -n "$ET_GIT_SHA" ] && [ "$ET_PY_SHA" != "$ET_GIT_SHA" ]; then
203+
echo " WARNING: $PYTHON has executorch built from ${ET_PY_SHA:0:12}, this tree"
204+
echo " is at ${ET_GIT_SHA:0:12}. Models exported with it may not match"
205+
echo " the library. Run ./install_executorch.sh, or see 'Keeping this"
206+
echo " working' in examples/arduino/README.md."
207+
fi
204208

205209
CODEGEN_OUT="$ET_SRC/codegen"
206210
CORTEX_M_YAML="$ET_ROOT/backends/cortex_m/ops/operators.yaml"
@@ -310,6 +314,30 @@ for candidate in \
310314
fi
311315
done
312316

317+
# Nothing vendors CMSIS-NN into a plain checkout -- the Cortex-M backend pulls
318+
# it with FetchContent at cmake time -- so fetch it at the revision that backend
319+
# pins. Without it the Cortex-M ops compile against headers that are not there.
320+
if [ -z "$CMSIS_NN" ]; then
321+
CMSIS_NN_PIN=$(sed -n '/set(CMSIS_NN_VERSION/,/)/p' \
322+
"$ET_ROOT/backends/cortex_m/CMakeLists.txt" | grep -oE '"[0-9a-f]{40}"' | tr -d '"')
323+
if [ -z "$CMSIS_NN_PIN" ]; then
324+
echo "ERROR: could not read CMSIS_NN_VERSION from backends/cortex_m/CMakeLists.txt"
325+
exit 1
326+
fi
327+
CMSIS_NN="$ET_ROOT/third-party/cmsis-nn"
328+
echo " Fetching CMSIS-NN at ${CMSIS_NN_PIN:0:12} (pinned by the Cortex-M backend)"
329+
rm -rf "$CMSIS_NN"
330+
git init -q "$CMSIS_NN"
331+
git -C "$CMSIS_NN" remote add origin https://github.com/ARM-software/CMSIS-NN.git
332+
if ! git -C "$CMSIS_NN" fetch -q --depth 1 origin "$CMSIS_NN_PIN"; then
333+
echo "ERROR: could not fetch CMSIS-NN $CMSIS_NN_PIN."
334+
echo " Clone it yourself to third-party/cmsis-nn, or point the build at"
335+
echo " an existing copy."
336+
exit 1
337+
fi
338+
git -C "$CMSIS_NN" checkout -q FETCH_HEAD
339+
fi
340+
313341
if [ -n "$CMSIS_NN" ]; then
314342
mkdir -p "$OUT_DIR/src/cmsis-nn"
315343
cp -r "$CMSIS_NN/Source" "$OUT_DIR/src/cmsis-nn/"
@@ -331,7 +359,8 @@ if [ -n "$CMSIS_NN" ]; then
331359
echo "[5/7] CMSIS-NN copied from $CMSIS_NN"
332360
else
333361
CMSIS_NN_REV="absent"
334-
echo "[5/7] WARNING: CMSIS-NN not found. Cortex-M ops will not link."
362+
echo "ERROR: CMSIS-NN unavailable; the Cortex-M ops cannot link without it."
363+
exit 1
335364
fi
336365

337366
# CMSIS Core headers (for arm_math_types.h)

‎examples/arduino/verify_models.py‎

Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
#!/usr/bin/env python3
2+
# Copyright (c) Meta Platforms, Inc. and affiliates.
3+
# All rights reserved.
4+
#
5+
# This source code is licensed under the BSD-style license found in the
6+
# LICENSE file in the root directory of this source tree.
7+
8+
"""Check the example models against the library that will run them.
9+
10+
python verify_models.py arduino_lib/ExecuTorch
11+
12+
A .pte records, per operator call, how many values it puts on the stack. The
13+
generated kernel wrappers check that count and reject anything else. The two
14+
only agree when the model and the library came from the same ExecuTorch
15+
commit, because Cortex-M operator schemas change between releases -- `scratch`
16+
was added to the conv operators in #19636 and #19825.
17+
18+
A mismatch is invisible until far too late: the program loads, every operator
19+
resolves, and then Method::execute returns InvalidProgram (0x23) naming
20+
nothing useful. This compares the two directly, in about a second, with no
21+
board and no toolchain.
22+
"""
23+
24+
import argparse
25+
import pathlib
26+
import re
27+
import sys
28+
29+
from executorch.exir._serialize._program import deserialize_pte_binary
30+
31+
KERNEL_RE = re.compile(r'Kernel\(\s*"([^"]+)"(.*?)stack\.size\(\)\s*==\s*(\d+)', re.S)
32+
33+
34+
def library_expectations(library: pathlib.Path) -> dict[str, int]:
35+
"""Stack size each registered kernel wrapper demands, by operator name."""
36+
generated = list((library / "src/executorch/codegen").glob("Register*Kernels*.cpp"))
37+
if not generated:
38+
sys.exit(f"no kernel registration found under {library}/src/executorch/codegen")
39+
expectations: dict[str, int] = {}
40+
for source in generated:
41+
for match in KERNEL_RE.finditer(source.read_text(encoding="utf-8")):
42+
name, size = match.group(1), int(match.group(3))
43+
previous = expectations.get(name)
44+
if previous is not None and previous != size:
45+
sys.exit(
46+
f"{name} is registered twice with different stack sizes "
47+
f"({previous} and {size}); the library is inconsistent"
48+
)
49+
expectations[name] = size
50+
if not expectations:
51+
sys.exit(
52+
f"parsed no stack-size checks out of {len(generated)} generated "
53+
"file(s). The codegen output format has probably changed, so this "
54+
"script needs updating -- treat that as a bug here, not a bad model."
55+
)
56+
return expectations
57+
58+
59+
def model_calls(pte: pathlib.Path) -> list[tuple[str, int]]:
60+
"""Operator name and stack size for every kernel call in a .pte."""
61+
parsed = deserialize_pte_binary(pte.read_bytes())
62+
plan = getattr(parsed, "program", parsed).execution_plan[0]
63+
calls = []
64+
for chain in plan.chains:
65+
for instruction in chain.instructions:
66+
args = getattr(instruction.instr_args, "args", None)
67+
index = getattr(instruction.instr_args, "op_index", None)
68+
if args is None or index is None:
69+
continue # not a kernel call
70+
op = plan.operators[index]
71+
name = f"{op.name}.{op.overload}" if op.overload else op.name
72+
calls.append((name, len(args)))
73+
return calls
74+
75+
76+
def main() -> int:
77+
parser = argparse.ArgumentParser(description=__doc__)
78+
parser.add_argument("library", type=pathlib.Path, help="Generated library root")
79+
args = parser.parse_args()
80+
81+
expectations = library_expectations(args.library)
82+
models = sorted(args.library.glob("examples/*/model.pte"))
83+
if not models:
84+
# The build converts each model.pte to model.h and removes it, so run
85+
# this against the source tree rather than the packaged library.
86+
models = sorted(pathlib.Path(__file__).parent.glob("examples/*/model.pte"))
87+
if not models:
88+
sys.exit("no example models found")
89+
90+
failures = 0
91+
for pte in models:
92+
problems = []
93+
for name, provided in model_calls(pte):
94+
expected = expectations.get(name)
95+
if expected is None:
96+
problems.append(f"{name}: not registered in the library")
97+
elif expected != provided:
98+
problems.append(
99+
f"{name}: model supplies {provided}, library expects {expected}"
100+
)
101+
if problems:
102+
failures += len(problems)
103+
print(f"FAIL {pte.parent.name}")
104+
for problem in dict.fromkeys(problems):
105+
print(f" {problem}")
106+
else:
107+
print(f"ok {pte.parent.name}")
108+
109+
if failures:
110+
print(
111+
"\nThe models and the library came from different ExecuTorch commits.\n"
112+
"Re-export the models from the same checkout that built the library;\n"
113+
"see the pin in extras/PROVENANCE.txt."
114+
)
115+
return 1
116+
print(f"\n{len(models)} models match the library")
117+
return 0
118+
119+
120+
if __name__ == "__main__":
121+
sys.exit(main())

0 commit comments

Comments
 (0)