Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 55 additions & 2 deletions .ci/scripts/test_espressif.sh
Original file line number Diff line number Diff line change
Expand Up @@ -34,13 +34,24 @@ source "${ESPRESSIF_TOOLS_DIR}/setup_path.sh"

python examples/espressif/export_smoke_model.py \
--output "${build_dir}/smoke.bpte" 2>&1 | tee "${build_dir}/export.log"
python examples/espressif/export_espnn_test_programs.py \
--output-dir "${build_dir}/espnn-tests" 2>&1 | tee "${build_dir}/espnn-export.log"

# One runtime serves the smoke test and the ESP-NN programs. quantized_ops_lib
# registers kernels outside the selected list, so the registry is sized
# explicitly instead of from the list. The quantized kernels' parallel_for
# runs inline without pthreadpool, which needs POSIX calls ESP-IDF lacks.
espnn_ops=$(tr -d '\n' < "${build_dir}/espnn-tests/operators.txt" \
| tr ',' '\n' | grep -v '^quantized_decomposed::' | paste -sd, -)

cmake --preset esp-baremetal -G Ninja -B "${build_dir}/runtime" \
-DCMAKE_TOOLCHAIN_FILE="${IDF_PATH}/tools/cmake/toolchain-esp32s3.cmake" \
-DCMAKE_BUILD_TYPE=Release \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=OFF \
-DEXECUTORCH_SELECT_OPS_LIST=aten::add.out,aten::mul.out \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_SELECT_OPS_LIST="aten::add.out,aten::mul.out,${espnn_ops}" \
-DMAX_KERNEL_NUM=256 \
-DEXECUTORCH_BUILD_PTHREADPOOL=OFF -DEXECUTORCH_BUILD_CPUINFO=OFF \
2>&1 | tee "${build_dir}/runtime-configure.log"
cmake --build "${build_dir}/runtime" \
2>&1 | tee "${build_dir}/runtime-build.log"
Expand All @@ -66,3 +77,45 @@ timeout 130s esp-emu --chip esp32s3 \
--net user --psram-size 8M --timeout 120s --exit-on 'Program complete.' \
2>&1 | tee "${build_dir}/emulator.log"
python examples/espressif/test_emulator.py "${build_dir}/emulator.log"

# ESP-NN backend: run every exported program through the ExecuTorch runtime
# and the ESP-NN delegate, and check it against the bundled host reference.
# The emulator does not implement the ESP32-S3 SIMD instructions, so ESP-NN is
# built with its ANSI C kernels.
printf 'CONFIG_NN_ANSI_C=y\n' > "${build_dir}/espnn-emulator.defaults"
# Runs a command with its output in a log, and shows the log's tail on failure.
logged() {
local log="${build_dir}/$1"
shift
if ! "$@" > "${log}" 2>&1; then
tail -n 60 "${log}"
echo "FAIL: see ${log}" >&2
return 1
fi
}
while read -r -u 3 name atol rtol; do
logged "espnn-${name}-configure.log" \
idf.py -C examples/espressif/project -B "${build_dir}/espnn-firmware" \
-DIDF_TARGET=esp32s3 \
-DSDKCONFIG="${build_dir}/espnn-sdkconfig" \
"-DSDKCONFIG_DEFAULTS=sdkconfig.defaults;${build_dir}/espnn-emulator.defaults" \
-DET_ESPNN_BACKEND=ON \
-DET_BUILD_DIR_PATH="${build_dir}/runtime" \
-DET_PTE_FILE_PATH="${build_dir}/espnn-tests/${name}.bpte" \
-DET_BUNDLE_IO=ON -DET_NUM_INFERENCES=1 \
-DET_ATOL="${atol}" -DET_RTOL="${rtol}" reconfigure
logged "espnn-${name}-build.log" cmake --build "${build_dir}/espnn-firmware"
logged "espnn-${name}-merge.log" \
idf.py -C examples/espressif/project -B "${build_dir}/espnn-firmware" merge-bin
logged "espnn-${name}-emulator.log" \
timeout 90s esp-emu --chip esp32s3 \
--firmware "${build_dir}/espnn-firmware/merged-binary.bin" \
--elf "${build_dir}/espnn-firmware/executorch_esp_runner.elf" \
--net user --psram-size 8M --timeout 60s --exit-on 'Program complete.'
echo -n "${name}: "
python examples/espressif/test_emulator.py "${build_dir}/espnn-${name}-emulator.log"
done 3< <(python -c '
import json, sys
for entry in json.load(open(sys.argv[1])):
print(entry["name"], entry["atol"], entry["rtol"])
' "${build_dir}/espnn-tests/manifest.json")
1 change: 1 addition & 0 deletions .github/workflows/test-espressif.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ on:
- release/*
pull_request:
paths:
- backends/espressif/**
- examples/espressif/**
- tools/cmake/preset/esp_baremetal.cmake
- CMakePresets.json
Expand Down
231 changes: 231 additions & 0 deletions backends/espressif/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,231 @@
# ESP-NN backend for Espressif chips

This backend is a **prototype under active development**.

This backend runs int8-quantized PyTorch models on Espressif microcontrollers.
It is tuned for the **ESP32-S3** and also runs on the original ESP32. It
delegates quantized subgraphs to [ESP-NN](https://github.com/espressif/esp-nn),
Espressif's kernel library. ESP-NN has SIMD kernels for the ESP32-S3 and
portable C kernels for the other chips.

A lowered model is a normal `.pte` program. `EspNNBackend` runs the int8
partitions with ESP-NN, and ExecuTorch's portable and quantized kernels run
everything else (float ops, unsupported layers, pre- and post-processing).
Each delegate payload is a compact int8 graph (XTS3) that the delegate's
interpreter executes. A host reference interpreter reproduces the device
arithmetic, so fully delegated programs are checked bit-exactly.

## Contents

```
backends/espressif/
espnn_preprocess.py EspNNBackend: BackendDetails.preprocess -> XTS3 graph
partition/ EspNNPartitioner: int8 regions -> delegate, rest -> CPU
quantizer/ EspNNQuantizer: PT2E int8 annotations ESP-NN can execute
lowering.py to_espnn_edge(): export -> edge -> lowered program + report
serialization/ XTS3 format writer and parser, op lowering rules
reference/ bit-exact host model of the ESP-NN kernels and the CPU fallback
espnn_runtime/ ESP-IDF component: XTS3 interpreter + ExecuTorch BackendInterface
test/ pytest suite (lowering, operators, CPU fallback, parser)
examples/espressif/
quantize_and_lower.py minimal quantize -> lower -> .pte example
export_espnn_test_programs.py the test models as BundleIO programs for the runner
camera_stream/ live camera classification on the ExecuTorch runtime
```

Programs run on the board with the existing
[`examples/espressif/project`](../../examples/espressif/project) runner, built
with `-DET_ESPNN_BACKEND=ON`.

## Supported operators

All delegated operators are int8 with per-tensor activations and per-channel
symmetric weights, as annotated by `EspNNQuantizer`.

| Group | Operators |
|---|---|
| Convolution | `conv2d` (any stride and padding, groups 1 or depthwise, dilation through phase decomposition), `conv_transpose2d`, `linear` |
| Activations | `relu`, `relu6`, `hardtanh`, plus table lookups for `hardswish`, `hardsigmoid`, `sigmoid`, `silu`, `tanh`, `gelu`, `elu`, `leaky_relu` |
| Elementwise | `add`, `mul` (with broadcasting), `softmax` (last dim) |
| Pooling | `max_pool2d`, `avg_pool2d`, `adaptive_avg_pool2d`, `mean` over H and W |
| Resize | `upsample_nearest2d`, `upsample_bilinear2d` (`align_corners` either way) |
| Data movement | `reshape`, `view`, `flatten`, `unflatten`, `permute`, `transpose`, `squeeze`, `unsqueeze`, `slice`, `select`, `expand`, `pad`, `cat`, `stack`, `split`, `chunk`, `unbind` |

Batch norm that follows a convolution is folded into it by PT2E
(`prepare_pt2e` / `prepare_qat_pt2e`); dropout in an exported eval-mode
model is a no-op.
Partitions can have several inputs and outputs. Anything else stays in the
program and runs on the CPU, including float ops, LSTM, layer norm and
attention matmuls. The partitioner reports what it delegated and what it left
behind. Pass `allow_cpu_fallback=False` to make leftover operators an error.

## Quantize and lower a model

[`examples/espressif/quantize_and_lower.py`](../../examples/espressif/quantize_and_lower.py)
contains the whole flow:

```python
import torch
import torchvision
from executorch.backends.espressif import EspNNQuantizer, to_espnn_edge
from torchao.quantization.pt2e.quantize_pt2e import convert_pt2e, prepare_pt2e

model = torchvision.models.mobilenet_v2(weights="DEFAULT").eval()
example_inputs = (torch.randn(1, 3, 224, 224),)

# 1. Capture the model and insert observers for the ESP-NN int8 scheme
exported = torch.export.export(model, example_inputs).module()
prepared = prepare_pt2e(exported, EspNNQuantizer())

# 2. Calibrate (use real samples from your dataset here)
with torch.no_grad():
for images in calibration_batches:
prepared(images)
quantized = convert_pt2e(prepared)

# 3. Lower: int8 regions -> EspNNBackend delegates, the rest -> CPU kernels
edge, report = to_espnn_edge(torch.export.export(quantized, example_inputs))
print(report["partitions"], report["delegated_ops"], report["cpu_ops"])

# 4. Serialize the ExecuTorch program
with open("mobilenet_v2_espnn.pte", "wb") as f:
f.write(edge.to_executorch().buffer)
```

```
$ python examples/espressif/quantize_and_lower.py
1 EspNN partition(s), 101 delegated operators, 0 operators left for the CPU
wrote mobilenet_v2_espnn.pte
```

Calibrate on a few hundred real samples; random data gives useless scales.

## Running on the ESP32-S3

You need ESP-IDF (CI uses v6.1) and an activated ESP-IDF shell.
[`examples/espressif/setup.sh`](../../examples/espressif/setup.sh) installs a
matching ESP-IDF and the emulator.

### 1. Cross-compile ExecuTorch

Build the runtime with the `esp-baremetal` preset. Select the portable CPU
operators your program needs. The quantized kernels provide the
quantize/dequantize operators at the delegate boundaries. The program lists
its operators:

```python
plan = edge.to_executorch().executorch_program.execution_plan[0]
print(",".join(sorted(
f"{op.name}.{op.overload}" for op in plan.operators
if not op.name.startswith("quantized_decomposed::")
)))
```

```bash
cmake --preset esp-baremetal -G Ninja -B cmake-out-esp \
-DCMAKE_TOOLCHAIN_FILE="${IDF_PATH}/tools/cmake/toolchain-esp32s3.cmake" \
-DCMAKE_BUILD_TYPE=Release \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_SELECT_OPS_LIST=<operators printed above> \
-DMAX_KERNEL_NUM=256 \
-DEXECUTORCH_BUILD_PTHREADPOOL=OFF -DEXECUTORCH_BUILD_CPUINFO=OFF
cmake --build cmake-out-esp --target install
```

- `quantized_ops_lib` registers kernels beyond the selected list, so
`MAX_KERNEL_NUM` is set explicitly instead of being sized from the
selection.
- pthreadpool needs POSIX calls that ESP-IDF doesn't provide. With it off,
the quantized kernels run single-threaded.
- The runner links `executorch_selected_kernels`, which only exists when a
selection is given. A fully delegated program with no CPU operators
still needs at least one selected operator, such as `aten::add.out`.

### 2. Build the runner with the ESP-NN backend

```bash
idf.py -C examples/espressif/project -B build-espnn \
-DIDF_TARGET=esp32s3 \
-DET_ESPNN_BACKEND=ON \
-DET_BUILD_DIR_PATH=$PWD/cmake-out-esp \
-DET_PTE_FILE_PATH=$PWD/mobilenet_v2_espnn.pte \
build flash monitor
```

`ET_ESPNN_BACKEND` adds
[`backends/espressif/espnn_runtime`](espnn_runtime) to the project. The
component pulls in `espressif/esp-nn` from the component registry, links
against the ExecuTorch targets that `executor_runner` imports, and registers
`EspNNBackend` at startup. It defines no platform layer; the runner's
`esp_pal.cpp` provides it.

For a BundleIO program (`.bpte`), add `-DET_BUNDLE_IO=ON` and the tolerances
`-DET_ATOL=... -DET_RTOL=...`. The runner then checks the outputs against the
bundled reference and logs `Test_result: PASS` or `FAIL`.

Other options, in menuconfig under *ExecuTorch ESP-NN backend*:

| Kconfig | Default | Effect |
|---|---|---|
| `ESPNN_DELEGATE_CONSTANTS_TO_PSRAM` | y | Copies weights from memory-mapped flash into PSRAM when they fit (faster than flash cache misses on large models) |
| `ESPNN_DELEGATE_DUAL_CORE` | n | Splits large convolutions across both cores |
| `ESPNN_DELEGATE_PROFILE` | n | Prints a per-operator time table after a delegate's first run |

### The original ESP32

The delegate also runs on the ESP32. ESP-NN has no SIMD kernels for it and
uses portable C, about 13 million int8 multiply-accumulates per second at
240 MHz, roughly an order of magnitude slower than the S3. Most ESP32 modules
have no PSRAM, so activations must fit in about 280 KB of internal RAM.

## Testing

```bash
pytest backends/espressif/test
```

The tests export small models for every operator group, lower them, run the
XTS3 graphs through the reference interpreter and compare the result against
the quantized PyTorch model. `test_xts3_parser_host.py` builds the device
parser for the host and feeds it corrupted graphs.

End to end, `.ci/scripts/test_espressif.sh` runs the same models through the
ExecuTorch runtime on an emulated ESP32-S3.
`examples/espressif/export_espnn_test_programs.py` exports each one as a
BundleIO program with the host reference's outputs. CI builds the runner with
`-DET_ESPNN_BACKEND=ON` for each program and runs it in `esp-emu`. Fully
delegated programs must match to float precision, which means bit-exact int8
results. Programs with CPU regions get a tolerance for the portable kernels'
float rounding. The emulator does not implement the S3 SIMD instructions, so
this build uses ESP-NN's ANSI C kernels (`CONFIG_NN_ANSI_C=y`).

The code follows the ExecuTorch coding style and passes the repository's
`lintrunner` checks:

```bash
lintrunner --paths-cmd "git ls-files backends/espressif examples/espressif"
```

## Limitations

- Only int8 is delegated. Float32 operators run on ExecuTorch's portable
kernels, which are slow on these chips.
- Activations are per-tensor and weights are per-channel symmetric. Dynamic
shapes are not supported, so each program has a fixed input shape.
- Attention matmuls, LSTM/GRU and normalization layers other than folded batch
norm fall back to the CPU.
- Dual-core execution only helps large convolutions, and it is off by default.
- The delegate handle comes from ExecuTorch's runtime allocator, but each
delegate allocates its activation arena, scratch buffers and optional PSRAM
weight copy with ESP-IDF's `heap_caps` allocator at `init()`, so it can
choose between internal RAM and PSRAM. Nothing is allocated during
`execute()`.

## License

Every source file in this backend and its examples is released under the
BSD 3-Clause license (`SPDX-License-Identifier: BSD-3-Clause`), the same
license as ExecuTorch. See the `LICENSE` file at the repository root. ESP-NN
is Apache-2.0 and is fetched from the Espressif component registry, not
vendored.
16 changes: 16 additions & 0 deletions backends/espressif/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Copyright 2026 Mwaniki Njagi.
# All rights reserved.
#
# This source code is licensed under the BSD-style license found in the
# LICENSE file in the root directory of this source tree.
#
# SPDX-License-Identifier: BSD-3-Clause

"""ESP-NN backend for Espressif ESP32 chips (ESP32-S3, ESP32, ESP32-P4, ...)."""

from .espnn_preprocess import EspNNBackend
from .lowering import to_espnn_edge
from .partition import EspNNPartitioner
from .quantizer import EspNNQuantizer

__all__ = ["EspNNBackend", "EspNNPartitioner", "EspNNQuantizer", "to_espnn_edge"]
61 changes: 61 additions & 0 deletions backends/espressif/espnn_preprocess.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# Copyright 2026 Mwaniki Njagi.
# All rights reserved.
#
# This source code is licensed under the BSD-style license found in the
# LICENSE file in the root directory of this source tree.
#
# SPDX-License-Identifier: BSD-3-Clause

from __future__ import annotations

import logging
from typing import List

from executorch.exir.backend.backend_details import BackendDetails, PreprocessResult
from executorch.exir.backend.compile_spec_schema import CompileSpec

from .serialization.xts3 import build_graph, serialize_graph

logger = logging.getLogger(__name__)

BACKEND_ID = "EspNNBackend"
INT8_FORMAT_SPEC = CompileSpec("espnn_format", b"int8")


class EspNNBackend(BackendDetails):
"""
Ahead-of-time half of the ESP-NN delegate.

Each partition produced by EspNNPartitioner is lowered to an XTS3 int8
graph: ESP-NN operator records, per-channel requantization parameters,
16-byte aligned int8/int32 constants and an input/output table. The
on-device EspNNBackend (backends/espressif/espnn_runtime) parses the blob
and executes it with ESP-NN kernels.

Compile specs: `espnn_format` selects the payload format; only `int8` is
defined, and any other value is rejected.
"""

@staticmethod
def preprocess(edge_program, compile_specs: List[CompileSpec]) -> PreprocessResult:
for spec in compile_specs:
if (
spec.key == INT8_FORMAT_SPEC.key
and spec.value != INT8_FORMAT_SPEC.value
):
raise ValueError(
f"Unsupported espnn_format {spec.value!r}; only b'int8' is defined"
)
graph = build_graph(edge_program)
blob = serialize_graph(graph)
logger.info(
"EspNN delegate: %d tensors, %d operations, %d constant bytes, %d byte blob",
len(graph.tensors),
len(graph.operations),
len(graph.constant_data),
len(blob),
)
return PreprocessResult(processed_bytes=blob, debug_handle_map={})


__all__ = ["BACKEND_ID", "EspNNBackend", "INT8_FORMAT_SPEC"]
Loading
Loading