Skip to content

Commit ba1c378

Browse files
authored
Zephyr: document classifying your own image, and correct stale references (pytorch#23300)
Follow-up to pytorch#23283, which added `gen_input.py` to the mv2-ethosu sample. Adds a step to the Alif tutorial covering how to classify your own photograph, including the two conditions needed for the recorded float32 prediction to be a valid reference: export with `--model_name=mv2` rather than `mv2_untrained`, and supply representative `--calibration_data`, since the default calibrates on the model's random example input. Also corrects the description of the bundled input, which is a fixed tensor rather than a test image. In the Zephyr skill, two corrections found while running this end to end on hardware: - `west sdk install` was unpinned. Zephyr 4.4.0 declares SDK 1.0.1 in `SDK_VERSION`; earlier SDKs do not work. - The Alif board row listed `alif_e8_dk` with 4M MRAM + 2M SRAM. The board target is `ensemble_e8_dk/ae822fa0e5597ls0/rtss_hp`, and the device reports 5.5 MB MRAM and 8 MB SRAM. Verified against a stock Zephyr 4.4.0 workspace with ExecuTorch v1.5.1 and SDK 1.0.1, building for the Alif Ensemble E8 DevKit. Authored with Claude.
1 parent d2cffec commit ba1c378

4 files changed

Lines changed: 98 additions & 16 deletions

File tree

‎.claude/skills/zephyr/SKILL.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -92,7 +92,7 @@ cd ../../..
9292
### 4. Install Zephyr SDK
9393

9494
```bash
95-
west sdk install --gnu-toolchains arm-zephyr-eabi
95+
west sdk install --version 1.0.1 --gnu-toolchains arm-zephyr-eabi
9696
```
9797

9898
### 5. Install Ethos-U tools (if targeting NPU boards)
@@ -194,4 +194,4 @@ When `CONFIG_ET_ARM_MODEL_PTE_DMA_ACCESSIBLE=y` is set in the board conf (which
194194
|-------|-----|--------|--------|
195195
| mps3/corstone300/fvp | Ethos-U55 | 512K ITCM + 2M ISRAM | hello-executorch (full), MV2 (build-only) |
196196
| mps4/corstone320/fvp | Ethos-U85 | 4M ISRAM + 2M SRAM | hello-executorch (full), MV2 (full) |
197-
| alif_e8_dk | Ethos-U55-256 | 4M MRAM + 2M SRAM | MV2 (model runs from MRAM in-place) |
197+
| ensemble_e8_dk/ae822fa0e5597ls0/rtss_hp | Ethos-U55-256 | 5.5M MRAM + 8M SRAM | MV2 (model runs from MRAM in-place) |

‎docs/source/zephyr_alif_tutorial.md‎

Lines changed: 59 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,8 @@ works on the Arm Corstone-320 FVP for development without hardware.
99

1010
- A quantized INT8 MobileNetV2 model with Ethos-U NPU delegation
1111
- A Zephyr RTOS application that loads the `.pte` model, runs inference on a
12-
static test image, and prints the top-5 ImageNet predictions over UART
12+
fixed input tensor, and prints the top-5 ImageNet predictions over UART
13+
- Optionally, classification of your own photograph (Step 7)
1314

1415
## Prerequisites
1516

@@ -65,7 +66,7 @@ in its own source tree.
6566
Install the Zephyr SDK (compiler toolchain):
6667

6768
```bash
68-
west sdk install --gnu-toolchains arm-zephyr-eabi
69+
west sdk install --version 1.0.1 --gnu-toolchains arm-zephyr-eabi
6970
```
7071

7172
## Step 3: Install ExecuTorch and Arm Tools
@@ -420,6 +421,62 @@ The reported inference time covers one `method->execute()` call, measured with
420421
outside that interval. For hardware comparisons, record the software versions,
421422
model, core/NPU configuration and clocks, and measurements over repeated runs.
422423

424+
## Step 7: Classify Your Own Image
425+
426+
The sample ships with a synthetic input tensor, which exercises the runtime but
427+
produces meaningless predictions. To classify a photograph, regenerate the
428+
input header:
429+
430+
```bash
431+
python modules/lib/executorch/zephyr/samples/mv2-ethosu/gen_input.py your_photo.jpg
432+
```
433+
434+
The script applies the resize and centre-crop that torchvision's default
435+
MobileNetV2 weights expect, and stores the result as uint8 RGB in NCHW order,
436+
matching the ImageNet normalisation the sample applies on device. It also records what float32 torchvision MobileNetV2 predicts for
437+
that image, so the device output has a reference to compare against.
438+
439+
For that reference to mean anything, export with `--model_name=mv2` rather than
440+
`mv2_untrained`, and supply representative calibration data.
441+
442+
`--calibration_data` takes a directory of `.pt` files, each holding one
443+
preprocessed input tensor of shape `[1, 3, 224, 224]`. Produce them with the
444+
same transform the model expects, so the quantized ranges reflect real images:
445+
446+
```python
447+
import glob
448+
import os
449+
450+
import torch
451+
import torchvision
452+
from PIL import Image
453+
454+
images = sorted(glob.glob("my_photos/*.jpg"))
455+
os.makedirs("calib", exist_ok=True)
456+
457+
weights = torchvision.models.MobileNet_V2_Weights.DEFAULT
458+
preprocess = weights.transforms()
459+
for i, path in enumerate(images):
460+
x = preprocess(Image.open(path).convert("RGB")).unsqueeze(0)
461+
torch.save(x, f"calib/{i:04d}.pt")
462+
```
463+
464+
Then export against that directory:
465+
466+
```bash
467+
python -m executorch.backends.arm.scripts.aot_arm_compiler \
468+
--model_name=mv2 --quantize --delegate \
469+
--target=ethos-u55-256 --calibration_data=calib \
470+
--output=mv2_ethosu.pte
471+
```
472+
473+
A few hundred images spread across classes is a reasonable starting point.
474+
475+
Then rebuild and reflash as in Steps 5 and 6b.
476+
477+
ImageNet has no "person" class, so portraits return an unrelated label with low
478+
confidence. Photographs of animals, objects, food and vehicles work well.
479+
423480
## Troubleshooting
424481

425482
| Symptom | Cause | Fix |

‎zephyr/samples/mv2-ethosu/README.md‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,9 @@ produces are meaningless. To classify something real, regenerate the header:
1616
python gen_input.py your_photo.jpg
1717
```
1818

19-
The script resizes to 256, centre-crops to 224x224, stores the result as uint8
20-
RGB in NCHW order, and records what float32 torchvision MobileNetV2 predicts.
19+
The script applies the same resize and centre-crop that torchvision's default
20+
MobileNetV2 weights expect, stores the result as uint8 RGB in NCHW order, and
21+
records what float32 torchvision MobileNetV2 predicts.
2122
Rebuild and reflash after regenerating.
2223

2324
Two things are required for that recorded prediction to be a valid reference.
@@ -29,10 +30,14 @@ calibrates on the model's example input, which is random:
2930
```bash
3031
python -m executorch.backends.arm.scripts.aot_arm_compiler \
3132
--model_name=mv2 --quantize --delegate \
32-
--target=ethos-u55-256 --calibration_data=<dir-of-pt-tensors> \
33+
--target=ethos-u55-256 --calibration_data=calib \
3334
--output=mv2_ethosu.pte
3435
```
3536

37+
`--calibration_data` takes a directory of `.pt` files, each holding one
38+
preprocessed input tensor of shape `[1, 3, 224, 224]`, produced with the same
39+
transform the model expects (`MobileNet_V2_Weights.DEFAULT.transforms()`).
40+
3641
ImageNet has no "person" class, so portraits return an unrelated label with low
3742
confidence. Photographs of animals, objects, food and vehicles work well.
3843

‎zephyr/samples/mv2-ethosu/gen_input.py‎

Lines changed: 29 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -12,10 +12,11 @@
1212
1313
python gen_input.py cat.jpg
1414
15-
The image is resized to 256, centre-cropped to 224x224 and stored as uint8 RGB
16-
in NCHW order, matching the normalisation main.cpp applies on device. The
17-
float32 torchvision prediction is recorded in a comment so the device output
18-
has a reference to be compared against.
15+
The image is resized and centre-cropped using the preset that ships with
16+
torchvision's default MobileNetV2 weights, then stored as uint8 RGB in NCHW
17+
order. The same weights produce the float32 prediction recorded in a comment,
18+
so the device output has a reference to be compared against. Normalisation is
19+
applied on device by main.cpp.
1920
2021
Requires: pillow, torch, torchvision.
2122
"""
@@ -38,18 +39,37 @@ def main() -> None:
3839
parser.add_argument("-o", "--output", default=HEADER, help="header to write")
3940
args = parser.parse_args()
4041

41-
weights = torchvision.models.MobileNet_V2_Weights.IMAGENET1K_V1
42+
# Match examples/models/mobilenet_v2, which exports MobileNet_V2_Weights.DEFAULT.
43+
# Resize and crop are taken from the weights so this cannot drift if the
44+
# default checkpoint changes.
45+
weights = torchvision.models.MobileNet_V2_Weights.DEFAULT
4246
model = torchvision.models.mobilenet_v2(weights=weights).eval()
4347
categories = weights.meta["categories"]
4448

45-
crop = transforms.Compose([transforms.Resize(256), transforms.CenterCrop(224)])
49+
preset = weights.transforms()
50+
resize_size = preset.resize_size[0]
51+
crop_size = preset.crop_size[0]
52+
53+
crop = transforms.Compose(
54+
[transforms.Resize(resize_size), transforms.CenterCrop(crop_size)]
55+
)
4656
normalize = transforms.Compose(
4757
[
4858
transforms.ToTensor(),
49-
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
59+
transforms.Normalize(mean=preset.mean, std=preset.std),
5060
]
5161
)
5262

63+
# main.cpp applies these constants on device; bail out if the weights
64+
# ever move to a different normalisation.
65+
device_mean = [0.485, 0.456, 0.406]
66+
device_std = [0.229, 0.224, 0.225]
67+
if list(preset.mean) != device_mean or list(preset.std) != device_std:
68+
raise SystemExit(
69+
f"weights expect mean={list(preset.mean)} std={list(preset.std)}, but "
70+
f"main.cpp applies mean={device_mean} std={device_std}; update both."
71+
)
72+
5373
image = crop(Image.open(args.image).convert("RGB"))
5474
with torch.no_grad():
5575
logits = model(normalize(image).unsqueeze(0))[0]
@@ -71,9 +91,9 @@ def main() -> None:
7191
" * LICENSE file in the root directory of this source tree.",
7292
" *",
7393
f" * Generated by gen_input.py from {os.path.basename(args.image)}.",
74-
" * Resized 256, centre-cropped 224, uint8 RGB in NCHW order.",
94+
f" * Resized {resize_size}, centre-cropped {crop_size}, uint8 RGB in NCHW order.",
7595
" *",
76-
f" * float32 torchvision MobileNetV2 predicts class {top1} "
96+
f" * float32 torchvision MobileNetV2 ({weights.name}) predicts class {top1} "
7797
f"({categories[top1]}) at {probs[top1].item() * 100:.1f}%.",
7898
" */",
7999
"",

0 commit comments

Comments
 (0)