Skip to content

onnx_deployment.md

Latest commit

 

History

History
150 lines (110 loc) · 4.19 KB

onnx_deployment.md

File metadata and controls

150 lines (110 loc) · 4.19 KB
Raw

ONNX Export and CPU Backend Benchmark

Purpose

This milestone converts the selected 40-epoch PyTorch detection checkpoint into a reproducible ONNX artifact and compares raw CPU inference performance between PyTorch and ONNX Runtime.

The workflow validates model structure, runtime compatibility, numerical parity, latency, throughput, and artifact provenance.

Selected model

Property Value
Architecture YOLO11n detection model
Checkpoint cpu_balanced_baseline_e40/weights/best.pt
Classes 10
Image size 320
Selected confidence threshold 0.20
PyTorch checkpoint size 5.18 MB

Reproducible ONNX export

The export command is:

edge-traffic-export-onnx --config configs/onnx_export.yaml

The controlled export configuration uses:

Setting Value
Batch size 1
Dynamic shapes false
Graph simplification false
ONNX opset 17
Export device CPU

The CLI performs the export in a temporary directory, validates the ONNX graph, opens it with ONNX Runtime, calculates SHA-256 metadata, and atomically publishes the model and manifest.

Exported artifact

Property Result
ONNX size 10,446,120 bytes (9.96 MB)
SHA-256 ae7f8c3340f39c2c21d07b6be4505c14fc01cbc34e67c839bfc19f4d9c4bbe52
Runtime provider CPUExecutionProvider
Input images, FLOAT, [1, 3, 320, 320]
Output output0, FLOAT, [1, 14, 2100]
ONNX checker Passed

Generated ONNX models and manifests are excluded from Git and must be regenerated from the selected checkpoint.

Numerical parity method

PyTorch and ONNX Runtime receive the same letterboxed, normalized, float32 NCHW tensor. Their raw pre-NMS detector outputs are compared before confidence filtering and non-maximum suppression.

The comparison criteria are:

Criterion Maximum allowed Observed
Box absolute delta 0.1 0.00367737
Score absolute delta 0.001 0.00000131
Mean absolute delta 0.0001 0.00000878

The raw output comparison passed.

Final post-NMS detection counts are not used as the primary export correctness test because small floating-point differences near a confidence or NMS boundary can change the final selected boxes.

Controlled raw-backend benchmark

Run the benchmark with:

edge-traffic-backend-benchmark --config configs/backend_benchmark.yaml

Controlled conditions:

Setting Value
Device CPU
Threads 8
Warm-up runs 10 per backend
Timed runs 50 per backend
Input tensor Identical float32 [1, 3, 320, 320]
Timing source time.perf_counter_ns()

Benchmark results

Metric PyTorch CPU ONNX Runtime CPU
Mean latency 20.604 ms 6.211 ms
Median latency 20.372 ms 6.539 ms
Standard deviation 1.446 ms 1.144 ms
P95 latency 23.314 ms 7.552 ms
Throughput 48.53 FPS 161.01 FPS

The measured mean-latency speedup is 3.317x in favor of ONNX Runtime for this controlled raw CPU inference workload.

Engineering decision

ONNX Runtime is selected as the preferred CPU deployment backend for the current model. PyTorch remains the training and reference correctness backend.

This decision is based on:

  • successful ONNX structural and runtime validation
  • raw numerical parity with the PyTorch checkpoint
  • lower mean and P95 latency
  • higher controlled raw-inference throughput

Limitations

This is a single-image raw-backend micro-benchmark. It excludes:

  • camera capture and video decoding
  • disk-based image loading
  • preprocessing time
  • confidence filtering and non-maximum suppression
  • visualization and display rendering
  • output serialization and network transmission
  • complete application orchestration

Therefore, the reported FPS is not an end-to-end camera or video application frame rate.

A later milestone must benchmark the complete video pipeline under real deployment conditions.

Generated reports

reports/metrics/backend_comparison/pytorch_vs_onnx.json
reports/metrics/backend_comparison/pytorch_vs_onnx_runs.csv

Generated benchmark reports are excluded from Git because timing measurements depend on hardware and system conditions.