Skip to content

Commit 45249b4

Browse files
NafizNoyonNafizNoyon
authored
feat(inference): add ONNX video pipeline (#15)
Add config-driven ONNX Runtime video and camera inference with preprocessing, NMS, annotation, reporting, tests, and documentation.
1 parent 47a5879 commit 45249b4

7 files changed

Lines changed: 1586 additions & 1 deletion

File tree

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,3 +110,6 @@ data/metadata/balanced_ingestion_manifest.csv
110110
reports/dataset_validation_balanced/**
111111
!reports/dataset_validation_balanced/**/
112112
!reports/dataset_validation_balanced/**/.gitkeep
113+
114+
# Video inference outputs
115+
reports/video_inference/**

‎README.md‎

Lines changed: 28 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@ performance, and deployment-oriented engineering.
4242
- Reproducible ONNX export with graph and runtime validation
4343
- Controlled PyTorch-vs-ONNX Runtime raw-backend benchmarking
4444
- Deterministic severity-based low-light, blur, and noise evaluation
45+
- Config-driven ONNX Runtime video and camera inference
4546

4647
## Dataset Ingestion
4748

@@ -177,6 +178,31 @@ Severe Gaussian noise reduced mAP50 by 39.71% and mAP50-95 by 41.24% relative to
177178
See [Deterministic Robustness Evaluation](docs/robustness_evaluation.md)
178179
for corruption definitions, reproducibility, results, and limitations.
179180

181+
## ONNX Video Inference
182+
183+
Validate the model and execution plan without opening the source:
184+
185+
~~~powershell
186+
edge-traffic-video --config configs/video_inference.yaml --preflight-only
187+
~~~
188+
189+
Run video or camera inference:
190+
191+
~~~powershell
192+
edge-traffic-video --config configs/video_inference.yaml
193+
~~~
194+
195+
A controlled 12-frame annotated-video smoke test verified decoding,
196+
ONNX Runtime inference, NMS, rendering, MP4 writing, JSON reporting,
197+
and output decoding.
198+
The smoke test measured 7.59 ms mean ONNX inference latency and 52.71 FPS processing throughput.
199+
200+
These measurements describe one repeated-frame local smoke test and
201+
must not be treated as complete real-road application performance.
202+
203+
See [Config-Driven ONNX Video Inference](docs/video_inference.md)
204+
for commands, architecture, smoke-test evidence, and limitations.
205+
180206
## Benchmarking
181207

182208
The project includes a reproducible repeated-run inference benchmark.
@@ -209,11 +235,12 @@ Completed milestones:
209235
11. Reproducible ONNX export and artifact validation
210236
12. Controlled PyTorch-vs-ONNX Runtime CPU benchmarking
211237
13. Deterministic severity-based robustness evaluation
238+
14. Config-driven ONNX Runtime video inference
212239

213240
Current milestone:
214241

215242
```text
216-
robustness evaluation completed -> real-time video inference -> end-to-end optimization
243+
video inference completed -> end-to-end optimization -> Docker packaging
217244
```
218245

219246
## Planned Technology Stack

‎configs/video_inference.yaml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# Config-driven ONNX Runtime video inference
2+
3+
model:
4+
onnx_model: models/onnx/edge_traffic_yolo11n_e40.onnx
5+
manifest: models/onnx/edge_traffic_yolo11n_e40_manifest.json
6+
dataset_yaml: configs/bdd100k_balanced.yaml
7+
image_size: 320
8+
class_count: 10
9+
10+
source:
11+
value: '0'
12+
13+
runtime:
14+
provider: CPUExecutionProvider
15+
threads: 8
16+
confidence: 0.20
17+
iou: 0.60
18+
max_detections: 300
19+
max_frames: 0
20+
21+
output:
22+
save_video: true
23+
video_path: reports/video_inference/camera_annotated.mp4
24+
report_path: reports/metrics/video_inference/video_summary.json
25+
codec: mp4v
26+
display: false
27+
draw_labels: true
28+
draw_fps: true
29+
line_thickness: 2
30+
overwrite: false

‎docs/video_inference.md‎

Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
# Config-Driven ONNX Video Inference
2+
3+
## Purpose
4+
5+
This milestone adds sequential ONNX Runtime object detection for
6+
video files and camera sources.
7+
8+
The workflow includes:
9+
10+
- ONNX model and manifest validation
11+
- Camera-index and video-file source parsing
12+
- OpenCV frame decoding and encoding
13+
- Letterbox preprocessing
14+
- ONNX Runtime CPU inference
15+
- Confidence filtering and non-maximum suppression
16+
- Bounding-box, class-label, confidence, and FPS rendering
17+
- Atomic JSON report and annotated-video output
18+
- Configurable frame limits and output behavior
19+
20+
## Command-line interface
21+
22+
Run model-only preflight without opening a camera or video:
23+
24+
~~~powershell
25+
edge-traffic-video --config configs/video_inference.yaml --preflight-only
26+
~~~
27+
28+
Run using the configured source:
29+
30+
~~~powershell
31+
edge-traffic-video --config configs/video_inference.yaml
32+
~~~
33+
34+
Override the source and maximum processed frames:
35+
36+
~~~powershell
37+
edge-traffic-video --config configs/video_inference.yaml --source input.mp4 --max-frames 300
38+
~~~
39+
40+
Override generated artifact paths:
41+
42+
~~~powershell
43+
edge-traffic-video --config configs/video_inference.yaml --source input.mp4 --output annotated.mp4 --report summary.json
44+
~~~
45+
46+
## Default configuration
47+
48+
| Setting | Value |
49+
|---|---|
50+
| Backend | ONNX Runtime |
51+
| Provider | CPUExecutionProvider |
52+
| Threads | 8 |
53+
| Model input | 1 x 3 x 320 x 320 |
54+
| Classes | 10 |
55+
| Confidence threshold | 0.20 |
56+
| NMS IoU threshold | 0.60 |
57+
| Maximum detections | 300 |
58+
| Default source | Camera index 0 |
59+
| Codec | mp4v |
60+
| Display window | Disabled |
61+
| Save annotated video | Enabled |
62+
| Draw labels | Enabled |
63+
| Draw FPS | Enabled |
64+
| Overwrite existing outputs | Disabled |
65+
66+
The confidence threshold of 0.20 is the selected operating point
67+
from the controlled 40-epoch model evaluation.
68+
69+
## Processing pipeline
70+
71+
Each decoded frame follows this sequence:
72+
73+
1. Validate the decoded BGR frame.
74+
2. Apply 320 x 320 letterbox resizing.
75+
3. Convert BGR to RGB.
76+
4. Convert HWC pixels to contiguous NCHW float32.
77+
5. Normalize pixel values to the range 0 to 1.
78+
6. Run the ONNX graph using CPUExecutionProvider.
79+
7. Normalize raw detector output into channel-first format.
80+
8. Apply confidence filtering and non-maximum suppression.
81+
9. Rescale boxes to the original video-frame dimensions.
82+
10. Render boxes, labels, confidence scores, and processing FPS.
83+
11. Write the annotated frame when video output is enabled.
84+
85+
## Controlled smoke test
86+
87+
A deterministic 640 x 360 MP4 source was created from one real
88+
validation image. The source contained 24 repeated frames, and the
89+
CLI was limited to the first 12 frames.
90+
91+
| Measurement | Result |
92+
|---|---:|
93+
| Frames processed | 12 |
94+
| Total detections | 13 |
95+
| Detections per frame | 1.0833 |
96+
| Mean preprocessing latency | 0.9389 ms |
97+
| Mean ONNX inference latency | 7.5867 ms |
98+
| Mean postprocessing latency | 7.3955 ms |
99+
| Mean rendering latency | 1.8829 ms |
100+
| Mean end-to-end processing latency | 18.9705 ms |
101+
| Measured processing throughput | 52.71 FPS |
102+
| Annotated frames decoded after writing | 12 |
103+
| Annotated-video size | 29856 bytes |
104+
| Completion reason | maximum_frames |
105+
| Manual visual inspection | Passed |
106+
107+
The annotated output was manually inspected. Bounding boxes,
108+
class labels, confidence text, and the processing-FPS overlay
109+
were visibly present, and the video played correctly.
110+
111+
## Generated outputs
112+
113+
Default outputs:
114+
115+
~~~text
116+
reports/video_inference/camera_annotated.mp4
117+
reports/metrics/video_inference/video_summary.json
118+
~~~
119+
120+
Generated annotated videos and machine-specific JSON reports are
121+
excluded from Git.
122+
123+
The JSON report includes:
124+
125+
- Environment and package versions
126+
- ONNX model metadata and SHA-256
127+
- Active execution providers
128+
- Source dimensions and frame rate
129+
- Processing settings
130+
- Frames and detection counts
131+
- Per-class detection counts
132+
- Completion reason
133+
- Preprocessing latency
134+
- ONNX inference latency
135+
- Postprocessing latency
136+
- Rendering latency
137+
- End-to-end processing latency and throughput
138+
139+
## Safety and output behavior
140+
141+
- Preflight never opens the configured source.
142+
- Existing outputs are rejected unless overwrite is enabled.
143+
- Temporary video output is removed after an unsuccessful run.
144+
- Completed output is moved atomically into its final path.
145+
- Camera capture, video writer, and display windows are released.
146+
- A zero-frame source is treated as a runtime failure.
147+
148+
## Limitations
149+
150+
- The controlled smoke source repeated one validation image.
151+
- The measured 52.71 FPS is processing throughput for this small,
152+
controlled local test and is not a complete real-road benchmark.
153+
- Video decoding and encoding performance depends on codec support.
154+
- Webcam behavior was not exercised during the controlled smoke test.
155+
- Detection counts across repeated frames are not unique object counts.
156+
- The pipeline performs frame-independent detection and does not track
157+
object identities across time.
158+
- The pipeline does not measure temporal consistency.
159+
- The current model retains its documented baseline-accuracy and
160+
robustness limitations.
161+
- Display-window responsiveness can vary by operating system.
162+
- Audio tracks are not preserved.

‎pyproject.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ dependencies = [
4040
[project.scripts]
4141
edge-traffic-ingest = "edge_traffic_vision.data.ingestion_cli:main"
4242
edge-traffic-baseline = "edge_traffic_vision.inference.baseline:main"
43+
edge-traffic-video = "edge_traffic_vision.inference.video:main"
4344
edge-traffic-benchmark = "edge_traffic_vision.evaluation.benchmark:main"
4445
edge-traffic-dataset-check = "edge_traffic_vision.data.cli:main"
4546
edge-traffic-train = "edge_traffic_vision.training_cli:main"

0 commit comments

Comments
 (0)