Skip to content

video_inference.md

Latest commit

 

History

History
173 lines (137 loc) · 5.37 KB

video_inference.md

File metadata and controls

173 lines (137 loc) · 5.37 KB
Raw

Config-Driven ONNX Video Inference

Purpose

This milestone adds sequential ONNX Runtime object detection for video files and camera sources.

The workflow includes:

  • ONNX model and manifest validation
  • Camera-index and video-file source parsing
  • OpenCV frame decoding and encoding
  • Letterbox preprocessing
  • ONNX Runtime CPU inference
  • Confidence filtering and non-maximum suppression
  • Bounding-box, class-label, confidence, and FPS rendering
  • Atomic JSON report and annotated-video output
  • Configurable frame limits and output behavior

Command-line interface

Run model-only preflight without opening a camera or video:

edge-traffic-video --config configs/video_inference.yaml --preflight-only

Run using the configured source:

edge-traffic-video --config configs/video_inference.yaml

Override the source and maximum processed frames:

edge-traffic-video --config configs/video_inference.yaml --source input.mp4 --max-frames 300

Override generated artifact paths:

edge-traffic-video --config configs/video_inference.yaml --source input.mp4 --output annotated.mp4 --report summary.json

Default configuration

Setting Value
Backend ONNX Runtime
Provider CPUExecutionProvider
Threads 8
Model input 1 x 3 x 320 x 320
Classes 10
Confidence threshold 0.20
NMS IoU threshold 0.60
Maximum detections 300
Warm-up frames 5
Default source Camera index 0
Codec mp4v
Display window Disabled
Save annotated video Enabled
Draw labels Enabled
Draw FPS Enabled
Overwrite existing outputs Disabled

The confidence threshold of 0.20 is the selected operating point from the controlled 40-epoch model evaluation.

Processing pipeline

Each decoded frame follows this sequence:

  1. Validate the decoded BGR frame.
  2. Apply 320 x 320 letterbox resizing.
  3. Convert BGR to RGB.
  4. Convert HWC pixels to contiguous NCHW float32.
  5. Normalize pixel values to the range 0 to 1.
  6. Run the ONNX graph using CPUExecutionProvider.
  7. Normalize raw detector output into channel-first format.
  8. Apply confidence filtering and non-maximum suppression.
  9. Rescale boxes to the original video-frame dimensions.
  10. Render boxes, labels, confidence scores, and processing FPS.
  11. Write the annotated frame when video output is enabled.

Controlled smoke test

A deterministic 640 x 360 MP4 source was created from one real validation image. The source contained 24 repeated frames, and the CLI was limited to the first 12 frames.

Measurement Result
Frames processed 12
Total detections 13
Detections per frame 1.0833
Mean preprocessing latency 0.9389 ms
Mean ONNX inference latency 7.5867 ms
Mean postprocessing latency 7.3955 ms
Mean rendering latency 1.8829 ms
Mean end-to-end processing latency 18.9705 ms
Measured processing throughput 52.71 FPS
Annotated frames decoded after writing 12
Annotated-video size 29856 bytes
Completion reason maximum_frames
Manual visual inspection Passed

The annotated output was manually inspected. Bounding boxes, class labels, confidence text, and the processing-FPS overlay were visibly present, and the video played correctly.

Generated outputs

Default outputs:

reports/video_inference/camera_annotated.mp4
reports/metrics/video_inference/video_summary.json

Generated annotated videos and machine-specific JSON reports are excluded from Git.

The JSON report includes:

  • Environment and package versions
  • ONNX model metadata and SHA-256
  • Active execution providers
  • Source dimensions and frame rate
  • Processing settings
  • Frames and detection counts
  • Per-class detection counts
  • Completion reason
  • Preprocessing latency
  • ONNX inference latency
  • Postprocessing latency
  • Rendering latency
  • End-to-end processing latency and throughput

Safety and output behavior

  • Preflight never opens the configured source.
  • Existing outputs are rejected unless overwrite is enabled.
  • Temporary video output is removed after an unsuccessful run.
  • Completed output is moved atomically into its final path.
  • Camera capture, video writer, and display windows are released.
  • A zero-frame source is treated as a runtime failure.

Optimization

The pipeline now excludes configurable warm-up frames, records decode and video-writing latency separately, reports complete source-to-output latency, and calculates running FPS using constant-time running totals.

See End-to-End Video Pipeline Optimization for the thread audit, validation, and remaining limitations.

Limitations

  • The controlled smoke source repeated one validation image.
  • The measured 52.71 FPS is processing throughput for this small, controlled local test and is not a complete real-road benchmark.
  • Video decoding and encoding performance depends on codec support.
  • Webcam behavior was not exercised during the controlled smoke test.
  • Detection counts across repeated frames are not unique object counts.
  • The pipeline performs frame-independent detection and does not track object identities across time.
  • The pipeline does not measure temporal consistency.
  • The current model retains its documented baseline-accuracy and robustness limitations.
  • Display-window responsiveness can vary by operating system.
  • Audio tracks are not preserved.