A real-time click-to-track webcam application using Siamese Region Proposal Network (SiamRPN) — a CVPR 2018 tracker that runs at ~160 FPS on GPU. Click on any object in the live feed and the tracker locks onto it immediately, no class labels needed.
Built as part of a drone tracking system where an operator watches a live video feed and clicks a target for the drone to follow.
SiamRPN tracking in action — template matched across frames in real time
Most people working on object tracking for the first time reach for OpenCV's built-in trackers — KCF, CSRT, MOSSE, MIL. Here's why SiamRPN is worth the extra setup:
| Tracker | OTB2015 AUC | Speed (GPU) | Speed (CPU) | Handles Occlusion | Class-agnostic |
|---|---|---|---|---|---|
| SiamRPN | 0.637 | ~160 FPS | ~30 FPS | ✅ Good | ✅ Yes |
| KCF | 0.477 | — | ~300 FPS | ❌ Poor | ✅ Yes |
| CSRT | 0.537 | — | ~25 FPS | ✅ Yes | |
| MOSSE | 0.338 | — | ~600 FPS | ❌ Poor | ✅ Yes |
| MIL | 0.357 | — | ~30 FPS | ❌ Poor | ✅ Yes |
| MedianFlow | 0.427 | — | ~40 FPS | ❌ Very poor | ✅ Yes |
KCF is fast on CPU but falls apart the moment the target is partially occluded or changes appearance (rotation, lighting shift). It tracks the correlation filter response, not the actual object — so when the filter drifts, it can't recover.
CSRT is OpenCV's best traditional tracker and handles scale changes reasonably well, but it's slow (~25 FPS CPU-only) and still fails on heavy occlusion.
SiamRPN uses a deep Siamese network to compare the target template against candidate regions in the current frame. Because it learned from millions of training pairs, it understands appearance variation — the same object under different lighting, partial occlusion, or at a different scale. It also has a Region Proposal Network (RPN) head that predicts a tight bounding box, not just a center point.
In short: if you care about real-world robustness, SiamRPN is the right baseline. KCF/CSRT are fine for controlled demos but break in production.
SiamRPN architecture — two-branch Siamese network with cross-correlation and RPN head
SiamRPN has two branches:
- Template branch — takes the 127×127 crop around the target (initialized at click time) and produces a feature embedding
- Search branch — takes a 255×255 crop centered on the last known position and produces a feature map
The two branches share weights (it's a Siamese network — same network, different inputs). The template embedding is cross-correlated with the search feature map to produce a response map. On top of this, an RPN head predicts classification scores and box regression offsets for multiple anchors at each position.
Anchor boxes at each location in the search region
The anchor with the highest classification score + its regression offset gives the predicted bounding box for the current frame.
Positive anchors (green) around the target
Template (what the tracker is looking for):
Detection output (bounding box predicted by RPN):
Bounding box regression in the search region:
Left: box in the 255×255 search crop. Right: box mapped back to the original frame.
Evaluated on OTB2015 and VOT2018 (numbers from the original CVPR 2018 paper and clean re-implementation):
| Dataset | Success (AUC) | Precision |
|---|---|---|
| OTB2013 | 0.641 | 0.855 |
| OTB2015 | 0.637 | 0.851 |
| UAV123 | 0.599 | 0.770 |
| DTB70 | 0.548 | 0.756 |
VOT2018: Accuracy 0.576, EAO competitive with trackers of the era.
git clone https://github.com/hassan-hfk/siamrpn-webcam-tracker.git
cd siamrpn-webcam-trackerpip install -r requirements.txtThe pretrained SiamRPN model (AlexNet backbone, trained on ILSVRC-VID + YouTube-BB):
Place the downloaded model.pth in the pretrained/ folder:
siamrpn-webcam-tracker/
└── pretrained/
└── model.pth
Open siam_rpn_tracker.py and update line 13:
self.tracker = SiameseRPNTracker(model_path='pretrained/model.pth')python siam_rpn_tracker.py| Action | What it does |
|---|---|
| Left click | Lock onto target at click point |
r |
Reset — drop current track, pick a new target |
q |
Quit |
When you click, the app grabs a 100×100 region around your click, initializes the tracker with it, and starts tracking. Two extra windows show the initial 100×100 template and what the tracker is currently looking at.
If you just want to test the UI without the pretrained weights, the app falls back to a DummyTracker automatically. It keeps the bounding box roughly in place — enough to verify click behavior and window layout.
siamrpn-webcam-tracker/
├── siam_rpn_tracker.py ← Webcam click-to-track app (main script)
├── code/ ← SiamRPN model and tracker (v2)
│ ├── net.py ← Siamese network definition
│ ├── siam_rpn_tracker.py ← Core tracker class
│ ├── run_SiamRPN.py ← OTB/VOT evaluation runner
│ └── ...
├── code_v1.0/ ← SiamRPN v1 (training pipeline)
│ ├── train_siamrpn.py ← Training script
│ └── ...
├── pretrained/ ← Put model.pth here (see Setup)
├── assets/ ← Architecture diagrams and demo GIFs
└── requirements.txt
If you want to train from scratch on your own dataset:
cd code_v1.0
python train_siamrpn.py \
--dataroot=/path/to/your/dataset \
--lr=0.001 \
--checkpoint_path=/path/to/save/weightsDataset format: VOT format — each sequence is a folder with img1.jpg, img2.jpg, ..., groundtruth.txt. If a sequence has a gap (no object visible), use 0,0,0,0 for that frame in the groundtruth file.
- Drone tracking — operator clicks a target on the GCS video feed; tracker outputs center coordinates fed to a PID controller
- Surveillance — pick a suspicious person and track them across the frame without needing a person detector
- Robotics — any task where you want to lock onto an arbitrary object without a predefined class label
- Generic click-to-track UI — as a building block for any vision pipeline that needs user-initiated tracking
Original paper: High Performance Visual Tracking with Siamese Region Proposal Network — Li et al., CVPR 2018
@InProceedings{Li_2018_CVPR,
author = {Li, Bo and Yan, Junjie and Wu, Wei and Zhu, Zheng and Hu, Xiaolin},
title = {High Performance Visual Tracking With Siamese Region Proposal Network},
booktitle = {CVPR},
year = {2018}
}MIT