SAM2 + OAK-D Real-time Segmentation

Real-time object segmentation and tracking using Meta's SAM 2.1 with a Luxonis OAK-D Lite depth camera, served through a FastAPI + WebSocket web interface.

---

Features

Mode	Description
Image	Live point-based segmentation — click on the camera feed to segment objects in real-time
Video	Record a clip → annotate the first frame → automatic multi-object tracking across all frames
Streaming	Click once → continuous real-time tracking on the live camera stream using SAM2 VideoPredictor

• Multi-object tracking with color-coded masks and contour overlays
• FP16 inference for optimal GPU performance
• Live FPS counter and VRAM usage monitoring
• Supports SAM 2.1 Tiny / Small / Base+ / Large model variants
• Includes FPS benchmark script for performance evaluation

Tested Environment

Note: This project was developed and tested on the following specific setup. Other configurations may work but are not guaranteed.

Component	Version
OS	Ubuntu 22.04.5 LTS (Jammy)
Kernel	6.8.0-90-generic
Python	3.10.12
GPU	NVIDIA GeForce RTX 4050 Laptop GPU
Camera	Luxonis OAK-D Lite
PyTorch	2.10.0
torchvision	0.25.0
DepthAI	3.4.0
NumPy	2.2.6
OpenCV	4.13.0.92
FastAPI	0.133.1

Prerequisites

• NVIDIA GPU with CUDA support
• Luxonis OAK-D Lite (or other OAK-D series camera)
• DepthAI SDK installed with USB rules configured

Installation

# Clone this repository (--recursive pulls the SAM 2 submodule automatically)
git clone --recursive https://github.com/warhammer50K/sam2-oakd-realtime.git
cd sam2-oakd-realtime

# Install SAM 2 in editable mode
cd sam2_repo && pip install -e . && cd ..

# Install dependencies
pip install -r requirements.txt

# Download the SAM 2.1 Tiny checkpoint
cd sam2_repo/checkpoints
bash download_ckpts.sh
cd ../..

Already cloned without --recursive? Run:

git submodule update --init --recursive

Usage

Web Application

# Start the server (default: http://localhost:8000)
uvicorn app:app --host 0.0.0.0 --port 8000

Open your browser and navigate to http://localhost:8000. Select a mode (Image / Video / Streaming) and interact by clicking on the video feed.

FPS Benchmark

# Benchmark with dummy images
python benchmark_fps.py --fp16

# Benchmark with OAK-D Lite camera
python benchmark_fps.py --fp16 --camera

Project Structure

.
├── app.py              # Main FastAPI web application (Image/Video/Streaming modes)
├── benchmark_fps.py    # FPS benchmark script for SAM2 inference
├── requirements.txt    # Python dependencies
└── sam2_repo/          # Meta SAM 2 (git submodule → facebookresearch/sam2)
    └── checkpoints/    # Model weights (downloaded separately)

How It Works

1. Image Mode — Each frame from the OAK-D camera is processed through SAM2's image predictor. Click points are used as prompts for instant segmentation.

2. Video Mode — Camera frames are recorded to disk, then SAM2's video predictor propagates annotations from the first frame across all recorded frames.

3. Streaming Mode — Extends SAM2's VideoPredictor for real-time use by dynamically injecting camera frames into the inference state and running per-frame tracking with memory management.

Acknowledgements

• SAM 2 by Meta AI (Apache 2.0 License)
• DepthAI by Luxonis

License

This project wraps Meta's SAM 2 (Apache 2.0). See sam2_repo/LICENSE for the original license terms.