Drone Vision Tuner

YOLO11 + QLoRA fine-tuning for drone vision: object detection, classification, and obstacle avoidance.

The Stack

Layer	Choice	Why
Model	YOLO11 (Ultralytics)	Industry standard for drones/robotics, real-time inference
Fine-tune	QLoRA (PEFT)	Lightweight, memory-efficient, shows modern ML skills
Dataset	VisDrone benchmark	Drone-specific dataset, object detection + classification
Cloud GPU	Modal (A40/A100)	Best price-to-setup ratio, Python-first, ships to your laptop
Tracking	WandB	Portfolio-ready experiment tracking
Export	ONNX/TFLite	Shows deployment thinking

This combo hits every signal a recruiter or reviewer looks for: real-world domain (drone autonomy), modern techniques (LoRA, PEFT), cloud-native training, and deployment-aware output.

Deep Dive: Why Each Layer

YOLO11 — The Industry Standard

You could pick a dozen object detection architectures. YOLO11 wins for drones because:

Real-time inference: 30+ FPS on edge hardware (Jetson Nano, drone flight controllers). Obstacle avoidance needs sub-100ms latency, period.
Single-shot: One forward pass, no region proposals. Faster and simpler than two-stage detectors (Faster R-CNN).
Ultralytics ecosystem: Training, validation, export, deployment all in one library. The industry standard toolchain.
Portfolio visibility: Recruiters know YOLO. It's the ResNet of the modern era.

YOLO11 variants (n/an/s/m/l/x) trade speed for accuracy. Start with yolo11n (nano) for fast iteration, scale up for final runs.

QLoRA — Parameter-Efficient Fine-Tuning

Full fine-tuning a YOLO model requires 24GB+ VRAM. That's A100 territory and expensive. QLoRA changes the math:

What QLoRA actually does:

Quantize the base model to 4-bit precision — reduces memory footprint by 4x with minimal accuracy loss
Freeze the quantized weights — you are not updating them
Attach small LoRA adapters — only these trainable parameters get updated
Result: A 3B parameter model fine-tuned in 16GB VRAM, trained 3x faster, with comparable accuracy to full fine-tune

Why this signals "modern ML engineer" to reviewers:

LoRA/PEFT is the dominant fine-tuning technique since 2023 (OpenAI, Meta, Google all use it)
It shows you understand memory-efficient training — critical for any production ML work
QLoRA specifically (quantized LoRA) is the cutting-edge variant

LoRA rank (r): Start at r=8. Higher rank = more expressive adapters, more VRAM. r=8 is enough for most domain adaptation tasks.

VisDrone Benchmark — Real Data, Real Domain

VisDrone2021 is the de facto benchmark for drone-mounted camera detection:

8,000+ images captured from drone platforms at varying altitudes, angles, and lighting conditions
10 object categories: pedestrian, person, bicycle, car, van, truck, bus, motor, knife, other
Aerial perspective: Objects appear small, occluded, at unusual angles — exactly what your drone will see
Challenge-aware: Dense scenes, small objects,恶劣天气 conditions are all represented

Training on generic datasets (COCO, Pascal VOC) gives you a general detector. VisDrone gives you a drone detector. The domain specificity is what makes the portfolio piece compelling.

Labeling your own drone data later: Roboflow Annotate + their API makes it trivial to add custom footage. VisDrone is your proof-of-concept; your own data is your final product.

Modal — Cloud GPU Without the Ops Headaches

Lambda Labs, Vast.ai, RunPod — they're all viable. Modal wins for this use case because:

Python-native: Define your GPU infrastructure in a Python file, not YAML or a web UI. modal run feels natural.
No idle cost: Modal provisions an A100 when your script starts, tears it down when it finishes. You're not paying for hours the GPU sits idle.
Free tier: 2 hours/month of A100 time — enough for several full training runs during development.
Spot instance handling: Modal auto-retries on interruption. Your 12-hour training won't vanish mid-run.

The portfolio angle: A recruiter reading your README sees modal run scripts/train_modal.py and immediately knows: this person has shipped ML to cloud infrastructure. That's a senior signal.

On your MacBook M3: Fine for prototyping, inference, and validation. Actual training with QLoRA needs CUDA or Modal. M3 MPS backend works for small models but YOLO11 + LoRA is more comfortable on an A100.

Weights & Biases — Experiment Tracking That Sells

Training without experiment tracking is like coding without version control. WandB gives you:

Hyperparameter logs: Every run with its LR, batch size, augmentation config
Metrics in real-time: mAP, precision, recall, loss curves during training
Artifact management: Store model weights, dataset versions, export files
Portfolio sharing: Public WandB reports show experiments in a clean UI

A clean WandB dashboard in a portfolio repo tells reviewers: this person runs structured experiments, not ad-hoc training scripts.

ONNX/TFLite Export — Deployment Thinking

Training is 10% of the ML lifecycle. Export shows you know this:

ONNX: Framework-agnostic inference graph. Deploy with ONNX Runtime on any platform.
TFLite: TensorFlow Lite for edge deployment (Raspberry Pi, mobile, drone flight controller).
ONNX Runtime: 2-3x faster inference than PyTorch vanilla in many cases.

The export step is what separates "I trained a model" from "I built a deployable system."

Quick Start

1. Clone & install

cd ~/project\ files/drone-vision-tuner
pip install -r requirements.txt

2. Get Roboflow API key

Free tier at roboflow.com. VisDrone is a public dataset.

export ROBOFLOW_API_KEY="your_key_here"

3. Download dataset

python scripts/download_dataset.py

4. Local training (MacBook M3)

# Fine-tune without LoRA on MPS (Apple Silicon)
python scripts/train_qlora.py

# Note: Full LoRA training requires CUDA or Modal cloud GPU

5. Cloud training (Modal A100)

# Install Modal CLI
pip install modal
modal setup

# Launch training on A100
modal run scripts/train_modal.py --epochs 50 --batch_size 16

6. Export for deployment

# After training, export to ONNX
from ultralytics import YOLO
model = YOLO("drone-vision-tuner/yolo11-visdrone-qlora-r8/weights/best.pt")
model.export(format="onnx", imgsz=640)

Project Structure

drone-vision-tuner/
├── configs/
│   └── train_config.yaml       # All hyperparameters
├── data/
│   └── VisDrone/               # Dataset (downloaded via Roboflow)
├── models/
│   └── (trained weights)
├── notebooks/
│   └── eda.ipynb              # Dataset exploration
├── scripts/
│   ├── train_qlora.py         # Local/MPS training script
│   ├── train_modal.py         # Modal cloud training script
│   └── download_dataset.py    # VisDrone downloader
├── requirements.txt
└── README.md

Key Decisions

Why YOLO11?

Real-time object detection at 30+ FPS on edge hardware (Jetson, drone flight controller). Drone obstacle avoidance needs sub-100ms inference. YOLO11 is the industry choice for this.

Why VisDrone?

Pre-labeled, drone-specific. Aerial perspective, small objects (people, cars from above), challenging lighting. It's the benchmark that matters for this domain.

Why QLoRA instead of full fine-tune?

3B parameter model fine-tuned with LoRA fits in 16GB VRAM
Full fine-tune of YOLO11n needs 24GB+ VRAM
QLoRA = 4-bit quantization + LoRA = fast + memory-efficient
Recruiter signal: you know modern PEFT techniques

Why Modal over Lambda Labs?

Python-native API, no YAML configs or web UI needed
Auto-scales, handles spot interruptions
Free tier: 2 hours/month A100
Portfolio-visible: modal run command is impressive in a README

Next Steps After Training

Add your own data: Record drone footage, label with Roboflow, fine-tune on your specific use case
Deploy to edge: ONNX Runtime on Jetson Nano or Raspberry Pi
Obstacle avoidance: Add depth estimation head (depth-anything-v2) + YOLO detection = full stack
Multi-modal: Combine RGB detection with thermal or LiDAR for nighttime avoidance

FilesExpand file tree

README.md

Latest commit

History