Two tools built on one perception stack:
| What it does | Entry point | |
|---|---|---|
| 🕳️ Pothole Vision | Detects and segments potholes, scores road condition | app.py — Gradio UI |
| 🚗 Driving Assistant | Full scene perception + driving alerts from dashcam video | assistant.py — CLI |
Everything runs locally on CPU; the assistant also runs on CUDA.
Requirements: Python 3.12 · Windows, Linux or macOS · ~2 GB disk for model weights · an NVIDIA GPU is optional (CPU works, just slower).
# Windows
python -m venv .venv
.venv\Scripts\python -m pip install -r requirements.txt# Linux / macOS
python3.12 -m venv .venv
.venv/bin/python -m pip install -r requirements.txtWindows note: always call
.venv\Scripts\pythonexplicitly. A barepip/pythonresolves to the Microsoft Store Python, where installing torch fails on the 260-character path limit.
.venv\Scripts\python app.py # pothole UI
.venv\Scripts\python assistant.py --source dashcam.mp4 # driving assistantModels download themselves on first use (~20 MB for the pothole app, ~200 MB for the full assistant) and are cached afterwards.
| Tab | Input | Output |
|---|---|---|
| 📷 Image | road photo | segmentation overlay, per-pothole severity, condition score |
| 🎬 Video | dashcam / phone video | annotated H.264 video, unique-pothole count, severity timeline, JSON report |
| 📹 Live camera | webcam / phone cam | ~1 fps live overlay with a running unique-pothole count |
Model. The default is a fine-tuned YOLO11s-seg
(models/pothole-yolo11s-seg.pt, trained on Colab on a 720-image road-level
dataset — see training.ipynb). On the same 60-image validation split it reaches
mask mAP50 0.745 against 0.072 for the pre-trained
keremberke
models — those were trained on 80 close-up shots and remain selectable, since
they still do better on overhead photographs.
How the analysis works.
- Severity — mask area as a fraction of the frame:
< 0.5 %minor ·0.5–2 %moderate ·> 2 %severe. - Counting — a greedy IoU tracker gives each pothole a stable id across frames, so one hole counts once rather than once per frame (confirmed after ≥ 2 analysed frames).
- Condition score — a deliberate heuristic:
100 − (2·minor + 5·moderate + 12·severe), floored at 0. ≥ 90 Excellent · ≥ 75 Good · ≥ 55 Fair · ≥ 35 Poor · else Very poor.
Model, confidence, frame stride and dashcam mode are adjustable in the UI.
Four perception layers run per frame and feed a risk engine:
flowchart LR
V["🎬 dashcam video"] --> E
subgraph E["perception layers"]
direction TB
O["YOLO11s COCO<br/>vehicles · pedestrians<br/>lights · signs"]
P["fine-tuned YOLO11s-seg<br/>potholes"]
S["YOLOPv2<br/>drivable area · lanes"]
D["Depth Anything V2<br/>metric depth"]
end
E --> SS["SceneState<br/><i>typed snapshot</i>"]
SS --> R["RiskEngine<br/><i>policy only</i>"]
R --> A["⚠️ alerts + HUD<br/>annotated.mp4 · alerts.json"]
Alerts: VEHICLE_CLOSE · PEDESTRIAN_AHEAD · POTHOLE_AHEAD
The split matters: models do the perceiving, code only does policy. Distance comes from the depth model rather than a box-size heuristic, and "is it in my lane?" comes from YOLOPv2's learned drivable mask rather than a hand-drawn trapezoid. What stays hand-written are the tuning knobs — alert thresholds, cooldowns, severity bands — which no model can choose for you.
.venv\Scripts\python assistant.py --source path\to\dashcam.mp4Outputs land in runs/assistant/<video>/ as annotated.mp4 + alerts.json:
frames: 823 wall: 482.3s fps: 1.71
alerts: {'POTHOLE_AHEAD': 3, 'VEHICLE_CLOSE': 1}
artifacts: /content/out
alerts.json is a flat list, one entry per alert raised — t_end marks when
the condition cleared:
[
{
"type": "POTHOLE_AHEAD",
"severity": "critical",
"message": "pothole #10 moderate 9m",
"track_id": 10,
"distance_m": 9.3,
"t_start": 19.52,
"t_end": 19.62
}
]That run is the project's acceptance case: on a 27 s highway clip the assistant flagged the pothole the driver actually hit (19.5 s) and the truck that closed to 8 m — four alerts total, no spam.
| Flag | Effect |
|---|---|
--device cuda |
run on GPU (auto-detected by default) |
--disable scene |
skip the slowest layer; still alerts, using distance only |
--stride-objects/-potholes/-scene/-depth N |
run a layer every Nth frame (≥ 1) |
--max-frames N |
stop early — handy for a quick look |
--no-window |
headless; write artifacts without a preview window |
⚠️ --disable depthswitches off all alerting, not just distance readouts: every alert rule needsdistance_m, and only the depth layer produces it. The run warns on stdout and the HUD showsLAYERS DOWN: depth. Use it to inspect detections, never for a real alerting run.
Measured, not estimated — the two segmentation/depth transformers dominate:
| Machine | Configuration | Throughput | 27 s clip |
|---|---|---|---|
| CPU (i7-1355U) | default strides | ~0.35 fps | ~40 min |
| Colab T4 | every layer, every frame | ~1.7 fps | ~8 min |
This is a replay/analysis tool at these speeds, not yet a live assistant. Real-time is a stage-3 goal (ONNX/CUDA work, prioritising the scene and depth layers — together 73 % of the CPU budget).
For a GPU run, scripts/make_colab_notebook.py generates
assistant_colab.ipynb: a notebook that clones this repo and runs the
assistant on a Colab T4.
app.py Gradio UI (3 tabs)
assistant.py Driving-assistant CLI
pothole/ Pothole detection
├── detector.py model loading + YOLO-seg inference + drawing
├── tracker.py greedy IoU tracker (unique ids)
├── analysis.py severity thresholds + condition score
├── video.py video pipeline → annotated MP4 + report
└── report.py JSON / Markdown reports
perception/ Driving perception
├── types.py SceneState, TrackedObject, Alert, PolicyConfig
├── sources.py frame sources (video file; camera later)
├── objects.py COCO detector + tracking
├── potholes.py pothole layer adapter
├── scene.py YOLOPv2 drivable area + lanes
├── depth.py Depth Anything V2 metric depth
├── engine.py stride scheduler → SceneState
├── risk.py alert policy (pure, no perception)
└── hud.py overlay renderer
scripts/ Colab notebook generator, smoke test, diagnostics
tests/ 88 tests (pytest)
.venv\Scripts\python -m pytest # all 88
.venv\Scripts\python -m pytest -m "not slow" # skip model-loading testsTests marked slow load real model weights; they skip rather than fail
when weights or network are unavailable.
Pothole detection
- Can false-positive on dark treelines and walls above the horizon. Dashcam mode (on by default for video/live) drops any detection whose box reaches above the top 45 % of the frame. Turn it off for overhead close-ups.
- At highway speed a pothole is often only detected in the last ~0.5 s before impact; a smaller frame stride helps counting.
- Severity assumes a roughly forward-facing camera — steep angles skew the mask-area calculation.
Driving assistant
- The car's own hood is sometimes detected as a nearby vehicle. The risk
engine ignores candidates closer than
PolicyConfig.min_distance_m(default 2.0 m); a mount where the hood intrudes further may need a higher value. - Depth tends to over-read distance for large objects that fill or are cut off by the frame — the adjacent-lane truck in the sample clip reads ~12 m at roughly 3–5 m real.
- Alerts fire on distance thresholds, not time-to-collision. TTC (the FCW industry standard, ~2.0–2.4 s) is planned for stage 2.
- Lane lines are detected and drawn but no lane-departure alert exists yet.
- No GPS — reports are per-video, not map-located.
| Document | Contents |
|---|---|
training.ipynb |
Colab notebook that fine-tunes the pothole model on a T4 |
docs/superpowers/specs/2026-07-29-driving-perception-design.md |
Driving-assistant design, staging, measured budgets |
docs/superpowers/specs/2026-07-28-pothole-detection-design.md |
Original pothole-app design |
docs/superpowers/plans/2026-07-29-perception-stage1.md |
Stage-1 implementation plan |
Questions and bug reports go to GitHub Issues. This is a personal research project — expect best-effort responses rather than SLAs.
AGPL-3.0 — chosen to match Ultralytics YOLO, which this project builds on and whose license carries over to the fine-tuned weights shipped here. If you run a modified version of this code as a network service, the AGPL requires you to offer users its source.
| Component | Source | License |
|---|---|---|
| Object + pothole detection | Ultralytics YOLO | AGPL-3.0 |
| Drivable area + lanes | YOLOPv2 | MIT |
| Metric depth | Depth Anything V2 | Apache-2.0 |
| Pothole training data | farzadnekouei | CC BY 4.0 |