Skip to content

README.md

Latest commit

 

History

History
369 lines (283 loc) · 11.8 KB

README.md

File metadata and controls

369 lines (283 loc) · 11.8 KB
Raw

cuda-local-vlm-video-captioning

A CUDA-first local video captioning workbench for MP4 files and open vision-language models, with first-class Jetson Orin support.

The MVP has two workflows:

  • A browser workspace: select a local MP4, press play, and watch frame captions appear as the local backend finishes each queued frame.
  • A CLI batch captioner: sample frames from one or more MP4 files and write structured JSONL for evaluation or downstream automation.

Example browser workspace showing local video playback with generated frame captions

Quickstart

Create a Python environment:

python3 -m venv .venv
. .venv/bin/activate
python -m pip install -U pip
python -m pip install -e '.[video]'

Run the environment check:

python -m cuda_local_vlm_video_captioning.env_check

Smoke-test the JSONL path without loading a model:

python scripts/make_smoke_video.py outputs/smoke.mp4
python -m cuda_local_vlm_video_captioning.cli \
  --video outputs/smoke.mp4 \
  --output outputs/stub-captions.jsonl \
  --backend stub \
  --sample-count 4

That command path only needs OpenCV. It verifies package installation, video decoding, frame sampling, and JSONL writing before any model weights are loaded.

CUDA Backend

transformers-cuda is the primary MVP backend. Install a CUDA-capable PyTorch build for your machine first, then install the Transformers extra:

python -m pip install -e '.[transformers]'
python -m cuda_local_vlm_video_captioning.env_check \
  --require-cuda \
  --require-transformers

On Jetson, generic PyPI PyTorch wheels are often not the right choice. For the Orin development system used for this MVP, JetPack 6.2 / CUDA 12.6 worked with:

python -m pip install \
  torch==2.8.0 \
  torchvision==0.23.0 \
  --index-url=https://pypi.jetson-ai-lab.io/jp6/cu126
python -m pip install -e '.[transformers]'

Run the CUDA CLI path:

python -m cuda_local_vlm_video_captioning.cli \
  --video outputs/smoke.mp4 \
  --output outputs/captions.jsonl \
  --backend transformers-cuda \
  --model-id Qwen/Qwen2-VL-2B-Instruct \
  --model-family auto \
  --sample-count 8 \
  --torch-dtype float16 \
  --max-new-tokens 768

First model use may download weights from Hugging Face unless they are already cached locally.

For a real validation image through the browser API, start the browser server and run:

python -m cuda_local_vlm_video_captioning.model_smoke \
  --server-url http://127.0.0.1:8765 \
  --browser-enabled-only \
  --output outputs/browser-ready-smoke.jsonl

By default, model_smoke downloads a small public image. Pass --image /path/to/image.jpg for offline or fully reproducible smoke tests.

Browser Workspace

Start the local server on the CUDA machine:

python -m cuda_local_vlm_video_captioning.web \
  --host 127.0.0.1 \
  --port 8765 \
  --backend transformers-cuda \
  --model-id Qwen/Qwen2-VL-2B-Instruct \
  --model-family auto \
  --torch-dtype float16 \
  --max-new-tokens 512 \
  --output outputs/browser-captions.jsonl

From another computer, forward the port over SSH:

ssh -L 8765:127.0.0.1:8765 user@CUDA_HOST

Open:

http://127.0.0.1:8765

Select an MP4 in the browser and press play. The browser captures decoded video frames in playback order and sends them to the local backend one at a time. Captions are appended to the transcript as they finish.

The direct server-side POST /api/caption path is disabled by default. Enable it only for trusted local videos by starting the server with --allowed-video-dir /path/to/videos; requests outside that directory are rejected.

The Performance drawer intentionally has only runtime safety controls:

  • VLM Model: chooses the local Transformers model for new frame captions. The server exposes its default model plus any candidates explicitly marked as browser-verified. Switching models opens a loading overlay, clears the previous Transformers runtime cache, and loads the selected model before new captions are queued.
  • Frame Stride: 1 attempts every decoded frame; N captures every Nth decoded frame.
  • Min Gap Sec: minimum playback-time spacing between queued captions. This prevents several decoded frames from producing visually duplicate timestamps.
  • Max Captions: caps the number of queued captions for a playback run.
  • Caption Detail: controls prompt verbosity for new frame captions. short asks for 1-2 concise sentences, normal asks for 2-4 sentences, and detailed asks for 4-6 sentences with more scene and behavior context.
  • Caption on play: disables automatic frame capture without changing server settings.

On Jetson-class hardware, caption generation can lag behind playback. That is expected for this MVP. For faster runs, increase frame stride, lower max captions, reduce caption detail, reduce --max-new-tokens, or use a larger CUDA GPU.

Scope Notes

  • Browser playback queues frame caption jobs; model output appears when each request completes. This is not token streaming or realtime video analytics.
  • max_visible_animals_hint and count_confidence are scene/context hints only. Use a detector/tracker if counts must be treated as source-of-truth data.
  • The repo is a standalone local workbench, not a private application or production integration.

Output Shape

Each CLI invocation writes one JSONL record per input video. Browser frame captioning appends one JSONL record per captured frame when --output is set.

See examples/example-output.jsonl for a compact example. The core fields are:

{
  "video_id": "example",
  "clip_summary": "A short scene summary.",
  "frame_observations": [],
  "behavior_labels": [],
  "interaction_hints": [],
  "max_visible_animals_hint": null,
  "count_confidence": "unknown",
  "uncertainty": [],
  "disagreements": [],
  "provenance": {}
}

Troubleshooting

No CUDA:

python -m cuda_local_vlm_video_captioning.env_check --require-cuda

If torch.cuda.is_available() is false, install a PyTorch build that matches your CUDA or JetPack version before installing the Transformers extra.

Model download or cache problems:

python -m cuda_local_vlm_video_captioning.env_check --require-transformers

Check network access, Hugging Face cache location, disk space, and model id. For private or gated checkpoints, set HF_TOKEN in the server environment.

Out of memory:

  • Use --torch-dtype float16.
  • Increase browser Frame Stride.
  • Increase browser Min Gap Sec.
  • Lower browser Max Captions.
  • Use browser Caption Detail: short.
  • Reduce --max-new-tokens.
  • Restart the server after a failed large run.

Video cannot be opened:

  • Verify the path exists for CLI usage.
  • Try a standard H.264 MP4.
  • Install OpenCV with python -m pip install -e '.[video]' if cv2 is missing.

Malformed model JSON:

The parser repairs a few common near-JSON responses, then stores non-JSON model text in clip_summary with an uncertainty note. Treat the structured fields as model-generated hints, not validated facts.

Development

Run tests:

python -m unittest discover -s tests

Run a syntax check:

python -m compileall -q src tests

Project layout:

  • src/cuda_local_vlm_video_captioning/cli.py: batch JSONL CLI.
  • src/cuda_local_vlm_video_captioning/web.py: dependency-free browser server.
  • src/cuda_local_vlm_video_captioning/backends/: backend adapter boundary.
  • src/cuda_local_vlm_video_captioning/sampler.py: OpenCV frame sampling.
  • src/cuda_local_vlm_video_captioning/schema.py: output normalization.

Backends

  • transformers-cuda: local PyTorch / Transformers baseline.
  • stub: development backend for validating video sampling and JSONL output without loading a model.

transformers-cuda supports a --model-family switch so model weights and model-specific inference APIs can vary independently:

  • auto: infer the family from --model-id.
  • chat, qwen-vl, smolvlm: chat-template image/text models. These use the full structured JSON prompt and can receive multiple sampled frames at once.
  • internvl: InternVL custom chat API. Current support captions one frame at a time and synthesizes the shared JSONL shape with behavior/count fields left untrusted.
  • moondream2: Moondream query/caption API. Caption-only one-frame loop.
  • florence2: Florence-2 task-prompt captioning. Caption-only one-frame loop.

The built-in candidate list is for benchmarking, not a guarantee that every model is browser-ready on every CUDA stack. The browser picker only exposes the server default plus candidates that have been verified locally. See docs/model-compatibility.md for the support matrix, commands used for validation, and current caveats.

Browser-verified in the main Transformers 5.7 runtime on the tested Orin stack:

  • Qwen/Qwen2-VL-2B-Instruct: current server default. Loads through the browser preload endpoint and returns structured single-frame captions.
  • HuggingFaceTB/SmolVLM2-2.2B-Instruct: loads through the browser preload endpoint and returns coherent single-frame captions; natural-language output is normalized into the shared JSONL shape when the model does not emit JSON.

Browser-verified through the isolated Transformers 4.49 worker overlay:

  • vikhyatk/moondream2: fails with malformed repeated text in the main Transformers 5.7 runtime, but returns coherent BF16 captions through the worker overlay.
  • OpenGVLab/InternVL2-4B: generation fails in the main Transformers 5.7 runtime because the checkpoint's remote language model does not expose generate(), but works through the worker overlay.
  • microsoft/Florence-2-large: loading fails in the main Transformers 5.7 runtime with a remote config attribute error, but works through the worker overlay.

Install the shared worker overlay without reinstalling CUDA PyTorch:

python -m pip install \
  --target /tmp/vlm-worker-overlays/transformers-4.49 \
  --no-deps \
  -r requirements/model-workers/transformers-4.49-overlay.txt

Enable those worker-backed candidates in the browser picker:

VLM_TRANSFORMERS_WORKER_PYTHONPATH=/tmp/vlm-worker-overlays/transformers-4.49 \
python -m cuda_local_vlm_video_captioning.web \
  --host 127.0.0.1 \
  --port 8765 \
  --backend transformers-cuda \
  --model-id Qwen/Qwen2-VL-2B-Instruct \
  --model-family auto \
  --torch-dtype float16 \
  --max-new-tokens 512 \
  --output outputs/browser-captions.jsonl

Worker-backed models run in a subprocess so their Transformers 4.x remote code does not downgrade or poison the main Qwen/SmolVLM runtime. First caption latency is high because each worker request loads the model in that subprocess.

List the built-in benchmark candidates:

python -m cuda_local_vlm_video_captioning.bench --list-candidates

Run one clip against one or more candidates:

python -m cuda_local_vlm_video_captioning.bench \
  --video /path/to/video.mp4 \
  --candidate qwen-vl=Qwen/Qwen2-VL-2B-Instruct \
  --candidate smolvlm=HuggingFaceTB/SmolVLM2-2.2B-Instruct \
  --candidate florence2=microsoft/Florence-2-large \
  --sample-count 2 \
  --max-new-tokens 384 \
  --output outputs/model-bench.jsonl

The benchmark JSONL records include status, latency, peak CUDA memory when available, errors, and the generated caption record.

Gate browser readiness through the same preload and frame-caption endpoints the browser uses:

python -m cuda_local_vlm_video_captioning.model_smoke \
  --browser-enabled-only \
  --output outputs/browser-ready-smoke.jsonl

See docs/model-compatibility.md for the browser readiness gate, current model status, and isolated worker environment pins for models that need a different Transformers runtime.

Additional service adapters can be added later once they are tested against a real local CUDA VLM server.

License

MIT