Skip to content

model-compatibility.md

Latest commit

 

History

History
147 lines (114 loc) · 5.75 KB

model-compatibility.md

File metadata and controls

147 lines (114 loc) · 5.75 KB
Raw

Model Compatibility

This project treats browser model selection as a verified runtime contract, not just a list of interesting Hugging Face repositories.

Browser-Ready Gate

A model is browser-ready only after it passes the same endpoints the browser uses:

  1. POST /api/preload-model succeeds.
  2. POST /api/frame-caption succeeds on a real image.
  3. The response contains coherent caption text.
  4. The response has at least one frame_observations entry.
  5. The server process does not crash.

Run the gate against the built-in candidates:

python -m cuda_local_vlm_video_captioning.model_smoke \
  --server-url http://127.0.0.1:8765 \
  --output outputs/model-smoke.jsonl

Run only models currently exposed in the browser picker:

python -m cuda_local_vlm_video_captioning.model_smoke \
  --browser-enabled-only \
  --output outputs/browser-ready-smoke.jsonl

Run one candidate:

python -m cuda_local_vlm_video_captioning.model_smoke \
  --candidate qwen-vl=Qwen/Qwen2-VL-2B-Instruct

The command returns non-zero if any candidate fails. By default, it downloads a small public image; pass --image /path/to/image.jpg for offline or fully reproducible smoke tests.

Verified On This Orin

Model Family Runtime Browser status Output mode Caveat
Qwen/Qwen2-VL-2B-Instruct qwen-vl main Transformers 5.7 Ready Structured JSON Current server default; browser endpoint smoke passed.
HuggingFaceTB/SmolVLM2-2.2B-Instruct smolvlm main Transformers 5.7 Ready Caption normalized into record Browser endpoint smoke passed; natural-language output may be wrapped.
vikhyatk/moondream2 moondream2 Transformers 4.49 worker overlay Ready with overlay Caption-only Main Transformers 5.7 output was malformed.
OpenGVLab/InternVL2-4B internvl Transformers 4.49 worker overlay Ready with overlay Caption-only Main Transformers 5.7 generation fails because remote code lacks generate().
microsoft/Florence-2-large florence2 Transformers 4.49 worker overlay Ready with overlay Caption-only Main Transformers 5.7 loading fails on remote config attributes.
OpenGVLab/InternVL2-2B internvl Not promoted Benchmark-only Unknown Needs the same endpoint gate before browser exposure.

Validation used python -m cuda_local_vlm_video_captioning.model_smoke against the browser server. Worker-overlay models passed individual endpoint smokes with --candidate moondream2=vikhyatk/moondream2, --candidate internvl=OpenGVLab/InternVL2-4B, and --candidate florence2=microsoft/Florence-2-large.

Isolated Worker Environments

Some model families ship custom Transformers remote code that is sensitive to Transformers version changes. Do not downgrade the main app just to make one family work. Use an isolated worker overlay or environment, validate it with the smoke gate, then promote the model only if it passes.

The shared lightweight overlay that passed on the tested Orin uses the main project virtual environment's CUDA PyTorch and overlays only the Hugging Face runtime packages:

python -m pip install \
  --target /tmp/vlm-worker-overlays/transformers-4.49 \
  --no-deps \
  -r requirements/model-workers/transformers-4.49-overlay.txt

Run the browser server with the overlay enabled:

VLM_TRANSFORMERS_WORKER_PYTHONPATH=/tmp/vlm-worker-overlays/transformers-4.49 \
python -m cuda_local_vlm_video_captioning.web \
  --host 127.0.0.1 \
  --port 8765 \
  --backend transformers-cuda \
  --model-id Qwen/Qwen2-VL-2B-Instruct \
  --model-family auto \
  --torch-dtype float16 \
  --max-new-tokens 512 \
  --output outputs/browser-captions.jsonl

By default, the overlay is used for internvl, moondream2, and florence2. Override that set with VLM_TRANSFORMERS_WORKER_FAMILIES, for example VLM_TRANSFORMERS_WORKER_FAMILIES=florence2.

The experimental requirement files are:

  • requirements/model-workers/transformers-4.49-overlay.txt
  • requirements/model-workers/internvl.txt
  • requirements/model-workers/florence2.txt
  • requirements/model-workers/moondream2.txt

If a full virtual environment is preferred, create it with the target machine's CUDA PyTorch first, then install the worker requirements. On the Orin used for this project:

python3 -m venv .venv-internvl
. .venv-internvl/bin/activate
python -m pip install -U pip
python -m pip install \
  torch==2.8.0 \
  torchvision==0.23.0 \
  --index-url=https://pypi.jetson-ai-lab.io/jp6/cu126
python -m pip install -r requirements/model-workers/internvl.txt
python -m pip install -e '.[video]'

Repeat with the Florence or Moondream requirement file as needed.

Current Failure Notes

Moondream2

The model card recommends revision="2025-06-21" and device_map={"": "cuda"}. The main app follows that path for Moondream2. On the tested Orin stack, the main Transformers 5.7 runtime loads Moondream2 but produces malformed repeated text, even with BF16. The Transformers 4.49 worker overlay returns coherent single-frame captions with BF16.

InternVL2

InternVL2-4B preloads in the main app after installing tokenizer and model-code dependencies, but generation fails under Transformers 5.7 because the remote Phi-3 language model does not expose generate(). The Transformers 4.49 worker overlay returns coherent single-frame captions for OpenGVLab/InternVL2-4B. OpenGVLab/InternVL2-2B has not been promoted yet.

Florence-2

Florence-2-large fails during remote config/processor loading under Transformers 5.7 with a Florence2LanguageConfig missing forced_bos_token_id. The Transformers 4.49 worker overlay loads the model with AutoModelForCausalLM, AutoProcessor, trust_remote_code=True, and float16, then returns coherent single-frame captions.