Vision-Language Models (VLMs) exhibit degraded OCR quality via Ollama vs. HuggingFace Transformers

[jimnoneill]

Related branch: all-ollama-attempt

Summary

When serving Qwen2.5-VL / Qwen3-VL models through Ollama versus HuggingFace Transformers, image OCR quality is significantly degraded. This is not a tokenization or output formatting issue—it's a fundamental architectural difference in how multimodal vision-language models are served between the two platforms.

Environment

• Ollama version: <!-- fill in your version -->
• Models tested:
  • qwen2.5-vl:7b (Ollama) — Note: Qwen2-VL-7B is not available on Ollama; only Qwen2.5-VL and newer
  • Qwen/Qwen2-VL-7B-Instruct (HuggingFace Transformers)
  • qwen3-vl:4b (Ollama)
  • llama3.1:8b-instruct (Ollama) — for JSON structuring comparison
• Hardware: Dual RTX 3090 (24GB VRAM each)

Observed Behavior

Platform	Model	OCR Quality	JSON Structuring
HuggingFace Transformers	Qwen2-VL-7B-Instruct	✅ High quality	✅ Compliant
Ollama	Qwen2.5-VL-7B / Qwen3-VL-4B	❌ Degraded	N/A
Ollama	Llama 3.1 8B Instruct	N/A (text-only)	✅ Compliant

This prevents full PyTorch removal from the Docker image for image-based poster extraction. The hybrid deployment is the current production approach:

• Ollama-only pipeline for PDFs → smaller Docker image, no PyTorch dependency
• HuggingFace Transformers pipeline for images → requires PyTorch, larger footprint

Potential Solutions

1. Accept hybrid deployment (current approach): PDFs use lightweight Ollama-only pipeline; images use full HF stack
2. Investigate alternative VLMs on Ollama: Test LLaVA 1.6, Moondream v2, or other vision models that may have better Ollama/GGUF support
3. Upstream investigation: File issue with Ollama/llama.cpp regarding VLM quantization fidelity for Qwen-VL family
4. Wait for improved GGUF VLM support: The llama.cpp multimodal support is actively developing

Related Links

• Qwen2-VL architecture paper
• llama.cpp multimodal discussion
• Ollama vision model docs

Acceptance Criteria

• Document the specific architectural components causing degradation
• Benchmark alternative Ollama-compatible VLMs (LLaVA 1.6, Moondream v2)
• Decision: accept hybrid deployment vs. pursue Ollama-native VLM solution

# Vision-Language Models (VLMs) exhibit degraded OCR quality via Ollama vs. HuggingFace Transformers

Labels: bug, investigation, ollama, vlm, architecture

Related branch: [all-ollama-attempt](https://github.com/fairdataihub/machine-actionable-posterextraction-beta/tree/all-ollama-attempt)

---

Summary

When serving Qwen2.5-VL / Qwen3-VL models through Ollama versus HuggingFace Transformers, image OCR quality is significantly degraded. This is not a tokenization or output formatting issue—it's a fundamental architectural difference in how multimodal vision-language models are served between the two platforms.

Environment

• Ollama version:
• Models tested:
  • qwen2.5-vl:7b (Ollama) — Note: Qwen2-VL-7B is not available on Ollama; only Qwen2.5-VL and newer
  • Qwen/Qwen2-VL-7B-Instruct (HuggingFace Transformers)
  • qwen3-vl:4b (Ollama)
  • llama3.1:8b-instruct (Ollama) — for JSON structuring comparison
• Hardware: Dual RTX 3090 (24GB VRAM each)

Observed Behavior

Platform	Model	OCR Quality	JSON Structuring
HuggingFace Transformers	Qwen2-VL-7B-Instruct	✅ High quality	✅ Compliant
Ollama	Qwen2.5-VL-7B / Qwen3-VL-4B	❌ Degraded	N/A
Ollama	Llama 3.1 8B Instruct	N/A (text-only)	✅ Compliant

Text-only models (Llama 3.1) perform equivalently across both platforms. The issue is isolated to vision-language multimodal models.

Root Cause Analysis

The Qwen2-VL architecture uses a ViT-based vision encoder coupled with the language model via cross-attention layers that fuse visual token embeddings into the LLM's attention mechanism. This multimodal fusion requires:

1. Vision encoder forward pass → patch embeddings
2. Projection layer mapping vision embeddings to LLM embedding space
3. Cross-attention or interleaved attention where visual tokens attend to/from text tokens

When Ollama converts/quantizes these models (via llama.cpp / GGUF format), the vision-language fusion architecture doesn't transfer with full fidelity. Specifically:

• Quantization artifacts in the projection layers degrade the vision→text embedding mapping
• Cross-attention weight quantization loses precision in the multimodal attention patterns
• Vision encoder handling may differ in how patch embeddings are processed and cached

HuggingFace Transformers runs the native PyTorch architecture with full FP16/BF16 precision on the vision components, preserving the intended multimodal fusion behavior.

Evidence

• Identical prompts, identical images
• HF Transformers output: accurate text extraction, correct spatial reading order
• Ollama output: missed text regions, incorrect reading order, hallucinated characters

Impact on posters.science

Our extraction pipeline uses a hybrid architecture:

Poster Type	OCR Method	JSON Structuring	PyTorch Required?
PDF posters	pdfalto text extraction	Llama 3.1 (Ollama)	❌ No
Image posters	Qwen2-VL (HuggingFace Transformers)	Llama 3.1 (Ollama)	✅ Yes

This prevents full PyTorch removal from the Docker image for image-based poster extraction. The hybrid deployment is the current production approach:

• Ollama-only pipeline for PDFs → smaller Docker image, no PyTorch dependency
• HuggingFace Transformers pipeline for images → requires PyTorch, larger footprint

Potential Solutions

1. Accept hybrid deployment (current approach): PDFs use lightweight Ollama-only pipeline; images use full HF stack
2. Investigate alternative VLMs on Ollama: Test LLaVA 1.6, Moondream v2, or other vision models that may have better Ollama/GGUF support
3. Upstream investigation: File issue with Ollama/llama.cpp regarding VLM quantization fidelity for Qwen-VL family
4. Wait for improved GGUF VLM support: The llama.cpp multimodal support is actively developing

Related Links

• [Qwen2-VL architecture paper](https://arxiv.org/abs/2409.12191)
• [llama.cpp multimodal discussion](https://github.com/ggerganov/llama.cpp/discussions)
• [Ollama vision model docs](https://ollama.com/blog/vision-models)

Acceptance Criteria

• Document the specific architectural components causing degradation
• Benchmark alternative Ollama-compatible VLMs (LLaVA 1.6, Moondream v2)
• Decision: accept hybrid deployment vs. pursue Ollama-native VLM solution

[fairdataihub-bot]

Hello! Thank you for opening this issue. Your input is valuable and helps improve the project. Can you please provide a detailed description of the problem you're encountering? Any additional information such as steps to reproduce the issue would be greatly appreciated. Thank you!