Vision Language Model

VLM submodule combines the abilities of vision and language models to handle both image and text on the **NXP i.MX9 applications processors.

Installation

cd vlm
./install.sh

Command to run the VLM and GUI.

# Run VLM
./launch.sh

It runs the chat_interface and the main vlm process. The first time you run the app it will take longer due to download of models.

-m, --model
Specifies the VLM to use. Available models are:
- smolvlm-256M
- smolvlm-500M
-im, --input_image
Path to the image to caption.

Default image delivery and industry in test/data

User can choose which part of the model is fp32 vs q8 by changing config.py

#Example
 ./launch.sh -m smolvlm-500M -im /path/to/your_image.png -p q8 -g

#Helper
 ./launch.sh --help

It is posible to run the code without the GUI interface.

python3 -m vlm

i.MX95	Precision	Vision Encoder	Decoder (TTFT)	Decoder
SmolVLM2-256M	FP32	6.66s	0.84s	0.13s - 0.16s
	INT8	3.31s	0.48s	0.08s - 0.09s
SmolVLM2-500M	FP32	6.76s	1.98s	0.21s - 0.25s
	INT8	3.34s	0.81s	0.12s - 0.19s

SmolVLM2-256 and SmolVLM2-500M share the same vision encoder so performance are the same.

i.MX95	Precision	Vision Encoder	Decoder (TTFT)	Decoder
SmolVLM2-500M	INT8	2.198s	0.58s	0.07s - 0.125s