---

Llama 3.2 11B Vision Radiology Fine-Tuning

Fine-tune Llama 3.2 11B Vision Instruct on a radiology image-caption dataset using Unsloth, 4-bit quantization, and LoRA.

Overview

The notebook covers the following workflow:

1. Loads the Llama 3.2 11B Vision model in 4-bit mode.
2. Adds trainable LoRA adapters to vision and language components.
3. Loads the unsloth/Radiology_mini dataset.
4. Converts each image-caption sample into a multimodal chat conversation.
5. Runs inference before fine-tuning.
6. Fine-tunes the model with supervised fine-tuning (SFT).
7. Runs inference after fine-tuning.
8. Saves the trained LoRA adapter and tokenizer locally.

---

Notebook

Llama3_2_(11B)_Vision.ipynb

Requirements

• Python notebook environment (e.g., Google Colab)
• CUDA-compatible NVIDIA GPU
• Sufficient GPU memory for an 11B vision-language model in 4-bit mode
• Internet access for installing packages and downloading the model/dataset

Automatic Dependencies

The notebook installs the main dependencies automatically:

unsloth
unsloth_zoo
transformers==4.56.2
trl==0.22.2
datasets==4.3.0
bitsandbytes
accelerate
peft
xformers
sentencepiece
protobuf

---

Model Configuration

• Base model: unsloth/Llama-3.2-11B-Vision-Instruct

Loading Configuration

The model is loaded using 4-bit quantization to significantly reduce GPU memory usage:

model, tokenizer = FastVisionModel.from_pretrained(
    "unsloth/Llama-3.2-11B-Vision-Instruct",
    load_in_4bit=True,
    use_gradient_checkpointing="unsloth",
)

---

LoRA Configuration

LoRA adapters are applied to both the visual and language components:

model = FastVisionModel.get_peft_model(
    model,
    finetune_vision_layers=True,
    finetune_language_layers=True,
    finetune_attention_modules=True,
    finetune_mlp_modules=True,
    r=16,
    lora_alpha=16,
    lora_dropout=0,
    bias="none",
    random_state=3407,
    use_rslora=False,
)

---

Dataset

The notebook utilizes the unsloth/Radiology_mini dataset and loads the training split:

dataset = load_dataset("unsloth/Radiology_mini", split="train")

Each sample contains:

• image: A radiology image
• caption: The expected image description

Training Format

Each dataset sample is structured into a multimodal conversation format:

[
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "You are an expert radiographer. Describe accurately what you see in this image."
            },
            {
                "type": "image",
                "image": sample["image"]
            }
        ]
    },
    {
        "role": "assistant",
        "content": [
            {
                "type": "text",
                "text": sample["caption"]
            }
        ]
    }
]

---

Training Configuration

The notebook uses SFTTrainer with the following hyperparameters:

Parameter	Value
Per-device batch size	2
Gradient accumulation steps	4
Effective batch size	8
Training steps	30
Warmup steps	5
Learning rate	2e-4
Optimizer	adamw_8bit
Weight decay	0.001
Scheduler	Linear
Maximum sequence length	2048
Random seed	3407
Output directory	outputs

Training is initiated with:

trainer_stats = trainer.train()

---

Running the Notebook

1. Open Llama3_2_(11B)_Vision.ipynb in a GPU-enabled notebook environment.
2. Select a CUDA-compatible GPU runtime.
3. Run the installation cell.
4. Run the remaining cells in order.
5. Review the generated description before training.
6. Run the training cell.
7. Review the generated description after training.
8. Save or download the generated llama_lora directory.

---

Inference

Inference is performed using a radiology image and the instruction prompt:

"You are an expert radiographer. Describe accurately what you see in this image."

Generation Settings

model.generate(
    **inputs,
    streamer=text_streamer,
    max_new_tokens=128,
    use_cache=True,
    temperature=1.5,
    min_p=0.1,
)

Note: The same dataset sample is used for both the pre-training and post-training demonstrations.

---

Saved Output

The trained LoRA adapter and tokenizer are saved locally:

model.save_pretrained("llama_lora")
tokenizer.save_pretrained("llama_lora")

This saves the adapter weights rather than a fully merged standalone model in the directory llama_lora/.

Reloading the Adapter

To load the adapter in a new session:

from unsloth import FastVisionModel

model, tokenizer = FastVisionModel.from_pretrained(
    model_name="llama_lora",
    load_in_4bit=True,
)

FastVisionModel.for_inference(model)

(Note: Change the block condition from if False: or move the code outside that block when running in a new session.)

---

Monitoring

The notebook reports the following metrics:

• GPU name & Total GPU memory
• Reserved memory before training
• Training runtime
• Peak reserved memory
• Approximate memory used by LoRA training
• Percentage of total GPU memory used

---

Limitations

• Demonstration only: Training runs for only 30 steps.
• No validation: No validation/test splits or quantitative evaluation metrics are used.
• Data leakage: Inference demonstration uses an image from the training set.
• Non-medical grade: Generated output must not be treated as a medical diagnosis.
• Adapters only: Saves LoRA adapter weights, not a merged full model.

---

⚠️ Disclaimer

This project is for experimentation and research purposes only. It is not a medical device and must not be used as a substitute for professional evaluation by a qualified healthcare professional.