Skip to content

Fine-tuning_Guide.md

Latest commit

 

History

History
94 lines (69 loc) · 4.18 KB

Fine-tuning_Guide.md

File metadata and controls

94 lines (69 loc) · 4.18 KB
Raw

LLaVA-NeXt Fine-tuning Guide

Overview

This pipeline facilitates the fine-tuning of LLaVA-v1.6-Mistral-7b to create a navigation assistant for Blind and Visually Impaired Persons (BVIP). By leveraging QLoRA (Quantized LoRA), the fine-tuning process is optimized for memory efficiency, enabling execution on consumer hardware.

  • Minimum Requirement: 12GB VRAM
  • Recommended: 16GB+ VRAM

Expected Output

Upon running this pipeline, the system will generate:

  1. LoRA Adapters: Lightweight fine-tuned weights saved in models/finetuned/. The current script will save three models for 3 epochs.
  2. Fine-tuning Logs: WandBlogs stored in logs/.

Project Structure

Ensure your directory contains the following before running:

.
├── main.py                      # Fine-tuning script
├── llava-v1.6-mistral-7b-hf/    # Base model directory (Local weights)
├── requirements.txt             # Python dependencies
├── src/                         # Custom modules
│   ├── data_collator.py
│   ├── trainer.py
│   ├── utils.py
├── data/                        # Dataset directory
│   ├── images/                  # RGB images
│   ├── masks/                   # Segmentation masks
│   ├── train.csv                # Training metadata
│   └── val.csv                  # Validation metadata
└── logs/                        # Output for WandB logs

Configuration & Parameters

You can customize the fine-tuning behavior by modifying specific variables in main.py.

1. Mask & Instructions Toggle

This parameter controls the input data format and system prompt.

Parameter Default Description
with_mask True True: The model uses both RGB images and Segmentation Masks.
False: The model fine-tunes on RGB images only.

Note: Changing this flag automatically updates the instructions variable to ensure the system prompt matches the input data (e.g., instructing the model to about the mask properties).

2. Fine-tuning Hyperparameters

These parameters in training_args = SFTConfig(...) control the optimization process.

Parameter Default Description
num_train_epochs 3 number of epochs.
learning_rate 2e-5 learning rate. Default set to LLaVA official recommendation. Lower values are generally more stable for fine-tuning.
per_device_train_batch_size 1 Batch size. Keep at 1 for 12GB VRAM; increase if you have more memory.
gradient_accumulation_steps 1 Number of steps to accumulate gradients before updating weights. Increase this to simulate larger batch sizes.
fp16 True Enables Mixed Precision fine-tuning (faster and uses less memory).

3. LoRA & Model Configuration

These parameters in lora_config and quantization_config control the model architecture and efficiency.

Parameter Default Description
r (Rank) 16 The dimension of the low-rank matrices. Higher values allow more expressivity but increase memory usage.
lora_alpha 32 Scaling factor for LoRA updates. Typically set to 2 * r.
lora_dropout 0.2 Dropout probability to prevent overfitting during adaptation.
load_in_4bit True Loads the base model in 4-bit precision to reduce VRAM usage.

How to Fine-tune LLaVa-NeXt

First of all, install dependencies and check for the required folders and its contents.

  1. Install dependencies:

    pip install -r requirements.txt

  2. Ensure the base model folder exists and contains the weights:

    • llava-v1.6-mistral-7b-hf/

  3. Simply start fine-tuning with:

    python main.py

Output

Upon successful execution, the script generates a new directory models/finetuned/ containing the fine-tuned LoRA adapter weights (checkpoints) and configuration files, along with a logs/ directory containing the training metrics.