Skip to content

API_Guide_With_Mask.md

Latest commit

 

History

History
79 lines (56 loc) · 2.62 KB

API_Guide_With_Mask.md

File metadata and controls

79 lines (56 loc) · 2.62 KB
Raw

LLaVA-Next FastAPI Guide

This API generates precise, localized, and contextually relevant captions that enhance situational awareness for BVIPs during outdoor navigation using LLaVA-Next (Mistral-7B) model.

The API processes:

  • An RGB image (scene)
  • A segmentation mask (grayscale)

The output is a caption describing sidewalks, crossings, signals, obstacles, and directional cues from the pedestrian’s perspective.

Project Structure

├── src/app.py                   # FastAPI application
├── llava-v1.6-mistral-7b-hf/    # base LLaVA-Next model
├── image_mask_model/            # PEFT/LoRA checkpoint folder
  • llava-v1.6-mistral-7b-hf/ → put the base model here (Download).
  • image_mask_model/ → put the fine-tuned PEFT checkpoint here (Download).
  • If these folders are missing, the app will raise an error at startup.

How to Use

First of all, install dependencies and check for the required folders and its contents.

  1. Install dependencies:

    pip install -r requirements.txt

  2. Check that model folders and its contents exist:

    • llava-v1.6-mistral-7b-hf/
    • image_mask_model/

  3. Running the API:

    Start the server with:

    uvicorn api.app:app --host 0.0.0.0 --port 8000 --reload &

Using the server endpoint

Parameters

  • input-api-key (required) — API key.
  • rgb (required) — RGB scene image (JPEG/PNG).
  • mask (required) — grayscale segmentation mask (same scene).
  • question (optional) — defaults to "What do you see in the image?".

Example — Using cURL

curl -X POST \
    "http://localhost:8000/generate-caption/" \
    -H "input-api-key: LLaVA_Next_API" \
    -F "rgb=@/path/to/scene.jpg" \
    -F "mask=@/path/to/mask.png" \
    -F "question=What do you see in the image?"

Example Response

"The sidewalk is centered in the view, bordered by a hedge on the left and a road on the right. A large yellow utility box is positioned on the right side of the sidewalk, partially obstructing the path."

⚠️ Notes & Tips

  • The corresponding mask input must be grayscale (NOT RGB segmentation mask).
  • GPU is strongly recommended. 4-bit quantization will not work well on CPU.
  • Every request must include an API key header:input_api_key: LLaVA_Next_API. (Default key is set in app.py → change API_KEY if you want your own.)