LLM Gateway

A production-grade API gateway for routing requests to multiple LLM providers — because putting all your eggs in one AI basket is a terrible idea.

LLM Gateway sits between your application and the chaos of LLM APIs. It handles provider switching, failure recovery, health checks, and real-time streaming — all without your users ever knowing something went sideways.

---

Key Features

• Multi-Provider Support — Unified interface for Groq and OpenAI. Switch providers with a config change, not a refactor.
• Resilient Fallback Routing — Primary provider down? The gateway automatically reroutes to the backup. No interruptions, no drama.
• Pre-flight Health Checks — Validates provider API health before attempting heavy streaming requests. No more sitting through a 30-second timeout just to learn the API was dead from the start.
• Real-Time Streaming — Streams tokens as they arrive using FastAPI's StreamingResponse and Python async generators. Fast, reactive, and your users will love it.
• Background Observability — Logs request metadata, latency, and provider usage asynchronously via FastAPI BackgroundTasks. The response ships first; logging happens quietly in the background.
• Clean Architecture — Decoupled service layer built on Abstract Base Classes (ABC) and FastAPI Dependency Injection. Swap providers without touching your routes.

---

Architecture

The gateway is organized into three clear layers:

Entry Point (main.py) FastAPI routes handle incoming HTTP requests, validate payloads via Pydantic, and wire up services through dependency injection.

Services (services/) An abstract LLMProvider base class is implemented separately by GroqProvider and OpenAIProvider. A FallbackProvider wraps both — it runs health checks and handles retry logic transparently so the route layer stays clean.

Core (core/) Manages configuration (loading secrets from .env) and background JSONL logging. Infrastructure concerns live here, away from business logic.

Models (models/) Pydantic schemas enforce strict request validation before anything touches the LLM APIs.

---

Tech Stack

Layer	Technology
Framework	FastAPI (Python)
Validation	Pydantic
Concurrency	asyncio, Async Generators
LLM SDKs	groq, openai

---

Project Structure

llm-gateway/
├── .env                        # API keys — keep this out of git
├── .gitignore
├── main.py                     # FastAPI app entry point
├── requirements.txt
├── core/
│   ├── __init__.py
│   ├── config.py               # Loads environment variables
│   └── logging.py              # Background JSONL logging
├── models/
│   ├── __init__.py
│   └── request.py              # Pydantic schemas
└── services/
    ├── __init__.py
    ├── base.py                 # Abstract LLMProvider interface
    ├── fallback_provider.py    # Fallback routing & health checks
    ├── groq_client.py          # Groq implementation
    └── openai_client.py        # OpenAI implementation

---

How to Run

1. Clone the repository

git clone https://github.com/your-username/llm-gateway.git
cd llm-gateway

2. Create and activate a virtual environment

python -m venv venv
source venv/bin/activate        # Windows: venv\Scripts\activate

3. Install dependencies

pip install -r requirements.txt

4. Configure environment variables

Create a .env file in the root directory:

GROQ_API_KEY=your_groq_api_key_here
OPENAI_API_KEY=your_openai_api_key_here
PRIMARY_PROVIDER=groq             # or openai

5. Start the server

uvicorn main:app --reload

The gateway will be live at http://localhost:8000. The --reload flag is your best friend during development.

---

🔌 API Endpoints

POST /chat

Send a chat request. The gateway handles provider selection, health checks, and streaming automatically.

curl -X POST http://localhost:8000/chat \
  -H "Content-Type: application/json" \
  -d '{"message": "Explain async generators in Python in one paragraph."}'

Response is streamed token-by-token.

---

GET /health

Check the health status of all configured providers. Useful for monitoring and debugging before you start yelling at the logs.

curl http://localhost:8000/health

{
  "groq": "healthy",
  "openai": "healthy"
}

---

Testing Fallback Logic

Want to see the fallback in action without waiting for an actual outage? Easy.

Option 1: Set an invalid API key

In your .env, replace your primary provider's key with a garbage value:

GROQ_API_KEY=this_is_not_a_real_key
PRIMARY_PROVIDER=groq

Restart the server and fire a /chat request. The pre-flight health check will flag Groq as unhealthy and the FallbackProvider will route to OpenAI automatically.

Option 2: Check the health endpoint first

curl http://localhost:8000/health

If the primary shows "unhealthy", any subsequent /chat call will already be routing through the backup. No manual intervention needed — that's kind of the whole point.

What to look for in logs:

INFO: Primary provider health check failed — routing to fallback
INFO: Request completed via openai | latency: 1.24s

---

Notes

• Logs are written in JSONL format for easy ingestion into any log aggregation tool.
• The FallbackProvider is the only component that knows multiple providers exist — everything above it is blissfully unaware.
• Health checks use lightweight requests to avoid burning tokens just to verify connectivity.

---

Built with FastAPI, a healthy distrust of API uptime guarantees, and an appreciation for clean abstractions.