LLM Inference Optimization on RTX 5070 Ti (16GB VRAM)

Production-ready benchmarks, quantization pipelines, and serving configs for NVIDIA RTX 5070 Ti (Blackwell architecture)

🎯 Overview

This repository provides optimized LLM inference configurations specifically tuned for the NVIDIA RTX 5070 Ti 16GB (Blackwell architecture, 5th gen Tensor Cores, native FP8 support).

Why RTX 5070 Ti Specific?

• FP8 Tensor Cores: 2× FP16 throughput for quantization-native inference
• 16GB VRAM: Fits 8B models at FP8 (32K context), 14B at FP8 (16K context)
• 5th Gen Tensor Cores: 2.5× Ada Lovelace performance per SM
• TMA (Tensor Memory Accelerator): Async memory copies overlap compute
• Cluster Launch Control: Reduced kernel launch overhead

📊 Benchmark Targets (RTX 5070 Ti 16GB)

Model	Quantization	Context	Throughput (tok/s)	VRAM Usage	Engine
Llama 3.1 8B	FP8	32K	180-220	~10 GB	vLLM / SGLang
Llama 3.1 8B	INT4 AWQ	32K	250-300	~6 GB	vLLM / llama.cpp
Qwen 2.5 14B	FP8	16K	120-150	~13 GB	vLLM / SGLang
Qwen 2.5 7B	FP8	32K	200-240	~9 GB	vLLM / SGLang
Nemotron 3 Ultra	FP8	16K	100-130	~14 GB	vLLM / TensorRT-LLM
Phi-3.5 Mini	INT4	128K	400+	~3 GB	llama.cpp / vLLM

Run python scripts/benchmark.py to generate your own benchmarks.

🚀 Quick Start

Prerequisites

• RTX 5070 Ti 16GB (or compatible Blackwell GPU)
• CUDA 12.4+
• Python 3.10+
• 20GB+ free disk space for models

Installation

# Clone the repo
git clone https://github.com/yourusername/llm-inference-rtx5070ti.git
cd llm-inference-rtx5070ti

# Create virtual environment
python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Install vLLM with CUDA 12.4 support
pip install vllm --extra-index-url https://download.pytorch.org/whl/cu124

# Or use Docker (recommended)
docker compose -f docker/docker-compose.yml up -d

Run Benchmarks

# Full benchmark suite (all models, quants, engines)
python scripts/benchmark.py --all

# Quick benchmark: Llama 3.1 8B FP8 on vLLM
python scripts/benchmark.py --model llama-3.1-8b --quant fp8 --engine vllm

# Compare engines for a model
python scripts/benchmark.py --model qwen-2.5-7b --quant fp8 --engines vllm,sglang,tensorrt

Start Optimized Serving

# vLLM OpenAI-compatible server (recommended for production)
python scripts/serve.py --engine vllm --model llama-3.1-8b --quant fp8 --port 8000

# SGLang server (better for long context)
python scripts/serve.py --engine sglang --model qwen-2.5-14b --quant fp8 --port 8001

# TensorRT-LLM (maximum throughput)
python scripts/serve.py --engine tensorrt --model llama-3.1-8b --quant fp8 --port 8002

📁 Repository Structure

llm-inference-rtx5070ti/
├── README.md
├── LICENSE
├── requirements.txt
├── configs/
│   ├── models.yaml           # Model configs with optimal RTX 5070 Ti settings
│   ├── quantization.yaml     # Quantization recipes (FP8, INT4 AWQ, GPTQ, GGUF)
│   └── serving.yaml          # Engine-specific serving configurations
├── scripts/
│   ├── benchmark.py          # Automated benchmarking suite
│   ├── quantize.py           # Quantization pipeline
│   ├── serve.py              # Multi-engine serving launcher
│   ├── profile.py            # NSight Compute/Perf profiling
│   └── convert.py            # HF → TensorRT-LLM / GGUF conversion
├── benchmarks/
│   ├── results/              # CSV/JSON benchmark outputs
│   └── plots/                # Generated comparison charts
├── docker/
│   ├── Dockerfile.vllm
│   ├── Dockerfile.sglang
│   ├── Dockerfile.tensorrt
│   └── docker-compose.yml
├── examples/
│   ├── openai_compatible.py  # OpenAI API compatible client
│   ├── streaming_chat.py     # Streaming chat demo
│   └── batch_inference.py    # High-throughput batch processing
└── docs/
    ├── quantization_guide.md
    ├── serving_comparison.md
    ├── blackwell_optimizations.md
    └── troubleshooting.md

⚙️ Configuration

Model Configs (configs/models.yaml)

Each model has engine-specific optimizations:

models:
  llama-3.1-8b:
    hf_id: "meta-llama/Llama-3.1-8B-Instruct"
    optimal_quant: "fp8"
    max_context: 32768
    vllm:
      tensor_parallel_size: 1
      gpu_memory_utilization: 0.92
      max_model_len: 32768
      enforce_eager: false
      disable_log_stats: false
    sglang:
      mem_fraction_static: 0.92
      chunked_prefill_size: 8192
      max_running_requests: 256

Quantization Recipes (configs/quantization.yaml)

quantization_recipes:
  fp8:
    method: "tensorrt_llm_fp8"
    calibration_samples: 512
    dtype: "fp8"
    description: "Native Blackwell FP8 - best quality/speed balance"
  
  int4_awq:
    method: "awq"
    group_size: 128
    zero_point: true
    version: "gemm"
    description: "INT4 AWQ - maximum compression, minimal quality loss"
  
  int4_gptq:
    method: "gptq"
    group_size: 128
    act_order: true
    desc_act: true
    description: "GPTQ - good alternative to AWQ"
  
  gguf_q4_k_m:
    method: "llama_cpp"
    quantization: "Q4_K_M"
    description: "GGUF Q4_K_M - best for llama.cpp"

🔬 Benchmarking

The benchmark suite measures:

• Prefill latency (time to first token)
• Decode latency (per-token generation time)
• Throughput (tokens/second)
• VRAM usage (peak allocated)
• GPU utilization (SM occupancy)
• Power draw (watts)

# Output formats
python scripts/benchmark.py --all --output json,csv,html

# Custom test matrix
python scripts/benchmark.py \
  --models llama-3.1-8b,qwen-2.5-7b \
  --quants fp8,int4_awq \
  --engines vllm,sglang \
  --context-lens 4096,16384,32768 \
  --output-lens 128,512,1024

🐳 Docker Deployment

# Build all images
docker compose -f docker/docker-compose.yml build

# Run vLLM server
docker compose -f docker/docker-compose.yml up vllm

# Run all engines
docker compose -f docker/docker-compose.yml up

# Production with GPU pinning
docker run --gpus '"device=0"' \
  -p 8000:8000 \
  -v $(pwd)/models:/models \
  llm-inference-rtx5070ti:vllm

📈 Blackwell-Specific Optimizations

FP8 Native Inference

# vLLM FP8 config for RTX 5070 Ti
from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct-FP8",
    tensor_parallel_size=1,
    gpu_memory_utilization=0.92,
    max_model_len=32768,
    quantization="fp8",  # Native FP8
    enforce_eager=False,  # Use CUDA graphs
    disable_log_stats=False,
)

CUDA Graphs + Chunked Prefill

# configs/serving.yaml
vllm:
  use_cuda_graph: true
  cuda_graph_max_batch_size: 256
  chunked_prefill_enabled: true
  max_num_batched_tokens: 8192

sglang:
  chunked_prefill_size: 8192
  max_running_requests: 256
  enable_flashinfer: true  # FlashInfer MLA for Blackwell

TensorRT-LLM FP8 Build

# Convert and build TensorRT-LLM engine
python scripts/convert.py \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --output ./engines/llama-3.1-8b-fp8 \
  --quantization fp8 \
  --max_batch_size 256 \
  --max_input_len 32768 \
  --max_output_len 4096

📚 Documentation

Doc	Description
docs/quantization_guide.md	FP8 vs INT4 vs GGUF - when to use each
docs/serving_comparison.md	vLLM vs SGLang vs TensorRT-LLM vs llama.cpp
docs/blackwell_optimizations.md	FP8, TMA, Cluster Launch, CUDA Graphs deep dive
docs/troubleshooting.md	OOM, slow inference, CUDA errors, model loading issues

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-optimization)
3. Run benchmarks to validate (python scripts/benchmark.py --model your-model)
4. Submit a PR with benchmark results

📄 License

MIT License - see LICENSE for details.

🙏 Acknowledgments

• NVIDIA for Blackwell architecture and TensorRT-LLM
• vLLM team for the excellent serving engine
• SGLang team for FlashInfer integration
• llama.cpp team for GGUF quantization
• The open-source LLM community

---

Built for RTX 5070 Ti 16GB | Tested on CUDA 12.4, Driver 560+ | Optimized for Production