quantization_guide.md

Quantization Guide for RTX 5070 Ti

Overview

The RTX 5070 Ti (Blackwell architecture) introduces native FP8 support through 5th generation Tensor Cores. This guide covers when to use each quantization method.

Quantization Methods Comparison

Method	VRAM Savings	Quality	Speed	Best For
FP8 (Native)	~50%	99%+	1.8-2.2x	8B/14B models, production
INT4 AWQ	~75%	95-98%	2.5-3.5x	32B+ models, max compression
INT4 GPTQ	~75%	94-97%	2.3-3.2x	AWQ alternative
GGUF Q4_K_M	~75%	96-98%	CPU/GPU	Edge, Apple Silicon, llama.cpp
FP16	0%	100%	1x	Reference, quality-critical

FP8 (Native Blackwell) - RECOMMENDED

Why FP8 on RTX 5070 Ti?

1. Hardware Native: 5th gen Tensor Cores have dedicated FP8 MMA instructions
2. No Calibration Needed: Dynamic FP8 works out-of-the-box in vLLM
3. KV Cache FP8: Additional 50% KV cache savings
4. Quality: Near-FP16 quality (99%+ retention)

When to Use FP8

• ✅ Models ≤ 14B parameters
• ✅ Production serving (vLLM, SGLang, TensorRT-LLM)
• ✅ When you want best quality/speed balance
• ✅ Long context (32K+) with KV cache FP8

FP8 Variants

Variant	Calibration	Quality	Speed	Use Case
Dynamic FP8	None	99%	Fastest	Default vLLM, quick deployment
Static FP8	512-1024 samples	99.5%+	Same	Maximum quality, TensorRT-LLM

vLLM FP8 Usage

from vllm import LLM

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    quantization="fp8",  # Dynamic FP8
    gpu_memory_utilization=0.92,
    max_model_len=32768,
    kv_cache_dtype="fp8",  # FP8 KV cache
)

TensorRT-LLM Static FP8

# Convert checkpoint
python -m tensorrt_llm.commands.convert_checkpoint \
  --model_dir meta-llama/Llama-3.1-8B-Instruct \
  --output_dir ./ckpt_fp8 \
  --dtype fp8

# Build engine with calibration
trtllm-build \
  --checkpoint_dir ./ckpt_fp8 \
  --output_dir ./engine_fp8 \
  --fp8 true \
  --calibrate true \
  --calibration_data ./calibration_data

INT4 AWQ - Maximum Compression

When to Use AWQ

• ✅ Models 32B+ (won't fit in 16GB at FP8)
• ✅ When VRAM is critical
• ✅ Batch inference where throughput matters most
• ✅ llama.cpp deployment

AWQ Configuration

# configs/quantization.yaml
int4_awq:
  group_size: 128      # 128 optimal for most models
  zero_point: true     # Slightly better quality
  version: "gemm"      # "gemm" faster on Blackwell

Quality Preservation Tips

1. Use 512-1024 calibration samples from target domain
2. Group size 128 is sweet spot for most models
3. Zero-point adds ~0.5% quality for minimal overhead
4. Test on your data - AWQ quality varies by model

INT4 GPTQ - Alternative

When to Use GPTQ

• AWQ not available for model architecture
• Want different quality/speed tradeoff
• Legacy model support

GPTQ Configuration

int4_gptq:
  group_size: 128
  act_order: true      # Important for quality
  desc_act: true       # Descending activation order
  dampening: 0.01      # Numerical stability

GGUF (llama.cpp) - Maximum Compatibility

GGUF Quantization Types

Type	Size	Quality	Speed	Use Case
Q2_K	Smallest	~90%	Fastest	Extreme compression
Q3_K_M	Small	~93%	Fast	Mobile/Edge
Q4_K_M	Medium	96-98%	Balanced	Default recommendation
Q5_K_M	Large	98-99%	Slower	Quality priority
Q6_K	Larger	99%+	Slow	Near-FP16
Q8_0	Largest	99.5%+	Slowest	Maximum quality

llama.cpp Conversion

# Convert HF to fp16 GGUF
python -m llama_cpp.convert_hf_to_gguf \
  meta-llama/Llama-3.1-8B-Instruct \
  --outfile llama-3.1-8b-fp16.gguf \
  --outtype f16

# Quantize to Q4_K_M
llama-quantize llama-3.1-8b-fp16.gguf llama-3.1-8b-q4_k_m.gguf Q4_K_M

RTX 5070 Ti llama.cpp Settings

# Optimal for RTX 5070 Ti
./llama-server \
  -m llama-3.1-8b-q4_k_m.gguf \
  -ngl 99 \              # All layers on GPU
  -c 32768 \             # Context
  -b 512 \               # Batch size
  -ub 512 \              # Micro batch
  --flash-attn \         # Flash attention
  --rope-scaling linear  # For long context

Model-Specific Recommendations

Llama 3.1 8B

• Best: FP8 (fits 32K context in 16GB)
• Alternative: INT4 AWQ (6GB, 32K context)
• Avoid: FP16 (16GB+, no headroom)

Qwen 2.5 7B/14B

• 7B: FP8 (32K context)
• 14B: FP8 (16K context) or INT4 AWQ (32K context)
• Qwen benefits more from AWQ than Llama

Nemotron 3 Ultra

• Best: FP8 (14B, 16K context)
• Large model, FP8 essential for 16GB

Phi-3.5 Mini (3.8B)

• Best: INT4 AWQ/Q4_K_M (tiny, 128K context)
• FP8 overkill for 3.8B model

Qwen 2.5 32B

• Only option: INT4 AWQ or GGUF Q4_K_M
• FP8 needs ~20GB, INT4 needs ~10GB

Decision Flowchart

START
  │
  ▼
Model ≤ 14B? ──Yes──▶ FP8 (Native) ──▶ DONE
  │
  No
  ▼
Model ≤ 32B? ──Yes──▶ INT4 AWQ ──▶ DONE
  │
  No
  ▼
Need CPU/Apple Silicon? ──Yes──▶ GGUF Q4_K_M ──▶ DONE
  │
  No
  ▼
FP8 with offload? ──Yes──▶ FP8 + CPU offload ──▶ DONE
  │
  No
  ▼
INT4 GPTQ / GGUF Q5_K_M ──▶ DONE

Benchmarking Your Quantization

# Quick quality check
python scripts/benchmark.py \
  --model llama-3.1-8b \
  --quant fp8 \
  --engine vllm \
  --input-lens 512 \
  --output-lens 256

# Compare quantization methods
python scripts/benchmark.py \
  --model llama-3.1-8b \
  --quants fp8,int4_awq \
  --engine vllm

Common Issues

OOM with FP8

• Reduce gpu_memory_utilization to 0.85
• Reduce max_model_len
• Enable swap_space: 4
• Try INT4 AWQ instead

Quality Degradation

• Use static FP8 with calibration
• Increase AWQ calibration samples to 1024
• Try GPTQ with act_order=True
• Use GGUF Q5_K_M

Slow Inference

• Enable CUDA graphs (enforce_eager: false)
• Use chunked prefill for long context
• Try SGLang with FlashInfer
• TensorRT-LLM for static workloads

References

• NVIDIA FP8 Quantization Whitepaper
• vLLM FP8 Documentation
• AWQ Paper
• GPTQ Paper
• GGUF Specification