Does VLLM support DP attention for deepseek?

[llc-kc]

Does VLLM support DP attention for deepseek?

[Shyboy0499]

Yes — vLLM (V1 engine) supports data-parallel attention for DeepSeek and other MLA-based MoE models.

Status

• Distributed EP/DP support landed in vLLM 0.8.0 (EP support for DeepSeek, EP/TP MoE + DP attention, data-parallel communication setup).
• vLLM 0.9.1 added further DP-attention + expert-parallelism hardening: CUDA-graph support for DP + all-to-all kernels, DeepEP dispatch/combine, and DeepGEMM kernels.

Why DP attention matters for DeepSeek

MLA keeps the KV cache small, but TP replicates that KV across ranks. DP attention instead keeps one KV copy per rank for the attention layers and pairs it with expert parallelism (or TP) for the MoE expert layers. That cuts KV memory replication and lets you scale across nodes using sparse dispatch/combine (DeepEP over NVSHMEM/RDMA) rather than TP all-reduces. The vLLM docs explicitly recommend this topology for DeepSeek-class models: "data parallel for the attention layers and expert or tensor parallel (EP or TP) for the expert layers."

How to enable it

# DP=4, TP=2 on a single 8-GPU node
vllm serve deepseek-ai/DeepSeek-V3 --data-parallel-size 4 --tensor-parallel-size 2

# Same, but run the expert layers with expert parallelism instead of TP
vllm serve deepseek-ai/DeepSeek-V3 --data-parallel-size 4 --tensor-parallel-size 2 --enable-expert-parallel

Multi-node and Ray are supported via --data-parallel-size-local, --data-parallel-address, --data-parallel-start-rank, and --data-parallel-backend=ray. DP+EP also works offline through the LLM class (see examples/features/data_parallel/data_parallel_offline.py).

When it pays off

DP attention is a throughput play — it shines under high concurrency. Budget for the coordination overhead: MoE forward passes must stay synchronized across DP ranks (dummy forward passes run in ranks with no requests), and --max-num-seqs applies per DP rank.

References: Data Parallel Deployment · Expert Parallel Deployment · tracking issue vllm-project/vllm#12871