Tuned w8a8 block-FP8 Triton configs for DeepSeek-V4-Flash-0731 on
NVIDIA GB10 / DGX Spark (sm_121), tensor-parallel 2.
vLLM looks these up by matrix shape and silently falls back to a generic default when one is missing, warning:
Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal!
Config files not found at ['.../N=4096,K=12288,device_name=NVIDIA_GB10,dtype=fp8_w8a8,block_shape=[128,128].json']
This repo carries the complete set of 7 shapes the model requests at TP=2.
Every published GB10 tuning run for this model covers six shapes and is
missing exactly one — N=4096,K=12288.
Prior art on the sibling device. jasl/vllm#37
(alexbi29, merged 2026-08-03) added this same shape for RTX PRO 6000 Blackwell
(SM120), tuned with the same upstream script over the same 18-point grid. That
file does not serve GB10: vLLM's _SM12X_TUNED_CONFIG_DEVICE_ALIASES maps
only the RTX PRO 6000 Max-Q and Server editions onto the Workstation name, and
NVIDIA_GB10 is absent, so GB10 looks for its own filename and finds nothing.
Hence a GB10-specific file is still required — this repo.
Their result independently corroborates ours: an order-controlled A/B on different SM12x silicon also found no throughput gain, with an apparent +5.9% traced to first-benchmark-of-session drift. Two separate parties, two devices, same conclusion.
It is the DSpark draft's mtp.0.main_proj, verified shape [4096, 12288],
F8_E4M3 with a [32, 96] F8_E8M0 block scale. 12288 = 3 × hidden_size
because dspark_target_layer_ids = [40, 41, 42]: main_proj fuses three
target layers' hidden states into one.
The inherited tuning run predates every DSpark-equipped checkpoint. DeepSeek
published DeepSeek-V4-Flash-DSpark on 2026-06-27, carrying the same DSpark
config as 0731 (dspark_target_layer_ids=[40,41,42], dspark_markov_rank=256,
dspark_block_size=5, hidden_size=4096) — so main_proj is not unique to
0731. What matters is the date:
| date | |
|---|---|
original DeepSeek-V4-Flash (MTP draft) |
2026-04-22 |
inherited tuning run gb10_20260512 |
2026-05-12 |
DeepSeek-V4-Flash-DSpark (first DSpark-equipped) |
2026-06-27 |
DeepSeek-V4-Flash-0731 (this deployment) |
2026-07-31 |
The May run predates all of them, so it cannot contain the shape. Any GB10 block-FP8 set tuned before 2026-06-27 has the same gap, regardless of which DSpark-equipped checkpoint you serve.
| Shapes | Source | License |
|---|---|---|
N=1536,K=4096, N=2048,K=4096, N=4096,K=1024, N=4096,K=4096, N=8192,K=1024, N=16384,K=1024 |
jasl/vllm-ds4-sm120-harness tuning_runs/gb10_20260512/ @ cd5b19e6 (2026-05-12), verified byte-identical — per-file SHA-256 in THIRD_PARTY_NOTICES.md |
MIT |
N=4096,K=12288 |
tuned here (see TUNING-REPORT.md) |
MIT |
Redistributed under MIT (copyright: DeepSeek V4 SM12x Validation Harness
contributors; notice preserved verbatim in THIRD_PARTY_NOTICES.md), with
thanks to @jasl, whose
vLLM fork this deployment is also built on.
Copy into vLLM's config directory inside your container/venv:
cp configs/*.json \
"$(python -c 'import vllm,os;print(os.path.dirname(vllm.__file__))')/model_executor/layers/quantization/utils/configs/"Or bind-mount them per file — mounting the directory masks the ~234 configs vLLM ships for other GPUs:
CFG_DST=/src/vllm/vllm/model_executor/layers/quantization/utils/configs
for f in configs/*.json; do
MOUNTS+=( -v "$PWD/$f:$CFG_DST/$(basename "$f"):ro" )
done
docker run "${MOUNTS[@]}" ...Confirm adoption in the log — one line per shape:
Using configuration from .../N=8192,K=1024,...json for W8A8 Block FP8 kernel.
tune-shape.py is the driver that supplied N=4096,K=12288 to vLLM's upstream
tuner (benchmarks/kernels/benchmark_w8a8_block_fp8.py), whose
get_weight_shapes() is hard-coded for DeepSeek-V3. Search space, timing and
selection are unmodified upstream code. Run it inside the same vLLM image.
Not yet included, and needed before treating this as a reproducible benchmark
artifact: the full server command, the prompt corpus and request parameters,
raw per-request throughput with DSpark acceptance, and winner-vs-default kernel
timings per M.
Adopting the six pre-existing configs moved single-stream decode 42.4 → 41.1 tok/s on a fixed six-prompt benchmark — no change, arguably marginally worse, i.e. within run-to-run noise.
The reason is structural, and it was subsequently measured with a torch profiler trace on this deployment rather than argued. GPU self-time during decode:
| kernel | share |
|---|---|
marlin_moe_wna16 (MoE) |
32.2% |
_w8a8_triton_block_scaled_mm (what these configs tune) |
22.8% |
cutlass_80_wmma (lm_head + DSpark markov_head) |
~15% |
ncclDevKernel_AllReduce (TP) |
9.2% |
sparse_mla_decode (attention) |
1.3% |
The MoE path dominates and is tuned separately — the GB10 E=256,N=384 /
E=256,N=512 fused-MoE configs already ship in vLLM, which is why that loader
emits no fallback warning. These block-FP8 configs address the 22.8% slice, and
the tuner had already been selecting near-optimal tiles for it. Decode at small
batch is additionally memory-bandwidth-bound (GB10: 128 GB LPDDR5X @ 273 GB/s).
Shapes were obtained with torch_profiler_record_shapes; the attribution above
is kernel self-time from that trace, not inference.
Publish/adopt these for correctness and to silence a real warning — not as a throughput fix. If you are chasing decode throughput on this model, measure DSpark acceptance first: it is content-dependent over a 2.1× range (measured 80.6 tok/s @ 94.5% acceptance vs 37.6 tok/s @ 34.4% on the same healthy tier, same minute). Throughput quoted without its prompt set is meaningless.
MIT — see LICENSE.