All numbers measured on the stack in the README's Verified environment table, on 2026-07-28.
Two scripts, measuring two different things:
scripts/sustained_load.py— steady-state throughput under sustained concurrent load. Use this for capacity planning.scripts/benchmark.py— burst behavior across a range of concurrency levels. Use this for latency expectations.
scripts/benchmark.py, driving
POST /v1/chat/completions over the LAN from a separate machine (so client
overhead is included — this is end-to-end, as an agent would see it).
- Prompts: short code-generation requests ("Write a Python function implementing <algorithm>. Code only."), one distinct algorithm per concurrent slot so responses differ.
max_tokens=250,temperature=0.2- Concurrency via
ThreadPoolExecutor, all requests dispatched together - Per-request tok/s = completion tokens ÷ that request's wall time
- Aggregate tok/s = all completion tokens ÷ wall time for the whole batch
- Non-streaming, so per-request timing includes TTFT
Limitations, stated plainly:
- Short prompts. Long-context agent workloads shift the bottleneck to KV cache and will scale worse.
- Steady-state after warmup; see the warmup note below.
- Single run per level, not averaged across repetitions — treat as indicative, not publication-grade.
~350 tok/s aggregate at 16 concurrent, confirmed two independent ways:
| Accounting | Result |
|---|---|
| Client-side (24,000 tokens / 68.1 s wall) | 352.4 tok/s |
| vLLM's own server-side logger, steady state | 342–381 tok/s (six consecutive 10 s windows, Running: 16 reqs throughout) |
Per-request during sustained load: ~22 tok/s each, 16 users in parallel.
Read the aggregate correctly. ~350 tok/s is 16 × ~22, summed. It is a capacity figure, not a speed anyone experiences. If you are the only user, the number that describes your experience is ~70 tok/s.
Method: 16 concurrent requests with max_tokens=1500 and prompts engineered to
force long generation (full red-black tree implementation with docstrings,
comments, and tests), so the load sustains well past vLLM's ~10 s logging
window and steady state dominates.
Why this differs from the burst numbers below. The short-prompt benchmark (
max_tokens=250, ~9 s wall) reports ~270 tok/s at 16 concurrent. That is a real measurement, but at that duration the prefill ramp-up and the ragged tail — requests completing at staggered times while the batch drains — dominate the wall clock and depress the average. It characterizes burst behavior, not throughput.Both are honest; they answer different questions. For "how much can this box serve 16 users," use ~350 tok/s. For "how fast does a short request come back under load," use the per-request figures.
The single most useful thing we learned benchmarking this stack.
Client-side "aggregate throughput" — total completion tokens ÷ total wall clock — is the obvious metric and the one most benchmark scripts report, including ours. It is also unstable by up to 20% between runs of an identical configuration, because it absorbs prefill ramp-up, the ragged tail as a batch drains, and network round-trip.
vLLM's own generation throughput logging is stable. Measured across a full
session of restarts, memory states, and configuration changes:
| Run | Client-side aggregate | Server-side generation (Running: 16) |
|---|---|---|
| Early session, gmu 0.80 | 352.4 tok/s | 355.1 / 347.1 / 345.5 / 347.1 / 339.2 → mean 346.8 |
| Late session, gmu 0.80 | 279.7 tok/s | 361.6 / 355.2 / 340.7 / 339.1 / 329.5 → mean 345.2 |
Same configuration. Client-side says the box lost 21% of its throughput. Server-side says nothing changed at all — 346.8 vs 345.2, a 0.5% difference.
The server was right. Nothing had degraded; the client measurement had drifted.
We chased two separate wrong conclusions from client-side numbers:
- "Raising
--gpu-memory-utilizationto 0.90 costs 23% throughput" — with a plausible mechanism (page cache collapsing from 24 GB to 1 GB, engine processes swapping). A control re-run at the original 0.80 reproduced the same "slow" number, killing it. - "The machine degrades ~20% over a session" — clearing swap and page cache didn't recover it, because there was nothing to recover.
Both were the measuring instrument, not the system. Both would have been published as confident findings with tidy explanations attached.
Read the server's own accounting, and require Running: N to match your
intended concurrency:
docker logs <container> 2>&1 | grep "generation throughput" | tail -10- Windows showing
Running: 16when you dispatched 16 → measuring steady state. - Windows showing
Running: 0or a smaller number → you are measuring ramp-up or drain, and your aggregate will understate.
Use client-side aggregate as a cross-check, not the headline. When the two disagree by more than a few percent, the client is wrong.
And run a control. If you change one flag and see a difference, re-run the original configuration before believing it. On this stack that single habit caught two false findings in one afternoon.
| Concurrency | Aggregate | Per-request (min/avg/max) | Wall | Scaling |
|---|---|---|---|---|
| 1 † | ~70 tok/s | 69.3 / 70.7 / 72.1 | — | 1.0× |
| 4 | 148.7 tok/s | 43.5 / 47.4 / 51.9 | 2.7 s | 2.1× |
| 8 | 196.3 tok/s | 32.6 / 36.9 / 45.2 | 5.5 s | 2.8× |
| 16 | 272.5 tok/s | 22.1 / 24.3 / 27.0 | 9.3 s | 3.9× |
† The concurrency-1 row is three separate sequential single requests
(69.3 / 70.1 / 72.1 tok/s), not one run of the script — at n=1 the script
emits min == avg == max by construction. Reported this way for an honest
spread. A later independent run of benchmark.py --levels 1,4 on the same
server produced 65.8 tok/s single and 153.6 tok/s at 4 concurrent, which is the
run-to-run variation you should expect.
Shape of the curve: near-linear to 4, solid gains to 8, real but diminishing returns at 16. Consistent with other operators' reports for 3B-active MoE models on this hardware — MoE routing widens as batch grows, and the box is memory-bandwidth-bound.
The very first request after startup or a long idle measured 4.9 tok/s (92 tokens / 18.9 s), then immediately 69.3 and 72.1 tok/s on identical follow-ups. Attributable to CUDA graph capture and prefix-cache population.
Discard the first request when benchmarking. In production it shows up as an occasional slow first interaction after idle.
Same model, same hardware, same prompts — only driver and CUDA graph state differ:
| Configuration | Single | 4 concurrent |
|---|---|---|
Driver 580.126.09 + --enforce-eager |
25 tok/s | 67 tok/s |
| Driver 595.84 + CUDA graphs | 70 tok/s | 149 tok/s |
| Improvement | 2.8× | 2.2× |
The driver upgrade is the single highest-impact change available on this hardware. Driver 580 cannot capture CUDA graphs on GB10 at all (see TROUBLESHOOTING #7).
| Runtime | Model / quant | Context | Single | Concurrent | Memory |
|---|---|---|---|---|---|
| vLLM (this repo) | int4-AutoRound | 131072 | 70 tok/s | ~350 tok/s @16 sustained | ~103 GB |
llama.cpp llama-server |
Q5_K_M GGUF | 16384, -np 2 |
46 tok/s | not measured | ~58 GB |
| vLLM + GGUF | Q5_K_M GGUF | — | won't load | — | needs ~160 GB |
Read this table carefully — it is not apples-to-apples. The two working rows differ in quantization (int4 vs Q5_K_M), context (131072 vs 16384, an 8× gap) and slot count. The memory figures in particular are not a fair comparison: much of vLLM's larger footprint is KV cache for 8× the context and 16 vs 2 sequences, plus the fact that vLLM pre-allocates while llama.cpp allocates on demand.
What we can say confidently: vLLM was faster single-stream (70 vs 46 tok/s)
on the same hardware and model family, and it sustained 16 concurrent requests
at ~350 tok/s sustained aggregate. We did not measure llama.cpp under concurrency, so
this table says nothing about how it would compare there —
llama-server does have continuous batching. A matched-context, matched-slot
comparison is an open contribution.
Context window and concurrent sequences compete for the same KV cache. We run 131072 rather than the model's full 262144 for exactly this reason — community reports indicate 262K admits only ~2–4 sequences before KV starvation.
If you need more than 131K context, --kv-cache-dtype fp8 roughly halves KV memory — but vLLM's own DGX Spark guidance advises against it: it "may affect model predictability and can carry a noticeable performance cost on Spark for some workloads; avoid it unless memory pressure requires it and quality checks pass" (vLLM blog). We have not benchmarked it. Treat as a last resort, and measure.
vLLM's DGX Spark blog uses 0.85, reasoning that you must "leave room for the operating system, kernel page cache, container runtime, KV cache growth, and any other process touching that same memory." Other guidance circulating suggests 0.90 or 0.93.
We swept it. It changes KV cache size exactly as advertised — and changes throughput almost not at all:
--gpu-memory-utilization |
GPU KV cache | Server-side generation @16 |
|---|---|---|
| 0.80 (ours) | 2,299,331 tokens | ~345 tok/s |
| 0.85 | 2,581,640 tokens | ~345 tok/s |
| 0.90 | 2,768,431 tokens | ~345 tok/s |
Server-side generation throughput was flat across all three (individual 10 s
windows at Running: 16 ranged 329–362 in every configuration — indistinguishable
run-to-run noise).
What the setting actually buys you is capacity, not speed: more KV cache means more sequences, or longer contexts, resident at once. Choose it by how many concurrent users and how much context you need, not to chase tok/s.
We keep 0.80 for OS headroom on a box that also runs other containers. 0.85 is a defensible default if the Spark is dedicated to serving.
⚠️ This is where we nearly published a false finding. Client-side measurements suggested 0.90 was 23% slower, with a tidy mechanism (page cache collapsing, engine swapping). A control re-run at 0.80 produced the same "slow" number — the effect was client-side measurement variance, not the flag. See the client-vs-server section.
Set to 16. vLLM's DGX Spark blog suggests 4, with this rationale:
"Above four concurrent decode streams the per-token bandwidth tax can outweigh continuous-batching gains, and time-to-first-token spikes."
That recommendation is given for NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 — a 12B-active MoE. Qwen3-Coder-Next activates 3B per token, roughly four times less memory traffic per decoded token, so the bandwidth wall they hit at 4 streams should land considerably further out. Our 16-concurrent results bear that out.
Supporting context from the same post: they state Spark suits MoE models with "roughly 10-15 billion active parameters." This model sits below that window — which may be exactly why it outperforms their published figures rather than despite it.
If CUDA graph capture fails with memory errors, lower it.
Raised to 16384 from the 8192 default, so long agent prompts aren't chunked excessively during prefill.
4–8 concurrent coding-agent users on one Spark, each seeing 37–47 tok/s — comfortably interactive. 16 concurrent works at ~22–24 tok/s each (~350 tok/s aggregate), usable but noticeably slower per user.
--max-num-seqs 16 is a value we chose to stress-test with. It is not a
hardware limit, and we never found the actual one.
Memory is not what stops you. vLLM reports ~2.3M tokens of KV cache
available at --gpu-memory-utilization 0.80. Its own startup log puts that at
"Maximum concurrency for 131,072 tokens per request: 17.54x" — but that
assumes every request uses the full 131K context. Real coding-agent requests
run a few thousand tokens:
| Tokens per request | Sequences the KV cache could hold |
|---|---|
| 131,072 (full context) | ~17 |
| 16,384 | ~140 |
| 8,192 | ~280 |
| 4,096 | ~560 |
So at realistic request sizes there is room for hundreds of concurrent sequences before KV cache binds.
Diminishing returns stop you long before that. From the burst sweep:
| Users | Aggregate | Speedup vs 1 user | Marginal gain |
|---|---|---|---|
| 1 | 70 tok/s | 1.0× | — |
| 4 | 149 tok/s | 2.1× | +113% for 4× users |
| 8 | 196 tok/s | 2.8× | +32% for 2× users |
| 16 | 273 tok/s | 3.9× | +39% for 2× users |
Every doubling of users adds less throughput than the last. Total output keeps climbing; each person's share keeps shrinking.
The mechanism is bandwidth, not capacity. The model's weights must be read from memory for every token generated. That pipe is finite regardless of how many conversations you stack on it — which is why aggregate throughput flattens while KV cache still has room to spare. Adding sequences past that point mostly redistributes the same tokens among more people.
So the useful question isn't "where does it break?" — it's "where does per-person speed stop being pleasant?" On this hardware that's around 8 concurrent users (37–47 tok/s each). 16 is usable. Beyond that we have no data, and the honest answer is that we stopped testing, not that it failed.
If you find the real ceiling, we'd like to know.
Monitor TTFT, not error rate. vLLM queues rather than rejecting, so saturation appears as growing latency, not failures. Other operators report TTFT staying acceptable through ~16 concurrent and degrading sharply past ~32.
Long-context agents will hit KV limits before compute limits — if you serve agents with large working contexts, expect fewer concurrent users than these short-prompt numbers suggest.
Burst curve across concurrency levels:
python3 scripts/benchmark.py --host <SPARK_IP> --port 8000 \
--model Intel/Qwen3-Coder-Next-int4-AutoRoundSustained steady-state throughput (the headline number):
python3 scripts/sustained_load.py <SPARK_IP> 8000 16Always cross-check against the server's own accounting rather than trusting client-side arithmetic alone — that discrepancy is exactly what surfaced the burst-vs-sustained distinction here:
docker logs <container> 2>&1 | grep "generation throughput" | tail -10Look for windows reporting Running: 16 reqs (your full concurrency). If every
window shows fewer, your load isn't sustaining and you're measuring ramp-up.
Sends a warmup request (discarded), then measures 1/4/8/16 concurrency.