Serving LLMs on an NVIDIA DGX Spark with vLLM

Complete, reproducible recipes for serving models as an OpenAI-compatible API on a single NVIDIA DGX Spark (GB10, 121 GB unified memory, aarch64) — with working tool-calling, vision, and real multi-user concurrency.

Three models are documented. The one at the top is what this box serves today. It is also the slowest of the three, and the reason why turns out to be the most useful thing in this repo.

Model	Active params / token	Single stream	Why it is here
Qwen3.8-27B (NVFP4, Unsloth) — primary	all 27B	23.3 tok/s	what runs here now: 262K context, vision, tool-calling
Qwen3.6-35B-A3B (NVFP4)	3B	102.3 tok/s	fastest single stream measured here
Qwen3-Coder-Next 80B-A3B (int4-AutoRound)	3B	~70 tok/s	multi-user coding agents, 16 concurrent

These numbers were measured different ways. 23.3 is the mean of 5 natural generations at short context; 102.3 is llama-benchy at a 30,000 token prompt, n=3, std 6.5; ~70 is a burst run with short prompts. At the same 30K prompt shape the primary model measures 17.4 tok/s, which is the figure to compare against 102.3. Details in each section.

The single most useful finding here: decode speed on GB10 tracks active parameters per token — not total parameters, not quantization, not tuning. The two 3B-active MoE checkpoints run 4–6x faster than a 27B with no expert routing, and a different vendor's 27B AWQ build lands in the same class on the same box. If you are choosing a model for this hardware, choose on active parameters first and everything else second.

If you read nothing else: --gpu-memory-utilization does not bound vLLM's allocation on this hardware. It sizes the KV cache from free memory, not from the grant, so 0.50 can still allocate ~101 GiB. --kv-cache-memory is the flag that binds. That mistake hard-crashed this box once — the write-up is here.

---

Qwen3.8-27B-NVFP4 (Unsloth): results, stated honestly

The slowest generation measured here, and the reason is architectural, not configuration. Worth publishing precisely because the number is disappointing: it is the cleanest demonstration in this repo that decode speed on GB10 tracks active parameters per token, and nothing else you can set.

	Qwen3.6-35B-A3B (NVFP4)	KAT-Coder V2.5 (NVFP4)	Qwen3.8-27B (NVFP4)
Expert routing	MoE, 3B active	MoE, 8 of 256	none
Active params / token	3B	small fraction	all 27B
Single stream generation	110.8 tok/s	68.1 tok/s	23.3 tok/s
Prefill (short ctx)	5,658 tok/s	6,450 tok/s	1,746 tok/s

Qwen3.8-27B is a hybrid-attention model — of its 64 layers, 16 are full attention and 48 are Gated DeltaNet, plus a vision tower — but it has no expert routing, so every one of its 27B parameters activates on every token. A 27B model with no routing lands in the low-20s on this box, and a 27B AWQ build from a different vendor measured in the same class on the same hardware. No serve flag moves that. The MoE checkpoints below are 4-7x faster because they activate a fraction of their weights, not because they are better tuned.

Measured

Measured three ways, because the first way was wrong and the difference is instructive. All runs use llama-benchy 0.4.1.dev1 except where noted.

Shape	Method	Generation
Natural, short context	5 chat completions, 400 tok, no ignore_eos	23.26 tok/s (21.1–25.8)
Long context, 30K prompt	benchy --pp 30000 --tg 2000 --runs 3 --no-cache	17.44 ± 0.51
Forced length, short context	benchy --pp 2048 --tg 128 --exact-tg	16.03 ± 1.27

The 16.03 figure is an artifact — do not quote it. --exact-tg forces output length with min_tokens + ignore_eos, which pushes generation past the model's natural stopping point. With MTP speculative decoding that appears to depress draft acceptance, and it understated real use by roughly 45%. --exact-tg is the right flag for comparing engines on identical output lengths; it is the wrong flag for answering "how fast does this feel."

23.3 tok/s is what you actually experience in chat and coding at normal context. Within that, code generation is consistently faster than prose — 25.8 versus 21.6 — which is what you would expect if speculative decoding accepts more drafts on structured, predictable output.

17.4 tok/s is the number comparable to this repo's other models, since the Qwen3.6 figure of 102.3 was measured at the same 30K prompt shape. Decode drops about 25% between short context and a 30,000-token prompt, and prefill at that depth runs 1,097.94 ± 0.20 tok/s for a TTFT of ~25 s.

Raw output: benchy-pp30000 · natural-generation · benchy --exact-tg.

Do not read an aggregate as throughput. Peak aggregate at 4 concurrent streams measured 67.67 tok/s, which is four requests' tokens summed. Per request it was 15.90 tok/s. Quoting an aggregate as generation speed is the easiest way to overstate any model on this box by 4x — it is the same trap as the "350 tok/s" figure at the top of this README.

The 4-concurrent numbers are unstable: TTFT 5,762 ± 3,017 ms is a ±52% spread across three runs. Recorded, not explained.

Three changes I made to the upstream recipe, and why

Upstream's defaults are tuned for a Spark with headroom. On this box they were not survivable. All three changes are recorded inline in the vendored start.sh with the upstream original kept beside it as a .bak.

Setting	Upstream	Here	Why
--gpu-memory-utilization	0.84	0.65	lowers the grant, but does NOT bind KV here
--max-model-len	1,000,000 (static YaRN)	262,144 (native)	see below
--restart	(absent)	unless-stopped	see below
--kv-cache-memory	(absent)	20 GiB	the flag that actually binds; freed 37.8 GiB
--compilation-config	(absent, defaults FULL_AND_PIECEWISE)	PIECEWISE	the default produces garbage output here; see below

Dropping the 1M context is the change that did the work — not the utilization number. Read the warning at the top of this README: --gpu-memory-utilization does not bound vLLM's allocation on this hardware. It sizes the KV cache from free memory, not from the grant. Lowering 0.84 to 0.65 therefore did not cap anything by itself; what actually reduced KV demand was returning --max-model-len from 1,000,000 to the native 262,144.

Upstream reaches 1M tokens with static YaRN through --hf-overrides, and its own README notes a single 1M-token sequence needs ~32 GB of KV cache. Measured occupancy on my workload was 1-4%. Reserving that much KV for a window nothing was using livelocked the machine. Upstream also notes static YaRN "can slightly impact short-context quality," so at normal lengths dropping it costs nothing and buys back the box.

That gap is now closed. --kv-cache-memory=21474836480 (20 GiB) was applied on 2026-08-17. vLLM had been sizing KV from free memory and took 53.28 GiB (1,529,669 tokens) against a genuine ceiling of max_num_seqs 4 x 262,144 = 1,048,576 — more KV than the concurrency limit can ever address, at a measured peak occupancy of 12.4% over 3,170 samples across 21 hours.

Pinning it at 20 GiB (573,440 tokens, still 3x the observed peak) took the process from 83,020 MiB to 44,266 MiB resident, freeing 37.8 GiB, with host available going 30 GB -> 67 GB. Decode speed was unchanged: 23.38 tok/s before, 23.76 after, which is what the roofline predicts, since KV read is ~3% of the decode byte budget at these context lengths. You give up capacity you could not address anyway.

--restart unless-stopped is my addition, not upstream's. Upstream's docker run carries no restart policy at all, so the container did not survive a reboot or an OOM kill — the served model simply vanished until someone noticed. unless-stopped rather than always on purpose: a deliberate ./stop.sh should stay stopped. Applied to an already-running container with docker update --restart=unless-stopped <name>, which takes effect with no downtime.

⚠ The default CUDA graph mode produces garbage output on this model

Symptom. The model returns degenerate multilingual token salad instead of English, deterministically, for particular requests:

".eth.ע\n\n3\n in\n to\nthe\nE\n\nE\n。\n压力.... Play the"
"优化E\n with.\n.\n。.J it, a..\n\n<tool_call>\n\n\n\n\nP..\n\n. with"

Note the second one leaking a literal <tool_call> token into the content stream while tool_calls[] stays empty. That is the fingerprint.

Cause. vllm-project/vllm#53051 (filed 2026-08-20, open at time of writing). GPUModelRunner._is_uniform_decode is a pure shape check, so a prefill whose scheduled token count equals (1 + num_speculative_tokens) x num_reqs is misclassified as a uniform-decode batch and dispatched into the FULL CUDA graph captured for speculative decode. On a hybrid model the Gated DeltaNet metadata builder only refreshes its persistent buffers when num_prefills == 0, so the graph replays with stale capture-time NULL_BLOCK_ID state indices, and the FLA kernels' null-block guards silently skip every recurrent state write. The request's conv/SSM state stays zero, so the first sampled token already diverges and everything after is computed from a wrong state.

This model is 48 Gated DeltaNet + 16 full-attention layers, which is exactly the class affected. cudagraph_mode defaults to FULL_AND_PIECEWISE — nobody opts in.

Fix. One flag, already in the serve command above:

--compilation-config '{"cudagraph_mode": "PIECEWISE"}'

The failure is state-dependent, and the A/B here is NOT controlled. Read this before quoting the result.

Same 33,294-token request body:

Engine state	cudagraph_mode	Output
Production-warm (2 days, 7+ consecutive occurrences)	FULL_AND_PIECEWISE	garbage every time
Freshly idle, no config change	FULL_AND_PIECEWISE	clean
Since switching	PIECEWISE	no garbage in any run

The post-switch runs were on a just-restarted engine, and the pre-switch idle-engine runs were also clean — so engine state is confounded with the flag. PIECEWISE is a mitigation that has held here. It is not proven by this data to be the cause. A related shape flipped 2/3 garbage to 0/4 clean between two sessions minutes apart with no config change at all.

Cost: decode measured 23.11 tok/s before and 23.94 after (mean of 3 natural chat completions, max_tokens 2048, temp 0.3) — inside run-to-run variance, so read it as "no measurable cost", not a gain. MTP stays enabled and the KV pin is unchanged at 573,440 tokens, so unlike disabling speculative decoding there is no decode penalty.

Scoring caveat that cost us a wrong conclusion: a "fewer than 3 English words" garbage heuristic false-positives on legitimately terse valid replies ("Got it.", "Test received.") and made a clean pre-fix run look like a failure. Flag \x00 bytes and CJK/Hebrew characters instead.

What made this hard to find, recorded because it will mislead the next person:

• It is not prompt length. 43,063 tokens of filler is clean; 33,254 tokens of the real payload is garbage.
• It is not tool count. 31 synthetic tools is clean. 30 real tools plus a duplicate, at a higher token count, is clean.
• It looks content-specific and position-dependent: the identical 31 tools at the identical 33,254 tokens are garbage with write last and clean with write first, and adding one trailing space to a tool description fixes it. That is because prefix caching means the number of tokens actually scheduled for prefill is the uncached residual, which depends on content rather than total length. Prefix caching is lossless and is not the defect — it is what determines the number that trips the shape check.
• Because agent clients sort tools alphabetically, write lands last every time, which is why this reproduces on every request rather than intermittently.

Caveat. The naive trigger in #53051 (a prefill of exactly 1+k tokens) does not reproduce on v0.27.2rc1.dev77+gac7509e2b; a 3-token prompt and cached residuals of 1-6 tokens were all clean both before and after. The exact offending shape on this build is unidentified. PIECEWISE eliminates the failure regardless, but the standing rule for this box is: replay the stored repro body after any image bump or flag change.

What is not measured

No quality evaluation, no head-to-head. 16 tok/s says nothing about whether the outputs are better than the faster checkpoints above. Video input is configured via --media-io-kwargs but untested here.

---

---

Why this repo exists

The goal is a vLLM configuration for DGX Spark that's actually easy to set up — and to keep it in one place.

Getting this working took piecing together information scattered across NVIDIA developer forum threads, GitHub issues in three different repos, Hugging Face model cards, a community Docker project, and a lot of trial and error. Several of the decisive details — which checkpoint actually performs well, that one environment variable the int4 kernels need, why the driver version matters, the Secure Boot trap — each lived somewhere different, and none of them were obvious from the official docs. Some of it I only learned by breaking things.

I didn't want the next person to repeat that search, so this is everything in one spot: the working configuration, the reasoning behind each setting, real benchmark numbers, and an honest account of every dead end.

I don't assume this is the best possible configuration. It's the best one I found, measured on my hardware, at a moment when this whole stack is moving fast. So:

• Know a better setup? Please tell me. A different checkpoint, better flags, a runtime I dismissed too quickly — open an issue or a PR.
• Something here wrong or outdated? Corrections are the most valuable contribution. Several claims in the first draft of this repo were wrong and got fixed before publishing; assume more will need fixing.
• Got numbers from your own Spark? Even one data point on a different model, driver, or concurrency level makes this more useful.

See CONTRIBUTING.md for what's especially wanted and how to report results comparably. Open questions I'd genuinely like answers to are tracked in Known gaps below.

---

Table of contents

• Qwen3.8-27B: results, stated honestly
• ⚠ The default CUDA graph mode produces garbage output
• Why this repo exists
• TL;DR
• Verified environment
• The critical thing to know first
• Setup
• Configuration explained
• Also covered: Qwen3.6-35B-A3B-NVFP4
• Also covered: Qwen3-Coder-Next 80B-A3B
• Connecting a client
• Known gaps
• Further reading

---

TL;DR

The primary model. For the other two, see their sections below.

# 1. Driver 590+ is REQUIRED for CUDA graphs on GB10 (2.8x single-stream speed).
#    ⚠ If Secure Boot is enabled, read §1 BEFORE running this — the driver
#    metapackage pulls DKMS, which hangs on an unanswerable password prompt.
#    You need Canonical's PRE-SIGNED modules for your target kernel.
sudo apt-get install -y nvidia-driver-595-open \
                        linux-modules-nvidia-595-open-<TARGET_KERNEL>
sudo reboot

# 2. Serve scripts (MiaAI-Lab, MIT). NOTE: this model runs on UPSTREAM's
#    nightly-aarch64 image, not the eugr/spark-vllm container the other two use.
git clone https://github.com/MiaAI-Lab/Qwen3.8-27B-DGX-Spark-RTX-6000
cd Qwen3.8-27B-DGX-Spark-RTX-6000

# 3. Weights: a NATIVE NVFP4 checkpoint. NOT a GGUF. (~22.6 GB)
./download.sh

# 4. Serve. Edit start.sh first — see "Three changes I made" below; upstream's
#    0.84 / 1M-YaRN defaults livelocked this box.
./start.sh

Full serve command that produces the benchmark numbers:

vllm serve unsloth/Qwen3.8-27B-NVFP4 \
  --served-model-name qwen38-27b-unsloth-nvfp4 \
  --host 0.0.0.0 --port 8000 \
  --tensor-parallel-size 1 \
  --trust-remote-code \
  --quantization compressed-tensors \
  --attention-backend triton_attn \
  --gpu-memory-utilization 0.65 \
  --max-model-len 262144 \
  --max-num-seqs 4 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill --enable-prefix-caching --skip-mm-profiling \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice \
  --kv-cache-memory=21474836480 \
  --media-io-kwargs '{"video": {"num_frames": -1}}' \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 2}' \
  --compilation-config '{"cudagraph_mode": "PIECEWISE"}'

(run in vllm/vllm-openai:nightly-aarch64 with --restart unless-stopped, which MiaAI-Lab's start.sh does once you apply the changes documented below)

---

Verified environment

Every number in this repo was measured on exactly this stack:

Component	Version
Hardware	NVIDIA DGX Spark, GB10, 121 GiB unified memory
Architecture	aarch64 (sm_121, compute capability 12.1)
OS	Ubuntu 24.04.4 LTS
Kernel	6.17.0-1029-nvidia
NVIDIA driver	595.84 (open kernel modules, Canonical-signed)
Docker	29.1.3
Container	eugr/spark-vllm @ 81a33f3
vLLM	0.26.1rc1.dev30+g5773c4e60
Model	Intel/Qwen3-Coder-Next-int4-AutoRound (~41 GB)
Secure Boot	Enabled (this matters — see below)

---

The critical thing to know first: GGUF + vLLM + this model

Do not try to serve this model in vLLM from a GGUF file. It cannot work, and it fails destructively.

The chain of reasons:

1. vLLM removed in-tree GGUF quantization (PR #39612, merged 2026-06-12, first absent in v0.24.0), moving it to a separate vllm-gguf-plugin package.
2. That plugin has no support for the Gated-DeltaNet hybrid architecture Qwen3-Coder-Next uses. As of 2026-07-28 the plugin tree contains no reference to qwen3next at all, and its tested-coverage table lists no GDN hybrid. The nearest tracking issue is #80, filed for the sibling Qwen3.5/3.6 MoE architectures — note these are distinct GGUF architecture ids (qwen3next vs qwen35/qwen35moe), so #80 does not cover Qwen3-Next specifically, but the maintainer's reply there ("It's on my plan but don't have enough bandwidth to implement it") indicates the general state of GGUF MoE work. Related: PR #87 (closed unmerged), PR #91 (open).
3. With no quantization method registered, vLLM's FusedMoE layer silently falls back to UnquantizedFusedMoEMethod — it dequantizes all 512 expert tensors to bf16 rather than erroring out.

The arithmetic is unforgiving:

Qwen3-Coder-Next	80B total params (3B active)
Q5_K_M on disk	~0.6875 bytes/param → ~56 GB
Dequantized to bf16	2 bytes/param → ~160 GB
DGX Spark capacity	121 GB

So it cannot fit, at any --gpu-memory-utilization or --max-model-len setting. Those flags govern KV cache and activations; this blows up earlier, while materializing model weights.

Worse, the failure is not graceful. In a Docker Compose setup with restart: unless-stopped, vLLM OOMs, Docker restarts it, and it climbs back to ~117 GB before dying again — indefinitely. Observed consequences on our box: SSH became unresponsive for hours, and every other container on the machine (vector DB, web UIs, an Ollama instance) was OOM-killed and restarted.

If you are here because that is happening to you right now:

docker compose down     # NOT `stop` — restart policy will resurrect it

The fix is to use a natively quantized checkpoint (int4-AutoRound, AWQ, FP8, NVFP4) instead of GGUF. That is what this repo does.

If you specifically want GGUF, use llama.cpp, which does genuine quantized inference — see docs/ALTERNATIVES.md.

---

Setup

1. NVIDIA driver — and the Secure Boot trap

Driver 590 or newer is required. On driver 580, CUDA graph capture crashes on GB10:

torch.AcceleratorError: CUDA error: an illegal instruction was encountered
  ... in _capture_cudagraphs -> _dummy_run

You can work around it with --enforce-eager, but that costs roughly 2.8× single-stream throughput (measured: 25 tok/s eager vs 70 tok/s with graphs). Upgrading the driver is the real fix.

⚠ Secure Boot: use pre-signed modules, not DKMS.

Installing nvidia-driver-595-open alone pulls in the DKMS package, which compiles kernel modules locally. Under Secure Boot those modules must be enrolled with a MOK password typed at the physical console during boot — impossible over SSH. The install will hang on an unanswerable debconf prompt:

Enter a password for Secure Boot. It will be asked again after a reboot.

If you let that complete and reboot anyway, the unsigned modules are rejected and your GPU does not come back.

Canonical ships pre-signed module packages that avoid this entirely.

First find which pre-signed module package exists. The driver metapackage may pull in a newer kernel, so check both your current kernel and the newest available -nvidia flavour:

apt-cache search 'linux-modules-nvidia-595-open' | grep -- '-nvidia'

Then install the driver and the matching pre-signed modules in one transaction, so the prebuilt signed modules are present for the kernel you will actually boot:

sudo apt-get install -y \
  nvidia-driver-595-open \
  linux-modules-nvidia-595-open-6.17.0-1029-nvidia   # ← your target kernel

If apt hangs on the Secure Boot password prompt anyway (the DKMS postinst runs update-secureboot-policy --enroll-key), it is waiting for input that can never arrive over SSH. Do not reboot. Recover with:

# find the PIDs — do NOT use `pkill -f`, it matches your own SSH command line
ps -eo pid,cmd | grep -E 'update-secureboot|apt-get install'
sudo kill -9 <PIDs>
sudo dpkg --configure -a
sudo apt-get check

Then install the pre-signed module package and verify signing (below) before rebooting. This is the path we actually took; installing both packages together is the cleaner sequence we'd recommend in hindsight.

Verify before rebooting — this check is the difference between a clean reboot and a dead GPU:

MOD=$(find /lib/modules/<TARGET_KERNEL> -name 'nvidia.ko*' | head -1)
modinfo -F signer  "$MOD"    # MUST print: Canonical Ltd. Kernel Module Signing
modinfo -F version "$MOD"    # MUST exactly match your installed userspace version

Userspace and kernel module versions must match exactly; a mismatch yields Failed to initialize NVML: Driver/library version mismatch and a non-functional GPU.

Also confirm you have a fallback kernel and that the initramfs was built:

dpkg -l | grep '^ii  linux-image-6'
ls -la /boot/initrd.img-<TARGET_KERNEL>

Then sudo reboot, and verify:

uname -r                                                   # target kernel
nvidia-smi --query-gpu=driver_version --format=csv,noheader # 595.84
cat /sys/module/nvidia/version                              # 595.84 — must match

Note: DKMS may have also built unsigned modules for your old kernel. That means booting the old kernel will now fail to load the GPU. Your fallback is the pre-signed combination, not the previous one.

2. The container — and why this model is the exception

The other two models in this repo need the community eugr/spark-vllm container, because upstream images were not built for GB10/sm_121. Qwen3.8-27B is the exception: it runs on upstream's own nightly, which now carries what GB10 needs.

git clone https://github.com/MiaAI-Lab/Qwen3.8-27B-DGX-Spark-RTX-6000
cd Qwen3.8-27B-DGX-Spark-RTX-6000
# start.sh pulls vllm/vllm-openai:nightly-aarch64 and sets CUTE_DSL_ARCH=sm_121a

If startup dies with CUDA or arch errors, confirm you are on the nightly-aarch64 tag — the container sets CUTE_DSL_ARCH=sm_121a for GB10's cutlass kernels, and a generic image will not.

3. The model

./download.sh          # unsloth/Qwen3.8-27B-NVFP4, ~22.6 GB

That is model.safetensors plus model_mtp.safetensors — the second file is what makes --speculative-config work. A GGUF build of the same model exists and is not interchangeable; see the GGUF section.

Resident weights run well above the on-disk figure. Do not size from du.

4. Serve

Edit start.sh before the first run. Upstream's defaults are tuned for a Spark with headroom and they livelocked this box — see Three changes I made.

./start.sh              # writes .vllm.pid and .vllm.log

First start takes several minutes (weight load + CUDA graph capture). Ready when:

curl -s http://<SPARK_IP>:8000/v1/models | jq '.data[].id'
# "qwen38-27b-unsloth-nvfp4"

Stop with ./stop.sh. The PID file holds the container id, so the configuration outlives the shell you started it from.

5. Start on boot

Upstream's docker run carries no restart policy, so the container does not survive a reboot or an OOM kill — the served model simply vanishes until someone notices. Add it to start.sh:

docker run -d \
  --name "${CONTAINER_NAME}" \
  --restart unless-stopped \
  ...

unless-stopped rather than always on purpose: a deliberate ./stop.sh should stay stopped. To apply it to an already-running container without downtime:

docker update --restart=unless-stopped qwen3.8-27b-nvfp4

This is the one place this deployment differs from upstream by adding something rather than dialling something down.

---

Configuration explained

Every flag in the primary serve command, and why it is there. Flags for the other two models are explained in their own sections.

Flag	Why
--quantization compressed-tensors	How Unsloth's NVFP4 checkpoint is packed. Not a GGUF, not AWQ.
--attention-backend triton_attn	Required, not chosen. FlashAttention-2 cannot serve an FP8 KV cache on GB10/sm_121 — vLLM wants FA3 on SM90 or FA4 on SM100. Only the 16 full-attention layers use it; the 48 Gated DeltaNet layers are unaffected and the vision tower still runs FA. Reverting to flash_attn forces --kv-cache-dtype bfloat16.
--gpu-memory-utilization 0.65	Lowered from upstream's 0.84. Read the warning at the top: on this hardware this flag does not bound the KV allocation, so treat it as a grant, not a cap.
--max-model-len 262144	The checkpoint's native window. Upstream reaches 1M with static YaRN; that reserved ~80 GB of KV for 1–4% measured occupancy and livelocked the box.
--max-num-seqs 4	Four concurrent sequences. This is a single-user config; the 80B section below is the one tuned for 16.
--enable-prefix-caching	Reuses KV across requests sharing a prefix. Large win for agent loops that resend a system prompt.
--skip-mm-profiling	Skips multimodal memory profiling at startup. Saves several minutes of boot on a vision-capable checkpoint.
--reasoning-parser qwen3	Separates thinking from the answer. Note: this vLLM build returns it in the reasoning field, not reasoning_content.
--tool-call-parser qwen3_coder + --enable-auto-tool-choice	Working tool calls. Verified against a two-tool schema.
--media-io-kwargs '{"video": {"num_frames": -1}}'	Video input, all frames. Configured but untested here.
--speculative-config '{"method": "mtp", "num_speculative_tokens": 2}'	Uses the MTP heads shipped inside the checkpoint. No external drafter exists for this model.

A flag that is conspicuously absent: --kv-cache-memory. Everywhere else in this repo that pin is the control that actually binds — on the Qwen3.6 recipe it took the box from 27 GB free to 68 GB free with no loss of usable capacity. This config inherits upstream's flag set, which omits it, and runs at 83 GB resident with roughly 4 GB free. It is stable at the native context, but stable by having less to cache, not because anything is bounding it.

---

Also covered: Qwen3.6-35B-A3B-NVFP4

Same box, different checkpoint, and the fastest single-stream generation measured here. Added 2026-08-03; superseded as the primary model on 2026-08-16 but kept in full because it is still the speed record on this hardware and its KV-cache section is the most important operational writing in this repo.

	Qwen3-Coder-Next 80B-A3B (int4-AutoRound)	Qwen3.6-35B-A3B (NVFP4)
Total / active params	80B / 3B	35B / 3B
Single stream generation	~70 tok/s	102.3 tok/s
How measured	scripts/benchmark.py, short prompts	llama-benchy 0.4.0, n=3, std 6.5
Weights on disk	~41 GB	22 GB

These are not equivalent measurements. The 70 tok/s figure is a burst run with short prompts; the 102.3 figure is a harness run at a 30,000 token prompt with three repetitions and a reported standard deviation. Both are honest, neither is directly comparable to the other, and we did not re-run Qwen3-Coder-Next under benchy. Treat the comparison as indicative.

Measured

llama-benchy 0.4.0 by Eugene Rakhmatulin, --pp 30000 --tg 2000 --concurrency 1 --latency-mode generation, prefix caching disabled for the run, 3 repetitions. Coherence test passed.

Metric	Mean	Std	Raw values
Generation	102.33 tok/s	6.50	110.72, 101.37, 94.89
Prefill	5,430.3 tok/s	71.1	5,481.4, 5,329.8, 5,479.8
Peak	127.0 tok/s	5.1	132, 129, 120
TTFT @ 30K prompt	5,583.9 ms	73.0	5,531.3, 5,687.2, 5,533.2

Raw output: benchmarks/raw/qwen36-35b-a3b-nvfp4-benchy.json.

Prefill runs the other way. Generation is faster than the 80B checkpoint, but prefill is not the strength here. If your workload sends long prompts, the generation advantage does not translate one for one. In a real application that sends 30K to 65K token prompts, a build measured 6.6 minutes against a 7.9 minute baseline on a cloud model, not the 4x the tok/s ratio implies.

Serve it

Recipe: recipes/qwen36-35b-a3b-nvfp4-solo.yaml.

vllm serve nvidia/Qwen3.6-35B-A3B-NVFP4 \
  --tensor-parallel-size 1 \
  --kv-cache-dtype fp8 \
  --kv-cache-memory 17179869184 \
  --attention-backend flashinfer \
  --moe-backend marlin \
  --gpu-memory-utilization 0.50 \
  --max-model-len 131072 \
  --max-num-seqs 4 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill --async-scheduling --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}' \
  --load-format fastsafetensors \
  --reasoning-parser qwen3 --tool-call-parser qwen3_xml --enable-auto-tool-choice

⚠ --gpu-memory-utilization did not bound the allocation

This is the part worth reading even if you never run this model.

On vLLM v0.26.1rc1.dev30+g5773c4e60, --gpu-memory-utilization 0.50 granted 60.84 GiB and vLLM then allocated ~101 GiB anyway, because it sizes the KV cache from free memory rather than from the grant. vLLM's own startup line says so:

Free memory on device (113.35/121.69 GiB) on startup.
Desired GPU memory utilization is (0.5, 60.84 GiB).
Actual usage is 46.14 GiB for consumed memory (weights + non-torch) ...
Current kv cache memory in use is 54.75 GiB.

--kv-cache-memory is the flag that actually binds. Pinning it to 16 GiB took the box from 27 GB free to 68 GB free with no loss of usable capacity.

Two more numbers that surprised us:

• Resident weights were 46.14 GiB against 22 GB on disk, roughly 2.1x. Do not size from du.
• KV cost was 14,188 bytes per token, not the ~40 KiB a naive calculation predicts from 40 layers x 2 KV heads x head_dim 256 at fp8. This checkpoint is hybrid attention, so only a subset of layers carry a KV cache. Derive it by dividing the logged GiB by the logged token count (54.75 GiB / 4,143,236 tokens), never from the layer count.

Sizing that follows from those measurements: max_num_seqs 4 x max_model_len 131072 = 524,288 tokens = 6.93 GiB of genuinely usable KV. The 16 GiB pin is 2.3x that, with the surplus serving the prefix cache. After pinning, vLLM reported 1,210,288 KV tokens and Maximum concurrency for 131,072 tokens per request: 9.23x.

⚠ We crashed the box getting here

First attempt set --gpu-memory-utilization 0.85, reasoned by analogy from a two node recipe where 0.4 was sized for two nodes sharing. The Spark went unresponsive during weight load. Ping answered, sshd could not accept connections, every service died, and it hard rebooted.

GB10 memory is unified. A GPU fraction comes from the same 121 GB pool the OS and every container uses. 0.85 reserves ~103 GB and leaves ~18 GB for the operating system and nine running containers.

The rule we now follow: never set a memory or capacity value by copying it from another recipe. Measure free memory on the target first, do the arithmetic, write it in the recipe comment, and start low when uncertain. See §6 "Free the memory first".

One tuning note that costs 50x

Qwen3.6 is a reasoning model. Measured on this box, a two token answer costs 101 completion tokens with thinking enabled and 2 with it disabled. Left unset against a 32,000 token budget, a stage can exhaust its budget and return empty content. Set it explicitly per request:

{"chat_template_kwargs": {"enable_thinking": false}}

---

Also covered: Qwen3-Coder-Next 80B-A3B

The original subject of this repo, and still the right choice for multi-user serving — it is the only model here measured across 16 concurrent users. Kept in full. Verified 2026-07-28.

What you get	Number	What it means
Solo user	~70 tok/s	The number that matters if it's just you. Comparable to hosted APIs.
Each of 16 concurrent users	~22 tok/s	What a person actually experiences under full load.
Whole-box aggregate at 16 users	~350 tok/s	16 × 22 added together. Capacity planning only.

Plus 131K context and working tool-calling, on one Spark.

Don't read "350 tok/s" as a speed. It's a sum across 16 people, not something anyone experiences. Aggregate throughput is a capacity measure — it answers "can this box serve my team?", not "how fast does it feel?" For reference: humans read at roughly 4–5 tok/s, so 22 tok/s outruns reading comfortably in chat, but a 300-line file (~4,000 tokens) still takes about three minutes to generate.

Concurrent figures are confirmed by vLLM's own server-side accounting, not just client-side arithmetic — and here's why that distinction matters more than we expected.

Verified 2026-07-28.

---

Setup (Qwen3-Coder-Next)

Steps 2 through 6. Step 1, the driver and Secure Boot, is shared and lives above.

2. The container

Upstream vllm/vllm-openai images are not built for GB10/sm_121. Use the community container, which carries the necessary patches:

git clone https://github.com/eugr/spark-vllm-docker
cd spark-vllm-docker
./build-and-copy.sh          # pulls a tested nightly image (~22 GB)

3. The model

python3 -c "from huggingface_hub import snapshot_download; \
  print(snapshot_download('Intel/Qwen3-Coder-Next-int4-AutoRound'))"

~41 GB. See docs/ALTERNATIVES.md for why this checkpoint over FP8/NVFP4/AWQ.

4. Serve

cp recipes/qwen3-coder-next-tuned.yaml <spark-vllm-docker>/recipes/
cd <spark-vllm-docker>
python3 run-recipe.py --dry-run --solo qwen3-coder-next-tuned   # inspect first
python3 run-recipe.py --solo qwen3-coder-next-tuned

First start takes several minutes (weight load + CUDA graph capture). Ready when:

curl -s http://<SPARK_IP>:8000/v1/models | jq '.data[].id'
# "Intel/Qwen3-Coder-Next-int4-AutoRound"

5. Start on boot

From this repo's root (not the spark-vllm-docker checkout you were in for step 4):

sudo cp systemd/vllm-coder.service /etc/systemd/system/
sudo sed -i "s/__USER__/$USER/g" /etc/systemd/system/vllm-coder.service
sudo systemctl daemon-reload
sudo systemctl enable --now vllm-coder.service

The unit's user must be in the docker group (sudo usermod -aG docker $USER). Edit WorkingDirectory/ExecStart if your spark-vllm-docker checkout isn't at /home/<user>/spark-vllm-docker.

6. Free the memory first ⚠

vLLM pre-allocates its --gpu-memory-utilization share (0.8 ≈ 97 GB) and refuses to start if that much isn't free:

ValueError: Free memory on device cuda:0 (24.7/121.69 GiB) on startup is less
than desired GPU memory utilization (0.8, 97.35 GiB)

On a 121 GB unified-memory box, nothing else substantial can be resident. Check for other model servers before starting:

nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
systemctl list-units --type=service --state=running | grep -iE 'llama|ollama|vllm'

Note that unified memory means CPU-side allocations count too. Also see vLLM issue #35313: on UMA systems vLLM can under-report free memory because it doesn't account for reclaimable page cache — dropping caches or lowering utilization to 0.72–0.85 helps.

---

Quick start (Qwen3-Coder-Next)

# 1. Driver 590+ is REQUIRED for CUDA graphs on GB10 (2.8x single-stream speed).
#    ⚠ If Secure Boot is enabled, read §1 BEFORE running this — the driver
#    metapackage pulls DKMS, which hangs on an unanswerable password prompt.
#    You need Canonical's PRE-SIGNED modules for your target kernel.
sudo apt-get install -y nvidia-driver-595-open \
                        linux-modules-nvidia-595-open-<TARGET_KERNEL>
sudo reboot

# 2. Community GB10-patched vLLM container
git clone https://github.com/eugr/spark-vllm-docker && cd spark-vllm-docker
./build-and-copy.sh

# 3. Model: a NATIVE quantized checkpoint. NOT a GGUF. (~41 GB)
python3 -c "from huggingface_hub import snapshot_download; \
  snapshot_download('Intel/Qwen3-Coder-Next-int4-AutoRound')"

# 4. Serve
cp /path/to/this-repo/recipes/qwen3-coder-next-tuned.yaml recipes/
python3 run-recipe.py --solo qwen3-coder-next-tuned

Full serve command that produces the benchmark numbers:

VLLM_MARLIN_USE_ATOMIC_ADD=1 \
vllm serve Intel/Qwen3-Coder-Next-int4-AutoRound \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --gpu-memory-utilization 0.8 \
  --host 0.0.0.0 --port 8000 \
  --load-format fastsafetensors \
  --enable-prefix-caching \
  --max-model-len 131072 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 16384

(run inside the eugr/spark-vllm container, with the mods/fix-qwen3-next-autoround mod applied — the recipe file does both)

---

Configuration explained (Qwen3-Coder-Next)

Every non-obvious flag, and why:

Flag	Value	Why
--load-format fastsafetensors		Much faster weight loading
--tool-call-parser qwen3_coder		Model-specific parser; required for structured tool_calls
--enable-auto-tool-choice		Lets the model decide when to call tools
--enable-prefix-caching		Big win for agents that resend a shared system prompt
--max-model-len	131072	Deliberately reduced from 262144. Context and concurrency compete for the same KV cache; 262K starves it and admits only ~2–4 sequences
--max-num-seqs	16	Concurrent sequence ceiling. vLLM's DGX Spark blog suggests 4, but for Nemotron-3-Super-120B-A12B — a 12B-active MoE. With only 3B active per token this model sustains far more concurrent decode streams
--max-num-batched-tokens	16384	Raised from the 8192 default for long agent prompts
--gpu-memory-utilization	0.8	≈97 GB. Leaves ~18 GB for the OS and other containers
VLLM_MARLIN_USE_ATOMIC_ADD=1		Required for int4-AutoRound Marlin kernels on this hardware
mods/fix-qwen3-next-autoround		Container mod fixing AutoRound weight loading
(no --enforce-eager)		CUDA graphs on — worth ~2.8× single-stream. Requires driver ≥590

The context ↔ concurrency tradeoff is the main tuning dial. If your agents genuinely need >131K context, either accept fewer concurrent users or add --kv-cache-dtype fp8 to roughly halve KV memory — but see the warning below before doing so.

---

Results (Qwen3-Coder-Next)

Sustained throughput — scripts/sustained_load.py, 16 concurrent long generations (max_tokens=1500, 24,000 tokens over ~67 s):

Accounting	Result
Client-side	352–360 tok/s (two runs)
vLLM's own server-side logger	342–381 tok/s steady state, Running: 16 reqs

Per-request under that load: ~22 tok/s each, 16 users in parallel.

Burst throughput — scripts/benchmark.py, short prompts (max_tokens=250, temperature=0.2):

Concurrency	Aggregate	Per-request	Scaling efficiency
1	~70 tok/s	70 tok/s	—
4	~150 tok/s	~47 tok/s	2.1×
8	~200 tok/s	~37 tok/s	2.8×
16	~270 tok/s	~24 tok/s	3.9×

These measure different things — don't conflate them. The burst numbers run only ~9 s, so prefill ramp-up and the ragged completion tail dominate the wall clock and understate throughput. Under genuinely sustained load the same box does ~350 tok/s at the same concurrency. Use the sustained figure for capacity planning; use the burst per-request figures for latency expectations. Details in BENCHMARKS.md.

Driver/mode comparison, same model and hardware:

Configuration	Single stream	4 concurrent
Driver 580 + --enforce-eager	25 tok/s	67 tok/s
Driver 595 + CUDA graphs	70 tok/s	149 tok/s
llama.cpp, Q5_K_M GGUF (reference)	46 tok/s	not measured

llama.cpp's llama-server does have continuous batching (--cont-batching, on by default; -np sets parallel slots) — we simply did not benchmark it under concurrency, because it was running with -np 2 as an interim single-user setup. Do not read the blank cell as "it can't." Contributions welcome.

Practical capacity: 4–8 concurrent coding-agent users is the comfortable range, each still seeing 37–47 tok/s. 16 works, at ~24 tok/s each.

16 is a number we chose, not a ceiling we hit. --max-num-seqs 16 was picked to stress-test with; we never found the real limit.

Memory isn't what stops you — vLLM reports ~2.3M tokens of KV cache, which at realistic 4–8K-token agent requests is room for 280–560 sequences. Diminishing returns stop you first: 1 user → 70 tok/s, 4 → 149, 8 → 196, 16 → 273. Each doubling adds less than the last, because the constraint is memory bandwidth (the weights must be read for every token generated), not memory capacity.

So the useful question isn't "where does it break" but "where does per-person speed stop being pleasant" — around 8 here. Full numbers in BENCHMARKS.md.

Caveats worth knowing:

• The first request after idle can be slow (~5 tok/s) — cache/graph warmup, not representative. Discard it when benchmarking.
• Watch TTFT, not error rate, as your saturation signal.
• vLLM has no batch-invariant kernel path for Gated DeltaNet (GDN_ATTN), so bitwise-reproducible output across batch sizes isn't available — VLLM_BATCH_INVARIANT=1 hard-fails at startup (vLLM #42960, open feature request). We did not attempt to quantify output variation across concurrency levels.

Full methodology and raw numbers: docs/BENCHMARKS.md.

---

---

Connecting a client (OpenCode etc.)

Standard OpenAI-compatible endpoint — http://<SPARK_IP>:8000/v1, model id Intel/Qwen3-Coder-Next-int4-AutoRound, no API key.

Tool calling verified working:

curl -s http://<SPARK_IP>:8000/v1/chat/completions \
  -H 'Content-Type: application/json' -d '{
  "model":"Intel/Qwen3-Coder-Next-int4-AutoRound",
  "messages":[{"role":"user","content":"What is the weather in Kansas City?"}],
  "tools":[{"type":"function","function":{"name":"get_weather",
    "parameters":{"type":"object","properties":{"city":{"type":"string"}},
    "required":["city"]}}}]}' | jq '.choices[0].message.tool_calls'

[{"id":"chatcmpl-tool-...","type":"function",
  "function":{"name":"get_weather","arguments":"{\"city\": \"Kansas City\"}"}}]

OpenCode provider config (~/.config/opencode/opencode.jsonc)

{
  "provider": {
    "spark-vllm": {
      "name": "Spark Qwen3 Coder (vLLM)",
      "npm": "@ai-sdk/openai-compatible",
      "options": { "baseURL": "http://<SPARK_IP>:8000/v1" },
      "models": {
        "Intel/Qwen3-Coder-Next-int4-AutoRound": {
          "name": "Qwen3 Coder Next (int4)",
          "tools": true,
          "limit": { "context": 131072, "output": 32768 }
        }
      }
    }
  }
}

Keep limit.context in sync with --max-model-len. Note OpenCode merges config.json and opencode.jsonc if both exist — a stale config.json will silently reintroduce old model ids.

---

Known gaps

Things this repo does not answer. If you can close any of these, that's the most useful contribution available — open an issue or PR.

Untested but promising

• MTP speculative decoding (qwen3_next_mtp). Potentially another 1.4–2×. The commonly-cited blocker (vLLM #40880) turns out to be closed and specific to the TurboQuant KV backend, which this config doesn't use — so it's probably not blocked here. Nobody has posted results on a Spark. This is the single most promising unexplored lead.
• --kv-cache-dtype fp8. Would roughly halve KV memory, buying either more context or more concurrency. Contested: NVIDIA's own recipe and several community setups enable it, but vLLM's DGX Spark blog explicitly advises against it on this hardware — "may affect model predictability and can carry a noticeable performance cost on Spark for some workloads; avoid it unless memory pressure requires it and quality checks pass." We have not measured it either way. If you try it, measure both speed and output quality.
• The real concurrency ceiling. We picked --max-num-seqs 16 arbitrarily and never found where it actually tops out. KV cache has room for hundreds of short sequences (~2.3M tokens), so the binding constraint is memory bandwidth and diminishing returns rather than memory capacity. Someone testing 32 and 64 would close this — the marginal gain per doubling was already shrinking (+113% at 4 users, +32% at 8, +39% at 16), so the interesting number is where per-user speed stops being usable, not where it errors.

Measured incompletely

• llama.cpp under concurrency. We only measured it single-stream (46 tok/s at -np 2). llama-server does have continuous batching, so the runtime comparison in BENCHMARKS.md is one-sided. A matched-context, matched-slot comparison would settle it.

• Long-context behavior. All benchmarks used short prompts. Real coding agents send large contexts, which shifts the bottleneck to KV cache — expect fewer concurrent users than our numbers imply, but we haven't quantified it.

• Output quality — the biggest gap in this repo. We verified correctness informally (the model writes working code and emits valid tool calls) but ran no benchmark comparing int4-AutoRound against Q5_K_M or the FP8 checkpoint. 4-bit is 4-bit; if your work is precision-sensitive, evaluate it yourself.

  Worth taking seriously: at least one practitioner running Qwen3.6 on a DGX Spark argues that 3B-active MoE models trade away too much quality for speed — "they only generate garbage faster... generating wrong code at a faster pace makes everything else slower" (video). That's one opinion about a different model, not a measurement of ours — but this model is also 3B-active, and everything in this repo optimizes throughput. A tokens/sec win is worthless if it costs you correctness, and we have no data either way. Quality benchmarks would be the most valuable contribution here.

• Tool-calling under a real agent loop. We validated tool calls with a single trivial function (a weather query) — which proves the parser works, not that the model is reliable across a long agentic session. The same source above shows this model class emitting malformed tool calls that a harness had to intercept and repair. If you're driving this with a coding agent, expect to need retry/repair logic, and treat our tool-calling check as a smoke test rather than validation.

• Prefix caching accuracy. Enabled here and it works, but it's flagged experimental for Mamba/GDN hybrids, and at least one community recipe disables it for accuracy reasons. We didn't A/B it.

Second-hand claims we couldn't verify

Some statements in ALTERNATIVES.md and TROUBLESHOOTING.md come from forum reports rather than our own testing — notably that TensorRT-LLM fails on GB10, that multi-Spark clustering underperforms single-node, and the throughput figures for checkpoints we didn't run. They're labeled as reported rather than measured, but treat them accordingly.

Moving targets

• Whether vllm-gguf-plugin gains GDN-hybrid support (would make GGUF viable on vLLM).
• Whether NVFP4 overtakes int4 once SM121 CUTLASS MoE kernels mature.
• SGLang on GB10 for this model class — plausible, unvalidated.

---

Further reading

• docs/TROUBLESHOOTING.md — every error we hit,
• docs/CONTEXT-262K.md — doubling context to 262,144: measurements, config drift found, and full rollback with root cause and fix. Start here when something breaks.
• docs/BENCHMARKS.md — methodology, raw numbers, expected scaling curves from other operators.
• docs/ALTERNATIVES.md — checkpoint comparison (int4-AutoRound vs FP8 vs NVFP4 vs AWQ), llama.cpp as an alternative, MTP and speculative decoding status, why TensorRT-LLM is a dead end here.
• docs/FEATURES.md — structured output (grammar engine), prefix caching, and n-gram speculative decoding: what each one actually speeds up, measured numbers, and when to enable which. Includes the silent-failure trap where old guided_* parameters are ignored.
• benchmarks/ — raw unedited output backing every throughput number here, including vLLM's own server-side logs. Don't take our word for the numbers; check the evidence.
• docs/SOURCES.md — every source consulted, with links and provenance labels (verified / reported-by-others / background). This is the "one spot" the repo exists to create.
• appendix/ — a monkey-patch that gets vLLM past its GGUF config-parsing bugs. Documented for completeness; it does not make GGUF serving viable (you still hit the 160 GB wall), but the two upstream bugs it works around affect any local-path model.

Credits

Author: dentity007 (Nathan Maine) — github.com/dentity007 · github.com/NathanMaine. The measurements, the hardware, the failures, and the conclusions are mine. Drafting and editing were AI assisted; the specification, the acceptance bar, and every number published here are not.

• Eugene Rakhmatulin — twice over, and this repo would not exist without either:
  • eugr/spark-vllm-docker — the GB10-patched vLLM container and recipe system this builds on.
  • eugr/llama-benchy — the benchmarking tool behind every Qwen3.6 number published here (v0.4.0, MIT). A llama-bench style harness that works across backends, reports standard deviation across repetitions, and runs a coherence check before it will report a result. The reason the 102.3 tok/s figure carries an n and a std instead of being one hopeful curl call.
• Unsloth — the unsloth/Qwen3.8-27B-NVFP4 quantization, including the model_mtp.safetensors MTP weights that the speculative decoding in that section depends on. The quantization is entirely their work; I only measured it.
• MiaAI-Lab — Qwen3.8-27B-DGX-Spark-RTX-6000 (MIT), the serve scripts behind the Qwen3.8 section. The container name, image, triton_attn backend, --skip-mm-profiling and MTP-2 config are theirs unchanged; I changed three settings and documented why in that section. Their README is also where the FP8-KV-on-SM121 constraint and the GB10-vs-RTX-6000 memory guidance are explained properly.
• The NVIDIA DGX Spark developer forum community, whose posts identified the int4-AutoRound checkpoint and the VLLM_MARLIN_USE_ATOMIC_ADD requirement.

License

MIT — see LICENSE.

Not affiliated with NVIDIA, Qwen/Alibaba, Intel, or the vLLM project.