Repository navigation
[NVIDIA][AgentX] Retune GB300 DSV4.1-Flash Dynamo + SGLang HiCache disagg points on the 20261030 nightly / [NVIDIA][AgentX] 在 20261030 nightly 上重新调整 GB300 DSV4.1-Flash Dynamo + SGLang HiCache 分离式点位 - #4673
Conversation
|
Thanks for the contribution!
中文感谢你的贡献！
|
Add 5P bs48 c3615, 6P bs64 c4370 and 7P bs96 c5540 with the decode cap at the P/D equilibrium batch, and 7P c5800 and 9P c7480 at cap 192 with total_requests decode balancing. Drop 8P c4096. Each point's bound recipe matches the measured run's resolved config apart from the unused Mooncake services.
…ys to the 20261030 nightly nightly-dev-cu13-20261030-302b78cc (index digest d4b7701d) bundles SGLang 52604e1d46, 181 commits after 586355a8, and Dynamo 1.7.0.dev20261029. Recipes are unchanged; the points are being re-measured on this image before the sweep.
In 3600 s runs on 586355a8, 7P BS 96 c5540 gave 1,031,199 tok/s/GPU at p90 interactivity 155.2 and 7P BS 192 c5800 gave 1,029,432 at 127.1.
8542d92 to
b8510e8
Compare
weireweire
changed the title
…t covers Root cause: #4679 added agg TP2/EP2 c16 back to fill the interactivity gap between agg TP4 c16 and disagg 5P c2884, at ~58k tok/s/GPU. Fix: add 3 x DEP2 prefill + DEP16 decode at c1024 (decode CUDA graph batch 20, max-running-requests 320, extra slots 30) and remove override_tp2_c16 and its search-space entry. Validation: 302b78cc fast run with this recipe: ~320k tok/s/GPU projected full at conv p90 288-294 (agg TP2/EP2 c16 published: 57,670 @ 292.4) and E2E-normalized p90 107.9 vs 91.4.
…-gb300-sgl-disagg-pareto # Conflicts: # inferencex-e2e/perf-changelog.yaml
…tivity range Root cause: the disagg key carried neighbouring points that add little to the curve (5P c3615 next to 5P c2884, 9P batch 160 next to 9P batch 192, and the new 3P c1024 next to 3P c1228). Fix: drop override_5p1d_dep2_dep16_b48_c3615, override_9p1d_dep2_dep16_b160_c7480 and override_3p1d_dep2_dep16_b20_c1024 with their search-space entries, and rewrite this PR's changelog entry for the final point set. Validation: rule 18(g) on mean interactivity and tput per GPU with the 302b 3600 s fulls: published 9P batch 160 lies under the 7P c5540 - 9P batch 192 line (+0.6%), agg TP2/EP2 c16 under the agg TP4 c16 - 3P c1228 line, and 8P c4096 under the 5P - 6P line; 5P bs48 cost 0.6% of curve at mean 199.7.
weireweire
added
the
full-sweep-enabled
…-gb300-sgl-disagg-pareto # Conflicts: # inferencex-e2e/perf-changelog.yaml
|
Claude finished @weireweire's task in 32s —— View job Review
LGTM - no blocking issues found What I checked
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
I reviewed this PR and found no bugs. nvidia-master.yaml and the gb300 srt-slurm recipes have designated CODEOWNERS, so one of them should still look over the retuned points.
What was reviewed:
- Every srt-recipe reference in the two master-config keys points to an override that exists. Prefill num-worker (6/7/9) matches the recipe workers, and the node counts (3/4/inherited 5) have room for those DEP2 workers.
- The removed overrides (override_tp2_c16, override_8p1d_dep2_dep16_c4096, b160_c7480) are no longer referenced anywhere. Image digests, the dynamo-router version and the recipe identity.frameworks pins agree across both keys.
- The perf-changelog block is appended only at the end, lists both config keys, and says the base extra-slots of 256 still applies, which matches the recipe.
Extended reasoning...
The change is benchmark config only: two GB300 DSV4.1-Flash Dynamo + SGLang srt-slurm variant recipes, their matching nvidia-master.yaml search-space rows and image/router pins, and one appended perf-changelog entry. It touches no security-sensitive code. The recipe and master config agree, and no dangling override references remain. I deferred instead of approving because .github/CODEOWNERS assigns nvidia-master.yaml and the gb300 srt-slurm recipe paths to specific NVIDIA owners. The tuning choices (decode batch caps, removed points, total_requests balancing) are also performance decisions those owners should confirm.
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=41581664540 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=41581664540 |
…-gb300-sgl-disagg-pareto # Conflicts: # inferencex-e2e/perf-changelog.yaml
3x DEP2 prefill + one DEP16 decode at c700 (decode CUDA graph batch 18, max-running-requests 288, extra slots 27) sits right of agg TP2/EP2 c16 on p90 interactivity: 299.2 vs 292.4, at about 3.8x its throughput per GPU (302b78cc, 1200 s fast run, 22 GPUs).
…-gb300-sgl-disagg-pareto # Conflicts: # inferencex-e2e/perf-changelog.yaml
The point between 5P c2884 and 7P c5540 is not needed for the curve. The published 8P c4096 (803,198 @ mean 191.4) stays covered: the line between 5P c2884 and 7P c5540 gives about 837,000 there.
|
@nvpohanh sweep finished, please have a look |
|
[by Claude Code] The SGLang cookbook integration of this PR's AgentX configs ( This PR has no authorized |
|
[by Claude Code] sgl-project/sglang#48392 has been merged. This PR's AgentX configs ( This PR has no authorized |
|
/use 41581664540 |
|
@nvpohanh staged run 41581664540. This run remains available across future |
|
As a PR reviewer and CODEOWNER, I have reviewed this and have: Checklist items
Additional detail section:
Signed: |
✅✅✅ Verdict: PASS ✅✅✅Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — ✅ Check 1 (Sweep on in-PR commit): PASS — Run Sweep 41581664540 ( ✅ Check 2 (Evals): PASS — ✅ Check 3 (Upstream recipe): PASS — sgl-project/sglang#48392 MERGED 2026-11-04T12:42:12Z publishes ✅ Check 4 (Reuse command): PASS — ✅ Check 5 (Checklist template): PASS — all 20 items from the current template are present and checked. ✅ Check 6 (Upstream image / ordering / provenance): PASS — ✅ Check 7 (Deprecated models): PASS — MODELS.md lists ✅ Check 8 (Benchmark hacks): PASS — no ✅ Check 9 (Chat template): PASS — the AgentX client replays through the Dynamo chat endpoint with ✅ Check 10 (Engine patches): PASS — at ✅ Check 11 (Golden AL): PASS — benchmark job logs show the connector applying ➖ Check 12 (Append-only): N/A — the new changelog entry has no ✅ Check 13 (Draft precision): PASS — the draft is the DSpark head bundled in ✅ Check 14 (Pareto coverage): PASS — one curve, dsv41flash / agentic-coding / gb300_dynamo-sglang / fp4 / run 41581664540 (agg and disagg, 8 ✅ Check 15 (PR description): PASS — the body's per-point prefill count, concurrency, decode BS, MRR, and DP-balancing table matches the pinned recipes; its throughput figures are clearly labeled as off-sweep cluster runs or published values. ✅ Check 16 (NVIDIA overview): PASS — the NVIDIA overview Framework Recipes table lists DeepSeek-V4.1-Flash with a populated SGLang cell that links the SGLang DeepSeek-V4.1 cookbook. ➖ Check 17 (Metrics publication): N/A — ✅ Check 18 (Recipe/image/sweep consistency): PASS — no generation errors; master and recipe agree (22/26/30/34 GPUs in results); the image digest resolves; changelog decode caps and the 7P base extra slots of 256 are accurate; on the combined agg+disagg curve, Assessed commit: |
|
[by Claude Code] The SGLang cookbook update of this PR's AgentX configs ( The commands mirror the launch measured in run 41581664540, pinned by this PR's |
|
[by Claude Code] sgl-project/sglang#48634 has been merged. This PR's AgentX configs ( The commands mirror the launch measured in run 41581664540, pinned by this PR's |
# Conflicts: # inferencex-e2e/perf-changelog.yaml
|
Ingest report — ✅ Results ingested into the Ingest workflow · Merge Ingest run · Benchmark source run
benchmarkFiles 9, rawBenchmarkRows 16, coveredByPerJobFiles 8, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 10552, gpuMetricSamplesNew 0, powerPublicationPoints 8, changelogEntriesWritten 1 Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun. 中文Ingest 报告 — ✅ 结果已写入 上表仅统计本次运行写入的数据，各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。 |
Move the GB300 DeepSeek-V4.1-Flash Dynamo + SGLang agg and prefill-HiCache disagg keys to digest-pinned
lmsysorg/sglang:nightly-dev-cu13-20261030-302b78cc(SGLang0.0.0.dev1+g52604e1d4, Dynamo1.7.0.dev20261029), and keep one point per interactivity range.Based on main after #4595 and #4679. The commits change
disagg-dep16-hicache-variants.yaml,agg-variants.yaml, the two keys' search spaces and image pins, and add one changelog entry that lists both keys, so the whole Dynamo + SGLang curve comes from this PR's sweep.Disagg decode is always one DEP16 worker. Removed: agg TP2/EP2 c16 (from #4679), 8 × DEP2 c4096 and 9 × DEP2 BS 160 c7480.
Why these points:
302b78cc(1200 s, 22 GPUs), c700 gives p90 299.2 vs the published 292.4 of agg TP2/EP2 c16, at 220,700-223,300 tok/s/GPU after the 1200 s → 3600 s factor (about 3.8x), with mean interactivity 290.2 and E2E-normalized p90 136.8. c800 (296.8) and c900 (294.5) were within 1.5% and 0.7% of TP2/EP2 c16, inside run-to-run noise, so c700 is the point with margin.total_requestsbalancing lets decode take every request prefill delivers.total_requestskeeps the per-rank running counts within 2-7 of each other (11-26 withtotal_tokens); the step is paced by the busiest rank.Removed published points (rule 18(g), mean interactivity and throughput per GPU; agg TP4 c16, 3 × DEP2 c1228 and the removed points as published, the other kept points from our 3600 s runs below):
Throughput gain at the same p90 interactivity (tok/s/GPU). Our points were measured on our GB300 cluster on
302b78ccwith the bound recipes of this PR and the AgentX client, not in this PR's sweep. The vLLM column linearly interpolates the currently published GB300 DeepSeek-V4.1-Flash Dynamo + vLLM and vLLM points on their Pareto frontier at our p90 interactivity. A blank cell means no gain.At p90 97.4, the vLLM frontier lies between 1,011,514 @ 75.5 and 993,521 @ 145.6, which gives 1,005,893. At p90 157.5 it lies between 993,521 @ 145.6 and 954,146 @ 196.5, which gives 984,316.
The 3 × DEP2 c1228 point keeps #4679's settings; on
302b78cca 1200 s run gave about 371,000 tok/s/GPU after the 1200 s → 3600 s factor, at p90 interactivity 281.2 and E2E-normalized p90 91.1, matching the published 383,313 @ 283.3 within the fast-run projection.Validation details
All runs used
nightly-dev-cu13-20261030-302b78ccthrough srt-slurm on GB300 with the AgentX client (reasoning_efforthigh, temperature 1) and the golden acceptance length 3.51. Each run's resolved recipe matches the recipe thatinfx generatebinds for this PR, apart from cluster NIC names, scheduler directives and local image/model packaging of the same digest.中文
将 GB300 DeepSeek-V4.1-Flash Dynamo + SGLang 的 agg 与 prefill-HiCache 分离式 key 换到固定 digest 的
nightly-dev-cu13-20261030-302b78cc（SGLang0.0.0.dev1+g52604e1d4、Dynamo1.7.0.dev20261029），每个交互性区间只保留一个点。基于 #4595 与 #4679 合入后的 main。 本 PR 新增一条同时列出两个 key 的 changelog，让整条 Dynamo + SGLang 曲线都来自本 PR 的 sweep。
total_requestsDP 均衡）。decode 都是一个 DEP16 worker。AI model disclosure
aws/anthropic/eccn-claude-opus-5-5) planned the point set from the P/D rate analysis, ran the cluster experiments through delegated agents on the same model, wrote the recipe overrides, master config, changelog entry and this description, and ran independent reviews of the diff through separate delegated agents on the same model.