Repository navigation
[NVIDIA][AgentX] GB300 DSV4.1-Flash Dynamo + SGLang: 20261111 nightly and a DEP32 mid-curve point / [NVIDIA][AgentX] GB300 DSV4.1-Flash Dynamo + SGLang:升级到 20261111 nightly 并加入 DEP32 中段点位 - #4997
Conversation
|
Thanks for the contribution!
中文感谢你的贡献!
|
… and a DEP32 c3018 mid-curve point
9066ae4 to
960ec83
Compare
…-gb300-sgl-dep32-midcurve # Conflicts: # inferencex-e2e/perf-changelog.yaml
|
Claude finished @weireweire's task in 37s —— View job LGTM - no blocking issues found |
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=43029773811 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=43029773811 |
On nightly-dev-cu13-20261111-314bce7c the NVIDIA megakernel plugin rejects FlashInfer's low-latency block-scaled GEMM at startup, which failed the agg TP4 c1 canary.
…-gb300-sgl-dep32-midcurve
|
[by Claude Code] The SGLang cookbook integration of this PR's AgentX configs ( This PR has no authorized |
|
[by Claude Code] sgl-project/sglang#50081 has been merged. This PR's AgentX configs ( This PR has no authorized |
|
/use 43029773811 |
|
@nvpohanh staged run 43029773811. This run remains available across future |
|
As a PR reviewer and CODEOWNER, I have reviewed this and have: Checklist items
Additional detail section:
Signed: |
✅✅✅ Verdict: PASS ✅✅✅Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — ✅ Check 1 (Sweep on in-PR commit): PASS — run 43029773811 (attempt 2, ✅ Check 2 (Evals): PASS — gsm8k em_strict 0.9735–0.9780 (n=1319) on all 6 points in ✅ Check 3 (Upstream recipe): PASS — sgl-project/sglang#50081 is MERGED (2026-11-12T11:25:20Z); ✅ Check 4 (Reuse command): PASS — ✅ Check 5 (Latest checklist): PASS — all 20 current-template items, including the recipe sub-item, are present and checked. ✅ Check 6 (Upstream images): PASS — both images are upstream ✅ Check 7 (Deprecated models): PASS — ✅ Check 8 (Benchmark hacks): PASS — no ✅ Check 9 (Chat template): PASS — the AgentX client replays through the Dynamo frontend with ✅ Check 10 (Engine patches): PASS — at ✅ Check 11 (Golden AL): PASS — the agg canary and DEP32 benchmark launches show ➖ Check 12 (Append-only): N/A — the new changelog entry does not set ✅ Check 13 (Draft precision): PASS — the draft is the DSpark head bundled in ✅ Check 14 (Pareto coverage): PASS — one curve ( ✅ Check 15 (PR description): PASS — the stated images, versions, point set, DEP32 settings, stream intervals, and sweep throughput/mean-interactivity values match the pinned recipes and ✅ Check 16 (NVIDIA overview): PASS — the NVIDIA overview Framework Recipes table lists DeepSeek-V4.1-Flash with a populated SGLang cell. ➖ Check 17 (Metrics publication): N/A — these are ✅ Check 18 (Recipe/image/sweep consistency): PASS — master 6×DEP2 + DEP32 matches the recipe (3+8 nodes, 44 GPUs in results); the new image resolves by digest; no orchestration or dispatch changes; the changelog is accurate; removed c1228 is dominated by DEP32 c3018, and c2884 and c700 lie under the head curve's segments (c700, 226,225 @ 295.0 by 1/ITL, sits under agg c16 to DEP32 c3018, about 469k; the facts' per-key Assessed commit: |
…-gb300-sgl-dep32-midcurve
|
Ingest report — ✅ Results ingested into the Ingest workflow · Merge Ingest run · Benchmark source run
benchmarkFiles 7, rawBenchmarkRows 12, coveredByPerJobFiles 6, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 7914, gpuMetricSamplesNew 853160, powerPublicationPoints 6, changelogEntriesWritten 1 Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun. 中文Ingest 报告 — ✅ 结果已写入 上表仅统计本次运行写入的数据,各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。 |
Move the GB300 DeepSeek-V4.1-Flash Dynamo + SGLang prefill-HiCache disagg key to digest-pinned
lmsysorg/sglang:nightly-dev-cu13-20261111-314bce7c(SGLang0.0.0.dev1+g314bce7cf, Dynamo1.7.0.dev20261110), and replace the middle of the disagg curve with one DEP32 decode point.The commits change
disagg-dep16-hicache-variants.yaml,agg-variants.yaml, the disagg key's search space and image pin, and add one changelog entry that lists both keys, so the whole Dynamo + SGLang curve comes from this PR's sweep.Removed: 3 × DEP2 + DEP16 at c700 and c1228, and 5 × DEP2 + DEP16 at c2884.
Why one DEP32 point: at the same prefill rate, spreading decode over 32 ranks keeps each rank at about 18 running requests, so the decode step stays short. One such point covers the middle of the curve: it dominates the two lowest-concurrency DEP16 points, and 5 × DEP2 c2884 falls under the line from it to 7 × DEP2 c5540.
DEP32 point settings: 32 redundant EPLB experts,
total_requestsDP balancing,disaggregation-decode-extra-slots80, andSGLANG_OPT_DSV41_BLOCKWISE_REJECTION=1(block-wise rejection for the sampled DSpark verify, exact in distribution). Everything else comes from the existing base.Stream interval: the agg role and the disagg prefill role now set
stream-interval: 32; both used the SGLang default of 1 before. Disagg decode already used 32.Agg image: the agg key stays on
nightly-dev-cu13-20261030-302b78cc. On the 20261111 image,--enable-nvidia-megakernelrejects the bundled FlashInfer low-latency block-scaled GEMM at startup (it is not the version the plugin's wo_b epilogue extends), so the agg canary could not start.Removed published points (rule 18(g): mean interactivity and throughput per GPU of the published points; DEP32 c3018 from our 3600 s run below):
DEP32 c3018 vs the published Dynamo + vLLM frontier (tok/s/GPU at the same p90 interactivity, the published
p90_intvtyfield). Measured on our GB300 cluster on314bce7c, 3600 s, with this PR's bound recipe and the AgentX client, not in this PR's sweep. The vLLM value interpolates linearly between its frontier points 461,600 @ 331.1 (DEP32 c3018) and 954,100 @ 196.5 (DEP32 c10888).The two runs sit on the vLLM frontier rather than clearly above it: +1.1 % in the first, no gain in the replicate.
This PR's sweep (run 43029773811, 3600 s, all six points succeeded). The vLLM column interpolates between its published frontier points as above (c5540 and c7480 between DEP32 c10888 954,100 @ 196.5, DEP16 c7480 993,500 @ 145.6 and DEP16 c9888 1,011,500 @ 75.5).
The removed points stay covered by the swept curve: c700 and c1228 are dominated by c3018 (474,138 @ mean 336.8), and c2884 (715,600 @ mean 220.9) lies under the swept c3018 to c5540 line (about 840,500 at mean 220.9).
Validation details
Runs use
nightly-dev-cu13-20261111-314bce7cthrough srt-slurm on GB300 with the AgentX client (reasoning_efforthigh, temperature 1) and the golden acceptance length 3.51. The DEP32 c3018 recipe bound byinfx generatefor this PR matches the measured run's resolved recipe, apart from benchmark variables injected by CI and cluster NIC names. The throughput above is total tokens over the 3629 s profile window divided by 44 engine GPUs. That run used the SGLang default stream interval on the prefill role; a prefill worker streams only the first token, so the change to 32 does not affect it.SGLANG_OPT_DSV41_BLOCKWISE_REJECTIONacts only on sampled verify. The GSM8K eval runs at temperature 0, so it does not exercise this path; its exactness rests on the upstream implementation (blockwise_rejection.py). Throughput runs use the golden acceptance length, so the measured AL is unaffected.中文
将 GB300 DeepSeek-V4.1-Flash Dynamo + SGLang 的 prefill-HiCache 分离式 key 换到固定 digest 的
nightly-dev-cu13-20261111-314bce7c(SGLang0.0.0.dev1+g314bce7cf、Dynamo1.7.0.dev20261110),并用一个 DEP32 decode 点替换分离式曲线的中段。total_requestsDP 均衡、block-wise rejection。nightly-dev-cu13-20261030-302b78cc(20261111 镜像上 NVIDIA megakernel 插件在启动时拒绝其自带的 FlashInfer low-latency block-scaled GEMM);分离式 7P c5540、9P c7480 除 stream interval 外只换镜像。AI model disclosure
aws/anthropic/eccn-claude-opus-5-5) chose the point set, ran the cluster experiments, wrote the recipe override, master config, changelog entry and this description.