Repository navigation
[NVIDIA][AgentX] Move GB300 DSV4.1-Flash SGLang to the 20261030 nightly with the tuned low-latency settings and TP4 c1 megakernel / [NVIDIA][AgentX] GB300 DSV4.1-Flash SGLang 升级到 20261030 nightly，采用调优后的低延迟配置并在 TP4 c1 启用 megakernel - #4603
weireweire wants to merge 12 commits into
Conversation
…61025-65633129 Move dsv41flash-fp4-gb300-sglang-agentic-dspark from nightly-dev-cu13-20261021-c1c287a4 to the digest-pinned nightly-dev-cu13-20261025-65633129, the image the Dynamo + SGLang agg key already uses. Recipes and search space are unchanged.
…-gb300-sgl-single-0025 # Conflicts: # inferencex-e2e/perf-changelog.yaml
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
…ned low-latency settings Use nightly-dev-cu13-20261028-586355a8, which includes sgl-project/sglang#46404 and #46891. TP4 c1 runs as the batch-1 NVIDIA megakernel point of the Dynamo + SGLang aggregated row, without SIM_DRAW_AHEAD so the simulated step does the same work as a real one. TP4 c4/c16 and TP2/EP2 c16/c32 pin the flashmla attention backend instead of auto (MegaAttention pads per-rank heads to 64).
weireweire
changed the title
The gate only moves the simulated-acceptance draw, a host-side sample plus a fill that real acceptance never runs, ahead of the draft launch. It does not skip work that a real-acceptance step performs.
weireweire
added
full-sweep-fail-fast
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=40809149952 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=40809149952 |
… 2 plugin nightly-dev-cu13-20261028-586355a8 bundles sgl-nvidia-mega 6fd7ac30 (megakernel API 1), while its SGLang requires API 2 since sgl-project/sglang#45266, so --enable-nvidia-megakernel fails at startup. TP4 c1 keeps its previous engine settings and pins flashmla like the other TP-sharded points.
…-gb300-sgl-single-0025
weireweire
added
the
full-sweep-fail-fast
…-gb300-sgl-single-0025 # Conflicts: # inferencex-e2e/perf-changelog.yaml
…run TP4 c1 on the megakernel nightly-dev-cu13-20261029-c10e0e17 bundles sgl-nvidia-mega 54b428921749 (megakernel API 2), so TP4 c1 returns to its megakernel config. The 16 runtime-optimization variables it set one by one are replaced by SGLANG_ENABLE_NV_MEGA_RUNTIME_OPTIMIZATION (sgl-project/sglang#47105).
weireweire
changed the title
|
[by Claude Code] The SGLang cookbook integration of this PR's AgentX config ( This PR has no authorized |
|
[by Claude Code] sgl-project/sglang#47214 has been merged. This PR's AgentX config ( This PR has no authorized |
|
[by Claude Code] The SGLang cookbook update of this PR's AgentX config ( This PR has no authorized |
|
/use 40809149952 |
|
@nvpohanh staged run 40809149952. This run remains available across future |
|
[by Claude Code] sgl-project/sglang#47464 has been merged. This PR's AgentX config ( This PR's |
|
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Additional detail section:
Signed: |
✅✅✅ Verdict: PASS ✅✅✅Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — ✅ Check 1 (Sweep on in-PR commit): PASS — Run 40809149952 ran on the pinned head ✅ Check 2 (Evals): PASS — ✅ Check 3 (Upstream recipe): PASS — sgl-project/sglang#47464 is MERGED (2026-10-30T11:23:25Z). On ✅ Check 4 (Reuse command): PASS — ✅ Check 5 (Latest checklist): PASS — All 20 items in the current ✅ Check 6 (Upstream image / provenance / ordering): PASS — The image is ✅ Check 7 (Deprecated models): PASS — ✅ Check 8 (No arch-changing hacks): PASS — (a) The recipe has no ✅ Check 9 (Spec-decode chat template): PASS — The AgentX aiperf client replays traces through its default chat endpoint (CLI in the TP4 c1 ✅ Check 10 (No engine patches): PASS — At ✅ Check 11 (Agentic golden AL): PASS — The run's srtctl overrides inject ➖ Check 12 (Append-only): N/A — The new changelog entry does not set ✅ Check 13 (Draft precision): PASS — The draft is the DSpark head bundled in ✅ Check 14 (Pareto coverage): PASS — One affected curve: dsv41flash / agentic-coding / GB300 sglang / fp4 / run 40809149952. From ✅ Check 15 (PR description): PASS — The body's claims match the evidence: the image tag and digest, the inclusion of sglang #46404, #46891, #47145 and #47105 (all merged ancestors of ✅ Check 16 (NVIDIA overview): PASS — The NVIDIA overview Framework Recipes table lists DeepSeek-V4.1-Flash with a populated SGLang cell, covering the GB300 SGLang pair. ➖ Check 17 (Metrics publication): N/A — This is single-node SGLang with no Dynamo TRT-LLM worker, so the Dynamo metrics controls do not apply. ✅ Check 18 (Recipe/image/sweep consistency): PASS — (a/b) No generation errors at base or head, and master tp/ep/dp-attn and conc lists match the recipe overrides' GPUs and parallelism. (c) The digest resolves. (d/e) No orchestration pin or dispatch changes. (f) The changelog states the image bump, megakernel, fp8_e4m3 KV, the runtime-opt env var and the Assessed commit: |
| env: | ||
| SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE: '0' | ||
| SGLANG_RAGGED_VERIFY_MODE: static | ||
| SGLANG_SPEC_TP_SYNC: rng,-dspark-accept-sample |
There was a problem hiding this comment.
what is SPEC_TP_SYNC=rng,-dspark-accept-sample? does this affect draft model accuracy?
There was a problem hiding this comment.
mainly wanna make sure it doesnt affect speculator perf (since we are simulating acc len )
There was a problem hiding this comment.
hi. this is used to reduce unnecessary communication between ranks, it won't change the draft model precision.
…-gb300-sgl-single-0025 # Conflicts: # inferencex-e2e/perf-changelog.yaml
…e simulated verify accept cost fix Root cause: nightly-dev-cu13-20261029-c10e0e17 predates sgl-project/sglang#46840, so simulated acceptance skipped the real verify accept cost and the TP4 c1 point was not measured on the methodology-conformant path. Fix: pin lmsysorg/sglang:nightly-dev-cu13-20261030-302b78cc by digest. It is 63 commits past the #46840 merge and bundles the same sgl-nvidia-mega 54b428921749 (megakernel API 2). Update this PR's changelog entry to name the image and #46840. Validation: Dynamo + SGLang TP4 c1 megakernel, c4 and c16 started and completed on 302b78cc with target and draft megakernels active on all four ranks.
…-gb300-sgl-single-0025 # Conflicts: # inferencex-e2e/perf-changelog.yaml
weireweire
changed the title
weireweire
added
full-sweep-enabled
Move
dsv41flash-fp4-gb300-sglang-agentic-dspark(single-node SGLang AgentX on GB300) to digest-pinnedlmsysorg/sglang:nightly-dev-cu13-20261030-302b78cc, and apply the low-latency settings tuned on the Dynamo + SGLang aggregated row.nightly-dev-cu13-20261021-c1c287a4tonightly-dev-cu13-20261030-302b78cc.nightly-dev-cu13-20261029-c10e0e17, was built before it.54b428921749([Fix] docker: update NVIDIA megakernel pin to tested commit sgl-project/sglang#47145), which implements megakernel API 2 as SGLang requires.--enable-nvidia-megakernel, fp8_e4m3 KV cache,mem-fraction-static 0.91, decode graph batch 1 and the Engram host table. Target and draft megakernels are active on all four TP ranks on this image.SGLANG_ENABLE_NV_MEGA_RUNTIME_OPTIMIZATION=1([Feat] [NVIDIA] runtime: Add a master switch for megakernel runtime optimizations sgl-project/sglang#47105). The switch also turns on the Qwen3.5/EAGLE runtime group, most of which is inactive on this path.SGLANG_OPT_DSV41_DSPARK_SIM_DRAW_AHEADstays explicit. It moves the simulated acceptance draw ahead of the draft launch and is inactive with real acceptance. The simulation settings, megakernel slots and NCCL settings also stay explicit.dsv4-attn-backend: flashmla, as the Dynamo + SGLang aggregated rows do.autoresolves to FlashMLA MegaAttention, which zero-pads per-rank attention heads to 64 (4x at TP4, 2x at TP2). The DP-attention points (c64/c128/c160) are unchanged.Risk: this image stores the DeepSeek-V4.1 KV cache in about 55 % of the previous bytes per token, so a given
mem-fraction-staticholds more tokens and commits more memory. On a DEP16 decode worker this left too little memory for CUDA graph capture at 0.85. The previous sweep of this PR onnightly-dev-cu13-20261028-586355a8started every point except the megakernel one. If a point fails to start on this image, its memory setting will be lowered.中文
将
dsv41flash-fp4-gb300-sglang-agentic-dspark（GB300 单节点 SGLang AgentX）更新到固定 digest 的lmsysorg/sglang:nightly-dev-cu13-20261030-302b78cc，并采用在 Dynamo + SGLang 聚合式配置上调好的低延迟参数。nightly-dev-cu13-20261021-c1c287a4升级到nightly-dev-cu13-20261030-302b78cc。nightly-dev-cu13-20261029-c10e0e17构建于它合入之前。54b428921749（[Fix] docker: update NVIDIA megakernel pin to tested commit sgl-project/sglang#47145），实现 SGLang 要求的 megakernel API 2。--enable-nvidia-megakernel、fp8_e4m3 KV cache、mem-fraction-static 0.91、decode graph batch 1、Engram host table。在该镜像上，4 个 TP rank 的 target 和 draft megakernel 均生效。SGLANG_ENABLE_NV_MEGA_RUNTIME_OPTIMIZATION=1（[Feat] [NVIDIA] runtime: Add a master switch for megakernel runtime optimizations sgl-project/sglang#47105）。该开关同时打开 Qwen3.5/EAGLE 那组优化，在此路径上大多不生效。SGLANG_OPT_DSV41_DSPARK_SIM_DRAW_AHEAD仍显式设置：它把模拟接受的采样提前到 draft launch 之前，真实接受时不起作用。模拟设置、megakernel slots 和 NCCL 设置也仍显式指定。dsv4-attn-backend: flashmla。auto会选 FlashMLA MegaAttention，它把每个 rank 的 attention head 零填充到 64（TP4 为 4 倍，TP2 为 2 倍）。DP attention 的点（c64/c128/c160）不变。风险： 该镜像中 DeepSeek-V4.1 每个 token 的 KV 只占原来约 55% 的字节，同样的
mem-fraction-static能放更多 token，实际占用的显存也更多。在 DEP16 decode worker 上，0.85 时剩余显存不够 CUDA graph capture。本 PR 上一轮在nightly-dev-cu13-20261028-586355a8上的 sweep 中，除 megakernel 点外的所有点都能启动；如在新镜像上有点无法启动，会调低其显存设置。AI model disclosure
aws/anthropic/eccn-claude-opus-5-5) chose the image, ported the tuned settings from the Dynamo + SGLang aggregated row, ran the changelog and recipe checks, and wrote this description. Cluster validation runs through a delegated agent on the same model.