Skip to content

[NVIDIA][AgentX] GB300 DSV4.1-Flash Dynamo + SGLang: 20261111 nightly and a DEP32 mid-curve point / [NVIDIA][AgentX] GB300 DSV4.1-Flash Dynamo + SGLang:升级到 20261111 nightly 并加入 DEP32 中段点位 - #4997

Merged
nvpohanh merged 7 commits into
mainfrom
weireweire/dsv41flash-gb300-sgl-dep32-midcurve
Nov 13, 2026

Conversation

@weireweire

@weireweire weireweire commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Move the GB300 DeepSeek-V4.1-Flash Dynamo + SGLang prefill-HiCache disagg key to digest-pinned lmsysorg/sglang:nightly-dev-cu13-20261111-314bce7c (SGLang 0.0.0.dev1+g314bce7cf, Dynamo 1.7.0.dev20261110), and replace the middle of the disagg curve with one DEP32 decode point.

The commits change disagg-dep16-hicache-variants.yaml, agg-variants.yaml, the disagg key's search space and image pin, and add one changelog entry that lists both keys, so the whole Dynamo + SGLang curve comes from this PR's sweep.

Point Prefill Decode Concurrency Decode CUDA graph max BS (per rank) Decode max-running-requests Change
agg TP4 c1 / c4 / c16 — — 1 / 4 / 16 — — stream interval 32
disagg 6 × DEP2 DEP32 (8 nodes) 3018 18 576 new
disagg 7 × DEP2 DEP16 5540 96 1536 image, prefill stream interval 32
disagg 9 × DEP2 DEP16 7480 192 3072 image, prefill stream interval 32

Removed: 3 × DEP2 + DEP16 at c700 and c1228, and 5 × DEP2 + DEP16 at c2884.

Why one DEP32 point: at the same prefill rate, spreading decode over 32 ranks keeps each rank at about 18 running requests, so the decode step stays short. One such point covers the middle of the curve: it dominates the two lowest-concurrency DEP16 points, and 5 × DEP2 c2884 falls under the line from it to 7 × DEP2 c5540.

DEP32 point settings: 32 redundant EPLB experts, total_requests DP balancing, disaggregation-decode-extra-slots 80, and SGLANG_OPT_DSV41_BLOCKWISE_REJECTION=1 (block-wise rejection for the sampled DSpark verify, exact in distribution). Everything else comes from the existing base.

Stream interval: the agg role and the disagg prefill role now set stream-interval: 32; both used the SGLang default of 1 before. Disagg decode already used 32.

Agg image: the agg key stays on nightly-dev-cu13-20261030-302b78cc. On the 20261111 image, --enable-nvidia-megakernel rejects the bundled FlashInfer low-latency block-scaled GEMM at startup (it is not the version the plugin's wo_b epilogue extends), so the agg canary could not start.

Removed published points (rule 18(g): mean interactivity and throughput per GPU of the published points; DEP32 c3018 from our 3600 s run below):

Removed point Published Covered by
3 × DEP2 c700 226,200 @ mean 295.7 dominated by DEP32 c3018 (474,900 @ mean 338.8)
3 × DEP2 c1228 383,600 @ mean 267.1 dominated by DEP32 c3018
5 × DEP2 c2884 715,600 @ mean 220.9 under the DEP32 c3018 to 7 × DEP2 c5540 (1,014,400 @ mean 150.6) line, about 812,900 at mean 220.9

DEP32 c3018 vs the published Dynamo + vLLM frontier (tok/s/GPU at the same p90 interactivity, the published p90_intvty field). Measured on our GB300 cluster on 314bce7c, 3600 s, with this PR's bound recipe and the AgentX client, not in this PR's sweep. The vLLM value interpolates linearly between its frontier points 461,600 @ 331.1 (DEP32 c3018) and 954,100 @ 196.5 (DEP32 c10888).

This recipe Run Throughput p90 interactivity Mean interactivity TTFT p95 vLLM at the same p90
6 × DEP2 + DEP32, BS 18, c3018 3600 s 474,900 328.9 338.8 3.77 s 469,700 (+1.1 %)
6 × DEP2 + DEP32, BS 18, c3018 (replicate) 3600 s 473,600 324.7 331.5 4.04 s 485,000

The two runs sit on the vLLM frontier rather than clearly above it: +1.1 % in the first, no gain in the replicate.

This PR's sweep (run 43029773811, 3600 s, all six points succeeded). The vLLM column interpolates between its published frontier points as above (c5540 and c7480 between DEP32 c10888 954,100 @ 196.5, DEP16 c7480 993,500 @ 145.6 and DEP16 c9888 1,011,500 @ 75.5).

Point Throughput p90 interactivity Mean interactivity TTFT p95 vLLM at the same p90
agg TP4 c1 8,246 943.4 965.3 0.43 s
agg TP4 c4 10,319 571.4 625.2 0.34 s
agg TP4 c16 29,254 423.7 523.2 2.46 s
6 × DEP2 + DEP32 c3018 474,138 328.9 336.8 3.90 s 469,700 (+0.9 %)
7 × DEP2 + DEP16 c5540 992,955 164.2 172.7 14.2 s 979,100 (+1.4 %)
9 × DEP2 + DEP16 c7480 1,071,943 106.6 120.3 15.6 s 1,003,500 (+6.8 %)

The removed points stay covered by the swept curve: c700 and c1228 are dominated by c3018 (474,138 @ mean 336.8), and c2884 (715,600 @ mean 220.9) lies under the swept c3018 to c5540 line (about 840,500 at mean 220.9).

Validation details

Runs use nightly-dev-cu13-20261111-314bce7c through srt-slurm on GB300 with the AgentX client (reasoning_effort high, temperature 1) and the golden acceptance length 3.51. The DEP32 c3018 recipe bound by infx generate for this PR matches the measured run's resolved recipe, apart from benchmark variables injected by CI and cluster NIC names. The throughput above is total tokens over the 3629 s profile window divided by 44 engine GPUs. That run used the SGLang default stream interval on the prefill role; a prefill worker streams only the first token, so the change to 32 does not affect it.

SGLANG_OPT_DSV41_BLOCKWISE_REJECTION acts only on sampled verify. The GSM8K eval runs at temperature 0, so it does not exercise this path; its exactness rests on the upstream implementation (blockwise_rejection.py). Throughput runs use the golden acceptance length, so the measured AL is unaffected.

中文

将 GB300 DeepSeek-V4.1-Flash Dynamo + SGLang 的 prefill-HiCache 分离式 key 换到固定 digest 的 nightly-dev-cu13-20261111-314bce7c(SGLang 0.0.0.dev1+g314bce7cf、Dynamo 1.7.0.dev20261110),并用一个 DEP32 decode 点替换分离式曲线的中段。

  • 新增: 6 × DEP2 prefill + 1 个 DEP32 decode(8 节点),c3018,decode BS 18、max-running-requests 576、32 个冗余 EPLB expert、total_requests DP 均衡、block-wise rejection。
  • 去掉: 3 × DEP2 + DEP16 的 c700 与 c1228(按平均交互性被 DEP32 c3018 支配),以及 5 × DEP2 c2884(在 DEP32 c3018 与 7 × DEP2 c5540 连线下方,约 812,900)。
  • Stream interval: agg 与分离式 prefill 角色显式设为 32(之前为 SGLang 默认值 1),decode 原本就是 32。
  • 其余点: agg TP4 c1 / c4 / c16 只加 stream interval,镜像保持 nightly-dev-cu13-20261030-302b78cc(20261111 镜像上 NVIDIA megakernel 插件在启动时拒绝其自带的 FlashInfer low-latency block-scaled GEMM);分离式 7P c5540、9P c7480 除 stream interval 外只换镜像。
  • 相对 vLLM: DEP32 c3018 为 474,900 @ p90 328.9(3600 s,我们的 GB300 集群),同 p90 下 vLLM 前沿约 469,700,约 +1.1%;复测为 473,600 @ p90 324.7,同 p90 下 vLLM 约 485,000,无优势。两次结果基本落在 vLLM 前沿上。本 PR 的 sweep(run 43029773811,3600 s,六个点全部成功):c3018 474,138 @ p90 328.9(vLLM 同 p90 约 469,700,+0.9%),c5540 992,955 @ 164.2(+1.4%),c7480 1,071,943 @ 106.6(+6.8%)。
  • 本 PR 的 changelog 同时列出两个 key,整条 Dynamo + SGLang 曲线都来自本 PR 的 sweep。
AI model disclosure
  • Claude Opus 5.5 (runtime identifier aws/anthropic/eccn-claude-opus-5-5) chose the point set, ran the cluster experiments, wrote the recipe override, master config, changelog entry and this description.

@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目,并且恰好带有一个主标签时,才会运行扫描:full-sweep-fail-fast(强烈推荐;canary 加逐矩阵 fail-fast)、full-sweep-enabled(有 canary;矩阵任务在失败后继续运行)或 non-canary-full-sweep-enabled(无 canary,也无 fail-fast)。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

@weireweire
weireweire force-pushed the weireweire/dsv41flash-gb300-sgl-dep32-midcurve branch from 9066ae4 to 960ec83 Compare November 11, 2026 03:10
@weireweire weireweire changed the title [NVIDIA][AgentX] GB300 DSV4.1-Flash Dynamo + SGLang: 20261111 nightly and DEP32 mid-curve points / [NVIDIA][AgentX] GB300 DSV4.1-Flash Dynamo + SGLang:升级到 20261111 nightly 并加入 DEP32 中段点位 [NVIDIA][AgentX] GB300 DSV4.1-Flash Dynamo + SGLang: 20261111 nightly and a DEP32 mid-curve point / [NVIDIA][AgentX] GB300 DSV4.1-Flash Dynamo + SGLang:升级到 20261111 nightly 并加入 DEP32 中段点位 Nov 11, 2026
@weireweire
weireweire marked this pull request as ready for review November 11, 2026 08:30
@weireweire
weireweire requested a review from a team November 11, 2026 08:30
@weireweire weireweire added the full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) label Nov 11, 2026
@claude

claude Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @weireweire's task in 37s —— View job


LGTM - no blocking issues found

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review found no issues

No high-confidence issues detected in this change.

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

On nightly-dev-cu13-20261111-314bce7c the NVIDIA megakernel plugin rejects
FlashInfer's low-latency block-scaled GEMM at startup, which failed the agg
TP4 c1 canary.
@nvpohanh

Copy link
Copy Markdown
Collaborator

[by Claude Code] The SGLang cookbook integration of this PR's AgentX configs (dsv41flash-fp4-gb300-dynamo-sglang-agentic-agg, dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg-hicache) will happen in sgl-project/sglang#50081, in the Agentic Long-Context Deployment section of the DeepSeek-V4.1 cookbook page.

This PR has no authorized /use <run_id> comment yet, so the cookbook shows the recipe at this PR's head (482df60e0) as "in progress". A later cookbook PR switches it to the measured run once a run is pinned or this PR merges.

@nvpohanh

Copy link
Copy Markdown
Collaborator

[by Claude Code] sgl-project/sglang#50081 has been merged. This PR's AgentX configs (dsv41flash-fp4-gb300-dynamo-sglang-agentic-agg, dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg-hicache) are now in the Agentic Long-Context Deployment section of the SGLang DeepSeek-V4.1 cookbook: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4_1#agentic-long-context-deployment

This PR has no authorized /use <run_id> comment yet, so the cookbook shows the recipe at this PR's head (482df60e0) as "in progress". A later cookbook PR switches it to the measured run once a run is pinned or this PR merges.

@nvpohanh

Copy link
Copy Markdown
Collaborator

/use 43029773811

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

@nvpohanh staged run 43029773811.

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

@nvpohanh

Copy link
Copy Markdown
Collaborator

As a PR reviewer and CODEOWNER, I have reviewed this and have:

Checklist items
  • Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
  • Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
  • Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
  • Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
  • Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
  • Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison.
  • For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
  • Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
  • Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
  • Verified that new models disable integer all-reduce, with SGLang ROCm explicitly setting ROCM_QUICK_REDUCE_QUANTIZATION=NONE. Existing models are grandfathered per MODELS.md.
  • For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
  • If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
  • For official InferenceX (non-preview), verified that established models use upstream engine images built from master (or the equivalent default development branch), with new models transitioning within a reasonable time after release. See MODELS.md.
  • If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
  • Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
    • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
  • Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
  • If this PR uses append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it.
  • If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
  • Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.

Additional detail section:

  • Assessed head: 482df60e0019e219baf2234400b649cff3b13efd, the current PR head. The pinned run is Run Sweep 43029773811 (pull_request, head 482df60e, conclusion success after attempt 2), named by /use 43029773811, the only /use on this PR. Its results_bmk, eval_results_all and 6 bmk_agentic_* artifacts are unexpired until 2027-02-09. No commit follows 482df60e.

  • Code quality:

    • The diff is config-only:
      • agg-variants.yaml adds stream-interval: 32 to the agg role.
      • disagg-dep16-hicache-variants.yaml bumps the framework identity to Dynamo 1.7.0.dev20261110 / SGLang 0.0.0.dev1+g314bce7cf, adds stream-interval: 32 to the prefill role, drops override_3p1d_dep2_dep16_c700, override_3p1d_dep2_dep16_c1228 and override_5p1d_dep2_dep16_c2884, and adds override_6p1d_dep2_dep32_c3018.
      • nvidia-master.yaml changes the disagg key's image and router pins and its search space to match.
    • It appends one perf-changelog.yaml entry at the end of the file and edits no existing entry. Against current main, git merge-tree reports a conflict only in perf-changelog.yaml, caused by other appends.
    • Master and recipe agree. The results report 4 GPUs for each agg point, 44 for 6P DEP32 (6 × DEP2 + DEP32 on 8 nodes), 30 for 7P and 34 for 9P. The served DEP32 decode resolves max_running_requests: 576 and stream_interval: 32, with DP/EP/TP 32, as in the override.
    • The changelog entry is accurate:
      • It names both images, the Dynamo and SGLang versions, the new and dropped points, and the DEP32 decode settings.
      • Its claim that disagg decode already used stream-interval: 32 matches the base recipe.
      • No quantization, KV dtype, checkpoint or speculative-method change is omitted.
  • PR validation:

    • In run 43029773811, all 6 benchmark jobs succeeded: multi-node canary / (agg TP4 c1) and the 5 multi-node agentic / jobs (agg TP4 c4 and c16, disagg 6P DEP32 c3018, 7P c5540 and 9P c7480).
    • All 6 multi-node agentic eval / jobs, check-changelog, collect-results and collect-evals also succeeded.
    • The skipped jobs are the fixed-sequence, single-node and video lanes that these keys do not generate.
  • Evals (gsm8k em_strict, n=1319, from eval_results_all on the PR images). The threshold is the default 0.90 in infx/evals/thresholds.yaml, which has no dsv41flash entry; every row also clears the stricter 0.91 dsv4 value.

    Config Image em_strict
    agg TP4 c1 20261030-302b78cc 0.9765
    agg TP4 c4 20261030-302b78cc 0.9742
    agg TP4 c16 20261030-302b78cc 0.9735
    disagg 6P DEP32 c3018 20261111-314bce7c 0.9757
    disagg 7P c5540 20261111-314bce7c 0.9780
    disagg 9P c7480 20261111-314bce7c 0.9765
  • Block-wise rejection (SGLANG_OPT_DSV41_BLOCKWISE_REJECTION=1, DEP32 decode only):

    • It is an upstream switch in the pinned image (environ.py at 314bce7cf, default EnvBool(SGLANG_ENABLE_NV_MEGA_RUNTIME_OPTIMIZATION)).
    • It changes only how sampled (non-greedy) DSpark verify accepts draws: it uses blockwise_rejection.py, which is exact in law, instead of the classic kernel. It changes no weights, precision or FLOPs.
    • gsm8k runs at temperature 0, so it takes the greedy path. The same DEP32 eval job's real-acceptance AL gate (temperature 1, thinking on) did run this path: the decode logs print DSpark: sampled verify accepts use block-wise rejection.
    • Measured AL there was 3.370, against 3.400 / 3.397 on the 7P / 9P DEP16 siblings that use the classic kernel (ratio 0.96 vs golden 3.51, pass). The benchmark decode logs also show the switch active.
  • Chat template: The AgentX client replays through the Dynamo chat endpoint. Both recipes set AIPERF_EXTRA_INPUTS={"chat_template_kwargs":{"reasoning_effort":"high","thinking":true},"temperature":1}, the thinking-on regime of the golden curve.

  • Draft precision:

    • The draft is the DSpark head bundled in deepseek-ai/DeepSeek-V4.1-Flash. Worker logs show speculative_draft_model_path: deepseek-ai/DeepSeek-V4.1-Flash and Draft checkpoint bundles a DSpark head; loading draft arch DeepseekV4ForCausalLMDSpark, so no checkpoint is substituted.
    • Shipped precision: The checkpoint has quant_method: fp8, weight_block_size: [32, 32], scale_fmt: ue8m0 and expert_dtype: fp4: MXFP8 dense weights and MXFP4 experts, with BF16 norms, gate and markov, and F32 hc_*/attn_sink. By default, the pinned images load the DSpark wo_a as BF16 and keep markov_w2 in BF16. SGLANG_DSPARK_OPT_MARKOV_W2_BF16 = EnvBool(True) in environ.py at both 314bce7cf (disagg) and 52604e1d (agg, unchanged image).
    • Effective precision: It matches the shipped baseline. Agg c1 and the DEP32 prefill and decode workers resolve dtype: auto, quantization: None, speculative_draft_model_quantization: None, speculative_draft_kv_cache_dtype: None and json_model_override_args: {}, so the draft KV inherits the target's fp8_e4m3.
    • None of the other settings lowers draft precision:
    • SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE: It is not enabled. It is absent from both recipes at 482df60e, from the generated config.yaml and sbatch_script.sh, and from every inspected worker log and config dump (agg c1 and c4, DEP32 benchmark and eval, 7P and 9P eval). environ.py defines it as EnvBool(False) at 314bce7cf.
  • Golden AL:

    • The agg role config and the DEP32 prefill and decode configs all show SGLANG_SIMULATE_ACC_LEN=3.51, SGLANG_SIMULATE_ACC_METHOD=match-expected and SGLANG_SIMULATE_ACC_TOKEN_MODE=real-draft-token.
    • This matches deepseek-v4.1-flash__dspark__native__probabilistic-block.yaml deepseek-v4.1-flash.thinking_on.5 = 3.51 for speculative-dspark-block-size: 5 / speculative-num-steps: 5 with thinking on. That is the only dsv41flash DSpark identity in identities.yaml.
    • The measured mean accept length on the DEP32 benchmark decode is 3.510 (36,673 decode lines).
    • The eval-only jobs run real acceptance (AL gate above).
  • MODELS.md: It lists dsv41flash Agentic coding (SGLang DSpark) as active, with no deprecated scenario.

  • Benchmark hacks and integer all-reduce:

    • There is no --hf-overrides, --json-model-override-args (it resolves to {}), model-config edit, layer skipping or indexer skipping.
    • SGLANG_OPT_DSV41_BF16_INDEX_LOGITS changes only the storage dtype of the target indexer's top-k scores, and the disagg evals cover it.
    • This is an NVIDIA CUDA path with no integer all-reduce control, and dsv41flash is an existing model.
  • Dynamo TRT-LLM metrics: N/A. These are dynamo-sglang configs, not Dynamo TRT-LLM.

  • Upstream images and main-branch provenance:

  • Engine-first ordering: These are SGLang (in-tree engine) entries served through Dynamo, so the ordering rule is satisfied.

  • Upstream recipe:

    • Merged cookbook PR sgl-project/sglang#50081 (MERGED 2026-11-12T11:25:20Z, merge commit 3df70c30) publishes this PR's head 482df60e for both keys. The file is docs/src/snippets/agentx/deepseek-ai/deepseek-v4_1.jsx on sglang main (same blob as the merge commit), shown in the SGLang DeepSeek-V4.1 cookbook, Agentic Long-Context Deployment section.
    • It covers every affected point:
      • agg override_tp4_c1, override_tp4_c4 and override_tp4_c16, with --stream-interval 32, on 302b78cc;
      • disagg override_6p1d_dep2_dep32_c3018, override_7p1d_dep2_dep16_b96_c5540 and override_9p1d_dep2_dep16_b192_c7480 on 314bce7c@sha256:387e9021….
    • The DEP32 entry gives:
      • the etcd and Dynamo frontend/router commands;
      • the 6 × DEP2 HiCache prefill launch with --stream-interval 32;
      • the 8-node DEP32 decode launch with --tensor/data/expert-parallel-size 32, --ep-num-redundant-experts 32, --max-running-requests 576, --cuda-graph-max-bs-decode 18, --disaggregation-decode-extra-slots 80, --load-balance-method total_requests and SGLANG_OPT_DSV41_BLOCKWISE_REJECTION=1;
      • the Mooncake transfer settings.
    • The 7P and 9P entries derive from it, with their own decode caps; block-wise rejection is removed there.
    • @fabric.rdma-devices and {node} appear in the cookbook as $IB_DEVICES and $NODE_IP.
    • Excluded InferenceX-only settings have no production serving effect:
      • logging (PYTHONUNBUFFERED, PYTHONNOUSERSITE)
      • cluster plumbing (HF_HUB_CACHE, Slurm/srun/sbatch options, placement, health checks, the tachometer and otel-collector services)
      • the AIPERF_* client env
      • the SGLANG_SIMULATE_ACC_* synthetic-acceptance controls
  • No engine patches: At 482df60e, neither recipe has a setup script, .patch, overlay, git apply, sed -i, site-packages edit or wheel install. Both use dynamo.install: false, and the generated sbatch_script.sh runs no setup step. The PR does not touch runners/srt-slurm/patches/, and there is no docs/waiver/4997.md.

  • Stream interval: It is 32 on every role of every rerun point: agg (explicit), disagg prefill (explicit, was the default 1) and disagg decode (base recipe). Server args confirm stream_interval: 32. That is within the ≤ 60 AgentX limit.

  • append-only: N/A. The new changelog entry does not use append-only: true.

  • Removed points (Check 18g): The published measurements of the removed points come from main's pinned run 41581664540 ([NVIDIA][AgentX] Retune GB300 DSV4.1-Flash Dynamo + SGLang HiCache disagg points on the 20261030 nightly / [NVIDIA][AgentX] 在 20261030 nightly 上重新调整 GB300 DSV4.1-Flash Dynamo + SGLang HiCache 分离式点位 #4673), given as tput_per_gpu @ mean_intvty. Every removed point is covered by the head curve:

    • 3P c700 (226,225 @ 295.7) and 3P c1228 (383,628 @ 267.1) are dominated by 6P DEP32 c3018 (474,138 @ 336.8).
    • 5P c2884 (715,623 @ 220.9) lies under the segment from DEP32 c3018 to 7P c5540 (992,956 @ 172.8), which gives about 840,700 at 220.9.
    • Re-tune deltas against run 41581664540 (throughput / mean interactivity):
      • agg c1: +0.02% / +1.49%
      • agg c4: −0.50% / +2.56%
      • agg c16: −0.34% / −0.88%
      • 7P c5540: −2.11% / +14.74%
      • 9P c7480: −3.72% / +16.55%
  • Pareto coverage:

    • There is one affected curve: dsv41flash / agentic-coding / gb300_dynamo-sglang / fp4 / run 43029773811, with agg and disagg points in one series.

    • Metrics are per_gpu.total_tput_tps against E2EL, from results_bmk (6 rows, matching the 6 bmk_agentic_* artifacts). Counts come from infx.workflows.pareto_coverage on main. Error-dropped records are 0–2 per point.

      Point Image GPUs P90 E2EL (s) P75 E2EL (s) tput/GPU
      agg TP4 c1 302b78cc 4 5.18 2.73 8,246
      agg TP4 c4 302b78cc 4 6.07 2.52 10,319
      agg TP4 c16 302b78cc 4 6.30 2.83 29,254
      6P1D DEP32 c3018 314bce7c 44 9.10 4.44 474,138
      7P1D c5540 314bce7c 30 21.84 13.11 992,956
      9P1D c7480 314bce7c 34 28.04 16.25 1,071,944
    • P90 E2EL: 6/6 frontier points.

    • P75 E2EL: 5/6 frontier points. Only agg TP4 c1 is off the frontier; agg TP4 c4 dominates it.

    • Both percentiles meet the 5-point recommendation.

Signed: nvpohanh

@github-actions

Copy link
Copy Markdown
Contributor

✅✅✅ Verdict: PASS ✅✅✅

Passed and not applicable checks

✅ Check 0 (CODEOWNER): PASS — @nvpohanh is a named owner of inferencex-e2e/configs/nvidia-master.yaml and of the **/gb300-*/ srt-slurm recipe paths; perf-changelog.yaml falls to the catch-all.

✅ Check 1 (Sweep on in-PR commit): PASS — run 43029773811 (attempt 2, pull_request) ran on the pinned head 482df60e; all 6 multi-node agentic //multi-node canary / and all 6 multi-node agentic eval / check-runs are success; no commit follows the pinned SHA.

✅ Check 2 (Evals): PASS — gsm8k em_strict 0.9735–0.9780 (n=1319) on all 6 points in eval_results_all, eval jobs ran on each config's image (302b78cc agg, 314bce7c disagg); every sibling and cohort gap (vs run 41581664540) is ok (max 0.0045 vs 2σ ≈ 0.012); the block-wise-rejection sampled path is covered by the DEP32 eval job's temperature-1 AL gate on decode workers with SGLANG_OPT_DSV41_BLOCKWISE_REJECTION=1 (AL 3.3698 vs golden 3.51, ratio 0.96, pass).

✅ Check 3 (Upstream recipe): PASS — sgl-project/sglang#50081 is MERGED (2026-11-12T11:25:20Z); docs/src/snippets/agentx/deepseek-ai/deepseek-v4_1.jsx on sglang main cites InferenceX 482df60e and covers agg TP4 c1/c4/c16 (--stream-interval 32, 302b78cc) and disagg 6P1D DEP32 c3018, 7P c5540, 9P c7480 (314bce7c) with etcd, Dynamo frontend, prefill and decode commands, including the DEP32 decode flags and the SGLANG_OPT_DSV41_BLOCKWISE_REJECTION=1 env; InferenceX-only exclusions (logging, Slurm plumbing, AIPERF_*, SGLANG_SIMULATE_ACC_*) are explained.

✅ Check 4 (Reuse command): PASS — /use 43029773811 posted by nvpohanh (permission write, role write), matching the Check 1 run.

✅ Check 5 (Latest checklist): PASS — all 20 current-template items, including the recipe sub-item, are present and checked.

✅ Check 6 (Upstream images): PASS — both images are upstream lmsysorg/sglang nightlies with digest pins; build commits 314bce7cf (disagg) and 52604e1d4 (agg) are ancestors of sgl-project/sglang main (159 and 1081 behind); dynamo-sglang is an open-source engine entry, so engine-first ordering does not apply.

✅ Check 7 (Deprecated models): PASS — dsv41flash agentic coding (SGLang DSpark) is active in MODELS.md on 2026-11-12.

✅ Check 8 (Benchmark hacks): PASS — no --hf-overrides, --json-model-override-args, or model-config edits in the recipes or launch logs; block-wise rejection changes only sampled verify acceptance (covered by the AL gate); dsv41flash is an existing model on an NVIDIA CUDA path with no integer all-reduce control.

✅ Check 9 (Chat template): PASS — the AgentX client replays through the Dynamo frontend with AIPERF_EXTRA_INPUTS chat_template_kwargs (thinking on, reasoning_effort high) in both recipes.

✅ Check 10 (Engine patches): PASS — at 482df60e neither recipe has a setup script, patch, pip, sed, or overlay; dynamo.install: false; the PR does not touch runners/srt-slurm/patches/.

✅ Check 11 (Golden AL): PASS — the agg canary and DEP32 benchmark launches show SGLANG_SIMULATE_ACC_LEN=3.51, match-expected, real-draft-token, matching deepseek-v4.1-flash__dspark__native__probabilistic-block.yaml thinking_on block size 5 = 3.51.

➖ Check 12 (Append-only): N/A — the new changelog entry does not set append-only: true.

✅ Check 13 (Draft precision): PASS — the draft is the DSpark head bundled in deepseek-ai/DeepSeek-V4.1-Flash; the launch logs have no draft quantization, dtype, or draft-path overrides; KV fp8_e4m3 is shared by target and draft; at 314bce7cf, environ.py defines SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE = EnvBool(False) and it is absent from both recipes and the inspected job logs; SGLANG_DSPARK_OPT_MARKOV_W2_BF16 defaults to True.

✅ Check 14 (Pareto coverage): PASS — one curve (dsv41flash/agentic-coding/gb300_dynamo-sglang/fp4/run 43029773811, agg and disagg together) from results_bmk, per_gpu.total_tput_tps vs E2EL: P90 6/6 frontier points, P75 5/6 (agg c4 dominates agg c1).

✅ Check 15 (PR description): PASS — the stated images, versions, point set, DEP32 settings, stream intervals, and sweep throughput/mean-interactivity values match the pinned recipes and results_bmk (agg c1/c16 p90 interactivity is quoted about 0.5% off, 943.4/423.7 vs 947.8/424.3, which is not material).

✅ Check 16 (NVIDIA overview): PASS — the NVIDIA overview Framework Recipes table lists DeepSeek-V4.1-Flash with a populated SGLang cell.

➖ Check 17 (Metrics publication): N/A — these are dynamo-sglang configs; the Dynamo TRT-LLM metrics controls do not apply.

✅ Check 18 (Recipe/image/sweep consistency): PASS — master 6×DEP2 + DEP32 matches the recipe (3+8 nodes, 44 GPUs in results); the new image resolves by digest; no orchestration or dispatch changes; the changelog is accurate; removed c1228 is dominated by DEP32 c3018, and c2884 and c700 lie under the head curve's segments (c700, 226,225 @ 295.0 by 1/ITL, sits under agg c16 to DEP32 c3018, about 469k; the facts' per-key undominated_removed ignores the agg points); re-tune deltas: agg c1 +0.02%/+1.92%, c4 −0.50%/+1.85%, c16 −0.34%/−0.52%, 7P −2.11%/+1.99%, 9P −3.72%/+8.61% (tput/mean intvty); stream interval is 32 on agg, prefill, and decode.

Assessed commit: 482df60e0019e219baf2234400b649cff3b13efd.

@nvpohanh nvpohanh removed the full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) label Nov 13, 2026
@nvpohanh
nvpohanh merged commit 7b64020 into main Nov 13, 2026
30 checks passed
@nvpohanh
nvpohanh deleted the weireweire/dsv41flash-gb300-sgl-dep32-midcurve branch November 13, 2026 02:27
@github-actions

Copy link
Copy Markdown
Contributor

Ingest report — ✅ Results ingested into the production database.

Ingest workflow · Merge Ingest run · Benchmark source run

Category new duplicate skipped unmapped failed
benchmark 6 0 6 0 0
eval 6 6 0 0 0
trace 6 0 0 0 0
power 31 0 0 0 0

benchmarkFiles 7, rawBenchmarkRows 12, coveredByPerJobFiles 6, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 7914, gpuMetricSamplesNew 853160, powerPublicationPoints 6, changelogEntriesWritten 1

Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun.

中文

Ingest 报告 — ✅ 结果已写入 production 数据库。

上表仅统计本次运行写入的数据,各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants