Skip to content

[Klaud Cold][NVIDIA][AgentX] GB300 Kimi-K3.1-Pro FP4 Dynamo+SGLang: DFLASH disaggregated and aggregated megakernel points / [Klaud Cold][NVIDIA][AgentX] GB300 Kimi-K3.1-Pro FP4 Dynamo+SGLang:DFLASH 分离式与聚合式 megakernel 配置点 - #4837

Closed
functionstackx wants to merge 3 commits into
mainfrom
klaud/dup-4811-kimik31-gb300-dflash

Conversation

@functionstackx

@functionstackx functionstackx commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

Re-sweep of the Kimi-K3.1-Pro GB300 Dynamo+SGLang DFLASH AgentX points from #4811 (nvpohanh), which merged into main at 17:41 UTC on 2026-11-05. Main already holds the recipes and master-config entries, so this PR only appends a perf-changelog entry that re-lists the two config keys and reruns every point unchanged. The description below is carried over from #4811 for reference.

Adds the first Kimi-K3.1-Pro GB300 Dynamo + SGLang AgentX config keys: kimik3.1-fp4-gb300-dynamo-sglang-agentic-dflash-disagg (disaggregated) and kimik3.1-fp4-gb300-dynamo-sglang-agentic-dflash-agg-mk (aggregated, low latency, with the NVIDIA megakernel). Both run lmsysorg/sglang:dev-kimi-k31-pro (multi-arch digest pinned) with the Dynamo 1.6.0 bundled in the image (dynamo.install: false), and speculate with the RadixArk/Kimi-K3.1-Dflash2-Preview DFLASH draft (revision pinned).

Golden AL. The srt driver injects the committed DFLASH curve from #4798, kimi-k3.1-pro__dflash__kimi-k3.1-dflash2-preview__default-block.yaml. DFLASH counts the verification width, so curve level N runs --speculative-num-draft-tokens N+1. The c300 point verifies 6 tokens (level 5, golden AL 3.43); every other point verifies 8 (level 7, golden AL 3.68). The curve was collected with block verification, so every role sets speculative-use-block-verification: true and the speculative-identity audit matches it with no findings.

Recipe override Prefill Decode or aggregated worker Conc Nodes Verify width (level, golden AL)
disagg-tp16-variants.yaml:override_3p1d_c300 3x TP8/DCP8, EP2 1x TP16 (DCP8 x attention DP2), EP16 MegaMoE 300 11 6 (5, 3.43)
disagg-tp16-variants.yaml:override_2p1d_c180 2x TP8/DCP8, EP2 1x TP16 (DCP8 x attention DP2), EP16 MegaMoE 180 9 8 (7, 3.68)
disagg-tp16-variants.yaml:override_1p1d_c76 1x TP8/DCP8, EP2 1x TP16 (DCP8 x attention DP2), EP16 MegaMoE 76 7 8 (7, 3.68)
disagg-tp8-variants.yaml:override_1p3d_c60 1x TP8/DCP8, EP2 3x TP8/DCP8, EP2 60 9 8 (7, 3.68)
disagg-tp8-variants.yaml:override_1p3d_c36 1x TP8/DCP8, EP2 3x TP8/DCP8, EP2 36 9 8 (7, 3.68)
disagg-tp8-variants.yaml:override_1p4d_c32 1x TP8/DCP8, EP2 4x TP8/DCP8, EP2 32 11 8 (7, 3.68)
agg-tp16-mk-variants.yaml:override_tp16_c1 none (aggregated) 1x TP16, EP1, megakernel 1 5 8 (7, 3.68)
agg-tp16-mk-variants.yaml:override_tp16_c2 none (aggregated) 1x TP16, EP1, megakernel 2 5 8 (7, 3.68)
agg-tp16-mk-variants.yaml:override_tp16_c4 none (aggregated) 1x TP16, EP1, megakernel 4 5 8 (7, 3.68)

Node counts include one node shared by the frontend, etcd and the AgentX client.

  • Speculation. DFLASH with block verification, cutedsl_mla draft attention and an FP8 draft KV cache; prefill and decode use the same verify width. No recipe sets SGLANG_SIMULATE_ACC_* or SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE. The draft is read from the shared Hub cache (HF_HUB_CACHE=/hf_hub_cache), so the pinned revision must be staged there or downloadable from the workers.
  • Prefill. TP8 with decode context parallel 8 and EP2, Prims-TS sparse attention for prefill and decode (dsa-*-backend: prims_ts), FP8 KV cache, 16k chunks and shortest-prefill-first scheduling. HiCache write_back holds 96 GB per rank, 384 GB per prefill node, inside the 0.80 dram-utilization share of the gb300-nv node budget. Prefill workers advertise KV routing with a 14400 s session-affinity TTL.
  • TP16 decode. DCP8 x attention DP2 over 16 GPUs with EP16 MegaMoE and a DP LM head, 384 running requests, decode CUDA graphs up to batch 192, mem-fraction 0.85 and ReplaySSM linear-attention speculation (enable-linear-replayssm-spec).
  • TP8 decode. TP8/DCP8 with EP2, 64 running requests and mem-fraction 0.88. The frontend's router-decode-active-request-weight sends each turn to the least-loaded decode worker.
  • Aggregated megakernel points. One TP16 EP1 worker over four nodes with --enable-nvidia-megakernel. max-running-requests equals the concurrency (1, 2 or 4), because the megakernel serves verify steps of 1, 2 or 4 requests and larger batches fall back to the eager path.
    • The megakernel spec requires the trtllm DSA prefill and decode backends, flashinfer_mxfp4 MoE, an FP8 KV cache with 64-token pages, ReplaySSM speculation, --language-only and an explicit --context-length (1048576, the model's maximum).
    • SGLANG_K3_SHARDED_MOE=0: at TP16 the sharded MoE front has no split-K tile for its per-rank width, so graph capture fails without it.
    • Megakernel environment levers keep the image defaults; no KV offload.
  • Serving. KV transfer uses Mooncake over MNNVL with the gb300-nv MC_TCP_BIND_ADDRESS and WITH_NVIDIA_PEERMEM settings of the sibling GB300 recipes. One Dynamo frontend uses the KV router and the SGLang chat processor. The AgentX client sends thinking on, temperature 1 and top-p 0.95, with Dynamo session IDs.
中文

新增第一批 Kimi-K3.1-Pro GB300 Dynamo + SGLang AgentX config key:kimik3.1-fp4-gb300-dynamo-sglang-agentic-dflash-disagg(分离式)与 kimik3.1-fp4-gb300-dynamo-sglang-agentic-dflash-agg-mk(聚合式低延迟配置,启用 NVIDIA megakernel)。两者均使用 lmsysorg/sglang:dev-kimi-k31-pro(固定 multi-arch digest),直接使用镜像自带的 Dynamo 1.6.0(dynamo.install: false),并以 RadixArk/Kimi-K3.1-Dflash2-Preview DFLASH draft(固定 revision)进行推测解码。

Golden AL:srt driver 注入 #4798 提交的 DFLASH 曲线 kimi-k3.1-pro__dflash__kimi-k3.1-dflash2-preview__default-block.yaml。DFLASH 计算的是验证宽度,因此曲线的 level N 对应 --speculative-num-draft-tokens N+1。c300 配置点验证 6 个 token(level 5,golden AL 3.43);其余配置点均验证 8 个 token(level 7,golden AL 3.68)。该曲线是在启用 block verification 的情况下采集的,因此每个角色都设置 speculative-use-block-verification: true,推测解码身份审计与之完全匹配,没有任何发现项。

  • 配置点:
    • disagg-tp16-variants.yaml:1 个 TP16 decode worker(DCP8 x attention DP2,EP16 MegaMoE),分别配 3/2/1 个 TP8/DCP8、EP2 prefill worker,并发 300/180/76,节点数 11/9/7。c300 的验证宽度为 6(level 5,golden AL 3.43),其余为 8(level 7,golden AL 3.68)。
    • disagg-tp8-variants.yaml:1 个 TP8/DCP8、EP2 prefill worker,配 3 个 TP8/DCP8、EP2 decode worker(并发 60 与 36,9 个节点)或 4 个(并发 32,11 个节点)。验证宽度均为 8(level 7,golden AL 3.68)。
    • agg-tp16-mk-variants.yaml:1 个聚合式 TP16、EP1 worker,启用 megakernel,并发 1/2/4,5 个节点。验证宽度为 8(level 7,golden AL 3.68)。
    • 节点数包含 frontend、etcd 与 AgentX 客户端共用的一个节点。
  • 推测解码:DFLASH,启用 block verification,draft attention 使用 cutedsl_mla,draft KV cache 为 FP8;prefill 与 decode 的验证宽度一致。recipe 中未设置 SGLANG_SIMULATE_ACC_* 或 SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE。draft 从共享 Hub 缓存(HF_HUB_CACHE=/hf_hub_cache)读取,因此固定的 revision 需要预先放入该缓存,或允许 worker 下载。
  • Prefill:TP8,decode context parallel 8,EP2;prefill 与 decode 的稀疏注意力均使用 Prims-TS(dsa-*-backend: prims_ts),FP8 KV cache,16k chunk,shortest-prefill-first 调度。HiCache write_back 每个 rank 96 GB,每个 prefill 节点 384 GB,在 gb300-nv 节点预算的 0.80 dram-utilization 份额之内。prefill worker 声明 KV 路由及 14400 秒 session affinity TTL。
  • TP16 decode:16 个 GPU 上 DCP8 x attention DP2,EP16 MegaMoE 与 DP LM head,384 个运行请求,decode CUDA graph 最大 batch 192,mem-fraction 0.85,启用 ReplaySSM 线性注意力推测(enable-linear-replayssm-spec)。
  • TP8 decode:TP8/DCP8,EP2,64 个运行请求,mem-fraction 0.88。frontend 的 router-decode-active-request-weight 将每一轮请求发往负载最低的 decode worker。
  • 聚合式 megakernel 配置点:1 个横跨 4 个节点的 TP16、EP1 worker,启用 --enable-nvidia-megakernel。max-running-requests 等于并发(1、2 或 4),因为 megakernel 只服务 1、2 或 4 个请求的验证步骤,更大的 batch 会回退到 eager 路径。
    • megakernel spec 要求 trtllm DSA prefill 与 decode backend、flashinfer_mxfp4 MoE、64 token page 的 FP8 KV cache、ReplaySSM 推测、--language-only,以及显式的 --context-length(1048576,即模型最大值)。
    • SGLANG_K3_SHARDED_MOE=0:在 TP16 下,分片 MoE front 的每 rank 宽度没有可用的 split-K tile,不设置该变量会导致 CUDA graph capture 失败。
    • megakernel 相关环境变量保持镜像默认值;不使用 KV offload。
  • 服务:KV 传输使用基于 MNNVL 的 Mooncake,并沿用同集群 GB300 recipe 的 MC_TCP_BIND_ADDRESS 与 WITH_NVIDIA_PEERMEM 设置。一个 Dynamo frontend 使用 KV router 与 SGLang chat processor。AgentX 客户端开启 thinking,temperature 1,top-p 0.95,并携带 Dynamo session ID。
AI model disclosure

🤖 Generated with Claude Code

…FLASH disaggregated and aggregated megakernel points / [Klaud Cold][NVIDIA][AgentX] GB300 Kimi-K3.1-Pro FP4 Dynamo+SGLang:DFLASH 分离式与聚合式 megakernel 配置点

Duplicate of #4811 (nvpohanh/kimik31-gb300-dynamo-sglang-dflash-disagg)
rebuilt on current main so the sweep runs from a same-repository branch.
The perf-changelog entry is appended after main's newest entries.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings marked 🟡 are optional suggestions and need no follow-up push.

Additional findings (outside the current diff — GitHub can't attach inline comments there):

  • 🔴 inferencex-e2e/perf-changelog.yaml — The changelog check on this PR's run will reject the new entry, so the sweep cannot be validated or ingested under this PR. The appended block still links #4811, the conflicting PR this one duplicates. validate_added_pr_link accepts a new entry only when its link is this PR's own URL or an XXX placeholder. Fix: set pr-link to this PR's URL, https://github.com/SemiAnalysisAI/InferenceX/pull/, or to the XXX placeholder.

    Why this was flagged

    This PR describes itself as a duplicate of #4811, rebuilt on main, so its PR number is not 4811. inferencex-e2e/perf-changelog.yaml:742 sets pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4811. On a PR run, infx/workflows/validate_perf_changelog.py:112-116 builds expected = f"https://github.com/SemiAnalysisAI/InferenceX/pull/{pr_number}" and raises new PR entry must use ... or an XXX placeholder for any other link. That fails the changelog validation for this PR. The ingest-recovery tooling also compares links to the expected one (infx/workflows/recover_failed_ingest.py:306-308), so results tied to this entry would point at the abandoned PR. Before this change there was no such entry, so nothing failed.

    Verification: Trigger: this PR is merged through the repository's merge-with-reuse flow. The appended block ends with pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4811 (inferencex-e2e/perf-changelog.yaml:12091). The PR-run check-changelog job will not reject this entry. merge_with_reuse.py:688 reaches validate_added_pr_link, which raises, so the reuse-merge aborts.

  • 🟡 inferencex-e2e/configs/nvidia-master.yaml — Results for the three TP16-decode points will be labeled as having no attention data parallelism. The decode worker actually runs attention DP2. The recipe decode role sets attn-dp-size: 2 (disagg-tp16-variants.yaml:158). The master rows declare decode: { ... dp-attn: false ... }. Fix: set decode dp-attn: true on the rows for override_3p1d_c300, override_2p1d_c180 and override_1p1d_c76, so the master matches what the recipe deploys.

    Why this was flagged

    The three disagg-tp16 rows at nvidia-master.yaml:7470, :7477 and :7484 set decode dp-attn: false. The recipe they select deploys tensor-parallel-size: 16, dcp-size: 8 and attn-dp-size: 2 (disagg-tp16-variants.yaml:325-327), and its own comment calls this "DCP8 x attention DP2". The repo treats attn-dp-size as the replacement for dp-size + enable-dp-attention (perf-changelog.yaml:10924). RECIPES.md:48 says the master supplies result labels and must stay in sync with the recipe. The master value flows to DECODE_DP_ATTN and into the result's decode_dp_attention field (infx/results/fixed_sequence.py:136-154, infx/srt_slurm/workload.py:72). Multi-node has no parallelism validator like single_node.py, so nothing catches the mismatch. Published points would therefore be grouped and compared as non-DP-attention TP16 decode.

    Verification: Severity: nit. Only the labels on the results are wrong. The decode role in disagg-tp16-variants.yaml sets attn-dp-size: 2 at lines 156-158, and the three overrides leave it unchanged. The master rows at nvidia-master.yaml:7470, 7477 and 7484 declare dp-attn: false. No multi-node validator compares the recipe against the master.

@functionstackx

Copy link
Copy Markdown
Collaborator Author

Closing: the original #4811 was merged into main at 17:41 UTC (merge commit 1cf1fa2) with its full sweep green (9 multi-node agentic + 7 agentic eval jobs), so this duplicate has no remaining purpose. Its branch is kept in case a re-sweep of the same config keys is wanted.

…1-gb300-dflash

# Conflicts:
#	inferencex-e2e/perf-changelog.yaml
@functionstackx functionstackx reopened this Nov 5, 2026
@functionstackx functionstackx added the non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim) label Nov 5, 2026

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review found no issues

No high-confidence issues detected in this change.

This review covers commit f7e60ec, which is no longer the latest commit on this pull request; later commits are not covered by it.

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

@cquil11 cquil11 closed this Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants