Repository navigation
[Klaud Cold][NVIDIA][AgentX] GB300 Kimi-K3.1-Pro FP4 Dynamo+SGLang: DFLASH disaggregated and aggregated megakernel points / [Klaud Cold][NVIDIA][AgentX] GB300 Kimi-K3.1-Pro FP4 Dynamo+SGLang:DFLASH 分离式与聚合式 megakernel 配置点 - #4837
functionstackx wants to merge 3 commits into
Conversation
…FLASH disaggregated and aggregated megakernel points / [Klaud Cold][NVIDIA][AgentX] GB300 Kimi-K3.1-Pro FP4 Dynamo+SGLang:DFLASH 分离式与聚合式 megakernel 配置点 Duplicate of #4811 (nvpohanh/kimik31-gb300-dynamo-sglang-dflash-disagg) rebuilt on current main so the sweep runs from a same-repository branch. The perf-changelog entry is appended after main's newest entries. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
There was a problem hiding this comment.
Findings marked 🟡 are optional suggestions and need no follow-up push.
Additional findings (outside the current diff — GitHub can't attach inline comments there):
-
🔴
inferencex-e2e/perf-changelog.yaml— The changelog check on this PR's run will reject the new entry, so the sweep cannot be validated or ingested under this PR. The appended block still links #4811, the conflicting PR this one duplicates.validate_added_pr_linkaccepts a new entry only when its link is this PR's own URL or an XXX placeholder. Fix: setpr-linkto this PR's URL, https://github.com/SemiAnalysisAI/InferenceX/pull/, or to the XXX placeholder.Why this was flagged
This PR describes itself as a duplicate of #4811, rebuilt on main, so its PR number is not 4811. inferencex-e2e/perf-changelog.yaml:742 sets
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4811. On a PR run, infx/workflows/validate_perf_changelog.py:112-116 buildsexpected = f"https://github.com/SemiAnalysisAI/InferenceX/pull/{pr_number}"and raisesnew PR entry must use ... or an XXX placeholderfor any other link. That fails the changelog validation for this PR. The ingest-recovery tooling also compares links to the expected one (infx/workflows/recover_failed_ingest.py:306-308), so results tied to this entry would point at the abandoned PR. Before this change there was no such entry, so nothing failed.Verification: Trigger: this PR is merged through the repository's merge-with-reuse flow. The appended block ends with
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4811(inferencex-e2e/perf-changelog.yaml:12091). The PR-runcheck-changelogjob will not reject this entry. merge_with_reuse.py:688 reachesvalidate_added_pr_link, which raises, so the reuse-merge aborts. -
🟡
inferencex-e2e/configs/nvidia-master.yaml— Results for the three TP16-decode points will be labeled as having no attention data parallelism. The decode worker actually runs attention DP2. The recipe decode role setsattn-dp-size: 2(disagg-tp16-variants.yaml:158). The master rows declaredecode: { ... dp-attn: false ... }. Fix: set decodedp-attn: trueon the rows for override_3p1d_c300, override_2p1d_c180 and override_1p1d_c76, so the master matches what the recipe deploys.Why this was flagged
The three disagg-tp16 rows at nvidia-master.yaml:7470, :7477 and :7484 set decode
dp-attn: false. The recipe they select deploystensor-parallel-size: 16,dcp-size: 8andattn-dp-size: 2(disagg-tp16-variants.yaml:325-327), and its own comment calls this "DCP8 x attention DP2". The repo treatsattn-dp-sizeas the replacement fordp-size+enable-dp-attention(perf-changelog.yaml:10924). RECIPES.md:48 says the master supplies result labels and must stay in sync with the recipe. The master value flows toDECODE_DP_ATTNand into the result'sdecode_dp_attentionfield (infx/results/fixed_sequence.py:136-154, infx/srt_slurm/workload.py:72). Multi-node has no parallelism validator like single_node.py, so nothing catches the mismatch. Published points would therefore be grouped and compared as non-DP-attention TP16 decode.Verification: Severity: nit. Only the labels on the results are wrong. The decode role in disagg-tp16-variants.yaml sets
attn-dp-size: 2at lines 156-158, and the three overrides leave it unchanged. The master rows at nvidia-master.yaml:7470, 7477 and 7484 declaredp-attn: false. No multi-node validator compares the recipe against the master.
…1-gb300-dflash # Conflicts: # inferencex-e2e/perf-changelog.yaml
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=42079634829 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=42079634829 |
Re-sweep of the Kimi-K3.1-Pro GB300 Dynamo+SGLang DFLASH AgentX points from #4811 (
nvpohanh), which merged into main at 17:41 UTC on 2026-11-05. Main already holds the recipes and master-config entries, so this PR only appends a perf-changelog entry that re-lists the two config keys and reruns every point unchanged. The description below is carried over from #4811 for reference.Adds the first Kimi-K3.1-Pro GB300 Dynamo + SGLang AgentX config keys:
kimik3.1-fp4-gb300-dynamo-sglang-agentic-dflash-disagg(disaggregated) andkimik3.1-fp4-gb300-dynamo-sglang-agentic-dflash-agg-mk(aggregated, low latency, with the NVIDIA megakernel). Both runlmsysorg/sglang:dev-kimi-k31-pro(multi-arch digest pinned) with the Dynamo 1.6.0 bundled in the image (dynamo.install: false), and speculate with theRadixArk/Kimi-K3.1-Dflash2-PreviewDFLASH draft (revision pinned).Golden AL. The srt driver injects the committed DFLASH curve from #4798,
kimi-k3.1-pro__dflash__kimi-k3.1-dflash2-preview__default-block.yaml. DFLASH counts the verification width, so curve level N runs--speculative-num-draft-tokens N+1. The c300 point verifies 6 tokens (level 5, golden AL 3.43); every other point verifies 8 (level 7, golden AL 3.68). The curve was collected with block verification, so every role setsspeculative-use-block-verification: trueand the speculative-identity audit matches it with no findings.disagg-tp16-variants.yaml:override_3p1d_c300disagg-tp16-variants.yaml:override_2p1d_c180disagg-tp16-variants.yaml:override_1p1d_c76disagg-tp8-variants.yaml:override_1p3d_c60disagg-tp8-variants.yaml:override_1p3d_c36disagg-tp8-variants.yaml:override_1p4d_c32agg-tp16-mk-variants.yaml:override_tp16_c1agg-tp16-mk-variants.yaml:override_tp16_c2agg-tp16-mk-variants.yaml:override_tp16_c4Node counts include one node shared by the frontend, etcd and the AgentX client.
cutedsl_mladraft attention and an FP8 draft KV cache; prefill and decode use the same verify width. No recipe setsSGLANG_SIMULATE_ACC_*orSGLANG_NVFP4_CKPT_FP8_NEXTN_MOE. The draft is read from the shared Hub cache (HF_HUB_CACHE=/hf_hub_cache), so the pinned revision must be staged there or downloadable from the workers.dsa-*-backend: prims_ts), FP8 KV cache, 16k chunks and shortest-prefill-first scheduling. HiCachewrite_backholds 96 GB per rank, 384 GB per prefill node, inside the 0.80dram-utilizationshare of the gb300-nv node budget. Prefill workers advertise KV routing with a 14400 s session-affinity TTL.enable-linear-replayssm-spec).router-decode-active-request-weightsends each turn to the least-loaded decode worker.--enable-nvidia-megakernel.max-running-requestsequals the concurrency (1, 2 or 4), because the megakernel serves verify steps of 1, 2 or 4 requests and larger batches fall back to the eager path.trtllmDSA prefill and decode backends,flashinfer_mxfp4MoE, an FP8 KV cache with 64-token pages, ReplaySSM speculation,--language-onlyand an explicit--context-length(1048576, the model's maximum).SGLANG_K3_SHARDED_MOE=0: at TP16 the sharded MoE front has no split-K tile for its per-rank width, so graph capture fails without it.MC_TCP_BIND_ADDRESSandWITH_NVIDIA_PEERMEMsettings of the sibling GB300 recipes. One Dynamo frontend uses the KV router and the SGLang chat processor. The AgentX client sends thinking on, temperature 1 and top-p 0.95, with Dynamo session IDs.中文
新增第一批 Kimi-K3.1-Pro GB300 Dynamo + SGLang AgentX config key:
kimik3.1-fp4-gb300-dynamo-sglang-agentic-dflash-disagg(分离式)与kimik3.1-fp4-gb300-dynamo-sglang-agentic-dflash-agg-mk(聚合式低延迟配置,启用 NVIDIA megakernel)。两者均使用lmsysorg/sglang:dev-kimi-k31-pro(固定 multi-arch digest),直接使用镜像自带的 Dynamo 1.6.0(dynamo.install: false),并以RadixArk/Kimi-K3.1-Dflash2-PreviewDFLASH draft(固定 revision)进行推测解码。Golden AL:srt driver 注入 #4798 提交的 DFLASH 曲线
kimi-k3.1-pro__dflash__kimi-k3.1-dflash2-preview__default-block.yaml。DFLASH 计算的是验证宽度,因此曲线的 level N 对应--speculative-num-draft-tokens N+1。c300 配置点验证 6 个 token(level 5,golden AL 3.43);其余配置点均验证 8 个 token(level 7,golden AL 3.68)。该曲线是在启用 block verification 的情况下采集的,因此每个角色都设置speculative-use-block-verification: true,推测解码身份审计与之完全匹配,没有任何发现项。disagg-tp16-variants.yaml:1 个 TP16 decode worker(DCP8 x attention DP2,EP16 MegaMoE),分别配 3/2/1 个 TP8/DCP8、EP2 prefill worker,并发 300/180/76,节点数 11/9/7。c300 的验证宽度为 6(level 5,golden AL 3.43),其余为 8(level 7,golden AL 3.68)。disagg-tp8-variants.yaml:1 个 TP8/DCP8、EP2 prefill worker,配 3 个 TP8/DCP8、EP2 decode worker(并发 60 与 36,9 个节点)或 4 个(并发 32,11 个节点)。验证宽度均为 8(level 7,golden AL 3.68)。agg-tp16-mk-variants.yaml:1 个聚合式 TP16、EP1 worker,启用 megakernel,并发 1/2/4,5 个节点。验证宽度为 8(level 7,golden AL 3.68)。cutedsl_mla,draft KV cache 为 FP8;prefill 与 decode 的验证宽度一致。recipe 中未设置SGLANG_SIMULATE_ACC_*或SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE。draft 从共享 Hub 缓存(HF_HUB_CACHE=/hf_hub_cache)读取,因此固定的 revision 需要预先放入该缓存,或允许 worker 下载。dsa-*-backend: prims_ts),FP8 KV cache,16k chunk,shortest-prefill-first 调度。HiCachewrite_back每个 rank 96 GB,每个 prefill 节点 384 GB,在 gb300-nv 节点预算的 0.80dram-utilization份额之内。prefill worker 声明 KV 路由及 14400 秒 session affinity TTL。enable-linear-replayssm-spec)。router-decode-active-request-weight将每一轮请求发往负载最低的 decode worker。--enable-nvidia-megakernel。max-running-requests等于并发(1、2 或 4),因为 megakernel 只服务 1、2 或 4 个请求的验证步骤,更大的 batch 会回退到 eager 路径。trtllmDSA prefill 与 decode backend、flashinfer_mxfp4MoE、64 token page 的 FP8 KV cache、ReplaySSM 推测、--language-only,以及显式的--context-length(1048576,即模型最大值)。SGLANG_K3_SHARDED_MOE=0:在 TP16 下,分片 MoE front 的每 rank 宽度没有可用的 split-K tile,不设置该变量会导致 CUDA graph capture 失败。MC_TCP_BIND_ADDRESS与WITH_NVIDIA_PEERMEM设置。一个 Dynamo frontend 使用 KV router 与 SGLang chat processor。AgentX 客户端开启 thinking,temperature 1,top-p 0.95,并携带 Dynamo session ID。AI model disclosure
claude-opus-5-5), via Claude Code.claude-fable-5-1), via Claude Code.🤖 Generated with Claude Code