Repository navigation
[NVIDIA][AgentX] GB300 Kimi-K3.1-Pro FP4 Dynamo+SGLang: retune disaggregated and megakernel points / [NVIDIA][AgentX] GB300 Kimi-K3.1-Pro FP4 Dynamo+SGLang:重新调优分离式与 megakernel 配置点 - #4893
Conversation
Replace the disaggregated and aggregated megakernel search spaces with nine points on the same image, draft and golden AL: - TP16 decode: 4P1D c520 and 3P1D c340 verify 5 tokens with a 132 GB per-rank prefill HiCache; 2P1D c180 verifies 6; 1P1D c60 verifies 8. - TP8 decode: 1P3D c50 verifies 8; 1P4D c36 and c16 verify 9. - Megakernel: c2 with max-running-requests 1; c4 with max-running-requests 2 and K31MK_* schedule options. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the contribution!
中文感谢你的贡献!
|
|
[by Claude Code] I've started babysitting the pipeline of this PR. I'll keep monitoring it until it passes. If a job fails, I may rerun it when the cause is infrastructure, or push a fix or a merge of the latest |
| - "TP8-decode rows: 1P3D c50 verifies 8 tokens; 1P4D c36 and c16 verify 9." | ||
| - "Aggregated megakernel rows: c2 with max-running-requests 1; c4 with max-running-requests 2 and K31MK_* megakernel schedule options." | ||
| - "Remove 3P1D c300, 1P1D c76, 1P3D c60 and c36, 1P4D c32 and the c1 megakernel row." | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXXX |
There was a problem hiding this comment.
🔴 Merging this PR through the changelog tooling will fail: the new entry's placeholder link is not one the merge step accepts. The pr-link is https://github.com/SemiAnalysisAI/InferenceX/pull/XXXX, which has four X's. PR_LINK_PLACEHOLDERS in validate_perf_changelog.py:21-24 only accepts XXX or .../pull/XXX. Fix: use https://github.com/SemiAnalysisAI/InferenceX/pull/XXX (three X's), as CONTRIBUTING.md:213 and other entries do, or the real PR number.
Why this was flagged
The appended entry at perf-changelog.yaml:12434 sets pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXXX. At merge time, merge_with_reuse imports prepare_perf_changelog_merge, and its canonicalize path calls compare_entries(..., pr_number) (prepare_perf_changelog_merge.py:88). That runs validate_added_pr_link (validate_perf_changelog.py:105-116). The link is neither the expected .../pull/<n> nor in PR_LINK_PLACEHOLDERS = {"XXX", ".../pull/XXX"}, so it raises "new PR entry must use ... or an XXX placeholder". The same check at prepare_perf_changelog_merge.py:108 raises "unexpected pr-link", and resolve-conflict hits it as well. The result is that the placeholder never becomes the real PR link and the merge or conflict-recovery flow stops. The PR-time check in run-sweep.yml (validate_perf_changelog main) only runs infx.matrix.plan and never calls compare_entries, so CI stays green on the PR and the error only appears at merge.
Verification: The new tail entry sets pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXXX (perf-changelog.yaml:12434, four X's). validate_added_pr_link (validate_perf_changelog.py:105-116) raises ChangelogValidationError when link not in PR_LINK_PLACEHOLDERS and link != expected. The error is not caught in canonicalize_changelog (merge_with_reuse.py:688), so the merge helper aborts.
| nodes: 4 | ||
| workers: 2 | ||
| args: | ||
| # Verifies the current token plus 5 drafts. |
There was a problem hiding this comment.
🟡 nit (optional): Two new recipe comments break the AGENTS.md comment rules. sweep:# (Verifies the current token plus 5 drafts\.$|Megakernel schedule options for the 2-request verify step) One is at disagg-tp16-variants.yaml:246. It only restates speculative-num-draft-tokens: 6, while the c520 version says why the value was chosen. The other is at agg-tp16-mk-variants.yaml:152 and is a header naming the K31MK_* block. Fix: delete both, or replace each with a one-line reason for the chosen values.
Why this was flagged
AGENTS.md "Comments (mandatory, every file)" bans header comments that describe a section. It also says comments may only explain why a value was chosen and must "never restate the code". At disagg-tp16-variants.yaml:246 the new comment "# Verifies the current token plus 5 drafts." only re-describes the value 6 on line 247. The matching comment in override_4p1d_c520 at line 220 adds a reason ("at this batch the shorter block is faster"), but this one does not. At agg-tp16-mk-variants.yaml:152 the comment "# Megakernel schedule options for the 2-request verify step." sits above the env block and labels it, which is a section header. Neither comment exists on the base branch, so merging adds two rule violations to recipes AGENTS.md says the rules apply to. No tooling enforces these rules, which is why they got through.
Verification: The rule: AGENTS.md:54-58 "Comments (mandatory, every file)". disagg-tp16-variants.yaml:246 reads # Verifies the current token plus 5 drafts. above speculative-num-draft-tokens: 6. It gives no reason for the choice. agg-tp16-mk-variants.yaml:152 reads # Megakernel schedule options for the 2-request verify step., which heads the 15 K31MK_* env lines. Nothing fails at runtime.
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=42376772667 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=42376772667 |
|
[by Claude Code] The SGLang cookbook integration of this PR's AgentX configs ( This PR has no authorized |
functionstackx
left a comment
There was a problem hiding this comment.
Approved for the K3.1 precision/architecture review at b5c7c8115289764641e6cb7afd64299edd732e16. I found no blocking violation in the effective megakernel paths reviewed against the K3.1 precision policy and draft-precision rules. This approval is not a merge or a completed CODEOWNER checklist.
What was audited
I inspected the five-file PR diff, the affected aggregated and disaggregated recipes, and the actual SGLang and sgl_nvidia_mega sources extracted from the unchanged pinned image, rather than substituting today's upstream source for the code that ran. The image and nine selected configurations are recorded in this PR and its exact-head run.
Image:
lmsysorg/sglang:dev-kimi-k31-pro@sha256:67fc77d1cbe67de27ca54d874c82757a4844743fea899ea7d0df547e30c2c391
SGLang source layer:
sha256:69d0671df9e1c16ed757f0d548da6a5457b6f1bcb6bd67784062c9db0d32e86f
Megakernel package layer:
sha256:efd970ca34afaae654b726d309034a70f9597795969152fb0194b06a75b8433b
Installed SGLang:
0.0.0.dev1+ge79de62d2.d20261104
Precision and computation
The following paths refer to the source extracted from that image, not unpinned upstream main.
| Area | Finding and implementation evidence |
|---|---|
| Linear and shared-expert operations | BF16 projection/shared-expert paths, with FP32 accumulation. Inspected kimi_k3/cutedsl/gemm/projection.py, moe/shared.py, the weight adapters and verify executor. Routed-expert MXFP4 is within the model policy. |
| Combine and activation collectives | BF16 combine/output and communication buffers in moe/tail.py, moe/tail_runtime.py, common/twoshot.py and oproj/prepared.py. The optional MXFP8 o-proj collective exists, but K31MK_OPROJ_FP8 defaults to 0 and is not enabled by the image/recipes reviewed. |
| KDA and convolution state | kimi_k3/providers.py requires FP32 temporal state and BF16 convolution state. kimi_k31/verify_kda.py passes these pools through to the verify provider; runtime startup also resolves mamba_ssm_dtype=float32. |
| Architecture | Full 93-layer execution, including 69 KDA and 24 MLA/DSA layers. The model checks preserve index top-k 2048 and indexer frequency 1; verify_executor.py iterates the layers and runs the MLA indexer, rather than skipping indexer layers. |
| Draft and draft head | The pinned draft's Hugging Face metadata reports all 2,784,811,008 parameters as BF16, with no quantized parameter group. The inspected draft projection/MLP/head paths use BF16 weights; there is no submission-side draft quantization override. Target and draft KV are both explicitly FP8, as allowed by the draft-KV rule. See the revision-specific metadata and draft config. |
| Prohibited draft flag | SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is absent from the effective recipe/image overrides reviewed and defaults to false in the pinned SGLang environment definitions. |
| Image integrity | No engine patch, replacement wheel, or image change is introduced by this PR. The plugin is shipped inside the pinned image. |
Two details deserve explicit treatment:
- c2 is not an optimization-free baseline.
kimi_k3/default_scope.pyenables the BEST defaults at TP16/DCP1/Q8/max-running-requests=1, including draft optimizations. c4 uses max-running-requests=2 and explicitly sets 15K31MK_*controls; I checked both selection mechanisms. - The two norm folds are not bitwise-preserving.
K31MK_OPROJ_NORM_FOLDandK31MK_TAIL_RMS_FOLDrearrange normalization/rounding while retaining BF16 computation, rather than reducing weight/cache precision or dropping model layers. The c4 startup log explicitly reports both non-bitwise paths; this approval does not call them mere scheduling changes or claim numerical identity with stock kernels (c4 job).
The long-context index split retains exact FP32-score top-2048 selection: K31MK_INDEX_SPLIT_COARSE_BITS=12 controls histogram bins, not model/cache precision. I followed kimi_k31/dsa/split.py through the pinned topk_v2.cuh wrapper: when the clustered threshold bin exceeds the tie buffer, it reruns the row with the exact streaming selector before emitting candidates. The distributed merge retains global top-2048; the bitonic draft-head option likewise changes the sorting implementation, not the requested top-k.
Recipe and run evidence
The affected recipes inherit SGLang's stream_interval=1, with no larger override; the c4 runtime log confirms 1. DFLASH uses block verification, and the harness maps verify widths 5/6/8/9 to the committed golden ALs 3.25/3.43/3.68/3.81 while removing synthetic acceptance for eval (run, policy/checklist).
The exact-head run has nine performance results and nine GSM8K eval results, each eval covering 1,319 questions; strict accuracy ranges from 97.877% to 98.256%, with c2 and c4 both 97.953%. The c4 real-acceptance artifact reports AL 3.6807 against golden 3.68, status pass, and report_only=true (run and artifacts). These are existing run results, not an independent GPU rerun; GSM8K is not the full KVV/TB4 suite and does not waive precision rules.
I also verified the merged cookbook update covers the aggregated and disaggregated production settings, including the c4 controls, rather than only linking an InferenceX recipe. SGLang #49260 merged on November 9.
Merge remains separate
At review time GitHub reports merge conflicts, the PR has no authorized /use pin, and the required eligible CODEOWNER checklist is still outstanding. Resolve those through the normal repository process; this technical approval does not replace them or authorize a merge (PR, review requirements).
中文
本次批准针对提交 b5c7c8115289764641e6cb7afd64299edd732e16 的 K3.1 精度与模型计算合规性审查,不代表已合并,也不替代 CODEOWNER checklist。
审查使用固定镜像内实际打包的 SGLang 和 sgl_nvidia_mega 源码,而非当前上游主分支。已检查 BF16 线性层、共享专家、combine 和激活通信路径,允许的 MXFP4 routed experts,FP32 KDA 状态、BF16 conv 状态,以及完整的 93 层计算与每个 MLA 层的 indexer。固定版本 draft 的存储参数全部为 BF16;未发现提交侧降低 draft/head 精度的设置。K31MK_OPROJ_FP8 和 SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE 均未启用。
c2 会通过镜像内的 BEST scope 自动启用优化,不能当作无优化基线。c4 显式设置了 15 个 K31MK_* 控制项;其中两个 norm fold 会改变舍入顺序,并非逐位一致,但没有降低规定的精度或跳过模型层。长上下文 index split 的 12-bit 参数控制直方图分桶;候选缓冲溢出时会重新执行精确 streaming top-k,而非截断候选后直接使用。
固定提交的运行包含九个性能点和九个 GSM8K eval,准确率为 97.877%–98.256%;c2/c4 均为 97.953%。c4 的真实 AL 为 3.6807,对应 golden 3.68,且 AL gate 标记为 report-only。上述证据不等于完整 KVV/TB4,也不是本次另行执行的 GPU 测试。有效 stream interval 为 1,性能测试采用 golden synthetic AL,eval 则移除 synthetic acceptance。官方 cookbook 更新 #49260 已合并,覆盖 aggregated/disaggregated 配置。
当前仍需处理合并冲突、授权的 /use 运行绑定,以及符合资格的 CODEOWNER checklist。本次仅批准所审查的技术内容,不执行合并。
Created with Perplexity
|
/use 42376772667 |
|
@nvpohanh staged run 42376772667. This run remains available across future |
|
@nvpohanh staging run 42376772667. Existing staged runs will be preserved; a completion link will be posted here. |
…0-dynamo-sglang-agentx-frontier-20261107 Resolve the perf-changelog.yaml conflict by keeping main's entries and re-appending this PR's entry at the end. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Ingest report — ✅ Results ingested into the Ingest workflow · Merge Ingest run · Benchmark source run
benchmarkFiles 10, rawBenchmarkRows 18, coveredByPerJobFiles 9, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 11871, gpuMetricSamplesNew 1834164, powerPublicationPoints 9, changelogEntriesWritten 1 Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun. 中文Ingest 报告 — ✅ 结果已写入 上表仅统计本次运行写入的数据,各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。 |
Retunes the two Kimi-K3.1-Pro GB300 Dynamo + SGLang AgentX config keys from #4811,
kimik3.1-fp4-gb300-dynamo-sglang-agentic-dflash-disaggandkimik3.1-fp4-gb300-dynamo-sglang-agentic-dflash-agg-mk. The image (lmsysorg/sglang:dev-kimi-k31-pro, same multi-arch digest), theRadixArk/Kimi-K3.1-Dflash2-PreviewDFLASH draft (same pinned revision) and the golden AL curve are unchanged. Only the search space and per-row settings change.What changes. Nine points replace the nine from #4811:
max-running-requestsat half the concurrency.Golden AL. The srt driver injects the committed DFLASH curve from #4798 (
kimi-k3.1-pro__dflash__kimi-k3.1-dflash2-preview__default-block.yaml). Curve level N runs--speculative-num-draft-tokens N+1. Every role keepsspeculative-use-block-verification: true.disagg-tp16-variants.yaml:override_4p1d_c520disagg-tp16-variants.yaml:override_3p1d_c340disagg-tp16-variants.yaml:override_2p1d_c180disagg-tp16-variants.yaml:override_1p1d_c60disagg-tp8-variants.yaml:override_1p3d_c50disagg-tp8-variants.yaml:override_1p4d_c36disagg-tp8-variants.yaml:override_1p4d_c16agg-tp16-mk-variants.yaml:override_tp16_c2max-running-requests1agg-tp16-mk-variants.yaml:override_tp16_c4max-running-requests2Node counts include one node shared by the frontend, etcd and the AgentX client.
Removed rows: 3P1D c300, 1P1D c76, 1P3D c60 and c36, 1P4D c32, and the c1 megakernel row.
dram-utilizationshare of the gb300-nv node budget. Rows not listed with 132 GB keep 96 GB per rank.K31MK_*megakernel schedule options for the 2-request verify step. They cover ordering and fusion of the attention, indexer and KDA stages. They are environment-only, and the c2 row keeps the image defaults. Both megakernel rows setweight-loader-drop-cache-after-load: false, as the disaggregated recipes already do.中文
重新调优 #4811 新增的两个 Kimi-K3.1-Pro GB300 Dynamo + SGLang AgentX config key:
kimik3.1-fp4-gb300-dynamo-sglang-agentic-dflash-disagg与kimik3.1-fp4-gb300-dynamo-sglang-agentic-dflash-agg-mk。镜像(lmsysorg/sglang:dev-kimi-k31-pro,相同的 multi-arch digest)、RadixArk/Kimi-K3.1-Dflash2-PreviewDFLASH draft(相同的固定 revision)与 golden AL 曲线均不变,只修改搜索空间与各配置点的设置。变更内容:用九个配置点替换 #4811 的九个配置点。
max-running-requests限制为并发的一半。Golden AL:srt driver 注入 #4798 提交的 DFLASH 曲线(
kimi-k3.1-pro__dflash__kimi-k3.1-dflash2-preview__default-block.yaml)。曲线 level N 对应--speculative-num-draft-tokens N+1。每个角色都保持speculative-use-block-verification: true。disagg-tp16-variants.yaml:1 个 TP16 decode worker(DCP8 x attention DP2,EP16 MegaMoE),分别配:disagg-tp8-variants.yaml:1 个 TP8/DCP8、EP2 prefill worker,分别配:agg-tp16-mk-variants.yaml:1 个启用 megakernel 的聚合式 TP16、EP1 worker,5 个节点,验证宽度 8(level 7,golden AL 3.68);并发 2 时max-running-requests为 1,并发 4 时为 2。dram-utilization份额之内。其余配置点保持每个 rank 96 GB。K31MK_*megakernel 调度选项,涉及 attention、indexer 与 KDA 阶段的执行顺序与融合方式。这些选项只通过环境变量设置;c2 配置点保持镜像默认值。两个 megakernel 配置点都设置weight-loader-drop-cache-after-load: false,与分离式 recipe 一致。AI model disclosure
claude-opus-5-5), via Claude Code.🤖 Generated with Claude Code