[kevin-mii]

Summary

This PR moves the Kimi K3.1 Pro SGLang MI355X AgentX configs to one image for every point, retunes the single-node points for P90 interactivity, and adds the 1P1D PD recipe.

Image, both keys: lmsysorg/sglang:dev-kimi-k31-pro-mi35x@sha256:5901db14 (was sha256:346a6d83).

• SGLang de0f491836: the #48427 branch with [AMD] Kimi-K3: pass gate_mode INTERLEAVE for GU-interleaved SiTU w13 sgl-project/sglang#49152 (SiTU gate) and the PD commits (MXFP4 DSA index keys sent with the KV, FP4 prefill logits scored per request).
• AITER [MoE] [FlyDSL] BM16 adaptive aux sort writes into the caller's output; retune kimik3 a8w4 token-16 inter-384 ROCm/aiter#7620 (5405ee0026, flydsl 0.4.2).
• sgl-amd-mono 219f40be9a (Previous run results are not shown on the website #114, Feature request: add historical comparison #116).

Single node (kimik3.1-fp4-mi355x-sglang-agentic-dflash):

• Concurrency 1: one running request instead of 4, so an agent's parallel tool calls queue instead of sharing decode steps.
• Concurrency 8 to 56: prefill-decode-interval goes from 16 to 32 (from 8 at 48 and 56). Prefills interrupt running requests' token streams half as often, which raised P90 interactivity at every DCP point with the same throughput. TTFT rises (p50 2–4 s).
• Concurrency 8, 16, 32: 6, 8 and 20 running requests. Concurrency 16 and 32 verify the current token plus 4 drafts.
• New concurrency 24 point: 13 running requests and 4 drafts, to fill the curve between 16 and 32.
• Concurrency 1 and 4: the current token plus 7 drafts.
• All points: SGLANG_ENABLE_ROCM_K3_ROW_SHARDED_ROUTED_FRONT, SGLANG_ENABLE_ROCM_K3_ROW_SHARDED_STREAM and SGLANG_OPT_DFLASH_RING_LOCS_FUSED, which were already on at 48 and 56.

PD, new (kimik3.1-fp4-mi355x-sglang-agentic-dflash-disagg):

• Layout: one TP8 DCP8 prefill node and one TP8 DCP8 decode node behind sglang-router 0.3.2.
• Transfer: KV and the MXFP4 index keys over Mooncake RDMA.
• Speculative decoding: five-token DFLASH (DFlash2) on both roles.
• Prefill: 16k chunks, shortest-prefill-first, a KDA checkpoint every 64 tokens, and a write-back HiCache host tier (ratio 4.0, KDA host ratio 5) sized under the node's 1,511 GiB pinned-memory limit.
• Decode: up to 144 running requests at concurrency 128 and 160 at concurrency 144, with matching decode graphs and a decode radix cache.

Concurrency	Total tok/s/GPU (16 GPUs)	P90 interactivity	TTFT p50
144	21,435	31.7	3.75 s
128	21,124	32.8	3.67 s

These points were measured over 3600 s on two MI355X nodes at golden acceptance 3.25 (5 draft tokens). Concurrency 144 is the highest concurrency that beat the one below it by more than 1%. GSM8K was 97.88% at native acceptance with the same configuration.

Accuracy on this stack: we checked a local build with the same SGLang, AITER and sgl-amd-mono commits (not the published digest itself):

• Single node: concurrency 56 scored GSM8K 97.50% with needles 3/3. Concurrency 4 scored GSM8K 97.80% with needles 3/3, and the monokernel launch sort is on.
• PD: GSM8K 97.42% with needles 3/3 through the router. On this digest itself: GSM8K 97.35% with needles 3/3.

Risks:

• PD needs a rebuilt image. This digest was built without ENABLE_MORI=1, so it has no ibverbs provider for the MI355X nodes' AMD Pollara (ionic) or Broadcom NICs. Mooncake finds 0 HCAs and falls back to TCP. Under AgentX load at concurrency 144, each KV transfer then took 50–120 s instead of about 1 s, and the prefill aborted during warmup. The PD key needs the same commits rebuilt with --build-arg ENABLE_MORI=1, and moves to that image in a follow-up commit. The PD throughput numbers above ran the same SGLang commits on the previous image's AITER (285579b), which had the providers.
• Single-node values tuned on MI350X. They were tuned locally on 8x MI350X with the same SGLang and plugin commits, the step55000 draft and simulated golden acceptance.
• Kernel compile at concurrency 1 and 4. With 7 drafts, the first server start compiles the monokernel's 8-token kernels (about 18 minutes), inside the configured 2-hour health check.
• PD model path not yet checked. The runner entry assumes Kimi-K3.1-Pro under it-share-data on mi355x-amds.

AI model disclosure: claude-opus-5-5 (Claude Code), working under Kevin Mi's direction, did the following:

• ran the tuning sweeps;
• wrote the configs, the changelog entries and this description.

PR checklist and change type

Type of Change

• Bug fix
• New feature
• Configuration change
• Documentation update
• Other (please describe)

Checklist

• I have completed the AI model disclosure and kept it current
• I have tested my changes locally
• I have updated documentation if necessary
• If python -m infx.matrix.changed --base origin/main lists affected config keys, I have appended one new entry covering them to the physical end of inferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entries
• If this PR appends an inferencex-e2e/perf-changelog.yaml entry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs): full-sweep-fail-fast (recommended), full-sweep-enabled, or non-canary-full-sweep-enabled. Optional modifiers all-evals, evals-only, and agentx-fast require a primary label; the last two block reuse while applied.
• Before merging via reuse, an authorized maintainer (OWNER/MEMBER/COLLABORATOR) has commented /use <run_id> (or the legacy /reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.

中文

概要

本 PR 将 Kimi K3.1 Pro SGLang MI355X AgentX 配置统一到同一个镜像，按 P90 交互性重新调优单节点配置点，并新增 1P1D PD 配置。

镜像（两个配置键）： lmsysorg/sglang:dev-kimi-k31-pro-mi35x@sha256:5901db14（原为 sha256:346a6d83）。

• SGLang de0f491836：#48427 分支，包含 [AMD] Kimi-K3: pass gate_mode INTERLEAVE for GU-interleaved SiTU w13 sgl-project/sglang#49152（SiTU gate）和 PD 提交（MXFP4 DSA 索引键随 KV 一起传输，FP4 预填充 logits 按请求计算）。
• AITER [MoE] [FlyDSL] BM16 adaptive aux sort writes into the caller's output; retune kimik3 a8w4 token-16 inter-384 ROCm/aiter#7620（5405ee0026，flydsl 0.4.2）。
• sgl-amd-mono 219f40be9a（Previous run results are not shown on the website #114、Feature request: add historical comparison #116）。

单节点（kimik3.1-fp4-mi355x-sglang-agentic-dflash）：

• 并发 1： 运行请求数由 4 改为 1，智能体的并行工具调用改为排队，而不是共享解码步。
• 并发 8–56： prefill-decode-interval 由 16 改为 32（并发 48、56 由 8 改为 32），预填充打断运行中请求 token 流的频率减半。在吞吐量不变的情况下，每个 DCP 点的 P90 交互性均有提升；TTFT 上升（p50 2–4 s）。
• 并发 8、16、32： 最大运行请求数分别为 6、8、20；并发 16 和 32 验证当前 token 加 4 个草稿 token。
• 新增并发 24： 13 个运行请求、4 个草稿 token，填补 16 与 32 之间的曲线。
• 并发 1 和 4： 验证当前 token 加 7 个草稿 token。
• 所有并发点： 开启 SGLANG_ENABLE_ROCM_K3_ROW_SHARDED_ROUTED_FRONT、SGLANG_ENABLE_ROCM_K3_ROW_SHARDED_STREAM 和 SGLANG_OPT_DFLASH_RING_LOCS_FUSED（原先仅在 48、56 开启）。

PD（新增）（kimik3.1-fp4-mi355x-sglang-agentic-dflash-disagg）：

• 布局： 一个 TP8 DCP8 预填充节点和一个 TP8 DCP8 解码节点，前端为 sglang-router 0.3.2。
• 传输： KV 和 MXFP4 索引键经 Mooncake RDMA 传输。
• 投机解码： 两个角色均使用 5 token DFLASH（DFlash2）。
• 预填充： 16k 分块、shortest-prefill-first、每 64 个 token 保存一次 KDA 检查点，write-back HiCache 主机层（比例 4.0，KDA 主机比例 5），按节点 1,511 GiB 锁页内存上限配置。
• 解码： 并发 128 时最多 144 个运行请求，并发 144 时最多 160 个，配套相应的解码图，并开启解码端 radix cache。

并发	总吞吐 tok/s/GPU（16 GPU）	P90 交互性	TTFT p50
144	21,435	31.7	3.75 s
128	21,124	32.8	3.67 s

以上为两台 MI355X 节点上 3600 s 的结果，golden 接受长度 3.25（5 个草稿 token）。并发 144 是吞吐量比下一档高出 1% 以上的最高并发。同一配置在真实接受长度下 GSM8K 为 97.88%。

该软件栈上的精度： 测试使用相同 SGLang、AITER、sgl-amd-mono 提交的本地构建（并非已发布的镜像本身）：

• 单节点： 并发 56 GSM8K 97.50%、needle 3/3；并发 4 GSM8K 97.80%、needle 3/3，monokernel launch sort 已开启。
• PD： 经 router 测得 GSM8K 97.42%、needle 3/3；在本镜像上为 GSM8K 97.35%、needle 3/3。

风险：

• PD 需要重新构建镜像。 本镜像构建时未设置 ENABLE_MORI=1，因此缺少 MI355X 节点 AMD Pollara（ionic）和 Broadcom 网卡的 ibverbs provider。Mooncake 找不到任何 HCA，回退到 TCP。在并发 144 的 AgentX 负载下，每次 KV 传输耗时 50–120 s（正常约 1 s），预填充在预热阶段中止。PD 配置键需要使用相同提交并加上 --build-arg ENABLE_MORI=1 重新构建的镜像，并将在后续提交中切换到该镜像。上述 PD 吞吐结果使用相同的 SGLang 提交，但 AITER 为上一版镜像的 285579b，该镜像包含这些 provider。
• 单节点取值在 MI350X 上调优。 所有取值均在本地 8x MI350X 上调优，使用相同的 SGLang 和插件提交、step55000 草稿模型和模拟 golden 接受长度。
• 并发 1 和 4 首次启动需要编译内核。 7 个草稿 token 时，服务首次启动会编译 monokernel 的 8-token 内核（约 18 分钟），在配置的 2 小时健康检查时限之内。
• PD 模型路径尚未确认。 runner 条目假定 mi355x-amds 上的模型位于 it-share-data 下的 Kimi-K3.1-Pro。

AI 模型使用说明：claude-opus-5-5（Claude Code）在 Kevin Mi 的指导下完成了以下工作：

• 运行调优扫描；
• 编写配置、changelog 条目和本说明。

[github-actions]

Thanks for the contribution!

• Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
• PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
• After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.

中文

感谢你的贡献！

• **审阅：**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER，请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，再通过 Slack 联系核心维护者。必须严格遵循模板，并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have，才能触发签核验证。
• **PR 验证：**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目，并且恰好带有一个主标签时，才会运行扫描：full-sweep-fail-fast（强烈推荐；canary 加逐矩阵 fail-fast）、full-sweep-enabled（有 canary；矩阵任务在失败后继续运行）或 non-canary-full-sweep-enabled（无 canary，也无 fail-fast）。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
• **合并后：**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复；参见重新运行失败任务的说明。

[claude]

Beyond the inline finding, I checked three more things and none is a problem. The changelog's "instead of 16" is correct, because the recipe's base args set prefill-decode-interval: 16. The three env flags were already set in the c48 and c56 overrides, as the entry says. The new conc 24 in amd-master.yaml has a matching override_tp8_dcp8_c24 in the recipe.

Extended reasoning...

I read the base args of agentic.yaml and the c48 and c56 overrides to check the changelog's "from 16" and "already on at 48 and 56" claims, and both hold. I also confirmed that conc 24 in the master config has a matching recipe override.

[functionstackx]

Does e2el improve tho

[kevin-mii]

should be flat on most cc

[github-actions]

View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=42910486568

View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=42910486568

[github-actions]

@kevin-mii staged run 42695979942.

• Kimi-K3.1-Pro · agentic-traces · FP4 · mi355x_mori-sglang · 2026-11-09
• Kimi-K3.1-Pro · agentic-traces · FP4 · mi355x_sglang · 2026-11-09

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

[cquil11]

/use 42910486568

[github-actions]

@cquil11 staged run 42910486568.

• Kimi-K3.1-Pro · agentic-traces · FP4 · mi355x_sglang · 2026-11-10

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

[github-actions]

Ingest report — ✅ Results ingested into the production database.

Ingest workflow · Merge Ingest run · Benchmark source run

Category	new	duplicate	skipped	unmapped	failed
benchmark	10	0	0	0	0
eval	4	4	2	0	0
trace	10	0	0	0	0
power	8	0	2	0	0

_{benchmarkFiles 10, rawBenchmarkRows 10, coveredByPerJobFiles 0, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 5276, gpuMetricSamplesNew 332312, powerPublicationPoints 10, changelogEntriesWritten 1}

Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun.

中文

Ingest 报告 — ✅ 结果已写入 production 数据库。

上表仅统计本次运行写入的数据，各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。