Repository navigation
[AMD][AgentX] MI355X Kimi-K3.1-Pro FP4 SGLang: DFLASH single-node points / [AMD][AgentX] MI355X Kimi-K3.1-Pro FP4 SGLang：DFLASH 单节点配置点 - #4823
Closed
kevin-mii wants to merge 2 commits into
Conversation
Contributor
|
Thanks for the contribution!
中文感谢你的贡献！
|
Add kimik3.1-fp4-mi355x-sglang-agentic-dflash: Kimi-K3.1-Pro MXFP4 AgentX on one 8x MI355X node with the SGLang day-0 image and the RadixArk/Kimi-K3.1-Dflash2-Preview draft at 5 draft tokens with block verification, so the srt driver simulates the committed DFLASH golden curve. TP8 monokernel at concurrency 1 and 4; TP8/DCP8 with the HiCache host tier at 8-48. Serve Kimi-K3.1-Pro from the shared HF cache on mi355x-amds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
|
Closing in favour of #4808, which already carries the Kimi K3.1 MI355X SGLang sweep. These configs are being moved into that PR. |
cquil11
added a commit
that referenced
this pull request
Nov 5, 2026
Take Kevin Mi's Kimi-K3.1-Pro MI355X SGLang DFLASH recipe and points: TP8 monokernel at c1 and c4, and TP8/DCP8 with the HiCache host tier at c8, c16, c32 and c48. Points above c48 are dropped because the node's GTT limit makes their host pools overflow. K3.1 is not staged in the shared HF cache on mi355x-amds, so the model keeps loading from the node-local cache, without HF_HUB_OFFLINE. Co-authored-by: Kevin Mi <mikevin920@yahoo.com>
cquil11
added a commit
that referenced
this pull request
Nov 5, 2026
* feat(kimik3.1): add Kimi K3.1 Pro SGLang AgentX on MI355X TP8 on the decode monokernel at c1 and c4 without KV offload, and TP8 DCP8 with HiCache from c4 to c60, both with the RadixArk DFLASH draft, following the SGLang cookbook's MI355X AgentX recipes. * chore(kimik3.1): link the MI355X SGLang changelog entry to its PR * feat(kimik3.1): adopt the MI355X SGLang recipe from #4823 Take Kevin Mi's Kimi-K3.1-Pro MI355X SGLang DFLASH recipe and points: TP8 monokernel at c1 and c4, and TP8/DCP8 with the HiCache host tier at c8, c16, c32 and c48. Points above c48 are dropped because the node's GTT limit makes their host pools overflow. K3.1 is not staged in the shared HF cache on mi355x-amds, so the model keeps loading from the node-local cache, without HF_HUB_OFFLINE. Co-authored-by: Kevin Mi <mikevin920@yahoo.com> --------- Co-authored-by: Kevin Mi <mikevin920@yahoo.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SGLang has no AMD Kimi-K3.1-Pro AgentX submission yet; the only K3.1 SGLang config is the GB300 disaggregated one in #4811. This adds the single-node MI355X curve on the public day-0 image, with the same DFLASH draft and verification as #4811, so the srt driver simulates the committed golden curve.
kimik3.1-fp4-mi355x-sglang-agentic-dflash(amd-master.yaml):lmsysorg/sglang:dev-kimi-k31-pro-mi35x@sha256:6af91676…,moonshotai/Kimi-K3.1-Pro, one 8x MI355X node, TP8.RadixArk/Kimi-K3.1-Dflash2-Preview@d0d8bc2e, 5 draft tokens (speculative-num-draft-tokens: 6),speculative-use-block-verification: true, FP8 draft KV. The driver selectskimi-k3.1-pro__dflash__kimi-k3.1-dflash2-preview__default-blockat 5 drafts (golden AL 3.43, thinking_on); no acceptance length is hard-coded, and the spec identity audit reports no findings. The width follows the SGLang cookbook's MI35x cells (block 6).kv-offloading: none).write_back,hicache-ratio 2.5, KDA-state host ratio 12). The host pools pin about 192 GB per GPU (KV 49 GB, KDA state 134 GB, indexer 10 GB), about 1.54 TB per node, inside the nodes' 1511 GiB GTT limit;dram-utilization: 0.513records that.AITER_OPUS_STAGE2_FP8=0) and all-reduce (ROCM_QUICK_REDUCE_QUANTIZATION=NONE); MXFP4 routed-expert weights with FP8 activations (A8W4); FP8 KV for target and draft; FP32 KDA state (ReplaySSM). The draft runs as shipped.infx/launch/drivers/srt/models.py): Kimi-K3.1-Pro AgentX on mi355x-amds loads from/it-share/hf-hub-cachewithHF_HUB_OFFLINE=1, as Muse Spark does; a cold 1.56 TB node-local download per node would also run into the Hub's 429 limit.Before the sweep:
/it-share/hf-hub-cachemust hold both repos. Seeding it needs cluster access:Known limits:
RUSTUP_TOOLCHAIN=stableon the HiCache points works around the image's Rust toolchain: without it some ranks fall back to the Python TreeCore, and write-back asserts on the mix. The Dockerfile fix is in [Feat] [AMD] model: Add Kimi-K3.1 MI35x support sgl-project/sglang#48427; the variable can go once the image is rebuilt.中文
SGLang 目前还没有 AMD 平台的 Kimi-K3.1-Pro AgentX 提交；唯一的 K3.1 SGLang 配置是 #4811 中的 GB300 分离式配置。本 PR 在公开的 day-0 镜像上新增 MI355X 单节点曲线，草稿模型与验证方式与 #4811 相同，因此 srt 驱动会模拟已提交的黄金接受长度曲线。
kimik3.1-fp4-mi355x-sglang-agentic-dflash（amd-master.yaml）：lmsysorg/sglang:dev-kimi-k31-pro-mi35x@sha256:6af91676…，moonshotai/Kimi-K3.1-Pro，单个 8x MI355X 节点，TP8。RadixArk/Kimi-K3.1-Dflash2-Preview@d0d8bc2e，5 个草稿 token（speculative-num-draft-tokens: 6），speculative-use-block-verification: true，草稿 KV 为 FP8。驱动选用kimi-k3.1-pro__dflash__kimi-k3.1-dflash2-preview__default-block曲线的 5 个草稿 token 档（黄金 AL 3.43，thinking_on）；没有硬编码接受长度，spec identity 审计没有发现问题。宽度沿用 SGLang cookbook 的 MI35x 配置（block 6）。kv-offloading: none）。write_back，hicache-ratio 2.5，KDA 状态主机比例 12）。主机内存池每个 GPU 固定约 192 GB（KV 49 GB、KDA 状态 134 GB、indexer 10 GB），每个节点约 1.54 TB，在节点 1511 GiB 的 GTT 上限之内；dram-utilization: 0.513记录了这一用量。AITER_OPUS_STAGE2_FP8=0）和 all-reduce（ROCM_QUICK_REDUCE_QUANTIZATION=NONE）均为 BF16；路由专家权重为 MXFP4，激活为 FP8（A8W4）；目标模型和草稿模型的 KV 均为 FP8；KDA 状态为 FP32（ReplaySSM）。草稿模型按发布时的精度运行。infx/launch/drivers/srt/models.py）：mi355x-amds 上的 Kimi-K3.1-Pro AgentX 与 Muse Spark 一样从/it-share/hf-hub-cache加载，并设置HF_HUB_OFFLINE=1；每个节点各自冷下载 1.56 TB 也会触发 Hub 的 429 限流。运行 sweep 之前：
/it-share/hf-hub-cache必须已包含上述两个仓库，需要有集群权限的人预先下载（命令见上文）。已知限制：
RUSTUP_TOOLCHAIN=stable，用于绕过镜像中的 Rust 工具链问题：否则部分 rank 会回退到 Python TreeCore，write-back 在两者混用时触发断言。Dockerfile 修复在 [Feat] [AMD] model: Add Kimi-K3.1 MI35x support sgl-project/sglang#48427 中；镜像重建后即可去掉该变量。AI model disclosure
claude-opus-5-5), running in Claude Code at Kevin Mi's request: wrote the recipe, master-config entry, shared-cache rule, changelog entry and this description, and ran the local matrix, spec-identity and test checks. No other models or delegated agents contributed.🤖 Generated with Claude Code