Skip to content

Refresh Kimi-K3 PD on ROCm nightly / 更新 Kimi-K3 PD nightly 配置 - #4391

Closed
YukioZzz wants to merge 19 commits into
mainfrom
yichaozhu/k3-pd-nightly-kernels
Closed

YukioZzz wants to merge 19 commits into
mainfrom
yichaozhu/k3-pd-nightly-kernels

Conversation

@YukioZzz

@YukioZzz YukioZzz commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Run Kimi-K3 FP4 PD on MI355X with vllm/vllm-openai-rocm:nightly-2b009fc6d1493a9a1ad19f4d0c7230b7e6b7e63a (was nightly-81198e97ba7eee2a22540caaa756b7fdddcb4d93), TP8/DCP8 workers, FP8 KV cache, and MoRIIO RDMA transfer. The image is pinned by digest and includes the eager SimpleCPUOffload CPU LRU refresh for GPU-hit prefixes and Mamba boundary state (vllm#62444). The recipe provides five configurations:

Topology Concurrency DSpark draft tokens P / per-D max sequences SSM state CPU offload
1P1D 8 7 1 / 1 FP32 None
1P1D 48 4 96 / 96 BF16 SimpleCPUOffload
1P2D 24 4 48 / 24 BF16 SimpleCPUOffload
1P2D 48 4 96 / 48 BF16 SimpleCPUOffload
1P3D 24 4 48 / 16 BF16 SimpleCPUOffload

The c8 decode worker uses latency_specialized with the V2 model runner and graph capture sizes [1, 8]. The four offload configurations use BF16 SSM state on both prefill and decode, with prefill-only SimpleCPUOffload at DRAM utilization 0.8 (2399 GB per prefill worker).

Compared with main, this removes 1P3D c12 (TP8/DCP1 decode) and 1P1D c56, adds 1P1D c8, switches the throughput points from FP32 to BF16 SSM state, and raises the offload budget from 1799 GB (DRAM utilization 0.6). The new image adds Kimi-K3 specialized decode on MI355X (vllm#60683), MoRIIO consumers with specialized decode (vllm#63031), BF16 KDA recurrent state (vllm#58743) and MoRIIO READ completion before the forward (vllm#60808).

DSpark uses /models/Inferact-Kimi-K3-DSpark with probabilistic drafting and block rejection; throughput runs use the selected golden acceptance curve, while evals use real verification.

The recipe uses vllm-router with consistent-hash routing and the standard eval selection. Recipe documentation: vllm-project/vllm#63726 (merged).

DSpark keeps BF16 weights and FP8 KV cache. ROCm QuickReduce is disabled on all prefill and decode workers, keeping draft and target collectives unquantized.

Validation: run 41285920098 on 7ffbdeca passed all five benchmark points and both evals on every point. kimi_tool_call_schema_full scores 0.9779–0.9828 over 408 cases per point; GSM8K em_strict scores 0.9674–0.9742 over 1,319 cases per point. Throughput uses golden AL 3.36 (K4) and 3.84 (K7). Four of the five points are on the throughput/E2EL Pareto frontier; 1P1D c8 is dominated.

AI model disclosure

AI assistance prepared the recipe and validation, reviewed upstream dependencies, and rebased the PR. A separate agent previously checked documentation discoverability. Exact model identifiers could not be verified from the runtime.

Claude Opus 5.5 (claude-opus-5-5, Claude Code) rewrote the perf-changelog block from the diff, checked the draft data types against the vLLM source at both image commits, and updated this description.

PR checklist and change type

Type of Change

  • Bug fix
  • New feature
  • Configuration change
  • Documentation update
  • Other

Checklist

  • I have completed the AI model disclosure and kept it current
  • I have tested my changes locally
  • I have updated documentation if necessary
  • If python -m infx.matrix.changed --base origin/main lists affected config keys, I have appended one new entry covering them to the physical end of inferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entries
  • If this PR appends an inferencex-e2e/perf-changelog.yaml entry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs): full-sweep-fail-fast (recommended), full-sweep-enabled, or non-canary-full-sweep-enabled. Optional modifiers all-evals, evals-only, and agentx-fast require a primary label; the last two block reuse while applied.
  • Before merging via reuse, an authorized maintainer (OWNER/MEMBER/COLLABORATOR) has commented /use <run_id> (or the legacy /reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.
中文

在 MI355X 上使用官方 vllm/vllm-openai-rocm:nightly-2b009fc6d1493a9a1ad19f4d0c7230b7e6b7e63a 运行 Kimi-K3 FP4 PD,P/D worker 均采用 TP8/DCP8、FP8 KV cache,通过 MoRIIO RDMA 传输 KV,镜像以 digest 固定,包含 GPU-hit prefix 及 Mamba 边界状态的 eager SimpleCPUOffload CPU LRU 刷新修复(vllm#62444)。

与 main 相比(原镜像为 nightly-81198e97ba7eee2a22540caaa756b7fdddcb4d93),本 PR 移除 1P3D c12(decode 为 TP8/DCP1)和 1P1D c56,新增 1P1D c8,吞吐点的 SSM 状态由 FP32 改为 BF16,offload 预算从 1799 GB(DRAM utilization 0.6)提高。新镜像包含 MI355X 上的 Kimi-K3 specialized decode(vllm#60683)、支持 specialized decode 的 MoRIIO consumer(vllm#63031)、BF16 KDA 循环状态(vllm#58743)以及在 forward 前等待 MoRIIO READ 完成(vllm#60808)。

五种配置见上表:1P1D c8 使用 FP32 SSM、K7 DSpark,不启用 CPU offload;decode 端使用 V2 model runner 的 latency_specialized 后端,graph capture sizes 为 [1, 8]。其余四个吞吐配置使用 BF16 SSM、K4 DSpark 和仅 P 侧启用的 SimpleCPUOffload,DRAM utilization 为 0.8,每个 P worker 的预算为 2399 GB。

DSpark draft 为 /models/Inferact-Kimi-K3-DSpark,使用 probabilistic drafting 和 block rejection。吞吐测试采用对应的 golden acceptance curve,eval 使用真实验证。路由采用 vllm-router consistent hash,评测沿用标准选择。上游 recipe:vllm-project/vllm#63726(已合并)。

DSpark 保持 BF16 权重和 FP8 KV cache。所有 P/D worker 关闭 ROCm QuickReduce,使 draft 和 target 的 collective 保持未量化。

验证:run 41285920098 在 7ffbdeca 上通过五个 benchmark 点,且每个点的两项 eval 均通过。kimi_tool_call_schema_full 每点 408 个用例,得分范围为 0.9779–0.9828;GSM8K em_strict 每点 1,319 个用例,得分范围为 0.9674–0.9742。吞吐测试使用 golden AL 3.36(K4)和 3.84(K7)。五个点中有四个位于吞吐/E2EL Pareto 前沿上;1P1D c8 被支配。

AI 协助配置、验证、依赖核对及 rebase;此前有独立 agent 检查文档可发现性。精确模型标识无法从运行环境核实。Claude Opus 5.5(claude-opus-5-5,Claude Code)根据 diff 重写了 perf-changelog 条目,对照两个镜像 commit 的 vLLM 源码核对了 draft 数据类型,并更新了本描述。

🤖 Generated with Claude Code

@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目,并且恰好带有一个主标签时,才会运行扫描:full-sweep-fail-fast(强烈推荐;canary 加逐矩阵 fail-fast)、full-sweep-enabled(有 canary;矩阵任务在失败后继续运行)或 non-canary-full-sweep-enabled(无 canary,也无 fail-fast)。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

@YukioZzz
YukioZzz force-pushed the yichaozhu/k3-pd-nightly-kernels branch from 31c06bb to dda9782 Compare October 22, 2026 04:23
@YukioZzz YukioZzz added the full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures label Oct 22, 2026
@YukioZzz
YukioZzz force-pushed the yichaozhu/k3-pd-nightly-kernels branch from dda9782 to cbfaaa9 Compare October 22, 2026 04:30
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

@YukioZzz
YukioZzz force-pushed the yichaozhu/k3-pd-nightly-kernels branch from cbfaaa9 to cec28da Compare October 22, 2026 04:38
更新五个 Kimi-K3 PD 点位:吞吐点启用 BF16 SSM,移除 1P1D c56 和 1P3D c12,增加 FP32 SSM 的 1P1D c1 K7 monokernel 点。临时 kernel 补丁单独提交。
导入 vLLM #60683、#58743、#60808、#63031 的公开提交,保留原始 SHA 与 cherry-pick 来源。安装器使用运行镜像对应的源码和 ROCm SDK 重建 HIP 扩展,便于 nightly 收录后整体移除。
官方 nightly 已包含四个依赖 PR,移除临时补丁、安装器及构建入口,仅保留 recipe 配置更新。
@YukioZzz
YukioZzz force-pushed the yichaozhu/k3-pd-nightly-kernels branch from eaeee53 to 0902c81 Compare October 24, 2026 14:43
固定包含所需上游功能的 nightly 提交和镜像 digest,确保各节点及重复运行使用同一镜像。
@YukioZzz YukioZzz added non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim) full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures and removed full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim) labels Oct 24, 2026

@billishyahao billishyahao left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a PR reviewer and CODEOWNER, I have reviewed this and have:

  • Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
  • Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
  • Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/40301194088
  • Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/40301194088
  • Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
  • Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison.
  • For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
  • Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
  • Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
  • For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
  • If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
  • If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
  • Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
    • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
  • Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
  • If this PR uses append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it.
  • If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
  • Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.

Additional detail section:

Signed: billishyahao

@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

@Oseltamivir staged run 40301194088.

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

@YukioZzz YukioZzz added full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures and removed full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures labels Oct 31, 2026
@YukioZzz YukioZzz removed the full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures label Oct 31, 2026
@YukioZzz YukioZzz added the full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures label Oct 31, 2026
@functionstackx functionstackx removed the full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures label Oct 31, 2026
Comment on lines 86 to 88
VLLM_ROCM_QUICK_REDUCE_QUANTIZATION: INT4
VLLM_ROCM_QUICK_REDUCE_MIN_SIZE_BYTES_MB: '16'
VLLM_ROCM_QUICK_REDUCE_MAX_SIZE_BYTES_MB: '256'

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doesnt this hurt long context evals at 500k? what is the KVV scores? https://github.com/MoonshotAI/Kimi-Vendor-Verifier

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @functionstackx , I think @YukioZzz put some KVV scores on the descriptions.
Previous configuration validation (before the QR minimum-size fix and image update): run 40645935622 on b49b522 passed all five points. Evals: kimi_tool_call_schema_full 0.973–0.9853 and gsm8k em_strict 0.9659–0.9727. Throughput uses golden AL 3.36 (K4) and 3.84 (K7). Four of the five points are on the throughput/E2EL Pareto frontier; 1P1D c8 is dominated.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we still keep this quick reduce INT4 enabled according to #4623 ?

 Existing models are grandfathered against an immutable snapshot of the model support matrix, 
including later recipe and image updates for those models.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@billishyahao if u pass the entire KVV suite, u def can

https://github.com/MoonshotAI/Kimi-Vendor-Verifier

KVV consists of the OCRBench MMMU Pro Vision BEAM (1M) DeepSWE

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI only runs an subsample of KVV, you would need to run the other parts of KVV such as OCRBench MMMU Pro Vision BEAM (1M) DeepSWE

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The full bench of kvv like MMMU/DeepSWE is a little bit heavy. I removed the QR env for this time.

@YukioZzz YukioZzz added full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures and removed full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures labels Nov 1, 2026
…ly-kernels

# Conflicts:
#	inferencex-e2e/perf-changelog.yaml
@YukioZzz YukioZzz added full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures and removed full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures labels Nov 1, 2026
@SemiAnalysisAI SemiAnalysisAI deleted a comment from billishyahao Nov 1, 2026
@SemiAnalysisAI SemiAnalysisAI deleted a comment from Oseltamivir Nov 1, 2026
@SemiAnalysisAI SemiAnalysisAI deleted a comment from billishyahao Nov 1, 2026
@functionstackx functionstackx added full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures evals-only Suppress throughput and run only eval jobs; combine with all-evals to expand selection and removed full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures labels Nov 1, 2026
@YukioZzz YukioZzz removed full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures evals-only Suppress throughput and run only eval jobs; combine with all-evals to expand selection labels Nov 1, 2026
…ly-kernels

# Conflicts:
#	inferencex-e2e/perf-changelog.yaml
@YukioZzz YukioZzz added the full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures label Nov 1, 2026
@functionstackx functionstackx added the all-evals Expand eval selection to every fixed-sequence config label Nov 1, 2026

functionstackx commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

As mentioned last week to everyone,we will deprecate it and not accept any more merges the moment k3.1 pro weights drop. this means existing open PRs will not be accepted anymore after k3.1 pro weights drops.

The PR may not have been merged due to various factors:

  • accuracy concerns that weren't addressed before the k3.1 pro weights drop
  • msging core maintainers in slack for review only after the k3.1 pro weights dropped (in accordance with the agreed upon policy for review/merge SLO MODELS.md )
  • having patches
  • or other assortment of issues

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

all-evals Expand eval selection to every fixed-sequence config full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

5 participants