Skip to content

[K3.1-Pro] Day-0 support vLLM on MI355X / [K3.1-Pro] MI355X 上 vLLM 的 Day-0 支持 - #4806

Merged
cquil11 merged 32 commits into
mainfrom
cam/kimik3.1-mi355x-vllm
Nov 6, 2026
Merged

cquil11 merged 32 commits into
mainfrom
cam/kimik3.1-mi355x-vllm

Conversation

@cquil11

@cquil11 cquil11 commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Adds Kimi K3.1 Pro day-0 vLLM AgentX on MI355X. It runs vLLM's AMD day-0 image (vllm/vllm-openai-rocm:kimi-k31@sha256:5f47cc10…, vLLM 0.0.0+mum.14d1ec2336, AITER 52846bc2) with the Inferact DSpark draft, at the concurrencies Kimi K3 currently shows on InferenceX for MI355X vLLM.

Flags follow vLLM's K3.1 recipe (recipes.vllm.ai): the MI355X override and the mi355x_agentx preset. The preset's three reduced-precision knobs are removed, and block verification is used.

The env keeps the image's HSA_NO_SCRATCH_RECLAIM=1. On this cluster's MEC firmware 38, the preset's HSA_NO_SCRATCH_RECLAIM=0 hung RCCL allgathers at 445k–700k context. gpu-memory-utilization is 0.90 through c24 and 0.91 at c32 and c48 (0.92 ran out of GPU memory at c48). VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1200, from the preset's runbook, covers long agentic prefills. c1 runs the latency_specialized decode backend with breakable graphs off.

Config key Topology Concurrency max-num-seqs DSpark draft tokens max-num-batched-tokens / gpu-memory-utilization KV offload
kimik3.1-fp4-mi355x-vllm-agentic-dspark 1 node, TP8 × DCP8 (a2a) 1 1 7 16384 / 0.90 none (GPU-resident)
4, 10 2, 6 5 16384 / 0.90 Simple CPU offload, 1799 GB
16, 24 10, 16 4 16384 / 0.90 Simple CPU offload, 1799 GB
32, 48 24, 48 3 10240 / 0.91 Simple CPU offload, 1799 GB

The points follow the reference apex-1104 sweep. max-num-seqs is below concurrency from c4 to c32, and the DSpark draft length steps down from 7 to 3 as concurrency rises. Decode CUDA graphs capture the max-num-seqs multiples of one plus the draft length, then a prefill ladder up to max-num-batched-tokens (c1 captures 2–16 only). dram-utilization: 0.6 resolves to 1799 GB of host DRAM (224.875 GB per TP rank), within the 3 TB per-server limit.

K3.1 week-1 precision policy

  • Linear ops and shared experts in BF16: the native checkpoint is served with no --quantization-config. The preset's online FP8 for the KDA input projections (--quantization-config '{"targets":{"*in_proj_qkvgfab":"fp8_per_channel"}}') is dropped.
  • MoE combine in BF16: the preset's AITER_FLYDSL_STAGE2_FP8=1, which sums MoE stage 2 through MXFP8, is dropped and set explicitly to 0.
  • AllReduce in BF16: the preset's VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 is dropped and set explicitly to NONE. This also satisfies the MODELS.md integer-quantized all-reduce ban. The image's inherited environment sets no quick-reduce or AITER quantization control.
  • MoE weights and activations: MXFP4 weights as shipped, with FP4 activations (A4W4). VLLM_ROCM_USE_AITER_MOE_SITUV2=1 selects the SiTUv2 FlyDSL MoE in a4w4 mode, the same as the image's auto default. The policy allows A4W4.
  • KV cache: kv-cache-dtype: fp8.
  • KDA cache: mamba-ssm-cache-dtype: float32.
  • Conv cache: engine default (BF16).
  • Indexer cache: FP4, via "indexer_kv_dtype":"mxfp4" in --attention-config. This is vLLM's own MI355X setting.
  • EPLB: none.
  • Draft: Inferact/Kimi-K3.1-Pro-DSpark-preview as shipped: BF16 weights, with FP8 draft KV that matches the FP8 target KV. The load-path evidence is in the draft data-type comment.

Verifier: the preset uses vLLM's default standard rejection sampler. This config sets rejection_sample_method: block with draft_sample_method: probabilistic, which matches the merged golden curve kimi-k3.1-pro__dspark__kimi-k3.1-pro-dspark-preview__probabilistic-block (as updated in #4851). Throughput runs get synthetic acceptance from that curve: 3.69 at 7 drafts (c1), 3.41 at 5, 3.19 at 4 and 2.88 at 3. The eval and AL-gate jobs verify for real.

First run per node downloads the checkpoint: K3.1 is not staged on the MI355X cluster, so there is no models.entries row. Each node's first run downloads the ~1.56 TB checkpoint into its node-local HF cache (/var/lib/hf-hub-cache). Every node currently has at least 11 TB free. No shared-cache lane routes this model to the shared Weka cache.

No disaggregated points: Kimi K3 also had MI355X vllm-disagg points. vLLM documents K3.1 prefill/decode disaggregation only with NixlConnector on NVIDIA, and needs VLLM_SSM_CONV_STATE_LAYOUT=DS there. No ROCm PD path for K3.1's KDA conv state is documented, so this PR has no PD points.

MI355X is in vLLM's supported hardware for K3.1 but is not marked verified.

Validation: run 42193891273 at 740ee083 passed all seven throughput points (c1–c48) and the c1 and c48 agentic evals. It is pinned with /use 42193891273.

中文

本 PR 在 MI355X 上新增 Kimi K3.1 Pro 的 Day-0 vLLM AgentX。它使用 vLLM 的 AMD Day-0 镜像(vllm/vllm-openai-rocm:kimi-k31@sha256:5f47cc10…,vLLM 0.0.0+mum.14d1ec2336,AITER 52846bc2)和 Inferact DSpark 草稿模型,并发点与 InferenceX 目前展示的 Kimi K3 MI355X vLLM 结果一致。

参数遵循 vLLM 的 K3.1 recipe(recipes.vllm.ai):MI355X override 和 mi355x_agentx 预设。去掉了预设中的三个降精度开关,并使用块验证(block verification)。

环境变量保留镜像自带的 HSA_NO_SCRATCH_RECLAIM=1。在本集群的 MEC 固件 38 上,预设的 HSA_NO_SCRATCH_RECLAIM=0 会让 RCCL allgather 在 445k–700k 上下文时挂起。gpu-memory-utilization 在 c24 及以下为 0.90,c32 和 c48 为 0.91(0.92 在 c48 时 GPU 显存不足)。VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1200 来自预设的运行手册,用于覆盖较长的 agentic prefill。c1 使用 latency_specialized decode 后端,并关闭 breakable graphs。

Config key 拓扑 并发 max-num-seqs DSpark 草稿 token 数 max-num-batched-tokens / gpu-memory-utilization KV offload
kimik3.1-fp4-mi355x-vllm-agentic-dspark 1 节点,TP8 × DCP8(a2a) 1 1 7 16384 / 0.90 无(KV 驻留 GPU)
4, 10 2, 6 5 16384 / 0.90 Simple CPU offload,1799 GB
16, 24 10, 16 4 16384 / 0.90 Simple CPU offload,1799 GB
32, 48 24, 48 3 10240 / 0.91 Simple CPU offload,1799 GB

这些并发点沿用参考的 apex-1104 sweep。c4 到 c32 的 max-num-seqs 低于并发数,DSpark 草稿长度随并发升高从 7 逐步降到 3。decode CUDA graph 捕获 (1 + 草稿长度) 的倍数,共 max-num-seqs 个,其上是直到 max-num-batched-tokens 的 prefill 梯度(c1 只捕获 2–16)。dram-utilization: 0.6 对应 1799 GB 主机内存(每个 TP rank 224.875 GB),在每台服务器 3 TB 的上限之内。

K3.1 第一周精度政策

  • Linear 与 shared experts 为 BF16: 原生 checkpoint 不带 --quantization-config。去掉了预设中对 KDA 输入投影的在线 FP8(--quantization-config '{"targets":{"*in_proj_qkvgfab":"fp8_per_channel"}}')。
  • MoE combine 为 BF16: 去掉预设的 AITER_FLYDSL_STAGE2_FP8=1(MoE stage 2 经 MXFP8 求和),并显式设为 0。
  • AllReduce 为 BF16: 去掉预设的 VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4,并显式设为 NONE。这同时满足 MODELS.md 对整数量化 all-reduce 的禁令。镜像继承的环境变量未设置任何 quick-reduce 或 AITER 量化控制。
  • MoE 权重与激活: MXFP4 权重保持原样,激活为 FP4(A4W4)。VLLM_ROCM_USE_AITER_MOE_SITUV2=1 选择 a4w4 模式的 SiTUv2 FlyDSL MoE,与镜像的 auto 默认值相同。该政策允许 A4W4。
  • KV cache: kv-cache-dtype: fp8。
  • KDA cache: mamba-ssm-cache-dtype: float32。
  • Conv cache: 引擎默认(BF16)。
  • Indexer cache: FP4,通过 --attention-config 中的 "indexer_kv_dtype":"mxfp4" 设置。这是 vLLM 自己的 MI355X 设置。
  • EPLB: 无。
  • 草稿模型: Inferact/Kimi-K3.1-Pro-DSpark-preview 按发布形式运行:BF16 权重,草稿 FP8 KV 与目标模型的 FP8 KV 一致。加载路径的证据见草稿数据类型评论。

验证器: 预设使用 vLLM 默认的标准拒绝采样器。本配置设置 rejection_sample_method: block 和 draft_sample_method: probabilistic,与已合并的黄金曲线 kimi-k3.1-pro__dspark__kimi-k3.1-pro-dspark-preview__probabilistic-block(经 #4851 更新)一致。吞吐测试使用该曲线的合成接受长度:7 个草稿(c1)为 3.69,5 个为 3.41,4 个为 3.19,3 个为 2.88。eval 和 AL gate 任务执行真实验证。

每个节点首次运行会下载 checkpoint: K3.1 未预置在 MI355X 集群上,因此没有 models.entries 条目。每个节点首次运行会把约 1.56 TB 的 checkpoint 下载到节点本地 HF 缓存(/var/lib/hf-hub-cache)。目前每个节点至少有 11 TB 空闲空间。没有共享缓存通道把该模型路由到共享 Weka 缓存。

没有 PD 分离点: Kimi K3 在 MI355X 上还有 vllm-disagg 点。vLLM 只在 NVIDIA 上用 NixlConnector 记录了 K3.1 的 prefill/decode 分离,并且需要 VLLM_SSM_CONV_STATE_LAYOUT=DS。ROCm 上 K3.1 KDA conv state 的 PD 路径没有文档,因此本 PR 不包含 PD 点。

MI355X 在 vLLM 的 K3.1 支持硬件列表中,但未标记为已验证。

验证: run 42193891273(740ee083)中七个吞吐点(c1–c48)以及 c1 和 c48 的 agentic eval 全部通过,已通过 /use 42193891273 固定。

AI model disclosure
  • claude-opus-5-5 (Claude Opus 5.5, in Claude Code): synced the perf-changelog block with the diff, checked the draft data types in the image's vLLM source, and revised this description.
  • The models used for the earlier commits on this branch are not recorded here and could not be verified.

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目,并且恰好带有一个主标签时,才会运行扫描:full-sweep-fail-fast(强烈推荐;canary 加逐矩阵 fail-fast)、full-sweep-enabled(有 canary;矩阵任务在失败后继续运行)或 non-canary-full-sweep-enabled(无 canary,也无 fail-fast)。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings marked 🟡 are optional suggestions and need no follow-up push.

Comment thread inferencex-e2e/perf-changelog.yaml Outdated
- kimik3.1-fp4-mi355x-vllm-agentic-dspark
description:
- "Add Kimi-K3.1-Pro day-0 vLLM AgentX on MI355X: TP8 x DCP8 with DSpark (Inferact/Kimi-K3.1-Pro-DSpark-preview), GPU-resident KV at conc 1 and 4 and Simple CPU KV offload at conc 4-60."
pr-link: TBD

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Merging this PR through the reuse-merge tool will fail. The new changelog entry's pr-link: TBD is not an accepted placeholder. validate_added_pr_link (validate_perf_changelog.py:113) accepts only XXX, https://github.com/SemiAnalysisAI/InferenceX/pull/XXX or this PR's own canonical URL. Fix: set the line to pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX, the form CONTRIBUTING.md:211 shows, or to this PR's real URL. The merge step then rewrites it to the real link.

Why this was flagged

The PR-time check in run-sweep.yml:105 only runs infx.matrix.plan. That check types pr-link as a plain str (validation.py:1143), so TBD passes the sweep. At merge, merge_with_reuse.py:688 calls canonicalize_changelog. That calls canonicalize_appended_links → compare_entries (prepare_perf_changelog_merge.py:88), which calls validate_added_pr_link("TBD", pr). It raises "new PR entry must use 'https://github.com/SemiAnalysisAI/InferenceX/pull/' or an XXX placeholder; found 'TBD'" because TBD is not in PR_LINK_PLACEHOLDERS (validate_perf_changelog.py:21-24). By then merge_with_reuse.py:658 has already posted /reuse-sweep-run on the PR. The changelog-conflict path, resolve_conflict_bytes (prepare_perf_changelog_merge.py:146), also goes through compare_entries and fails the same way. No other entry in perf-changelog.yaml uses TBD.

Verification: The diff adds pr-link: TBD at inferencex-e2e/perf-changelog.yaml:12020. merge_with_reuse.py:688 calls canonicalize_changelog(pr, repo) unconditionally, which reaches validate_added_pr_link (validate_perf_changelog.py:112-116) and raises for any link that is not a placeholder or this PR's canonical URL. run-sweep.yml:105 never calls compare_entries, so TBD passes CI and fails only at merge.

Comment on lines +17 to +20
# A node's first run downloads ~1.56 TB of MXFP4 shards into its local HF cache before loading.
health_check:
interval_seconds: 10
max_attempts: 720

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 On every MI355X node without the ~1.56 TB checkpoint already cached, the first K3.1 point can fail its readiness check. The download, load and graph capture all fit in the 7200 s budget only if the node downloads at about 217 MB/s or faster. Nothing routes this model to the shared cache. The budget is max_attempts: 720 x 10 s, matching VLLM_ENGINE_READY_TIMEOUT_S: '7200'. Fix: put the checkpoint where each node does not re-download it before serving. For example, add moonshotai/Kimi-K3.1-* to SHARED_HF_CACHE_LANES["mi355x-amds"], or stage a models.entries row. Alternatively, size the readiness budget for a 1.56 TB cold download.

Why this was flagged

The trigger is a sweep point landing on an mi355x-amds node whose node-local /var/lib/hf-hub-cache (runners.yaml:789) does not yet hold Kimi-K3.1-Pro. Every node starts that way, because no models.entries row exists for this model on mi355x-amds (runners.yaml:776-780). single_node_hf_cache (infx/launch/drivers/srt/models.py:198-201) only picks shared-hf-hub-cache for models matched in SHARED_HF_CACHE_LANES, and models.py:187-194 has no Kimi rule. As a result, vLLM pulls about 1.56 TB from HF inside server startup. The sibling MI355X recipe uses the same 720 x 10 s health check and notes at minimaxm31flash/vllm/mi355x-fp4/agentic.yaml:26-27 that a cold load of only 240 GB can use up the whole VLLM_ENGINE_READY_TIMEOUT_S. On a slow link, the first point on each node exceeds 7200 s and the job fails. Under full-sweep-fail-fast, that one failure cancels the sweep. The 10 points spread across nodes each pay the download separately. The recipe's comment at line 17 acknowledges the download, but nothing bounds or avoids it.

Verification: The readiness budget is 7200 s: agentic.yaml:18-20 sets interval_seconds: 10 and max_attempts: 720. No shared-cache rule matches moonshotai/Kimi-K3.1-Pro (models.py:187-194), so it falls through to node-local /var/lib/hf-hub-cache (runners.yaml:789), and vLLM downloads it during startup. 1.56 TB / 7200 s ≈ 217 MB/s. A node below that rate fails its first point.

Comment on lines +87 to +89

# Admission equals CONC: each breakable graph size costs graph-pool memory taken from the KV cache.
# Graphs cover every decode batch, CONC x (1 + drafts) in steps of 1 + drafts, then the prefill sizes;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 nit (optional): Several new comments restate the YAML beneath them, which the AGENTS.md comment rule forbids. sweep:^\s*# (Graphs cover every decode batch|.*c1 and c4 keep the KV cache on the GPUs|Verify the draft tokens with) For example, "Graphs cover every decode batch, CONC x (1 + drafts) in steps of 1 + drafts, then the prefill sizes" only repeats the cudagraph_capture_sizes lists. Fix: drop the restating sentences and keep only the reasons, such as why admission equals CONC and why block rejection gives way to golden AL.

Why this was flagged

AGENTS.md:58 allows only a short technical reason for a line or block and says "never restate the code". At agentic.yaml:88 the comment spells out the capture-size pattern that the compilation-config lists below it already show. At amd-master.yaml:913, "c1 and c4 keep the KV cache on the GPUs" repeats kv-offloading: none, conc-list: [1, 4]. At agentic.yaml:64, "Verify the draft tokens with the AITER assembly DCP kernel" repeats VLLM_ROCM_AITER_MLA_DCP_VERIFY: asm. The impact is limited to the repo convention, and runtime behaviour does not change.

Verification: AGENTS.md:58, which the root CLAUDE.md hands guidance to, says "never restate the code." In amd-master.yaml, "c1 and c4 keep the KV cache on the GPUs." only repeats kv-offloading: none, conc-list: [1, 4] on the row directly below. In agentic.yaml, "Verify the draft tokens with the AITER assembly DCP kernel." mostly spells out VLLM_ROCM_AITER_MLA_DCP_VERIFY: asm. Nothing breaks at runtime.

@cquil11
cquil11 force-pushed the cam/kimik3.1-mi355x-vllm branch 2 times, most recently from 833d26f to 691b84c Compare November 4, 2026 23:05
Single-node TP8 x DCP8 with the Inferact DSpark draft and block verification,
following vLLM's MI355X K3.1 recipe without its three reduced-precision knobs.
GPU-resident KV at conc 1 and 4; Simple CPU KV offload at conc 4-60.
Write the per-concurrency points as plain override variants instead of zip overrides.
@cquil11
cquil11 force-pushed the cam/kimik3.1-mi355x-vllm branch from 691b84c to bc5c564 Compare November 4, 2026 23:11
@cquil11 cquil11 added the non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim) label Nov 4, 2026
# Conflicts:
#	inferencex-e2e/perf-changelog.yaml
The image keeps HSA_NO_SCRATCH_RECLAIM=1, which the cluster's MEC firmware 38
needs, so large FP4-indexer prefill scratch stays pinned outside vLLM's memory
budget. At 0.92 the DRAM-offload points ran out of GPU memory in long-context
prefill (HSA_STATUS_ERROR_OUT_OF_RESOURCES). Use 0.90, as the Kimi K3 MI355X
recipe did on the same nodes.
@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

billishyahao and others added 8 commits November 5, 2026 01:40
The MI355X AgentX lane and the reference apex-1104 sweep differed in the
points they ran and in batching, so a comparison between them could not
attribute anything to the engine build.

Align the parts that shape the workload: seven points instead of ten
(drop the GPU-resident conc 4 and the conc 52/60 tail), max-num-seqs
below concurrency as that sweep sets it, max-num-batched-tokens 16384
except 10240 at conc 48, its cudagraph capture sizes, and 2577 GB of
host DRAM for the Simple CPU offload.

Keep this lane's own precision and speculative settings: 5 DSpark tokens
with block rejection at every point, the published
Inferact/Kimi-K3.1-Pro-DSpark-preview draft, FP32 KDA recurrent state,
BF16 all-reduce, BF16 MoE stage 2, the AITER FA MLA prefill backend, and
no FP8 indexer override or KDA input-projection quantization. Those are
deliberate here, and copying them would change what the runs measure --
synthetic acceptance in particular reports a modelled throughput and
makes the output nondeterministic by construction.

The dram-utilization comment claimed 0.6 resolves to 1799 GB. The nodes
report 3000 GB, so the pinned 2577 GB is 86% of host DRAM, not 60% --
the recipe's cpu_bytes_to_use_per_rank wins. Say so instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The planner derives TOTAL_CPU_DRAM_GB from dram-utilization and rejected the
recipe's 2577 GB against the 1799 GB that 0.6 resolves to. 0.8592 resolves to
2577 GB, within the 3 TB per-server limit.

Align comments with the settings they describe and correct the changelog:
throughput points still run synthetic acceptance at the golden 3.08.
The conc-1 latency point runs the reference sweep's latency_specialized decode
backend with 7 draft tokens; the throughput points stay at 5.

K3.1 has no torch.compile submodule, so FULL_AND_PIECEWISE captures decode only
and breakable graphs are the only thing that would add the prefill sizes. Turn
them off and leave scratch reclaim on, as the image sets it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The apex-1104 sweep sets mla_prefill_backend on its conc-48 point and leaves
conc 1-32 on the default. Move it out of base into that point's override.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The apex-1104 sweep turns both off at conc 1 and on at conc 4-48. Move the
conc-1 values into that point's override and leave the base on the sweep's
throughput setting.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
# Conflicts:
#	inferencex-e2e/perf-changelog.yaml
…hput servers

Synthetic acceptance streams unverified draft tokens. The kimi_k3 tool and reasoning parsers stop streaming on some of them, which empties or truncates the response the client sees. Evals and acceptance gates verify for real and keep their parsers.
…nc 16-32

With HSA_NO_SCRATCH_RECLAIM=1 the kernel scratch of these points filled the VRAM left outside the 0.90-0.91 budget during warmup and stalled RCCL, even on freshly rebooted nodes. Their GPU KV use peaks well under the smaller pool, and DRAM offload backs the rest.
@YukioZzz YukioZzz mentioned this pull request Nov 6, 2026
3 of 11 tasks
billishyahao and others added 2 commits November 6, 2026 09:14
… huge pages

THP_MEM_ALLOC_ENABLE=1 is what makes torch's CPU allocator call
madvise(MADV_HUGEPAGE). The hosts run THP in madvise mode, so without it the
224 GB per-rank offload pool is backed entirely by 4 KiB pages: about 55
million of them to register and keep mapped, against 107 thousand 2 MiB pages.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
seungrokj added a commit that referenced this pull request Nov 6, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@seungrokj

Copy link
Copy Markdown
Collaborator

/reuse-sweep-run 42193891273

@billishyahao

Copy link
Copy Markdown
Collaborator

/use 42193891273

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

@billishyahao staged run 42193891273.

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

…skip-sweep]

- Add Kimi-K3.1-Pro vLLM AgentX on one 8x MI355X node with vLLM's day-0 image vllm/vllm-openai-rocm:kimi-k31@sha256:5f47cc1004d9a7ac65f8a913725edb20c62c99a07e31cbf4ba21741bd0b4f9b1 (vLLM 0.0.0+mum.14d1ec2336, AITER 52846bc2).
- TP8 x DCP8 (a2a DCP backend) with the Inferact/Kimi-K3.1-Pro-DSpark-preview draft: GPU-resident KV at concurrency 1, and SimpleCPUOffloadConnector DRAM offload at concurrency 4, 10, 16, 24, 32 and 48 (224.875 GB per rank, 1799 GB per node, which is what dram-utilization 0.6 resolves to).
- Per point at concurrency 1/4/10/16/24/32/48: max-num-seqs 1/2/6/10/16/24/48 and DSpark draft tokens 7/5/5/4/4/3/3. max-num-batched-tokens is 16384 and gpu-memory-utilization 0.90 through concurrency 24; both change to 10240 and 0.91 at concurrency 32 and 48.
- Every point compiles in mode 3 with FULL_AND_PIECEWISE CUDA graphs. Concurrency 1 runs the latency_specialized decode backend, captures sizes 2-16 and turns breakable CUDA graphs off; the offload points keep VLLM_USE_BREAKABLE_CUDAGRAPH=1 and capture max-num-seqs multiples of (1 + draft tokens) plus a prefill ladder up to max-num-batched-tokens.
- Common settings: FP8 KV cache, FP32 KDA state (mamba-ssm-cache-dtype float32), MXFP4 indexer cache (indexer_kv_dtype mxfp4), fused KDA prefill, prefix caching with prefix-match-unit 64, max-model-len 1048576, fastsafetensors loading, stream-interval 10, AITER ASM DCP verify, AITER DCP allgather, FlyDSL sparse MLA, the image's HSA_NO_SCRATCH_RECLAIM=1, transparent huge pages for the offload pool (THP_MEM_ALLOC_ENABLE=1), and 1200 s execute-model and 7200 s engine-ready timeouts.
- K3.1 week-1 precision policy: linear layers and shared experts stay BF16 (no quantization-config); routed experts run A4W4, MXFP4 weights with FP4 activations, through the SiTUv2 FlyDSL MoE (VLLM_ROCM_USE_AITER_MOE_SITUV2=1, the same as the image's auto default); MoE stage 2 stays BF16 (AITER_FLYDSL_STAGE2_FP8=0); all-reduce stays BF16 (VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=NONE).
- Verification is block rejection with probabilistic draft sampling. Throughput runs replace it with the golden acceptance length for the draft length: 3.69 at 7 tokens, 3.41 at 5, 3.19 at 4 and 2.88 at 3. Evals and AL gates verify for real.
- No change to the DSpark draft's precision: Inferact/Kimi-K3.1-Pro-DSpark-preview (revision 6a2ef20a) ships all-BF16 weights with no quantization_config, the speculative config sets no quantization, so get_draft_quant_config() returns None and K3DSparkModel loads its layers in BF16 with the target's BF16 embedding and LM head. Its KV cache is FP8, the same as the target's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@seungrokj seungrokj changed the title [K3.1-Pro] Day-0 support vLLM on MI355X [K3.1-Pro] Day-0 support vLLM on MI355X / [K3.1-Pro] MI355X 上 vLLM 的 Day-0 支持 Nov 6, 2026
@seungrokj

seungrokj commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Accuracy-sensitive optimizations (#4705 MODELS.md items): none enabled; the MXFP4 indexer cache needs a ruling. I traced each item through the recipe, the image's Config.Env, which sets no quantization or quick-reduce controls, and the pinned source. The paths are in the same image as the draft section below. AITER 52846bc2 is not public either, and every vLLM file cited matches provenance.json. Each finding covers both the target and the DSpark draft worker, because the draft uses the same TP communicators.

#4705 item This config Evidence
Reduced-precision all-reduce / reduce-scatter Not enabled Detailed below the table.
Online quantization of linear layers Not enabled No quantization-config. vllm/model_executor/model_loader/weight_utils.py:216-229 composes online quantization only from that argument, and base_config.py:313-315 otherwise keeps the checkpoint method. The checkpoint's ignore list keeps self_attn, shared_experts, dense mlp, block_sparse_moe.gate, routed_expert_(up|down)_proj and lm_head in BF16. routed_expert_down_proj takes online_quantization_config, which is None, and routed_expert_up_proj takes quant_config=None (vllm/models/kimi_k3/amd/linear.py:364-414). The AttnRes FP8 fusion (linear.py:259) fires only when the next layer declares an FP8 input, and BF16 layers declare none (model_executor/layers/fusion/quant_activation.py:65-69).
Quantization of shared experts Not enabled re:.*shared_experts.* is on the checkpoint's ignore list. VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS defaults to off (vllm/envs.py:1433-1436), and neither the recipe nor the image sets it. So the shared experts are not folded into the FP4 MoE kernel (vllm/_aiter_ops.py:2361-2362). VLLM_KIMI_K3_SHARD_SP_SHARED_EXPERT is used only in kimi_k3/nvidia/model.py.
Cross-indexer sharing Not used The checkpoint sets dsa_config.index_topk_freq: 1. vllm/models/kimi_k3/amd/dsa.py:753-759 builds an indexer in every DSA layer and rejects index_topk_freq != 1.
FP4 KV cache Not used for the KV cache; the indexer cache is FP4 Target kv-cache-dtype: fp8; draft speculative-config kv_cache_dtype: fp8. Detailed below the table.
SSM/KDA/Mamba state below FP32 Not reduced mamba-ssm-cache-dtype: float32 sets the KDA recurrent state to FP32 (vllm/model_executor/layers/mamba/mamba_utils.py:142-143; auto would also give FP32 at :140-141). The short-conv state follows mamba_cache_dtype: auto, which is the model dtype, BF16 (:139). That is the native dtype, as in the week-1 "Conv cache: Native (BF16)" row. vllm/models/kimi_k3/amd/kda.py:109-116 uses these dtypes, and :230 selects the AITER decode kernel only for an FP32 state. The draft has no KDA layers.

All-reduce and reduce-scatter

  • The recipe sets VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=NONE, which is also the default (vllm/envs.py:1478-1482).
  • vllm/distributed/device_communicators/quick_all_reduce.py:186-192 returns on NONE before :263 clears disabled, so quick reduce stays disabled (:92).
  • The fused AITER QR+RMSNorm path needs an enabled quick-reduce communicator (vllm/_aiter_ops.py:1227-1233). So the default-on BF16→FP16 quick-reduce cast never runs either.
  • All-reduces go through the AITER custom all-reduce or RCCL in BF16. vLLM calls custom_all_reduce(inp) (vllm/distributed/device_communicators/aiter_custom_all_reduce.py:73-74) without open_fp8_quant, which defaults to False (aiter/dist/device_communicators/custom_all_reduce.py:1853-1854).
  • The MoE output all-reduce is BF16 (vllm/models/kimi_k3/amd/latent_moe_runner.py:98), and AITER_FLYDSL_STAGE2_FP8=0 keeps MoE stage 2 in BF16.
  • No other quick-reduce or GEMM-fused collective is active:
    • Neither the recipe nor the image sets AITER_QUICK_REDUCE_QUANTIZATION or ROCM_QUICK_REDUCE_QUANTIZATION.
    • vLLM never reads ROCM_QUICK_REDUCE_QUANTIZATION.
    • Only AITER's own quick-reduce communicator reads AITER_QUICK_REDUCE_QUANTIZATION (default NONE), and vLLM does not import that communicator.
    • The GEMM-AR and GEMM-RS fusions (VLLM_KIMI_K3_GEMM_AR, VLLM_ENABLE_GEMM_RS) exist only in kimi_k3/nvidia/model.py:184.

Disclosed, but outside the six items

  • MXFP4 MoE dispatch all-gather. At TP8 × DCP8 on gfx950, prefill batches of at least 8192 tokens shard routed_expert_down_proj across ranks and all-gather its output (vllm/models/kimi_k3/amd/linear.py:160-194). With A4W4 SiTUv2, each rank first quantizes its shard to per-1x32 MXFP4 and then all-gathers the FP4 values and E8M0 scales (linear.py:207-225, enabled by latent_moe_runner.py:245-287). These are the activations the A4W4 routed experts consume. This is an all-gather, not an all-reduce or reduce-scatter. The MODELS.md week-1 policy on main allows it ("MoE dispatch: MX/NV-style quantization allowed").
  • A4W4 routed experts. The checkpoint stores routed experts as weight-only MXFP4 (config_groups.group_0: 4-bit, group 32, no input_activations). VLLM_ROCM_USE_AITER_MOE_SITUV2=1 resolves to a4w4 (vllm/_aiter_ops.py:2100-2110), so their activations are quantized to FP4 at runtime. These are routed MoE experts, not linear layers, and the week-1 policy allows them ("MoE activations: A4W4 allowed").

Needs a ruling: the MXFP4 indexer cache. attention-config '{"indexer_kv_dtype":"mxfp4"}' stores the DSA indexer key cache in FP4. The ROCm default that picks MXFP4 automatically applies only to deepseek_v4 (vllm/platforms/rocm.py:385-418), so this is an explicit choice. #4705 names "FP4 KV cache on models that do not natively support FP4 KV cache" but does not say whether indexer caches count. The week-1 table on main lists "Indexer cache: FP4 allowed" as its own row. The K3.1-Pro config.json declares no indexer cache dtype. If reviewers count the indexer cache as KV cache under #4705, this config needs the three-arm KVV evidence before it qualifies. Otherwise it is covered by the week-1 row.

No three-arm evidence is attached, because none of the six optimizations is enabled.


Draft data types: no change. The DSpark draft Inferact/Kimi-K3.1-Pro-DSpark-preview (revision 6a2ef20a, the revision the golden AL curve was measured on) is a standalone 5-layer K3DSparkModel checkpoint with no quantization_config. All 67 tensors in its safetensors header are BF16:

Part Data type
Attention projections (q_a_proj, q_b_proj, kv_a_proj_with_mqa, kv_b_proj, o_proj) BF16
Dense MLP (gate_proj, up_proj, down_proj) BF16
context_proj, Markov head, confidence head BF16
Norms BF16
Embedding and LM head not in the draft checkpoint; shared from the target, whose embed_tokens and lm_head are stored BF16 (and lm_head is on the target's ignore list)
KV cache FP8, set by speculative-config kv_cache_dtype: fp8, the same as the target's kv-cache-dtype: fp8

The image's vLLM is a non-public build (vllm-0.0.0+mum.14d1ec2336, commit 14d1ec23363ae7495b7734d88983a478375d3ff8, which is not in vllm-project/vllm), so there are no GitHub permalinks. The paths below are under /opt/runtime-venv/lib/python3.12/site-packages/ in vllm/vllm-openai-rocm:kimi-k31@sha256:5f47cc1004d9a7ac65f8a913725edb20c62c99a07e31cbf4ba21741bd0b4f9b1. Each file's sha256 matches the image's /opt/runtime-provenance/provenance.json.

  • vllm/config/speculative.py:1308-1345: the draft inherits the target's quantization (compressed-tensors MXFP4) only when speculative-config gives no model. The recipe names the draft repo, so quantization stays unset.
  • vllm/config/speculative.py:1495: the draft ModelConfig is built with that unset quantization. :1620 excludes K3DSparkModel from the DeepSeek DSpark branch that copies the target's quantization onto the draft (:1636).
  • vllm/config/vllm.py:937,965: with no quantization, the quantization config is None.
  • vllm/model_executor/models/utils.py:1062-1084: get_draft_quant_config() resolves the draft's own config, which is None.
  • vllm/models/kimi_k3/nvidia/dspark_mla.py:72,145: every draft linear layer takes get_draft_quant_config(), so the layers load unquantized in BF16. :433-445: embed_tokens and lm_head are dropped from the draft checkpoint and shared from the target. K3DSparkModel is registered to this class at vllm/model_executor/models/registry.py:702-705 on ROCm as well.
  • vllm/config/speculative.py:423-430: only speculative-config's own attention_backend (ROCM_AITER_MLA) and kv_cache_dtype (fp8) override the draft's attention and KV dtype.

The recipe sets no quantization-config or --quantization, and no quantization inside speculative-config. The draft therefore runs as shipped: BF16 weights and activations, with an FP8 KV cache that matches the target's FP8 KV. CONTRIBUTING.md "Draft-model precision" allows that KV dtype. The c1 latency_specialized decode backend (vllm/models/kimi_k3/amd/specialized_decode/) replaces only the target's MoE decoder layers on verify steps and does not touch the draft. This config is new, so there is no earlier framework commit to compare.

中文

精度敏感优化(#4705 MODELS.md 条目):均未启用;MXFP4 indexer cache 需要裁定。 我逐项追踪了 recipe、镜像的 Config.Env(未设置任何量化或 quick-reduce 控制项)以及固定版本的源码。路径与下方草稿部分所用镜像相同。AITER 52846bc2 同样未公开,引用的所有 vLLM 文件都与 provenance.json 一致。每项结论同时适用于目标模型和 DSpark 草稿 worker,因为草稿使用相同的 TP 通信器。

#4705 条目 本配置 依据
降精度 all-reduce / reduce-scatter 未启用 详见表格下方。
线性层在线量化 未启用 未设置 quantization-config。vllm/model_executor/model_loader/weight_utils.py:216-229 只根据该参数组合在线量化,否则 base_config.py:313-315 保留 checkpoint 的方法。checkpoint 的 ignore 列表使 self_attn、shared_experts、dense mlp、block_sparse_moe.gate、routed_expert_(up|down)_proj 和 lm_head 保持 BF16。routed_expert_down_proj 使用 online_quantization_config,其值为 None;routed_expert_up_proj 使用 quant_config=None(vllm/models/kimi_k3/amd/linear.py:364-414)。AttnRes FP8 融合(linear.py:259)只在下一层声明 FP8 输入时触发,而 BF16 层不声明(model_executor/layers/fusion/quant_activation.py:65-69)。
共享专家量化 未启用 re:.*shared_experts.* 在 checkpoint 的 ignore 列表中。VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS 默认关闭(vllm/envs.py:1433-1436),recipe 和镜像都未设置,因此共享专家不会并入 FP4 MoE kernel(vllm/_aiter_ops.py:2361-2362)。VLLM_KIMI_K3_SHARD_SP_SHARED_EXPERT 只在 kimi_k3/nvidia/model.py 中使用。
跨 indexer 共享 未使用 checkpoint 设置 dsa_config.index_topk_freq: 1。vllm/models/kimi_k3/amd/dsa.py:753-759 为每个 DSA 层构建独立的 indexer,并拒绝 index_topk_freq != 1。
FP4 KV cache KV cache 未使用;indexer cache 为 FP4 目标模型 kv-cache-dtype: fp8;草稿 speculative-config 中 kv_cache_dtype: fp8。详见表格下方。
SSM/KDA/Mamba 状态低于 FP32 未降低 mamba-ssm-cache-dtype: float32 将 KDA 递归状态设为 FP32(vllm/model_executor/layers/mamba/mamba_utils.py:142-143;auto 在 :140-141 同样得到 FP32)。short-conv 状态跟随 mamba_cache_dtype: auto,即模型 dtype BF16(:139)。这是原生 dtype,与 week-1 "Conv cache: Native (BF16)" 一行一致。vllm/models/kimi_k3/amd/kda.py:109-116 使用这些 dtype,:230 只在状态为 FP32 时选用 AITER decode kernel。草稿没有 KDA 层。

All-reduce 与 reduce-scatter

  • recipe 设置 VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=NONE,这也是默认值(vllm/envs.py:1478-1482)。
  • vllm/distributed/device_communicators/quick_all_reduce.py:186-192 在 NONE 时于 :263 清除 disabled 之前返回,因此 quick reduce 保持禁用(:92)。
  • 融合的 AITER QR+RMSNorm 路径需要一个已启用的 quick-reduce 通信器(vllm/_aiter_ops.py:1227-1233),因此默认开启的 BF16→FP16 quick-reduce 转换也不会执行。
  • All-reduce 通过 AITER custom all-reduce 或 RCCL 以 BF16 执行。vLLM 调用 custom_all_reduce(inp)(vllm/distributed/device_communicators/aiter_custom_all_reduce.py:73-74)时不传 open_fp8_quant,其默认值为 False(aiter/dist/device_communicators/custom_all_reduce.py:1853-1854)。
  • MoE 输出的 all-reduce 为 BF16(vllm/models/kimi_k3/amd/latent_moe_runner.py:98),AITER_FLYDSL_STAGE2_FP8=0 使 MoE stage 2 保持 BF16。
  • 没有其他 quick-reduce 或 GEMM 融合集合通信处于启用状态:
    • recipe 和镜像都未设置 AITER_QUICK_REDUCE_QUANTIZATION 或 ROCM_QUICK_REDUCE_QUANTIZATION。
    • vLLM 从不读取 ROCM_QUICK_REDUCE_QUANTIZATION。
    • 只有 AITER 自己的 quick-reduce 通信器读取 AITER_QUICK_REDUCE_QUANTIZATION(默认 NONE),而 vLLM 不导入该通信器。
    • GEMM-AR 和 GEMM-RS 融合(VLLM_KIMI_K3_GEMM_AR、VLLM_ENABLE_GEMM_RS)只存在于 kimi_k3/nvidia/model.py:184。

需要披露、但不属于这六项

  • MXFP4 MoE dispatch all-gather。 在 gfx950 上的 TP8 × DCP8 配置中,至少 8192 token 的 prefill 批次会把 routed_expert_down_proj 切分到各 rank,再对其输出做 all-gather(vllm/models/kimi_k3/amd/linear.py:160-194)。使用 A4W4 SiTUv2 时,每个 rank 先把本地分片量化为 per-1x32 MXFP4,再 all-gather FP4 值和 E8M0 scale(linear.py:207-225,由 latent_moe_runner.py:245-287 启用)。这些正是 A4W4 路由专家所消费的激活。这是 all-gather,不是 all-reduce 或 reduce-scatter。main 上 MODELS.md 的 week-1 策略允许这种做法("MoE dispatch: MX/NV-style quantization allowed")。
  • A4W4 路由专家。 checkpoint 以仅权重 MXFP4 存储路由专家(config_groups.group_0:4 bit,group 32,无 input_activations)。VLLM_ROCM_USE_AITER_MOE_SITUV2=1 解析为 a4w4(vllm/_aiter_ops.py:2100-2110),因此其激活在运行时被量化为 FP4。它们是路由 MoE 专家而非线性层,week-1 策略允许这种做法("MoE activations: A4W4 allowed")。

需要裁定:MXFP4 indexer cache。 attention-config '{"indexer_kv_dtype":"mxfp4"}' 以 FP4 存储 DSA indexer 的 key cache。自动选择 MXFP4 的 ROCm 默认逻辑只适用于 deepseek_v4(vllm/platforms/rocm.py:385-418),因此这是显式选择。#4705 列出了"在不原生支持 FP4 KV cache 的模型上使用 FP4 KV cache",但没有说明 indexer cache 是否计入。main 上的 week-1 表格把 "Indexer cache: FP4 allowed" 单独列为一行。K3.1-Pro 的 config.json 没有声明 indexer cache dtype。如果评审认为 indexer cache 属于 #4705 所说的 KV cache,该配置在达到要求前需要三组对照的 KVV 证据;否则它由 week-1 那一行覆盖。

由于六项优化均未启用,本 PR 未附三组对照证据。


草稿数据类型:无变化。 DSpark 草稿模型 Inferact/Kimi-K3.1-Pro-DSpark-preview(revision 6a2ef20a,即黄金 AL 曲线测量时所用的版本)是一个独立的 5 层 K3DSparkModel checkpoint,没有 quantization_config。其 safetensors header 中的 67 个张量全部为 BF16:

部分 数据类型
注意力投影(q_a_proj、q_b_proj、kv_a_proj_with_mqa、kv_b_proj、o_proj) BF16
Dense MLP(gate_proj、up_proj、down_proj) BF16
context_proj、Markov head、confidence head BF16
Norm BF16
Embedding 和 LM head 不在草稿 checkpoint 中;与目标模型共享,目标模型的 embed_tokens 和 lm_head 以 BF16 存储(lm_head 在目标模型的 ignore 列表中)
KV cache FP8,由 speculative-config 的 kv_cache_dtype: fp8 设置,与目标模型的 kv-cache-dtype: fp8 相同

镜像中的 vLLM 是非公开构建(vllm-0.0.0+mum.14d1ec2336,commit 14d1ec23363ae7495b7734d88983a478375d3ff8,不在 vllm-project/vllm 中),因此无法给出 GitHub 永久链接。以下路径位于 vllm/vllm-openai-rocm:kimi-k31@sha256:5f47cc1004d9a7ac65f8a913725edb20c62c99a07e31cbf4ba21741bd0b4f9b1 的 /opt/runtime-venv/lib/python3.12/site-packages/ 下。每个文件的 sha256 都与镜像中 /opt/runtime-provenance/provenance.json 的记录一致。

  • vllm/config/speculative.py:1308-1345:只有在 speculative-config 未指定 model 时,草稿才会继承目标模型的量化(compressed-tensors MXFP4)。本 recipe 指定了草稿仓库,因此 quantization 保持未设置。
  • vllm/config/speculative.py:1495:草稿的 ModelConfig 使用这个未设置的 quantization 构建。:1620 将 K3DSparkModel 排除在 DeepSeek DSpark 分支之外,该分支会把目标模型的量化复制给草稿(:1636)。
  • vllm/config/vllm.py:937,965:没有量化时,量化配置为 None。
  • vllm/model_executor/models/utils.py:1062-1084:get_draft_quant_config() 解析草稿自己的配置,结果为 None。
  • vllm/models/kimi_k3/nvidia/dspark_mla.py:72,145:草稿的每个线性层都使用 get_draft_quant_config(),因此以未量化的 BF16 加载。:433-445:embed_tokens 和 lm_head 从草稿 checkpoint 中丢弃,改为与目标模型共享。在 ROCm 上,K3DSparkModel 同样通过 vllm/model_executor/models/registry.py:702-705 注册到这个类。
  • vllm/config/speculative.py:423-430:只有 speculative-config 自身的 attention_backend(ROCM_AITER_MLA)和 kv_cache_dtype(fp8)会覆盖草稿的注意力后端和 KV dtype。

本 recipe 没有设置 quantization-config 或 --quantization,speculative-config 中也没有量化设置。因此草稿按发布形式运行:BF16 权重和激活,FP8 KV cache 与目标模型的 FP8 KV 一致。CONTRIBUTING.md 的 "Draft-model precision" 允许这种 KV dtype。c1 的 latency_specialized decode 后端(vllm/models/kimi_k3/amd/specialized_decode/)只在验证步骤中替换目标模型的 MoE decoder 层,不涉及草稿。该配置是新增的,因此没有可对比的旧框架 commit。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@seungrokj seungrokj left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a PR reviewer and CODEOWNER, I have reviewed this and have:

Checklist items
  • Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
  • Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
  • Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
  • Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
  • Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
  • Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison.
  • For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
  • Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
  • Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
  • Verified that new models disable integer all-reduce, with SGLang ROCm explicitly setting ROCM_QUICK_REDUCE_QUANTIZATION=NONE. Existing models are grandfathered per MODELS.md.
  • For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
  • If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
  • For official InferenceX (non-preview), verified that established models use upstream engine images built from master (or the equivalent default development branch), with new models transitioning within a reasonable time after release. See MODELS.md.
  • If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
  • Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
    • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
  • Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
  • If this PR uses append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it.
  • If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
  • Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.

Additional detail section:

Signed: seungrokj

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

❌❌❌ REJECTED ❌❌❌

@seungrokj Blocking: the published vLLM Kimi-K3.1-Pro recipe is unchanged since 2026-11-06T00:43Z (7045518494). It still does not cover three production-useful settings in this MI355X config, and the sign-off's additional detail section gives no reason for excluding them. Either get the upstream recipe updated, or record and justify each exclusion in the sign-off, then sign off again.

❌ Check 3 (Upstream recipe): FAIL — For the single-node agg worker of kimik3.1-fp4-mi355x-vllm-agentic-dspark, three production-useful settings are missing from, or contradict, the published mi355x_agentx preset (source): (1) every point's --speculative-config sets rejection_sample_method: block, but the recipe documents only "vLLM's default (standard) rejection sampler"; (2) HSA_NO_SCRATCH_RECLAIM=1 contradicts the recipe's HSA_NO_SCRATCH_RECLAIM=0, and the recipe's HSA_SCRATCH_SINGLE_LIMIT_ASYNC=25769803776 is dropped (the firmware-38 reason appears only in the PR body, not in the sign-off); (3) THP_MEM_ALLOC_ENABLE=1 for the Simple offload pool is not in the recipe. The additional detail section contains only the recipe link and a precision-trace comment that does not address these settings. Covered by the recipe: dropping the three reduced-precision knobs ("Drop all three for BF16 precision"), Simple offload at 224.875 GB per rank, DSpark with FP8 draft KV and ROCM_AITER_MLA, DCP8 a2a, the AITER/DCP environment, the 1200 s execute-model timeout, and the default MLA prefill backend.

Passed and not applicable checks

✅ Check 0 (CODEOWNER): PASS — @seungrokj is a listed owner of inferencex-e2e/configs/amd-master.yaml and /inferencex-e2e/benchmarks/*_node/srt-slurm-recipes/**/mi355x-*/; perf-changelog.yaml falls under the catch-all.

✅ Check 1 (Sweep on in-PR commit): PASS — Run 42193891273 (attempt 2, head 740ee083) has all 7 agentic / jobs (c1–c48) and both agentic eval / jobs (c1, c48) at success. The later commits are non-material: cf9f6666 changes only changelog text, merge 312a967b brings main changes for other configs (GLM-5.3-Flash, H3, GB200 K3.1, a GLM cache lane) with no golden-AL change, b1c4fcea is empty, and the PR's recipe and master entry are identical between 740ee083 and the pinned SHA.

✅ Check 2 (Evals): PASS — The evals ran on the PR image @sha256:5f47cc10…. GSM8K em_strict is 0.9803±0.0038 at c1 and 0.9765±0.0042 at c48. The sibling gap of 0.0038 is under the 0.0113 threshold. Cohort gaps against B300 vLLM run 42150239586 (0.9773) are -0.0030 and 0.0008, under the 0.0112 and 0.0117 thresholds. The evals cover both the c1 latency_specialized GPU-KV path and the c48 offload path.

✅ Check 4 (Reuse command): PASS — /use 42193891273 (billishyahao, COLLABORATOR) and the newest command, /reuse-sweep-run 42193891273 (cquil11, COLLABORATOR, 2026-11-06T14:51Z), both name the Check 1 run.

✅ Check 5 (Checklist template): PASS — All 20 items of the current PR_REVIEW_CHECKLIST.md template are present and checked.

✅ Check 6 (Upstream image/ordering/provenance): PASS — vllm/vllm-openai-rocm:kimi-k31@sha256:5f47cc10… comes from the upstream vLLM org with framework: vllm, so the ordering rule does not apply. The model-enablement build 0.0.0+mum.14d1ec2336 falls under the temporary new-model allowance for Kimi-K3.1-Pro (added 2026-11-04) and is the recipe's own day-0 AMD image.

✅ Check 7 (Deprecated models): PASS — kimik3.1 agentic coding is an active row in MODELS.md (added 2026-11-04 via #4795).

✅ Check 8 (Architecture/accuracy hacks): PASS — There are no --hf-overrides, architecture overrides or --quantization-config. The settings follow the K3.1 week-1 policy: BF16 linear layers and shared experts, A4W4 MoE, BF16 combine (AITER_FLYDSL_STAGE2_FP8=0), FP32 KDA state, FP8 KV and the allowed FP4 indexer cache, and both evals cover them. K3.1 is not grandfathered, and VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=NONE disables quick reduce on this vLLM path.

✅ Check 9 (Chat template): PASS — Agentic throughput replays traces through the aiperf chat client, and the AL gate and evals use chat-template encoding.

✅ Check 10 (Engine patches): PASS — No patches, setup scripts, source edits or wheel installs can be reached from this recipe or master entry.

✅ Check 11 (Golden AL): PASS — The run's submission logs show the injected rejection_sample_method: synthetic with synthetic_acceptance_length 3.41 at c4 (5 tokens) and 2.88 at c32 (3 tokens). These match kimi-k3.1-pro__dspark__kimi-k3.1-pro-dspark-preview__probabilistic-block.yaml, which gives 3.69/3.41/3.19/2.88 for 7/5/4/3 tokens (block verify, thinking_on). The recipe hard-codes no AL value.

➖ Check 12 (Append-only): N/A — The new changelog entry does not set append-only: true.

✅ Check 13 (Draft precision): PASS — Inferact/Kimi-K3.1-Pro-DSpark-preview@6a2ef20a ships all-BF16 weights with no quantization_config, and the speculative config sets no quantization, so the draft loads in BF16. Its FP8 KV matches the FP8 target KV (allowed). SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE does not apply to this vLLM path and is not set.

✅ Check 14 (Pareto coverage): PASS — Curve kimik3.1/agentic-coding/mi355x_vllm/fp4/r42193891273/P90 E2EL, from the bmk_* artifacts at 740ee083, has 5 frontier points out of 7: c10 (27.43 s, 6599), c16 (27.47 s, 9095), c24 (41.17 s, 13304), c32 (48.73 s, 15958) and c48 attempt 2 (67.78 s, 18716 tok/s/GPU). c1 and c4 are dominated. pareto_coverage confirms 5.

✅ Check 15 (PR description): PASS — The per-point max-num-seqs, draft tokens, batched tokens, memory utilization, precision settings, golden AL values, validation run and commit, and the /use pin all match the pinned code and the run evidence.

➖ Check 16 (NVIDIA overview): N/A — The only affected hardware is AMD MI355X.

➖ Check 17 (Metrics publication): N/A — This is a direct vLLM config, so no Dynamo TRT-LLM metrics settings apply.

✅ Check 18 (Recipe/image/sweep consistency): PASS — All 7 points generate with per-concurrency overrides, and TP8×DCP8 on 8 GPUs matches the master entry. The digest-pinned image resolves, no points are removed, and the changelog accurately states the image, precision policy, FP8 KV, MXFP4 indexer cache and BF16 DSpark draft.

Assessed commit: b1c4fceae3a5bedcb93e000abc7a326eed11ea47.

cquil11 commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator Author

/reuse-sweep-run 42193891273

@cquil11
cquil11 merged commit ddf781d into main Nov 6, 2026
33 checks passed
@cquil11
cquil11 deleted the cam/kimik3.1-mi355x-vllm branch November 6, 2026 14:57
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Ingest report — ✅ Results ingested into the production database.

Ingest workflow · Merge Ingest run · Benchmark source run

Category new duplicate skipped unmapped failed
benchmark 7 0 0 0 0
eval 2 2 2 0 0
trace 7 0 0 0 0
power 7 0 0 0 0

benchmarkFiles 7, rawBenchmarkRows 7, coveredByPerJobFiles 0, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 2638, gpuMetricSamplesNew 284272, powerPublicationPoints 7, changelogEntriesWritten 1

Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun.

中文

Ingest 报告 — ✅ 结果已写入 production 数据库。

上表仅统计本次运行写入的数据,各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants