Skip to content

[K3.1-Pro] Day-0 support vLLM on MI355X - #4858

Open
YukioZzz wants to merge 37 commits into
mainfrom
yiczhu/k31-c16-c48-10k-gmu090
Open

YukioZzz wants to merge 37 commits into
mainfrom
yiczhu/k31-c16-c48-10k-gmu090

Conversation

@YukioZzz

@YukioZzz YukioZzz commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

Based on #4806 at 14a3dc883ad9708e4c7343486937953c251a716d, this PR retains the Kimi K3.1 Pro MI355X vLLM integration and caps c16/c24/c32/c48 at 10,240 batch/capture tokens and GMU 0.90 to leave more runtime headroom. This recipe adjustment is not a demonstrated fix for the prior collective hangs.

Relative to #4806: c16/c24 remove only capture sizes 12288, 14336 and 16384; all decode sizes remain covered. c32/c48 retain their lists and lower GMU from 0.91 to 0.90. c1/c4/c10 are unchanged.

Concurrency max-num-seqs Batch tokens Graph cap Buckets BCG DSpark k Golden AL GMU
1 1 16384 16 8 OFF 7 3.69 0.90
4 2 16384 16384 44 ON 5 3.41 0.90
10 6 16384 16384 45 ON 5 3.41 0.90
16 10 10240 10240 45 ON 4 3.19 0.90
24 16 10240 10240 50 ON 4 3.19 0.90
32 24 10240 10240 57 ON 3 2.88 0.90
48 48 10240 10240 78 ON 3 2.88 0.90

Actual serving configuration:

  • Image: vllm/vllm-openai-rocm:kimi-k31@sha256:4930e37b377b3973384ed5e274dd3cfc18789cfd1f8a98c6cdb5c2dd359ab175.
  • Target moonshotai/Kimi-K3.1-Pro, draft Inferact/Kimi-K3.1-Pro-DSpark-preview; single-node TP8/DCP8/EP1, DCP backend a2a, max context 1048576.
  • Graph mode FULL_AND_PIECEWISE, compilation mode 3, +fused_rms_norm_gated; c1 additionally uses latency_specialized and disables BCG.
  • c4-c48: SimpleCPUOffloadConnector, lazy_offload=false, 1799 GB total / 224.875 GB per rank (209.43 GiB/rank). c1 has no CPU KV offload.
  • FP8 target/draft KV, MXFP4 indexer cache, FP32 KDA state, fused KDA prefill. No online linear FP8 quantization. VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=NONE, AITER_FLYDSL_STAGE2_FP8=0, AITER_BF16_FP8_MOE_BOUND=0.
  • HSA_NO_SCRATCH_RECLAIM=1, no explicit async scratch limit. AITER/DCP/sparse attention settings and timeouts remain those of [K3.1-Pro] Day-0 support vLLM on MI355X / [K3.1-Pro] MI355X 上 vLLM 的 Day-0 支持 #4806.
  • Recipe uses probabilistic drafting and block rejection. AgentX throughput substitutes the current golden synthetic AL values above and removes Kimi tool/reasoning parser arguments; evaluation verifies for real. These launcher behaviors are inherited, not introduced here.
AI model disclosure

OpenAI model via Codex; the exact runtime model/version identifier could not be verified. Role: recipe editing, configuration checks and PR description. No sub-agents used.

PR checklist and change type

Type of Change

  • Bug fix
  • New feature
  • Configuration change
  • Documentation update
  • Other (please describe)

Checklist

  • I have completed the AI model disclosure and kept it current
  • I have tested my changes locally
  • I have updated documentation if necessary
  • If python -m infx.matrix.changed --base origin/main lists affected config keys, I have appended one new entry covering them to the physical end of inferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entries
  • If this PR appends an inferencex-e2e/perf-changelog.yaml entry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs): full-sweep-fail-fast (recommended), full-sweep-enabled, or non-canary-full-sweep-enabled. Optional modifiers all-evals, evals-only, and agentx-fast require a primary label; the last two block reuse while applied.
  • Before merging via reuse, an authorized maintainer (OWNER/MEMBER/COLLABORATOR) has commented /use <run_id> (or the legacy /reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.
中文

基于 #4806 的14a3dc883ad9,保留模型接入,将c16/c24/c32/c48统一为10240 batch/capture和GMU 0.90。c16/c24仅删除超过10240的三个桶,c32/c48保留原桶列表,decode覆盖完整;最终45/50/57/78桶。c1/c4/c10不变。此配置调整旨在增加显存余量,尚不能称为此前collective hang的已验证修复。

实际参数见上表。沿用原镜像、TP8/DCP8/EP1、FP8 KV、MXFP4 indexer、FP32 KDA和1799 GB总CPU offload(209.43 GiB/rank);QR量化、Linear FP8、FlyDSL Stage2 FP8关闭,scratch reclaim关闭。吞吐使用最新golden synthetic AL,精度评测真实验证。上述继承配置并非本次新增。

配置调整由OpenAI模型经Codex完成编辑、配置检查和说明,精确运行时模型版本无法确认,未使用子agent。

cquil11 and others added 28 commits November 4, 2026 17:11
Single-node TP8 x DCP8 with the Inferact DSpark draft and block verification,
following vLLM's MI355X K3.1 recipe without its three reduced-precision knobs.
GPU-resident KV at conc 1 and 4; Simple CPU KV offload at conc 4-60.
Write the per-concurrency points as plain override variants instead of zip overrides.
# Conflicts:
#	inferencex-e2e/perf-changelog.yaml
The image keeps HSA_NO_SCRATCH_RECLAIM=1, which the cluster's MEC firmware 38
needs, so large FP4-indexer prefill scratch stays pinned outside vLLM's memory
budget. At 0.92 the DRAM-offload points ran out of GPU memory in long-context
prefill (HSA_STATUS_ERROR_OUT_OF_RESOURCES). Use 0.90, as the Kimi K3 MI355X
recipe did on the same nodes.
The MI355X AgentX lane and the reference apex-1104 sweep differed in the
points they ran and in batching, so a comparison between them could not
attribute anything to the engine build.

Align the parts that shape the workload: seven points instead of ten
(drop the GPU-resident conc 4 and the conc 52/60 tail), max-num-seqs
below concurrency as that sweep sets it, max-num-batched-tokens 16384
except 10240 at conc 48, its cudagraph capture sizes, and 2577 GB of
host DRAM for the Simple CPU offload.

Keep this lane's own precision and speculative settings: 5 DSpark tokens
with block rejection at every point, the published
Inferact/Kimi-K3.1-Pro-DSpark-preview draft, FP32 KDA recurrent state,
BF16 all-reduce, BF16 MoE stage 2, the AITER FA MLA prefill backend, and
no FP8 indexer override or KDA input-projection quantization. Those are
deliberate here, and copying them would change what the runs measure --
synthetic acceptance in particular reports a modelled throughput and
makes the output nondeterministic by construction.

The dram-utilization comment claimed 0.6 resolves to 1799 GB. The nodes
report 3000 GB, so the pinned 2577 GB is 86% of host DRAM, not 60% --
the recipe's cpu_bytes_to_use_per_rank wins. Say so instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The planner derives TOTAL_CPU_DRAM_GB from dram-utilization and rejected the
recipe's 2577 GB against the 1799 GB that 0.6 resolves to. 0.8592 resolves to
2577 GB, within the 3 TB per-server limit.

Align comments with the settings they describe and correct the changelog:
throughput points still run synthetic acceptance at the golden 3.08.
The conc-1 latency point runs the reference sweep's latency_specialized decode
backend with 7 draft tokens; the throughput points stay at 5.

K3.1 has no torch.compile submodule, so FULL_AND_PIECEWISE captures decode only
and breakable graphs are the only thing that would add the prefill sizes. Turn
them off and leave scratch reclaim on, as the image sets it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The apex-1104 sweep sets mla_prefill_backend on its conc-48 point and leaves
conc 1-32 on the default. Move it out of base into that point's override.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The apex-1104 sweep turns both off at conc 1 and on at conc 4-48. Move the
conc-1 values into that point's override and leave the base on the sweep's
throughput setting.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
# Conflicts:
#	inferencex-e2e/perf-changelog.yaml
On this cluster's MEC firmware 38, HSA_NO_SCRATCH_RECLAIM=0 hung RCCL
allgathers at 445k-700k context on conc 32 and conc 48, and conc 48 ran out
of GPU memory at 0.92. Keep the image's reclaim-off default with 0.90 at
conc 4-48 and no async scratch limit there; conc 1 keeps its own settings.
Run 41954410041's conc-48 point died on a 600 s RCCL allgather timeout with no
successful request. Cut the per-step working set at the two largest points:
max-num-batched-tokens 10240, gpu-memory-utilization 0.91 and 3 DSpark tokens,
with the capture lists rebuilt from those. The other five points keep the
reference apex-1104 sweep's settings.

Also run MoE stage 2 in MXFP8 as that sweep does, and leave the MLA prefill
backend at its default everywhere.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MEC firmware 38 on this cluster cannot reclaim RCCL scratch, and the release
blocks until the running dispatches finish, which is how the conc-16 and
conc-48 points hung on a 600 s allgather. Set HSA_NO_SCRATCH_RECLAIM=1 at
every point and drop HSA_SCRATCH_SINGLE_LIMIT_ASYNC, which only bounds a
reclaim that no longer happens.

Step the DSpark token count down with concurrency, 7/5/4/3, and rebuild each
capture list from it. Size the CPU offload at 224.875 GB per rank, the 1799 GB
that dram-utilization 0.6 resolves to.

Describe the points on their own terms rather than against another sweep.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…hput servers

Synthetic acceptance streams unverified draft tokens. The kimi_k3 tool and reasoning parsers stop streaming on some of them, which empties or truncates the response the client sees. Evals and acceptance gates verify for real and keep their parsers.
…nc 16-32

With HSA_NO_SCRATCH_RECLAIM=1 the kernel scratch of these points filled the VRAM left outside the 0.90-0.91 budget during warmup and stalled RCCL, even on freshly rebooted nodes. Their GPU KV use peaks well under the smaller pool, and DRAM offload backs the rest.
@YukioZzz
YukioZzz requested a review from a team November 6, 2026 07:13
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目,并且恰好带有一个主标签时,才会运行扫描:full-sweep-fail-fast(强烈推荐;canary 加逐矩阵 fail-fast)、full-sweep-enabled(有 canary;矩阵任务在失败后继续运行)或 non-canary-full-sweep-enabled(无 canary,也无 fail-fast)。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

@billishyahao billishyahao added non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim) agentx AgentX benchmarks, recipes, and infrastructure labels Nov 6, 2026
@YukioZzz YukioZzz changed the title Tune Kimi c16-c48 to 10K / 调整 Kimi c16-c48 至10K [K3.1-Pro] Day-0 support vLLM on MI355X Nov 6, 2026
Comment on lines +12038 to +12047
- config-keys:
- kimik3.1-fp4-mi355x-vllm-agentic-dspark
description:
- "Add Kimi-K3.1-Pro day-0 vLLM AgentX on MI355X: TP8 x DCP8 with DSpark, GPU-resident KV at conc 1 and Simple CPU KV offload at conc 4-48."
- "Admit below concurrency on the day-0 image: max-num-seqs 1/2/6/10/16/24/48, max-num-batched-tokens 16384 up to conc 24 and 10240 at conc 32-48, gpu-memory-utilization 0.90 up to conc 24 and 0.91 at conc 32-48, the latency_specialized decode backend at conc 1, and breakable cudagraphs off at conc 1 only. The two largest points carry the smaller batch and the tighter utilization because run 41954410041's conc-48 point died on a 600 s RCCL allgather timeout with no successful request."
- "Step the DSpark token count down with concurrency - 7 at conc 1, 5 at conc 4-10, 4 at conc 16-24, 3 at conc 32-48 - rather than holding one count everywhere. The cudagraph capture list follows from the point: the decode sizes are the multiples of one plus the token count, max-num-seqs of them, and the prefill ladder above them stops at max-num-batched-tokens."
- "Keep the day-0 image's HSA_NO_SCRATCH_RECLAIM=1 at every point, and drop HSA_SCRATCH_SINGLE_LIMIT_ASYNC, which only bounds a reclaim that no longer happens. Every mi355x-amds node reports MEC firmware 38, and vLLM's Kimi-K3 recipe asks for this below firmware 177 because that firmware cannot reclaim RCCL scratch; the release blocks until the running dispatches finish, which is the shape of the allgather hangs seen here."
- "Size the Simple CPU offload at 224.875 GB per rank, 1799 GB across the 8 TP ranks, which is what dram-utilization 0.6 resolves to on these nodes."
- "Precision and speculative settings on the day-0 image: block rejection with the published Inferact/Kimi-K3.1-Pro-DSpark-preview draft (throughput points replace it with the golden acceptance length for their token count, 3.21 at conc 1, 3.08 at conc 4-10, 2.92 at conc 16-24 and 2.70 at conc 32-48; evals and AL gates verify for real), FP32 KDA recurrent state, BF16 all-reduce (no INT4 quick reduce, so the AITER AR+RMSNorm fusion keeps every payload), BF16 MoE stage 2 (AITER_FLYDSL_STAGE2_FP8=0, so the lossy MXFP8 path is off at every concurrency), the default MLA prefill backend at every point, and no FP8 indexer override or KDA input-projection quantization."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4806

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The changelog check will fail this PR, and the history would record settings the recipe does not use. The block at perf-changelog.yaml:12038 is inserted mid-file before the sglang-dflash entry, and a second block is also appended at the tail. That breaks the append-only, one-block-per-PR rule. The mid-file block also claims max-num-batched-tokens 16384 up to conc 24 and gpu-memory-utilization 0.91 at conc 32-48. Fix: delete the mid-file block and keep a single tail block that describes the final recipe (10240 at c16-c48, 0.90 everywhere), folding in its still-accurate content.

Why this was flagged

compare_entries in infx/workflows/validate_perf_changelog.py:129-138 compares base entry N with head entry N. Inserting a block at line 12038 shifts every later entry, so the first base entry after the insertion no longer matches. The check then raises "entry N changed; existing entries are immutable except for pr-link-only corrections". validate_raw_change (line 172) would also reject it because head_raw no longer starts with base_raw. On the base branch the file is untouched, so this failure is new. Even if the check were bypassed, the PR adds two blocks for kimik3.1-fp4-mi355x-vllm-agentic-dspark, against the AGENTS.md:88 rule of at most one block per PR. The inserted block's "max-num-batched-tokens 16384 up to conc 24" and "0.91 at conc 32-48" contradict the recipe's override_dram_c16/c24 (10240) and override_dram_c32/c48 (0.90).

Verification: The diff inserts one block before the existing kimik3.1-fp4-mi355x-sglang-agentic-dflash entry. compare_entries (validate_perf_changelog.py:129-138) compares base_entries[index] with head_entries[index]. That raises "entry N changed; existing entries are immutable except for pr-link-only corrections".

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

billishyahao and others added 7 commits November 6, 2026 09:29
… huge pages

THP_MEM_ALLOC_ENABLE=1 is what makes torch's CPU allocator call
madvise(MADV_HUGEPAGE). The hosts run THP in madvise mode, so without it the
per-rank offload pool is backed entirely by 4 KiB pages: tens of millions of
them to register and keep mapped, against about a five-hundredth as many
2 MiB pages.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Raise max-num-batched-tokens from 10240 to 16384 at conc 16, 24, 32 and 48,
so every throughput point now runs the same budget, and extend each capture
list through the 12288, 14336 and 16384 rungs to match.

Add conc 56 and conc 64 on the same recipe with 2 DSpark tokens, the golden
acceptance length 2.44 that pairs with it, and the DRAM offload pool the
other throughput points use.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PR 4806 landed the MI355X K3.1-Pro recipe upstream, including the day-0
image digest and THP_MEM_ALLOC_ENABLE this branch had carried, so those
commits are now no-ops and the recipe resolves to the upstream base.

Resolutions: take this branch's concurrency list and recipe, which only add
to what 4806 shipped, and keep 4806's changelog entry as merged, describing
this branch's own changes in a separate entry under PR 4858.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two remote commits are merges of main only, and this side already has a
newer main, so the result is this side's tree.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agentx AgentX benchmarks, recipes, and infrastructure non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim)

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

3 participants