Repository navigation
[K3.1-Pro] Day-0 support vLLM on MI355X / [K3.1-Pro] MI355X 上 vLLM 的 Day-0 支持 - #4806
Conversation
|
Thanks for the contribution!
中文感谢你的贡献!
|
| - kimik3.1-fp4-mi355x-vllm-agentic-dspark | ||
| description: | ||
| - "Add Kimi-K3.1-Pro day-0 vLLM AgentX on MI355X: TP8 x DCP8 with DSpark (Inferact/Kimi-K3.1-Pro-DSpark-preview), GPU-resident KV at conc 1 and 4 and Simple CPU KV offload at conc 4-60." | ||
| pr-link: TBD |
There was a problem hiding this comment.
🔴 Merging this PR through the reuse-merge tool will fail. The new changelog entry's pr-link: TBD is not an accepted placeholder. validate_added_pr_link (validate_perf_changelog.py:113) accepts only XXX, https://github.com/SemiAnalysisAI/InferenceX/pull/XXX or this PR's own canonical URL. Fix: set the line to pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX, the form CONTRIBUTING.md:211 shows, or to this PR's real URL. The merge step then rewrites it to the real link.
Why this was flagged
The PR-time check in run-sweep.yml:105 only runs infx.matrix.plan. That check types pr-link as a plain str (validation.py:1143), so TBD passes the sweep. At merge, merge_with_reuse.py:688 calls canonicalize_changelog. That calls canonicalize_appended_links → compare_entries (prepare_perf_changelog_merge.py:88), which calls validate_added_pr_link("TBD", pr). It raises "new PR entry must use 'https://github.com/SemiAnalysisAI/InferenceX/pull/' or an XXX placeholder; found 'TBD'" because TBD is not in PR_LINK_PLACEHOLDERS (validate_perf_changelog.py:21-24). By then merge_with_reuse.py:658 has already posted /reuse-sweep-run on the PR. The changelog-conflict path, resolve_conflict_bytes (prepare_perf_changelog_merge.py:146), also goes through compare_entries and fails the same way. No other entry in perf-changelog.yaml uses TBD.
Verification: The diff adds pr-link: TBD at inferencex-e2e/perf-changelog.yaml:12020. merge_with_reuse.py:688 calls canonicalize_changelog(pr, repo) unconditionally, which reaches validate_added_pr_link (validate_perf_changelog.py:112-116) and raises for any link that is not a placeholder or this PR's canonical URL. run-sweep.yml:105 never calls compare_entries, so TBD passes CI and fails only at merge.
| # A node's first run downloads ~1.56 TB of MXFP4 shards into its local HF cache before loading. | ||
| health_check: | ||
| interval_seconds: 10 | ||
| max_attempts: 720 |
There was a problem hiding this comment.
🔴 On every MI355X node without the ~1.56 TB checkpoint already cached, the first K3.1 point can fail its readiness check. The download, load and graph capture all fit in the 7200 s budget only if the node downloads at about 217 MB/s or faster. Nothing routes this model to the shared cache. The budget is max_attempts: 720 x 10 s, matching VLLM_ENGINE_READY_TIMEOUT_S: '7200'. Fix: put the checkpoint where each node does not re-download it before serving. For example, add moonshotai/Kimi-K3.1-* to SHARED_HF_CACHE_LANES["mi355x-amds"], or stage a models.entries row. Alternatively, size the readiness budget for a 1.56 TB cold download.
Why this was flagged
The trigger is a sweep point landing on an mi355x-amds node whose node-local /var/lib/hf-hub-cache (runners.yaml:789) does not yet hold Kimi-K3.1-Pro. Every node starts that way, because no models.entries row exists for this model on mi355x-amds (runners.yaml:776-780). single_node_hf_cache (infx/launch/drivers/srt/models.py:198-201) only picks shared-hf-hub-cache for models matched in SHARED_HF_CACHE_LANES, and models.py:187-194 has no Kimi rule. As a result, vLLM pulls about 1.56 TB from HF inside server startup. The sibling MI355X recipe uses the same 720 x 10 s health check and notes at minimaxm31flash/vllm/mi355x-fp4/agentic.yaml:26-27 that a cold load of only 240 GB can use up the whole VLLM_ENGINE_READY_TIMEOUT_S. On a slow link, the first point on each node exceeds 7200 s and the job fails. Under full-sweep-fail-fast, that one failure cancels the sweep. The 10 points spread across nodes each pay the download separately. The recipe's comment at line 17 acknowledges the download, but nothing bounds or avoids it.
Verification: The readiness budget is 7200 s: agentic.yaml:18-20 sets interval_seconds: 10 and max_attempts: 720. No shared-cache rule matches moonshotai/Kimi-K3.1-Pro (models.py:187-194), so it falls through to node-local /var/lib/hf-hub-cache (runners.yaml:789), and vLLM downloads it during startup. 1.56 TB / 7200 s ≈ 217 MB/s. A node below that rate fails its first point.
|
|
||
| # Admission equals CONC: each breakable graph size costs graph-pool memory taken from the KV cache. | ||
| # Graphs cover every decode batch, CONC x (1 + drafts) in steps of 1 + drafts, then the prefill sizes; |
There was a problem hiding this comment.
🟡 nit (optional): Several new comments restate the YAML beneath them, which the AGENTS.md comment rule forbids. sweep:^\s*# (Graphs cover every decode batch|.*c1 and c4 keep the KV cache on the GPUs|Verify the draft tokens with) For example, "Graphs cover every decode batch, CONC x (1 + drafts) in steps of 1 + drafts, then the prefill sizes" only repeats the cudagraph_capture_sizes lists. Fix: drop the restating sentences and keep only the reasons, such as why admission equals CONC and why block rejection gives way to golden AL.
Why this was flagged
AGENTS.md:58 allows only a short technical reason for a line or block and says "never restate the code". At agentic.yaml:88 the comment spells out the capture-size pattern that the compilation-config lists below it already show. At amd-master.yaml:913, "c1 and c4 keep the KV cache on the GPUs" repeats kv-offloading: none, conc-list: [1, 4]. At agentic.yaml:64, "Verify the draft tokens with the AITER assembly DCP kernel" repeats VLLM_ROCM_AITER_MLA_DCP_VERIFY: asm. The impact is limited to the repo convention, and runtime behaviour does not change.
Verification: AGENTS.md:58, which the root CLAUDE.md hands guidance to, says "never restate the code." In amd-master.yaml, "c1 and c4 keep the KV cache on the GPUs." only repeats kv-offloading: none, conc-list: [1, 4] on the row directly below. In agentic.yaml, "Verify the draft tokens with the AITER assembly DCP kernel." mostly spells out VLLM_ROCM_AITER_MLA_DCP_VERIFY: asm. Nothing breaks at runtime.
833d26f to
691b84c
Compare
Single-node TP8 x DCP8 with the Inferact DSpark draft and block verification, following vLLM's MI355X K3.1 recipe without its three reduced-precision knobs. GPU-resident KV at conc 1 and 4; Simple CPU KV offload at conc 4-60.
Write the per-concurrency points as plain override variants instead of zip overrides.
691b84c to
bc5c564
Compare
# Conflicts: # inferencex-e2e/perf-changelog.yaml
The image keeps HSA_NO_SCRATCH_RECLAIM=1, which the cluster's MEC firmware 38 needs, so large FP4-indexer prefill scratch stays pinned outside vLLM's memory budget. At 0.92 the DRAM-offload points ran out of GPU memory in long-context prefill (HSA_STATUS_ERROR_OUT_OF_RESOURCES). Use 0.90, as the Kimi K3 MI355X recipe did on the same nodes.
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=42193891273 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=42193891273 |
The MI355X AgentX lane and the reference apex-1104 sweep differed in the points they ran and in batching, so a comparison between them could not attribute anything to the engine build. Align the parts that shape the workload: seven points instead of ten (drop the GPU-resident conc 4 and the conc 52/60 tail), max-num-seqs below concurrency as that sweep sets it, max-num-batched-tokens 16384 except 10240 at conc 48, its cudagraph capture sizes, and 2577 GB of host DRAM for the Simple CPU offload. Keep this lane's own precision and speculative settings: 5 DSpark tokens with block rejection at every point, the published Inferact/Kimi-K3.1-Pro-DSpark-preview draft, FP32 KDA recurrent state, BF16 all-reduce, BF16 MoE stage 2, the AITER FA MLA prefill backend, and no FP8 indexer override or KDA input-projection quantization. Those are deliberate here, and copying them would change what the runs measure -- synthetic acceptance in particular reports a modelled throughput and makes the output nondeterministic by construction. The dram-utilization comment claimed 0.6 resolves to 1799 GB. The nodes report 3000 GB, so the pinned 2577 GB is 86% of host DRAM, not 60% -- the recipe's cpu_bytes_to_use_per_rank wins. Say so instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The planner derives TOTAL_CPU_DRAM_GB from dram-utilization and rejected the recipe's 2577 GB against the 1799 GB that 0.6 resolves to. 0.8592 resolves to 2577 GB, within the 3 TB per-server limit. Align comments with the settings they describe and correct the changelog: throughput points still run synthetic acceptance at the golden 3.08.
The conc-1 latency point runs the reference sweep's latency_specialized decode backend with 7 draft tokens; the throughput points stay at 5. K3.1 has no torch.compile submodule, so FULL_AND_PIECEWISE captures decode only and breakable graphs are the only thing that would add the prefill sizes. Turn them off and leave scratch reclaim on, as the image sets it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The apex-1104 sweep sets mla_prefill_backend on its conc-48 point and leaves conc 1-32 on the default. Move it out of base into that point's override. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The apex-1104 sweep turns both off at conc 1 and on at conc 4-48. Move the conc-1 values into that point's override and leave the base on the sweep's throughput setting. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
# Conflicts: # inferencex-e2e/perf-changelog.yaml
…hput servers Synthetic acceptance streams unverified draft tokens. The kimi_k3 tool and reasoning parsers stop streaming on some of them, which empties or truncates the response the client sees. Evals and acceptance gates verify for real and keep their parsers.
…nc 16-32 With HSA_NO_SCRATCH_RECLAIM=1 the kernel scratch of these points filled the VRAM left outside the 0.90-0.91 budget during warmup and stalled RCCL, even on freshly rebooted nodes. Their GPU KV use peaks well under the smaller pool, and DRAM offload backs the rest.
… huge pages THP_MEM_ALLOC_ENABLE=1 is what makes torch's CPU allocator call madvise(MADV_HUGEPAGE). The hosts run THP in madvise mode, so without it the 224 GB per-rank offload pool is backed entirely by 4 KiB pages: about 55 million of them to register and keep mapped, against 107 thousand 2 MiB pages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
/reuse-sweep-run 42193891273 |
|
/use 42193891273 |
|
@billishyahao staged run 42193891273. This run remains available across future |
…skip-sweep] - Add Kimi-K3.1-Pro vLLM AgentX on one 8x MI355X node with vLLM's day-0 image vllm/vllm-openai-rocm:kimi-k31@sha256:5f47cc1004d9a7ac65f8a913725edb20c62c99a07e31cbf4ba21741bd0b4f9b1 (vLLM 0.0.0+mum.14d1ec2336, AITER 52846bc2). - TP8 x DCP8 (a2a DCP backend) with the Inferact/Kimi-K3.1-Pro-DSpark-preview draft: GPU-resident KV at concurrency 1, and SimpleCPUOffloadConnector DRAM offload at concurrency 4, 10, 16, 24, 32 and 48 (224.875 GB per rank, 1799 GB per node, which is what dram-utilization 0.6 resolves to). - Per point at concurrency 1/4/10/16/24/32/48: max-num-seqs 1/2/6/10/16/24/48 and DSpark draft tokens 7/5/5/4/4/3/3. max-num-batched-tokens is 16384 and gpu-memory-utilization 0.90 through concurrency 24; both change to 10240 and 0.91 at concurrency 32 and 48. - Every point compiles in mode 3 with FULL_AND_PIECEWISE CUDA graphs. Concurrency 1 runs the latency_specialized decode backend, captures sizes 2-16 and turns breakable CUDA graphs off; the offload points keep VLLM_USE_BREAKABLE_CUDAGRAPH=1 and capture max-num-seqs multiples of (1 + draft tokens) plus a prefill ladder up to max-num-batched-tokens. - Common settings: FP8 KV cache, FP32 KDA state (mamba-ssm-cache-dtype float32), MXFP4 indexer cache (indexer_kv_dtype mxfp4), fused KDA prefill, prefix caching with prefix-match-unit 64, max-model-len 1048576, fastsafetensors loading, stream-interval 10, AITER ASM DCP verify, AITER DCP allgather, FlyDSL sparse MLA, the image's HSA_NO_SCRATCH_RECLAIM=1, transparent huge pages for the offload pool (THP_MEM_ALLOC_ENABLE=1), and 1200 s execute-model and 7200 s engine-ready timeouts. - K3.1 week-1 precision policy: linear layers and shared experts stay BF16 (no quantization-config); routed experts run A4W4, MXFP4 weights with FP4 activations, through the SiTUv2 FlyDSL MoE (VLLM_ROCM_USE_AITER_MOE_SITUV2=1, the same as the image's auto default); MoE stage 2 stays BF16 (AITER_FLYDSL_STAGE2_FP8=0); all-reduce stays BF16 (VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=NONE). - Verification is block rejection with probabilistic draft sampling. Throughput runs replace it with the golden acceptance length for the draft length: 3.69 at 7 tokens, 3.41 at 5, 3.19 at 4 and 2.88 at 3. Evals and AL gates verify for real. - No change to the DSpark draft's precision: Inferact/Kimi-K3.1-Pro-DSpark-preview (revision 6a2ef20a) ships all-BF16 weights with no quantization_config, the speculative config sets no quantization, so get_draft_quant_config() returns None and K3DSparkModel loads its layers in BF16 with the target's BF16 embedding and LM head. Its KV cache is FP8, the same as the target's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Accuracy-sensitive optimizations (#4705
All-reduce and reduce-scatter
Disclosed, but outside the six items
Needs a ruling: the MXFP4 indexer cache. No three-arm evidence is attached, because none of the six optimizations is enabled. Draft data types: no change. The DSpark draft
The image's vLLM is a non-public build (
The recipe sets no 中文精度敏感优化(#4705
All-reduce 与 reduce-scatter
需要披露、但不属于这六项
需要裁定:MXFP4 indexer cache。 由于六项优化均未启用,本 PR 未附三组对照证据。 草稿数据类型:无变化。 DSpark 草稿模型
镜像中的 vLLM 是非公开构建(
本 recipe 没有设置 |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Checklist items
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that
SGLANG_NVFP4_CKPT_FP8_NEXTN_MOEis not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison. - For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- Verified that new models disable integer all-reduce, with SGLang ROCm explicitly setting
ROCM_QUICK_REDUCE_QUANTIZATION=NONE. Existing models are grandfathered per MODELS.md. - For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- For official InferenceX (non-preview), verified that established models use upstream engine images built from
master(or the equivalent default development branch), with new models transitioning within a reasonable time after release. See MODELS.md. - If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
- Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.
Additional detail section:
- insert any additional info here
- recipe https://recipes.vllm.ai/moonshotai/Kimi-K3.1-Pro?hardware=mi355x&features=tool_calling%2Creasoning%2Cspec_decoding%2Ctext_only&preset=mi355x_agentx&strategy=single_node_tp&frontend=python&kv_offload=simple
- #4806 (comment)
Signed: seungrokj
❌❌❌ REJECTED ❌❌❌@seungrokj Blocking: the published vLLM Kimi-K3.1-Pro recipe is unchanged since 2026-11-06T00:43Z ( ❌ Check 3 (Upstream recipe): FAIL — For the single-node agg worker of Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — @seungrokj is a listed owner of ✅ Check 1 (Sweep on in-PR commit): PASS — Run 42193891273 (attempt 2, head ✅ Check 2 (Evals): PASS — The evals ran on the PR image ✅ Check 4 (Reuse command): PASS — ✅ Check 5 (Checklist template): PASS — All 20 items of the current PR_REVIEW_CHECKLIST.md template are present and checked. ✅ Check 6 (Upstream image/ordering/provenance): PASS — ✅ Check 7 (Deprecated models): PASS — ✅ Check 8 (Architecture/accuracy hacks): PASS — There are no ✅ Check 9 (Chat template): PASS — Agentic throughput replays traces through the aiperf chat client, and the AL gate and evals use chat-template encoding. ✅ Check 10 (Engine patches): PASS — No patches, setup scripts, source edits or wheel installs can be reached from this recipe or master entry. ✅ Check 11 (Golden AL): PASS — The run's submission logs show the injected ➖ Check 12 (Append-only): N/A — The new changelog entry does not set ✅ Check 13 (Draft precision): PASS — ✅ Check 14 (Pareto coverage): PASS — Curve kimik3.1/agentic-coding/mi355x_vllm/fp4/r42193891273/P90 E2EL, from the ✅ Check 15 (PR description): PASS — The per-point max-num-seqs, draft tokens, batched tokens, memory utilization, precision settings, golden AL values, validation run and commit, and the ➖ Check 16 (NVIDIA overview): N/A — The only affected hardware is AMD MI355X. ➖ Check 17 (Metrics publication): N/A — This is a direct vLLM config, so no Dynamo TRT-LLM metrics settings apply. ✅ Check 18 (Recipe/image/sweep consistency): PASS — All 7 points generate with per-concurrency overrides, and TP8×DCP8 on 8 GPUs matches the master entry. The digest-pinned image resolves, no points are removed, and the changelog accurately states the image, precision policy, FP8 KV, MXFP4 indexer cache and BF16 DSpark draft. Assessed commit: |
|
/reuse-sweep-run 42193891273 |
|
Ingest report — ✅ Results ingested into the Ingest workflow · Merge Ingest run · Benchmark source run
benchmarkFiles 7, rawBenchmarkRows 7, coveredByPerJobFiles 0, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 2638, gpuMetricSamplesNew 284272, powerPublicationPoints 7, changelogEntriesWritten 1 Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun. 中文Ingest 报告 — ✅ 结果已写入 上表仅统计本次运行写入的数据,各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。 |
Adds Kimi K3.1 Pro day-0 vLLM AgentX on MI355X. It runs vLLM's AMD day-0 image (
vllm/vllm-openai-rocm:kimi-k31@sha256:5f47cc10…, vLLM0.0.0+mum.14d1ec2336, AITER52846bc2) with the Inferact DSpark draft, at the concurrencies Kimi K3 currently shows on InferenceX for MI355X vLLM.Flags follow vLLM's K3.1 recipe (recipes.vllm.ai): the MI355X override and the
mi355x_agentxpreset. The preset's three reduced-precision knobs are removed, and block verification is used.The env keeps the image's
HSA_NO_SCRATCH_RECLAIM=1. On this cluster's MEC firmware 38, the preset'sHSA_NO_SCRATCH_RECLAIM=0hung RCCL allgathers at 445k–700k context.gpu-memory-utilizationis 0.90 through c24 and 0.91 at c32 and c48 (0.92 ran out of GPU memory at c48).VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1200, from the preset's runbook, covers long agentic prefills. c1 runs thelatency_specializeddecode backend with breakable graphs off.kimik3.1-fp4-mi355x-vllm-agentic-dsparkThe points follow the reference apex-1104 sweep. max-num-seqs is below concurrency from c4 to c32, and the DSpark draft length steps down from 7 to 3 as concurrency rises. Decode CUDA graphs capture the max-num-seqs multiples of one plus the draft length, then a prefill ladder up to max-num-batched-tokens (c1 captures 2–16 only).
dram-utilization: 0.6resolves to 1799 GB of host DRAM (224.875 GB per TP rank), within the 3 TB per-server limit.K3.1 week-1 precision policy
--quantization-config. The preset's online FP8 for the KDA input projections (--quantization-config '{"targets":{"*in_proj_qkvgfab":"fp8_per_channel"}}') is dropped.AITER_FLYDSL_STAGE2_FP8=1, which sums MoE stage 2 through MXFP8, is dropped and set explicitly to0.VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4is dropped and set explicitly toNONE. This also satisfies the MODELS.md integer-quantized all-reduce ban. The image's inherited environment sets no quick-reduce or AITER quantization control.VLLM_ROCM_USE_AITER_MOE_SITUV2=1selects the SiTUv2 FlyDSL MoE in a4w4 mode, the same as the image'sautodefault. The policy allows A4W4.kv-cache-dtype: fp8.mamba-ssm-cache-dtype: float32."indexer_kv_dtype":"mxfp4"in--attention-config. This is vLLM's own MI355X setting.Inferact/Kimi-K3.1-Pro-DSpark-previewas shipped: BF16 weights, with FP8 draft KV that matches the FP8 target KV. The load-path evidence is in the draft data-type comment.Verifier: the preset uses vLLM's default standard rejection sampler. This config sets
rejection_sample_method: blockwithdraft_sample_method: probabilistic, which matches the merged golden curvekimi-k3.1-pro__dspark__kimi-k3.1-pro-dspark-preview__probabilistic-block(as updated in #4851). Throughput runs get synthetic acceptance from that curve: 3.69 at 7 drafts (c1), 3.41 at 5, 3.19 at 4 and 2.88 at 3. The eval and AL-gate jobs verify for real.First run per node downloads the checkpoint: K3.1 is not staged on the MI355X cluster, so there is no
models.entriesrow. Each node's first run downloads the ~1.56 TB checkpoint into its node-local HF cache (/var/lib/hf-hub-cache). Every node currently has at least 11 TB free. No shared-cache lane routes this model to the shared Weka cache.No disaggregated points: Kimi K3 also had MI355X
vllm-disaggpoints. vLLM documents K3.1 prefill/decode disaggregation only withNixlConnectoron NVIDIA, and needsVLLM_SSM_CONV_STATE_LAYOUT=DSthere. No ROCm PD path for K3.1's KDA conv state is documented, so this PR has no PD points.MI355X is in vLLM's supported hardware for K3.1 but is not marked verified.
Validation: run 42193891273 at
740ee083passed all seven throughput points (c1–c48) and the c1 and c48 agentic evals. It is pinned with/use 42193891273.中文
本 PR 在 MI355X 上新增 Kimi K3.1 Pro 的 Day-0 vLLM AgentX。它使用 vLLM 的 AMD Day-0 镜像(
vllm/vllm-openai-rocm:kimi-k31@sha256:5f47cc10…,vLLM0.0.0+mum.14d1ec2336,AITER52846bc2)和 Inferact DSpark 草稿模型,并发点与 InferenceX 目前展示的 Kimi K3 MI355X vLLM 结果一致。参数遵循 vLLM 的 K3.1 recipe(recipes.vllm.ai):MI355X override 和
mi355x_agentx预设。去掉了预设中的三个降精度开关,并使用块验证(block verification)。环境变量保留镜像自带的
HSA_NO_SCRATCH_RECLAIM=1。在本集群的 MEC 固件 38 上,预设的HSA_NO_SCRATCH_RECLAIM=0会让 RCCL allgather 在 445k–700k 上下文时挂起。gpu-memory-utilization在 c24 及以下为 0.90,c32 和 c48 为 0.91(0.92 在 c48 时 GPU 显存不足)。VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1200来自预设的运行手册,用于覆盖较长的 agentic prefill。c1 使用latency_specializeddecode 后端,并关闭 breakable graphs。kimik3.1-fp4-mi355x-vllm-agentic-dspark这些并发点沿用参考的 apex-1104 sweep。c4 到 c32 的 max-num-seqs 低于并发数,DSpark 草稿长度随并发升高从 7 逐步降到 3。decode CUDA graph 捕获 (1 + 草稿长度) 的倍数,共 max-num-seqs 个,其上是直到 max-num-batched-tokens 的 prefill 梯度(c1 只捕获 2–16)。
dram-utilization: 0.6对应 1799 GB 主机内存(每个 TP rank 224.875 GB),在每台服务器 3 TB 的上限之内。K3.1 第一周精度政策
--quantization-config。去掉了预设中对 KDA 输入投影的在线 FP8(--quantization-config '{"targets":{"*in_proj_qkvgfab":"fp8_per_channel"}}')。AITER_FLYDSL_STAGE2_FP8=1(MoE stage 2 经 MXFP8 求和),并显式设为0。VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4,并显式设为NONE。这同时满足 MODELS.md 对整数量化 all-reduce 的禁令。镜像继承的环境变量未设置任何 quick-reduce 或 AITER 量化控制。VLLM_ROCM_USE_AITER_MOE_SITUV2=1选择 a4w4 模式的 SiTUv2 FlyDSL MoE,与镜像的auto默认值相同。该政策允许 A4W4。kv-cache-dtype: fp8。mamba-ssm-cache-dtype: float32。--attention-config中的"indexer_kv_dtype":"mxfp4"设置。这是 vLLM 自己的 MI355X 设置。Inferact/Kimi-K3.1-Pro-DSpark-preview按发布形式运行:BF16 权重,草稿 FP8 KV 与目标模型的 FP8 KV 一致。加载路径的证据见草稿数据类型评论。验证器: 预设使用 vLLM 默认的标准拒绝采样器。本配置设置
rejection_sample_method: block和draft_sample_method: probabilistic,与已合并的黄金曲线kimi-k3.1-pro__dspark__kimi-k3.1-pro-dspark-preview__probabilistic-block(经 #4851 更新)一致。吞吐测试使用该曲线的合成接受长度:7 个草稿(c1)为 3.69,5 个为 3.41,4 个为 3.19,3 个为 2.88。eval 和 AL gate 任务执行真实验证。每个节点首次运行会下载 checkpoint: K3.1 未预置在 MI355X 集群上,因此没有
models.entries条目。每个节点首次运行会把约 1.56 TB 的 checkpoint 下载到节点本地 HF 缓存(/var/lib/hf-hub-cache)。目前每个节点至少有 11 TB 空闲空间。没有共享缓存通道把该模型路由到共享 Weka 缓存。没有 PD 分离点: Kimi K3 在 MI355X 上还有
vllm-disagg点。vLLM 只在 NVIDIA 上用NixlConnector记录了 K3.1 的 prefill/decode 分离,并且需要VLLM_SSM_CONV_STATE_LAYOUT=DS。ROCm 上 K3.1 KDA conv state 的 PD 路径没有文档,因此本 PR 不包含 PD 点。MI355X 在 vLLM 的 K3.1 支持硬件列表中,但未标记为已验证。
验证: run 42193891273(
740ee083)中七个吞吐点(c1–c48)以及 c1 和 c48 的 agentic eval 全部通过,已通过/use 42193891273固定。AI model disclosure
claude-opus-5-5(Claude Opus 5.5, in Claude Code): synced the perf-changelog block with the diff, checked the draft data types in the image's vLLM source, and revised this description.