Repository navigation
Conversation
|
Thanks for the contribution!
中文感谢你的贡献!
|
31c06bb to
dda9782
Compare
dda9782 to
cbfaaa9
Compare
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=41285920098 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=41285920098 |
cbfaaa9 to
cec28da
Compare
更新五个 Kimi-K3 PD 点位:吞吐点启用 BF16 SSM,移除 1P1D c56 和 1P3D c12,增加 FP32 SSM 的 1P1D c1 K7 monokernel 点。临时 kernel 补丁单独提交。
导入 vLLM #60683、#58743、#60808、#63031 的公开提交,保留原始 SHA 与 cherry-pick 来源。安装器使用运行镜像对应的源码和 ROCm SDK 重建 HIP 扩展,便于 nightly 收录后整体移除。
官方 nightly 已包含四个依赖 PR,移除临时补丁、安装器及构建入口,仅保留 recipe 配置更新。
eaeee53 to
0902c81
Compare
固定包含所需上游功能的 nightly 提交和镜像 digest,确保各节点及重复运行使用同一镜像。
Resolve perf-changelog with the repository merge helper.
billishyahao
left a comment
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/40301194088
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/40301194088
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that
SGLANG_NVFP4_CKPT_FP8_NEXTN_MOEis not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison. - For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
- Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.
Additional detail section:
- Recipe: vllm-project/vllm#63726
Signed: billishyahao
|
@Oseltamivir staged run 40301194088. This run remains available across future |
| VLLM_ROCM_QUICK_REDUCE_QUANTIZATION: INT4 | ||
| VLLM_ROCM_QUICK_REDUCE_MIN_SIZE_BYTES_MB: '16' | ||
| VLLM_ROCM_QUICK_REDUCE_MAX_SIZE_BYTES_MB: '256' |
There was a problem hiding this comment.
doesnt this hurt long context evals at 500k? what is the KVV scores? https://github.com/MoonshotAI/Kimi-Vendor-Verifier
There was a problem hiding this comment.
Hi @functionstackx , I think @YukioZzz put some KVV scores on the descriptions.
Previous configuration validation (before the QR minimum-size fix and image update): run 40645935622 on b49b522 passed all five points. Evals: kimi_tool_call_schema_full 0.973–0.9853 and gsm8k em_strict 0.9659–0.9727. Throughput uses golden AL 3.36 (K4) and 3.84 (K7). Four of the five points are on the throughput/E2EL Pareto frontier; 1P1D c8 is dominated.
There was a problem hiding this comment.
Can we still keep this quick reduce INT4 enabled according to #4623 ?
Existing models are grandfathered against an immutable snapshot of the model support matrix,
including later recipe and image updates for those models.
There was a problem hiding this comment.
@billishyahao if u pass the entire KVV suite, u def can
https://github.com/MoonshotAI/Kimi-Vendor-Verifier
KVV consists of the OCRBench MMMU Pro Vision BEAM (1M) DeepSWE
There was a problem hiding this comment.
CI only runs an subsample of KVV, you would need to run the other parts of KVV such as OCRBench MMMU Pro Vision BEAM (1M) DeepSWE
There was a problem hiding this comment.
The full bench of kvv like MMMU/DeepSWE is a little bit heavy. I removed the QR env for this time.
…ly-kernels # Conflicts: # inferencex-e2e/perf-changelog.yaml
…ly-kernels # Conflicts: # inferencex-e2e/perf-changelog.yaml
|
As mentioned last week to everyone,we will deprecate it and not accept any more merges the moment k3.1 pro weights drop. this means existing open PRs will not be accepted anymore after k3.1 pro weights drops. The PR may not have been merged due to various factors:
|
Run Kimi-K3 FP4 PD on MI355X with
vllm/vllm-openai-rocm:nightly-2b009fc6d1493a9a1ad19f4d0c7230b7e6b7e63a(wasnightly-81198e97ba7eee2a22540caaa756b7fdddcb4d93), TP8/DCP8 workers, FP8 KV cache, and MoRIIO RDMA transfer. The image is pinned by digest and includes the eager SimpleCPUOffload CPU LRU refresh for GPU-hit prefixes and Mamba boundary state (vllm#62444). The recipe provides five configurations:The c8 decode worker uses
latency_specializedwith the V2 model runner and graph capture sizes[1, 8]. The four offload configurations use BF16 SSM state on both prefill and decode, with prefill-only SimpleCPUOffload at DRAM utilization 0.8 (2399 GB per prefill worker).Compared with
main, this removes 1P3D c12 (TP8/DCP1 decode) and 1P1D c56, adds 1P1D c8, switches the throughput points from FP32 to BF16 SSM state, and raises the offload budget from 1799 GB (DRAM utilization 0.6). The new image adds Kimi-K3 specialized decode on MI355X (vllm#60683), MoRIIO consumers with specialized decode (vllm#63031), BF16 KDA recurrent state (vllm#58743) and MoRIIO READ completion before the forward (vllm#60808).DSpark uses
/models/Inferact-Kimi-K3-DSparkwith probabilistic drafting and block rejection; throughput runs use the selected golden acceptance curve, while evals use real verification.The recipe uses
vllm-routerwith consistent-hash routing and the standard eval selection. Recipe documentation: vllm-project/vllm#63726 (merged).DSpark keeps BF16 weights and FP8 KV cache. ROCm QuickReduce is disabled on all prefill and decode workers, keeping draft and target collectives unquantized.
Validation: run 41285920098 on
7ffbdecapassed all five benchmark points and both evals on every point.kimi_tool_call_schema_fullscores 0.9779–0.9828 over 408 cases per point; GSM8Kem_strictscores 0.9674–0.9742 over 1,319 cases per point. Throughput uses golden AL 3.36 (K4) and 3.84 (K7). Four of the five points are on the throughput/E2EL Pareto frontier; 1P1D c8 is dominated.AI model disclosure
AI assistance prepared the recipe and validation, reviewed upstream dependencies, and rebased the PR. A separate agent previously checked documentation discoverability. Exact model identifiers could not be verified from the runtime.
Claude Opus 5.5 (
claude-opus-5-5, Claude Code) rewrote the perf-changelog block from the diff, checked the draft data types against the vLLM source at both image commits, and updated this description.PR checklist and change type
Type of Change
Checklist
python -m infx.matrix.changed --base origin/mainlists affected config keys, I have appended one new entry covering them to the physical end ofinferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entriesinferencex-e2e/perf-changelog.yamlentry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs):full-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied.OWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.中文
在 MI355X 上使用官方
vllm/vllm-openai-rocm:nightly-2b009fc6d1493a9a1ad19f4d0c7230b7e6b7e63a运行 Kimi-K3 FP4 PD,P/D worker 均采用 TP8/DCP8、FP8 KV cache,通过 MoRIIO RDMA 传输 KV,镜像以 digest 固定,包含 GPU-hit prefix 及 Mamba 边界状态的 eager SimpleCPUOffload CPU LRU 刷新修复(vllm#62444)。与
main相比(原镜像为nightly-81198e97ba7eee2a22540caaa756b7fdddcb4d93),本 PR 移除 1P3D c12(decode 为 TP8/DCP1)和 1P1D c56,新增 1P1D c8,吞吐点的 SSM 状态由 FP32 改为 BF16,offload 预算从 1799 GB(DRAM utilization 0.6)提高。新镜像包含 MI355X 上的 Kimi-K3 specialized decode(vllm#60683)、支持 specialized decode 的 MoRIIO consumer(vllm#63031)、BF16 KDA 循环状态(vllm#58743)以及在 forward 前等待 MoRIIO READ 完成(vllm#60808)。五种配置见上表:1P1D c8 使用 FP32 SSM、K7 DSpark,不启用 CPU offload;decode 端使用 V2 model runner 的
latency_specialized后端,graph capture sizes 为[1, 8]。其余四个吞吐配置使用 BF16 SSM、K4 DSpark 和仅 P 侧启用的 SimpleCPUOffload,DRAM utilization 为 0.8,每个 P worker 的预算为 2399 GB。DSpark draft 为
/models/Inferact-Kimi-K3-DSpark,使用 probabilistic drafting 和 block rejection。吞吐测试采用对应的 golden acceptance curve,eval 使用真实验证。路由采用vllm-routerconsistent hash,评测沿用标准选择。上游 recipe:vllm-project/vllm#63726(已合并)。DSpark 保持 BF16 权重和 FP8 KV cache。所有 P/D worker 关闭 ROCm QuickReduce,使 draft 和 target 的 collective 保持未量化。
验证:run 41285920098 在
7ffbdeca上通过五个 benchmark 点,且每个点的两项 eval 均通过。kimi_tool_call_schema_full每点 408 个用例,得分范围为 0.9779–0.9828;GSM8Kem_strict每点 1,319 个用例,得分范围为 0.9674–0.9742。吞吐测试使用 golden AL 3.36(K4)和 3.84(K7)。五个点中有四个位于吞吐/E2EL Pareto 前沿上;1P1D c8 被支配。AI 协助配置、验证、依赖核对及 rebase;此前有独立 agent 检查文档可发现性。精确模型标识无法从运行环境核实。Claude Opus 5.5(
claude-opus-5-5,Claude Code)根据 diff 重写了 perf-changelog 条目,对照两个镜像 commit 的 vLLM 源码核对了 draft 数据类型,并更新了本描述。🤖 Generated with Claude Code