[mispa-ms]

Update the Kimi-K3.1-Pro B300 vLLM aggregated AgentX configs (kimik3.1-fp4-b300-vllm-agentic-dspark, one B300 node, image kimi-k31-mc@sha256:f8199876…). Six points (c1, c6, c12, c24, c48, c64; c1 is not on the throughput/E2EL frontier):

• c1, c6: TP8 x DCP8, fused Kimi-K3 decode on FULL_DECODE_ONLY graphs, DSpark 4 tokens (agentic.yaml).
• c12, c24: the [K3.1-Pro] vLLM B300: per-concurrency TP8 / DCP8 / TEP recipe on kimi-k31-mc + TEP shared-expert TP shard / [K3.1-Pro] vLLM B300：按并发选择 TP8 / DCP8 / TEP，基于 kimi-k31-mc 并对 TEP shared expert 做 TP 切分 #4878 recipe, which also graphs prefill and mixed steps up to 4096 tokens, DSpark 3 tokens (agentic-mixed-graphs.yaml).
• c48, c64: routed experts EP8 on MegaMoE, FULL_DECODE_ONLY graphs, DSpark 4 tokens (agentic-ep.yaml).
• c12 and up admit prefill once every 8 engine steps (prefill-schedule-interval: 8).

Consolidates #4878 (Bowen Wang co-authored the config commit). No setup script or source patch; precision unchanged. vLLM recipe: vllm-project/vllm#66433.

AI assistance: the recipe search, the runs and this description were prepared with an AI coding assistant.

Corrected by Claude Code for @Ankur-singh: c72 was dropped in ec76df1, so the head runs six points; the changelog entry now says "six points".

[github-actions]

Thanks for the contribution!

• Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
• PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
• After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.

中文

感谢你的贡献！

• **审阅：**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER，请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，再通过 Slack 联系核心维护者。必须严格遵循模板，并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have，才能触发签核验证。
• **PR 验证：**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目，并且恰好带有一个主标签时，才会运行扫描：full-sweep-fail-fast（强烈推荐；canary 加逐矩阵 fail-fast）、full-sweep-enabled（有 canary；矩阵任务在失败后继续运行）或 non-canary-full-sweep-enabled（无 canary，也无 fail-fast）。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
• **合并后：**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复；参见重新运行失败任务的说明。

[claude]

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

[github-actions]

View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=42847854838

View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=42847854838

[Ankur-singh]

/use 42847854838

[by Claude Code]

[Ankur-singh]

As a PR reviewer and CODEOWNER, I have reviewed this and have:

Checklist items

• Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
• Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
• Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
• Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
• Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
• Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison.
• For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
• Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
• Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
• Verified that new models disable integer all-reduce, with SGLang ROCm explicitly setting ROCM_QUICK_REDUCE_QUANTIZATION=NONE. Existing models are grandfathered per MODELS.md.
• For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
• If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
• For official InferenceX (non-preview), verified that established models use upstream engine images built from master (or the equivalent default development branch), with new models transitioning within a reasonable time after release. See MODELS.md.
• If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
• Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
  • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
• Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
• If this PR uses append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it.
• If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
• Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.

Additional detail section:

• Assessed head: 0211db699b83ffc9eaf212aee3b36c3cf0da1df1. The sweep ran on 75b1a3c7; the only later commit, 0211db69, changes the perf-changelog wording ("six frontier points" → "six points", since c1 is not on the frontier) — no config, recipe or matrix change; validate_perf_changelog passes. Config kimik3.1-fp4-b300-vllm-agentic-dspark, one B300 node, three recipes: agentic.yaml (c1, c6: TP8 x DCP8, FULL_DECODE_ONLY, DSpark 4), agentic-mixed-graphs.yaml (c12, c24: prefill/mixed graphs to 4096, DSpark 3, prefill-schedule-interval 8), agentic-ep.yaml (c48, c64: EP8 MegaMoE, DSpark 4, prefill-schedule-interval 8), all with vllm-simple host-DRAM KV offload.
• PR validation + evals: run 42847854838 (attempt 2, success). All six benchmark points and both eval jobs (c24, c64) succeed on their latest attempts with 0 error drops. c24 failed in attempt 1 on 10.025% aiperf client errors (261 ClientConnectorError, just over the 10% limit) and passed cleanly in attempt 2. /use 42847854838 is posted and acknowledged. gsm8k em_strict 0.979–0.981, within 2σ of the previous run for this key (run 42150239586, c70 0.9773). The eval server logs show the changed paths ran: c24 applies prefill_schedule_interval=8, max_cudagraph_capture_size 4096 and the latency-specialized decode; c64 runs deep_gemm_mega_moe with expert parallelism (DeepGEMM loaded) and TOKENSPEED_MLA. The c1/c6 variant has no eval of its own; its decode path is covered by the c24 and c64 evals.
• Team TTFT gate: p90 TTFT below 5 s on every point (max c64 3.13 s).
• Golden AL: thinking_on, block rejection: 4 draft tokens = 3.19 (c1, c6, c48, c64) and 3 draft tokens = 2.88 (c12, c24); job logs show rejection_sample_method: synthetic with those values. The report-only AL gate on real verification measured c24 2.887/2.883 vs 2.88 and c64 3.200/3.176 vs 3.19.
• Draft precision: the draft is Inferact/Kimi-K3.1-Pro-DSpark-preview @ 6a2ef20a (equal to HF main), stored BF16 with no quantization_config. In the pinned build the drafter's GEMMs use BF16 operands with FP32 accumulation on its unquantized BF16 linears; the draft KV cache is FP8, the same as the target (allowed); TOKENSPEED_MLA_FP16_PARTIALS=0 keeps partials in FP32. No draft quantization, dtype override or checkpoint substitution. SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is SGLang-only and absent.
• Upstream recipe: merged vllm-project/vllm#66433, published at recipes.vllm.ai/moonshotai/Kimi-K3.1-Pro as presets b300_agentx (c1, c6), b300_agentx_mixed_graphs (c12, c24) and b300_agentx_ep (c48, c64), pinned to the same image digest and source revision 75b1a3c7. They match every production setting per point (DCP8 a2a, fp8 KV, FP32 KDA state, use-replayssm, max-num-batched-tokens 16384, stream-interval 10, graph capture sizes, max-num-seqs, gpu-memory-utilization, prefill-schedule-interval, latency-specialized decode, EP8 MegaMoE, DSpark 4/3 at revision 6a2ef20a, Simple host-DRAM offload budgets and the VLLM_*/TOKENSPEED_* env). Excluded as InferenceX-only: PYTHONHASHSEED, PYTHONUNBUFFERED, served-model-name, synthetic acceptance and aiperf settings.
• Image: vllm/vllm-openai:kimi-k31-mc@sha256:f8199876…, upstream vllm org, pinned by digest. It was built from vLLM commit 78e9a0c28 (server log 0.1.dev24218+g78e9a0c28), a Kimi-K3.1 enablement-branch commit that is not yet on vLLM main. Kimi-K3.1-Pro is a new model (added to MODELS.md on 2026-11-04), so this uses the new-model transition allowance; the features it relies on (latency-specialized decode, deep_gemm_mega_moe, prefill_schedule_interval, use_replayssm, TOKENSPEED_MLA) are on vLLM main, and the config should move to a main-built image once one carries the K3.1 support. No setup script, patch or overlay.
• Not deprecated: kimik3.1 agentic coding is active in MODELS.md. No architecture overrides; new model on the CUDA path with no integer/quick all-reduce controls. Not append-only. Dynamo TRT-LLM metrics: N/A (plain vLLM).
• Pareto (run 42847854838, latest attempt per point, total tput/GPU vs p90 E2EL, one curve B300 vLLM kimik3.1 agentic): 5 of 6 points on the frontier (c1 is dominated by c12), also 5 at p75.
• Removed points: the 9 points of the previous run (run 42150239586) that are not re-measured are all dominated by head points. Host-DRAM budgets (2,249 GB and 2,999 GB per server) are within the 3 TB limit.
• Description: the PR body has been corrected to six points (c72 was dropped in ec76df1c). The PR conflicts with main in perf-changelog.yaml only.

Signed: Ankur-singh

[by Claude Code]

[github-actions]

✅✅✅ Verdict: PASS ✅✅✅

Passed and not applicable checks

✅ Check 0 (CODEOWNER): PASS — @Ankur-singh is a named owner of inferencex-e2e/configs/nvidia-master.yaml and the srt-slurm-recipes/**/b300-*/ recipes; perf-changelog.yaml falls under the catch-all.

✅ Check 1 (Passing sweep on an in-PR commit): PASS — in-PR commit 75b1a3c7 has all six agentic / jobs and both agentic eval / jobs green in run 42847854838 (attempt 2); the only later commit, 0211db69, rewords one perf-changelog.yaml description line ("six frontier points" to "six points"), which is non-material.

✅ Check 2 (Evals pass): PASS — gsm8k em_strict on vllm/vllm-openai:kimi-k31-mc@sha256:f8199876… scored c24 (TP8/DCP8) 0.9788/0.9803 and c64 (EP8 MegaMoE) 0.9810/0.9803, all within 2σ of each other and of the run 42150239586 cohort (0.9773). The eval job logs show --moe-backend deep_gemm_mega_moe, --prefill-schedule-interval 8 and decode_backend latency_specialized in the launch arguments, and both evals' real-verification AL gates passed. The runtime-dispatch evidence in the eval server-log tarballs is recorded in the sign-off; this verifier could not extract those tarballs.

✅ Check 3 (Upstream recipe): PASS — vllm-project/vllm#66433 was merged on 2026-11-11 and is published at recipes.vllm.ai/moonshotai/Kimi-K3.1-Pro. Its presets b300_agentx, b300_agentx_mixed_graphs and b300_agentx_ep match every production setting per point: DCP8 a2a, FP8 KV/MXFP4 indexer, FP32 KDA state, replayssm, EP8 MegaMoE, prefill-schedule-interval, graph sizes, offload budgets, VLLM_*/TOKENSPEED_* env and DSpark at revision 6a2ef20a. Exclusions are justified as InferenceX-only.

✅ Check 4 (Reuse command): PASS — /use 42847854838 was posted on its own line by Ankur-singh (permission write, role write); it names the Check 1 run.

✅ Check 5 (Latest checklist template): PASS — every item of the current PR_REVIEW_CHECKLIST.md template is present and checked.

✅ Check 6 (Upstream images / provenance / ordering): PASS — the image is from upstream vllm/vllm-openai and pinned by digest; this is a vLLM-only entry, so engine-first ordering does not apply. The image's source commit 78e9a0c28 is on a Kimi-K3.1 enablement branch (diverged from vLLM main). That is allowed under the new-model transition: kimik3.1 was added on 2026-11-04.

✅ Check 7 (No deprecated models/scenarios): PASS — kimik3.1 agentic coding is active in MODELS.md.

✅ Check 8 (No architecture hacks): PASS — no --hf-overrides or other model-config edits. kimik3.1 is not grandfathered, and on the CUDA vLLM path no integer or quick-reduce all-reduce controls are set (FlashInfer all-reduce stays BF16).

✅ Check 9 (Chat templates for spec decode): PASS — AgentX replays the semianalysis_cc_traces_weka_062126 traces through aiperf's chat endpoint.

✅ Check 10 (No engine patches): PASS — the three recipes and their launch path have no setup script, patch, overlay or engine reinstall.

✅ Check 11 (Golden AL for agentic spec decode): PASS — the job logs show rejection_sample_method: synthetic, with synthetic_acceptance_length 3.19 for 4 draft tokens (c1, c6, c48, c64) and 2.88 for 3 draft tokens (c12, c24). Both match the kimi-k3.1-pro__dspark__kimi-k3.1-pro-dspark-preview__probabilistic-block golden curve for thinking_on with block verification.

➖ Check 12 (Append-only): N/A — no append-only: true entry.

✅ Check 13 (Draft precision): PASS — the draft Inferact/Kimi-K3.1-Pro-DSpark-preview at revision 6a2ef20a is stored as torch_dtype: bfloat16 with no quantization_config. No draft quantization, dtype override or checkpoint substitution is applied. The draft's FP8 KV cache matches the target's, which is allowed. SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is absent (vLLM path).

✅ Check 14 (Pareto coverage): PASS — one curve (B300 vLLM kimik3.1 agentic, p90 E2EL vs total tput/GPU) from run 42847854838's latest-attempt artifacts. It has 5 of 6 points on the frontier: c6, c12, c24, c48 and c64 (c1 is dominated by c12). Counted by hand from the artifacts because the repo helper could not read its input file in this sandbox; the sign-off reports the same 5.

✅ Check 15 (PR description): PASS — the body's six points, per-recipe layouts, prefill-schedule-interval scope, image and upstream recipe link all match the pinned code.

✅ Check 16 (NVIDIA overview): PASS — Kimi-K3.1-Pro has a populated vLLM cell in the Framework Recipes table of the NVIDIA overview.

➖ Check 17 (Metrics publication): N/A — plain vLLM aggregated serving; no Dynamo TRT-LLM path.

✅ Check 18 (Recipe/image/sweep consistency): PASS — master TP/EP/DCP and concurrency lists match the recipe overrides. Host DRAM: 0.75 gives 2249 GB, which equals 8 × 281.125 GB; 1.0 gives 2999 GB, which equals 8 × 374.875 GB; both are within the 3 TB limit. The image resolves. The changelog is accurate. All 11 removed points are dominated. Re-tuned deltas: c1 +7.1% tput/+7.3% intvty, c24 +25.1%/+57.2%. Stream interval is 10 in every recipe.

Assessed commit: 0211db699b83ffc9eaf212aee3b36c3cf0da1df1.

[Ankur-singh]

/use 42847854838

[github-actions]

@Ankur-singh staged run 42847854838.

• Kimi-K3.1-Pro · agentic-traces · FP4 · b300_vllm · 2026-11-10

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

[github-actions]

Ingest report — ✅ Results ingested into the production database.

Ingest workflow · Merge Ingest run · Benchmark source run

Category	new	duplicate	skipped	unmapped	failed
benchmark	6	0	0	0	0
eval	2	6	4	0	0
trace	6	0	0	0	0
power	6	0	0	0	0

_{benchmarkFiles 6, rawBenchmarkRows 6, coveredByPerJobFiles 0, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 2638, gpuMetricSamplesNew 301032, powerPublicationPoints 6, changelogEntriesWritten 1}

Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun.

中文

Ingest 报告 — ✅ 结果已写入 production 数据库。

上表仅统计本次运行写入的数据，各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。