Skip to content

[K3.1-Pro] Re-sweep vLLM on GB200 with the updated DSpark draft - #4864

Merged
cquil11 merged 4 commits into
mainfrom
cam/kimik3.1-gb200-vllm-new-draft
Nov 8, 2026
Merged

cquil11 merged 4 commits into
mainfrom
cam/kimik3.1-gb200-vllm-new-draft

Conversation

@cquil11

@cquil11 cquil11 commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

Re-sweeps Kimi-K3.1-Pro GB200 vLLM AgentX, both the aggregated and disaggregated rows from #4799, with the updated DSpark draft.

@cquil11
cquil11 requested a review from a team November 6, 2026 10:31
@cquil11 cquil11 added the non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim) label Nov 6, 2026
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目,并且恰好带有一个主标签时,才会运行扫描:full-sweep-fail-fast(强烈推荐;canary 加逐矩阵 fail-fast)、full-sweep-enabled(有 canary;矩阵任务在失败后继续运行)或 non-canary-full-sweep-enabled(无 canary,也无 fail-fast)。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Besides the inline finding, I checked whether the disaggregated row also needs the keep-alive setting. It doesn't: disagg-variants.yaml:48 already sets VLLM_HTTP_TIMEOUT_KEEP_ALIVE: "600", so this change brings the aggregated recipe in line with it.

Extended reasoning...

The keep-alive env var is the only change to the disaggregated side that I examined and ruled out. The disaggregated recipe already had the same value before this PR.

Comment on lines +12289 to +12295
- config-keys:
- kimik3.1-fp4-gb200-vllm-agentic-dspark-tp8pp2-agg
- kimik3.1-fp4-gb200-vllm-agentic-dspark-tp16-disagg
scenario-type:
- agentic-coding
description:
- "Re-sweep Kimi-K3.1-Pro GB200 vLLM AgentX with the updated DSpark draft (Inferact/Kimi-K3.1-Pro-DSpark-preview @ 6a2ef20a) and the #4851 golden curve, so throughput forces AL 3.41 at 5 draft tokens and 3.19 at 4."

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) The changelog says this sweep uses draft Inferact/Kimi-K3.1-Pro-DSpark-preview @ 6a2ef20a, but no recipe pins that revision. Both rows load whatever the Hub default branch serves. The vLLM speculative-config JSON in agg-tp8pp2-variants.yaml:89-137 and disagg-variants.yaml:79,107 has no revision key. So the recorded points are tied to an unpinned draft, and nothing in either config key actually changes for this re-sweep. Fix: add "revision":"<full 6a2ef20a sha>" to every DSpark speculative-config in both GB200 recipes. This matches the speculative-draft-model-revision pin in the sglang K3.1 recipes.

Why this was flagged

Both listed keys start vLLM with speculative-config {"model":"Inferact/Kimi-K3.1-Pro-DSpark-preview",...} and no revision (agg-tp8pp2-variants.yaml:89-137, disagg-variants.yaml:79,107). The draft that actually loads is whatever the nodes' HF cache or the Hub default branch gives at run time. Per the PR description, that already changed once without any config change. If Inferact pushes again, or a node keeps a stale cache, the results under this changelog entry and the 3.41/3.19 golden-curve forcing describe a different draft than the one that ran. The mismatch is silent. The base branch has the same unpinned load, but this entry is the first to claim a specific revision 6a2ef20a. The sglang K3.1 recipes do pin their draft with speculative-draft-model-revision (e.g. sglang/gb300-fp4/agentx/disagg-tp16-variants.yaml:110), so the vLLM ones are the odd ones out.

Verification: The new changelog block (perf-changelog.yaml:12295) says the re-sweep uses "Inferact/Kimi-K3.1-Pro-DSpark-preview @ 6a2ef20a", but neither recipe pins that revision. Every DSpark speculative-config in agg-tp8pp2-variants.yaml and disagg-variants.yaml has no revision key. The unpinned config is pre-existing in the recipes.

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

@cquil11

cquil11 commented Nov 8, 2026

Copy link
Copy Markdown
Collaborator Author

/use 42203512960

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

@cquil11 staged run 42203512960.

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

@cquil11

cquil11 commented Nov 8, 2026

Copy link
Copy Markdown
Collaborator Author

/reuse-sweep-run 42203512960

# Conflicts:
#	inferencex-e2e/perf-changelog.yaml
@cquil11
cquil11 merged commit d211dba into main Nov 8, 2026
28 checks passed
@cquil11
cquil11 deleted the cam/kimik3.1-gb200-vllm-new-draft branch November 8, 2026 21:47
@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Ingest report — ✅ Results ingested into the production database.

Ingest workflow · Merge Ingest run · Benchmark source run

Category new duplicate skipped unmapped failed
benchmark 10 0 0 0 0
eval 7 7 0 0 0
trace 10 0 0 0 0
power 52 0 0 0 0

benchmarkFiles 10, rawBenchmarkRows 10, coveredByPerJobFiles 0, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 9233, gpuMetricSamplesNew 1532268, powerPublicationPoints 10, changelogEntriesWritten 1

Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun.

中文

Ingest 报告 — ✅ 结果已写入 production 数据库。

上表仅统计本次运行写入的数据,各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant