Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ base:
env:
VLLM_ENGINE_READY_TIMEOUT_S: "3600"
VLLM_RPC_TIMEOUT: "600000"
# AIPerf reuses a session's connection across turns; outlast uvicorn's 5 s idle timeout.
VLLM_HTTP_TIMEOUT_KEEP_ALIVE: "600"
NCCL_CUMEM_ENABLE: "1"
NCCL_MNNVL_ENABLE: "1"
NCCL_NVLS_ENABLE: "1"
Expand Down
10 changes: 10 additions & 0 deletions inferencex-e2e/perf-changelog.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12471,3 +12471,13 @@
description:
- "The H3 client polls job status every 0.1 s instead of every 0.5 s. A job counts as done only at the next poll, so each published latency carried up to 0.5 s of polling delay; it now carries up to 0.1 s."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4904

- config-keys:
- kimik3.1-fp4-gb200-vllm-agentic-dspark-tp8pp2-agg
- kimik3.1-fp4-gb200-vllm-agentic-dspark-tp16-disagg
scenario-type:
- agentic-coding
description:
- "Re-sweep Kimi-K3.1-Pro GB200 vLLM AgentX with the updated DSpark draft (Inferact/Kimi-K3.1-Pro-DSpark-preview @ 6a2ef20a) and the #4851 golden curve, so throughput forces AL 3.41 at 5 draft tokens and 3.19 at 4."
Comment on lines +12475 to +12481

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) The changelog says this sweep uses draft Inferact/Kimi-K3.1-Pro-DSpark-preview @ 6a2ef20a, but no recipe pins that revision. Both rows load whatever the Hub default branch serves. The vLLM speculative-config JSON in agg-tp8pp2-variants.yaml:89-137 and disagg-variants.yaml:79,107 has no revision key. So the recorded points are tied to an unpinned draft, and nothing in either config key actually changes for this re-sweep. Fix: add "revision":"<full 6a2ef20a sha>" to every DSpark speculative-config in both GB200 recipes. This matches the speculative-draft-model-revision pin in the sglang K3.1 recipes.

Why this was flagged

Both listed keys start vLLM with speculative-config {"model":"Inferact/Kimi-K3.1-Pro-DSpark-preview",...} and no revision (agg-tp8pp2-variants.yaml:89-137, disagg-variants.yaml:79,107). The draft that actually loads is whatever the nodes' HF cache or the Hub default branch gives at run time. Per the PR description, that already changed once without any config change. If Inferact pushes again, or a node keeps a stale cache, the results under this changelog entry and the 3.41/3.19 golden-curve forcing describe a different draft than the one that ran. The mismatch is silent. The base branch has the same unpinned load, but this entry is the first to claim a specific revision 6a2ef20a. The sglang K3.1 recipes do pin their draft with speculative-draft-model-revision (e.g. sglang/gb300-fp4/agentx/disagg-tp16-variants.yaml:110), so the vLLM ones are the odd ones out.

Verification: The new changelog block (perf-changelog.yaml:12295) says the re-sweep uses "Inferact/Kimi-K3.1-Pro-DSpark-preview @ 6a2ef20a", but neither recipe pins that revision. Every DSpark speculative-config in agg-tp8pp2-variants.yaml and disagg-variants.yaml has no revision key. The unpinned config is pre-existing in the recipes.

- "Set VLLM_HTTP_TIMEOUT_KEEP_ALIVE to 600 s on the aggregated server so AIPerf's reused session connections outlive uvicorn's 5 s idle timeout."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4864
Loading