Repository navigation
Conversation
|
Thanks for the contribution!
中文感谢你的贡献！
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| description: | ||
| - "Add Kimi K3.1 Pro day-0 GB300 Dynamo SGLang AgentX on lmsysorg/sglang:dev-kimi-k31-pro with the RadixArk DFLASH draft and block verification." | ||
| - "Aggregated TP8 DCP8 at concurrency 1; disaggregated TP8 DCP8 prefill with HiCache and TP8 DCP8 decode (1P3D, 1P2D, 2P3D, 3P4D) plus 3P1D with a TP16 DCP8 attention-DP2 EP16 decode." | ||
| pr-link: TBD |
There was a problem hiding this comment.
🔴 The changelog check will fail on this PR, so CI stays red and the merge-prep step cannot rewrite the link. The new entry uses pr-link: TBD, which is not one of the accepted placeholders. validate_added_pr_link accepts only XXX, https://github.com/SemiAnalysisAI/InferenceX/pull/XXX, or this PR's own URL. Fix: set pr-link to https://github.com/SemiAnalysisAI/InferenceX/pull/XXX or to this PR's canonical URL.
Why this was flagged
The appended block at inferencex-e2e/perf-changelog.yaml:12016-12025 ends with pr-link: TBD. On a PR run, inferencex-e2e/infx/workflows/validate_perf_changelog.py:113 checks the link against PR_LINK_PLACEHOLDERS (lines 21-24: XXX and https://github.com/SemiAnalysisAI/InferenceX/pull/XXX) and against the expected PR URL. TBD matches neither, so it raises "new PR entry must use ... or an XXX placeholder; found 'TBD'". inferencex-e2e/infx/workflows/prepare_perf_changelog_merge.py:108-111 rejects the entry the same way ("appended entry N has unexpected pr-link 'TBD'"), so the merge-time link rewrite cannot run either. The changelog entry is what triggers result collection for the three new config keys, so they get no results until the link is fixed.
Verification: Trigger: any PR CI run of the changelog validator on this PR. The PR's own appended entry at inferencex-e2e/perf-changelog.yaml:12025 is pr-link: TBD. validate_added_pr_link accepts a link only if it is in PR_LINK_PLACEHOLDERS or equals https://github.com/SemiAnalysisAI/InferenceX/pull/{pr_number}. The merge-prep step therefore cannot rewrite the link to the canonical URL.
| ep: 4 | ||
| dp-attn: false | ||
|
|
||
| kimik3.1-fp4-gb300-dynamo-sglang-agentic-dflash-agg: |
There was a problem hiding this comment.
🔴 If this merges as is, every point of the three new kimik3.1 DFLASH config keys fails, because no matching golden AL curve exists in the tree. infx/golden_al_distribution/ has only the DSpark curve, and identities.yaml has no DFLASH identity. The perf-changelog entry still triggers sweeps for these keys. Fix: land the kimi-k3.1-pro__dflash__kimi-k3.1-dflash2-preview__default-block curve and its identity in this PR, or gate the merge on #4798, before keeping the changelog entry that schedules these points.
Why this was flagged
Every search-space row of kimik3.1-fp4-gb300-dynamo-sglang-agentic-dflash-agg, -disagg and -wideep-disagg (nvidia-master.yaml:6533-6677) sets spec-decoding: draft_model with the RadixArk DFLASH draft. infx/golden_al_distribution/ contains only kimi-k3.1-pro__dspark__kimi-k3.1-pro-dspark-preview__probabilistic-block.yaml. identities.yaml:105 lists only that DSpark identity. The PR description says the srt driver fails a throughput point with no matching curve. The changelog block appended at perf-changelog.yaml:12016 schedules collection for all three keys, so each dispatched point fails. The description notes the dependency on #4798, but nothing in this diff stops the entry from scheduling these points before the curve lands.
Verification: Trigger: this PR merges while its own commits carry no DFLASH curve, i.e. before #4798 lands. Its appended perf-changelog block then schedules the three keys and every speculative AgentX throughput point fails before it is submitted. In identity.py:216-260, select_curve raises ValueError("No golden curve for model prefix 'kimik3.1' and method 'dflash'").
| DRAFT_MODEL="RadixArk/Kimi-K3.1-Dflash2-Preview" | ||
| MODEL_KEY="$(basename "$SERVE_MODEL" | tr '[:upper:]' '[:lower:]')" | ||
| # The draft card fits 32 running requests at verification width 8 on B300; AL does not | ||
| # depend on batch size. | ||
| CONCURRENCY="32" | ||
| SAMPLING='{"temperature": 1.0, "top_p": 0.95, "repetition_penalty": 1.0}' | ||
|
|
||
| SPEEDBENCH_DIR="/workspace/speed_bench_data" | ||
| RESULTS_DIR="/workspace/speedbench_results" | ||
| mkdir -p "$RESULTS_DIR" | ||
|
|
||
| nvidia-smi | ||
| GPU_NAME="$(nvidia-smi --query-gpu=name --format=csv,noheader | head -1)" | ||
|
|
||
| # Pyxis containers share the host network, so a stray listener on the node fails every cell's | ||
| # bind; stop before the downloads and name the holder. | ||
| if [[ -n "$(ss -Hltn "sport = :${PORT}" 2>/dev/null)" ]] \ | ||
| || (exec 3<>"/dev/tcp/127.0.0.1/${PORT}") 2>/dev/null; then | ||
| echo "CRITICAL: port ${PORT} on $(hostname) is already in use by a process outside this job." | ||
| ss -ltnp "sport = :${PORT}" 2>&1 || true | ||
| exit 1 | ||
| fi | ||
|
|
||
| # An unstaged checkpoint lands in the cluster's writable download root; hf download resumes and | ||
| # verifies a partial copy, which a non-empty-directory check would accept as complete. | ||
| if [[ ! -e "$MODEL_PATH" || -w "$MODEL_PATH" ]]; then | ||
| hf download "$MODEL" --local-dir "$MODEL_PATH" | ||
| fi | ||
| hf download "$DRAFT_MODEL" |
There was a problem hiding this comment.
🔴 The golden AL curve from this collector may be measured on a different draft than the one the benchmarks serve. The recipes pin speculative-draft-model-revision: d0d8bc2e3c6d0a195d2e00a49017e71ca8d881f1. This script downloads and serves RadixArk/Kimi-K3.1-Dflash2-Preview at whatever its default branch holds. Fix: pin the same revision in one place in the script and pass it to both hf download --revision and --speculative-draft-model-revision, so the collected AL matches the draft the three configs run.
Why this was flagged
The trigger is the RadixArk preview repo publishing new draft weights on its default branch after the recipes pinned d0d8bc2. kimik3.1_fp4_b300_sglang.sh:45 runs hf download "$DRAFT_MODEL" without --revision, and the serve_args near line 154 pass --speculative-draft-model-path "$DRAFT_MODEL" without --speculative-draft-model-revision. So the collector measures acceptance length on the newest draft. All three recipes (agg-tp8-dcp8-dflash7.yaml, disagg-variants.yaml, disagg-3p1d-p-tp8-d-tp16-dpa2-ep16.yaml) set speculative-draft-model-revision: d0d8bc2e3c6d0a195d2e00a49017e71ca8d881f1. The srt driver injects that golden AL into the throughput runs, so reported spec-decode results would rest on an AL that does not match the pinned draft. The output YAML header records only the draft name and no revision, so the mismatch would not show up afterwards. Nothing in the script checks that the two drafts are the same.
Verification: In kimik3.1_fp4_b300_sglang.sh, the serve_args, lines 157-162, pass --speculative-draft-model-path "$DRAFT_MODEL" with no --speculative-draft-model-revision. If the draft moves, the injected acceptance lengths come from different weights than the pinned draft the benchmarks serve, so throughput results are silently wrong. The PR adds this script, so the base branch does not have this gap.
9c1f18e to
0509883
Compare
Aggregated TP8 DCP8 at c1, TP8 DCP8 1P3D/1P2D/2P3D/3P4D disaggregation with prefill HiCache, and 3P1D with a TP16 DCP8 attention-DP2 EP16 decode, all on the RadixArk DFLASH draft with block verification.
0509883 to
42eb420
Compare
cquil11
added
the
non-canary-full-sweep-enabled
Published Dynamo wheels cannot load the kimi_k31 tokenizer and their SGLang argument check rejects DCP decode for K3.1's DSA layers. ai-dynamo/dynamo#18124 fixes both; pin it and record it as the router version.
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=41976585823 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=41976585823 |
Dynamo #18124 does not build, so the GB300 K3.1 SGLang workers run as sglang.launch_server: the disaggregated points behind the SGLang Model Gateway (cache-aware, one gateway) and the aggregated point as a direct server. Engine args, topologies and concurrencies are unchanged; Dynamo-only worker flags give way to SGLang's own kimi_k3 parsers, and the config keys drop dynamo from their names.
Port Po-Han Huang's GB300 Dynamo+SGLang DFLASH disaggregated points from #4811: 3P1D c300, 2P1D c180 and 1P1D c76 with a TP16 (DCP8 x attention DP2, EP16 MegaMoE) decode, and 1P3D c60, 1P3D c36 and 1P4D c32 with TP8 DCP8 decodes, served by the Dynamo 1.6.0 the image ships with its KV router and the SGLang chat processor. They replace the SGLang-router disaggregated and wide-EP points; the aggregated c1 point stays a direct SGLang server. Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
The SGLang day-0 container already has the Dynamo fix. and the cookbook commands are also with Dynamo: https://docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-K3.1-Pro?ax_hw=gb300&ax_ckpt=kimi-k3.1-pro&ax_cfg=disagg-2p1d-p-tp8-dcp8-ep2-2n-d-tp16-dcp8-dpa2-ep16-4n&ax_c=160&ax_kv=hicache&ax_router=dynamo&ax_tab=env#agentic-long-context-deployment We should use dynamo router. Or the routing will be bad, leading to bad perf |
|
Closing in favour of #4811 for the Kimi K3.1 GB300 SGLang submission. |
Day-0 Kimi K3.1 Pro AgentX on GB300 with SGLang and the RadixArk DFLASH drafter (
RadixArk/Kimi-K3.1-Dflash2-Preview, pinned to revisiond0d8bc2).The disaggregated points are Po-Han Huang's (NVIDIA) from #4811, moved here. They run Dynamo with its KV router and the SGLang chat processor, using the Dynamo 1.6.0 that
lmsysorg/sglang:dev-kimi-k31-proships (dynamo.install: false). That build accepts DCP decode for K3.1's DSA layers. The aggregated point is a direct SGLang server.kimik3.1-fp4-gb300-sglang-agentic-dflash-aggkimik3.1-fp4-gb300-dynamo-sglang-agentic-dflash-disagg--speculative-use-block-verificationon the verifying roles, which matches the curve's block verifier.--speculative-num-draft-tokensis the verification width, the drafts plus one, and both PD roles draft at the same width. The driver injects the golden AL; nothing is hard-coded, and eval-only runs use real block verification.dram-utilization: 0.80). That is inside the MODELS.md rule for a role using all four GPUs of a tray.K3.1 week-1 precision policy:
fp8_e4m3. KDA state FP32 (--mamba-ssm-dtype float32on the aggregated server; SGLang's float32 default and the ReplaySSM decode roles on the disaggregated ones). Conv state at the engine default (BF16).--enable-symm-mem, which deadlocks with it.Risks:
lmsysorg/sglang:dev-kimi-k31-prohas not been checked against these commands. The SGLang cookbook's GB300 snapshot ran a custom image with a pinned FlashInfer autotune cache that is not published.