Repository navigation
Conversation
Single-node TP8 x DCP8 with the Inferact DSpark draft and block verification, following vLLM's MI355X K3.1 recipe without its three reduced-precision knobs. GPU-resident KV at conc 1 and 4; Simple CPU KV offload at conc 4-60.
Write the per-concurrency points as plain override variants instead of zip overrides.
# Conflicts: # inferencex-e2e/perf-changelog.yaml
The image keeps HSA_NO_SCRATCH_RECLAIM=1, which the cluster's MEC firmware 38 needs, so large FP4-indexer prefill scratch stays pinned outside vLLM's memory budget. At 0.92 the DRAM-offload points ran out of GPU memory in long-context prefill (HSA_STATUS_ERROR_OUT_OF_RESOURCES). Use 0.90, as the Kimi K3 MI355X recipe did on the same nodes.
The MI355X AgentX lane and the reference apex-1104 sweep differed in the points they ran and in batching, so a comparison between them could not attribute anything to the engine build. Align the parts that shape the workload: seven points instead of ten (drop the GPU-resident conc 4 and the conc 52/60 tail), max-num-seqs below concurrency as that sweep sets it, max-num-batched-tokens 16384 except 10240 at conc 48, its cudagraph capture sizes, and 2577 GB of host DRAM for the Simple CPU offload. Keep this lane's own precision and speculative settings: 5 DSpark tokens with block rejection at every point, the published Inferact/Kimi-K3.1-Pro-DSpark-preview draft, FP32 KDA recurrent state, BF16 all-reduce, BF16 MoE stage 2, the AITER FA MLA prefill backend, and no FP8 indexer override or KDA input-projection quantization. Those are deliberate here, and copying them would change what the runs measure -- synthetic acceptance in particular reports a modelled throughput and makes the output nondeterministic by construction. The dram-utilization comment claimed 0.6 resolves to 1799 GB. The nodes report 3000 GB, so the pinned 2577 GB is 86% of host DRAM, not 60% -- the recipe's cpu_bytes_to_use_per_rank wins. Say so instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The planner derives TOTAL_CPU_DRAM_GB from dram-utilization and rejected the recipe's 2577 GB against the 1799 GB that 0.6 resolves to. 0.8592 resolves to 2577 GB, within the 3 TB per-server limit. Align comments with the settings they describe and correct the changelog: throughput points still run synthetic acceptance at the golden 3.08.
The conc-1 latency point runs the reference sweep's latency_specialized decode backend with 7 draft tokens; the throughput points stay at 5. K3.1 has no torch.compile submodule, so FULL_AND_PIECEWISE captures decode only and breakable graphs are the only thing that would add the prefill sizes. Turn them off and leave scratch reclaim on, as the image sets it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The apex-1104 sweep sets mla_prefill_backend on its conc-48 point and leaves conc 1-32 on the default. Move it out of base into that point's override. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The apex-1104 sweep turns both off at conc 1 and on at conc 4-48. Move the conc-1 values into that point's override and leave the base on the sweep's throughput setting. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
# Conflicts: # inferencex-e2e/perf-changelog.yaml
On this cluster's MEC firmware 38, HSA_NO_SCRATCH_RECLAIM=0 hung RCCL allgathers at 445k-700k context on conc 32 and conc 48, and conc 48 ran out of GPU memory at 0.92. Keep the image's reclaim-off default with 0.90 at conc 4-48 and no async scratch limit there; conc 1 keeps its own settings.
Run 41954410041's conc-48 point died on a 600 s RCCL allgather timeout with no successful request. Cut the per-step working set at the two largest points: max-num-batched-tokens 10240, gpu-memory-utilization 0.91 and 3 DSpark tokens, with the capture lists rebuilt from those. The other five points keep the reference apex-1104 sweep's settings. Also run MoE stage 2 in MXFP8 as that sweep does, and leave the MLA prefill backend at its default everywhere. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MEC firmware 38 on this cluster cannot reclaim RCCL scratch, and the release blocks until the running dispatches finish, which is how the conc-16 and conc-48 points hung on a 600 s allgather. Set HSA_NO_SCRATCH_RECLAIM=1 at every point and drop HSA_SCRATCH_SINGLE_LIMIT_ASYNC, which only bounds a reclaim that no longer happens. Step the DSpark token count down with concurrency, 7/5/4/3, and rebuild each capture list from it. Size the CPU offload at 224.875 GB per rank, the 1799 GB that dram-utilization 0.6 resolves to. Describe the points on their own terms rather than against another sweep. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…hput servers Synthetic acceptance streams unverified draft tokens. The kimi_k3 tool and reasoning parsers stop streaming on some of them, which empties or truncates the response the client sees. Evals and acceptance gates verify for real and keep their parsers.
…nc 16-32 With HSA_NO_SCRATCH_RECLAIM=1 the kernel scratch of these points filled the VRAM left outside the 0.90-0.91 budget during warmup and stalled RCCL, even on freshly rebooted nodes. Their GPU KV use peaks well under the smaller pool, and DRAM offload backs the rest.
… huge pages THP_MEM_ALLOC_ENABLE=1 is what makes torch's CPU allocator call madvise(MADV_HUGEPAGE). The hosts run THP in madvise mode, so without it the 224 GB per-rank offload pool is backed entirely by 4 KiB pages: about 55 million of them to register and keep mapped, against 107 thousand 2 MiB pages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…2, capture cap 8192 Shrink the Simple CPU offload pool to 2700 blocks per TP rank (cpu_bytes_to_use_per_rank 112251571200, dram-utilization 0.2994 -> 898 GB) to cut the per-GPU VRAM the pinned pool's page tables take, keep THP_MEM_ALLOC_ENABLE=1, and lower max-num-batched-tokens and the cudagraph capture cap to 8192 at conc 4-48. Conc 1 is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the contribution!
中文感谢你的贡献！
|
bae55b0 to
f7d2b0e
Compare
Raise num_speculative_tokens from 3 to 4 at conc 32 and 48 and rebuild their capture lists on multiples of 5 (up to 120 and 240) below the 8192 prefill ladder. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
seungrokj
added
the
full-sweep-enabled
|
Claude finished @seungrokj's task in 33s —— View job Review summary
🔴 1 blocking issue: the new I found no other issues:
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| - "Add Kimi-K3.1-Pro day-0 vLLM AgentX on B300: TP8 x DCP8 with DSpark (Inferact/Kimi-K3.1-Pro-DSpark-preview) and Simple CPU KV offload, conc 1-70." | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4796 | ||
|
|
||
| - config-keys: |
There was a problem hiding this comment.
🔴 BLOCKING: Add new inferencex-e2e/perf-changelog.yaml entries at the END of the file. This entry is at line ~12038, but the file now has 12,304 lines. The merge from main added newer entries below it (#4833, #4852, #4857, #4799, #4720).
Why it matters: The file is read in chronological order, oldest at the top and newest at the bottom. An entry inserted in the middle breaks that order and may not be picked up as this PR's appended entry when sweeps are triggered.
Fix: Move this 9-line block (- config-keys: kimik3.1-fp4-mi355x-vllm-agentic-dspark … pr-link: .../pull/4867) from here to the end of the file, after the #4720 entry.
seungrokj
added
non-canary-full-sweep-enabled
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=42221940480 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=42221940480 |
| - config-keys: | ||
| - kimik3.1-fp4-mi355x-vllm-agentic-dspark | ||
| description: | ||
| - "Shrink the Simple CPU offload to 2700 blocks per TP rank, 112,251,571,200 bytes (112.25 GB) per rank and 898 GB across the 8 ranks, via dram-utilization 0.2994. The image registers the pool with cudaHostRegister, which costs VRAM for page tables on every GPU in proportion to its size, so a smaller pool leaves more of each GPU outside the gpu-memory-utilization budget. A replay of the conc-48 trace through a model of the offload puts the loss at about half a point of overall prefix-cache hit rate against the 1799 GB pool; the model is optimistic, and the hit rate falls steeply below about 2150 blocks." | ||
| - "Keep THP_MEM_ALLOC_ENABLE=1 so the pool is backed by transparent huge pages." | ||
| - "Lower max-num-batched-tokens to 8192 and cap the cudagraph capture list at 8192 (max_cudagraph_capture_size 8192) at conc 4-48. Conc 1 keeps max-num-batched-tokens 16384 and its decode-only capture list." | ||
| - "Raise the DSpark token count at conc 32-48 from 3 to 4, matching conc 16-24, so those points get the 4-token golden acceptance length; their decode capture sizes become the multiples of 5 up to max-num-seqs of them (120 at conc 32, 240 at conc 48), followed by the prefill ladder." | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4867 |
There was a problem hiding this comment.
🔴 The changelog tooling and later PRs get an entry inserted mid-file rather than at the tail, which breaks the append-only changelog contract. The new block sits at line 12038, ahead of the existing kimik3.1-fp4-mi355x-sglang-agentic-dflash entry (#4808) and every entry after it. Fix: move the whole #4867 block to the physical end of perf-changelog.yaml, leaving every existing byte and separator unchanged.
Why this was flagged
The diff adds the kimik3.1-fp4-mi355x-vllm-agentic-dspark block at perf-changelog.yaml:12038-12045. Existing entries follow it, starting with the #4808 entry at line 12047, then #4801, #4722, #4804 and more. CONTRIBUTING.md:30 says a PR must append its entry to the physical end of the file. AGENTS.md:88 says the file is append-only and byte-sensitive, and entries are appended only at the tail. Changelog-diff tooling that reads only the new tail, and the conflict helper, will not see this entry where they expect it. On the base branch these entries are contiguous.
Verification: The trigger is any run of the changelog validation against base. The diff hunk @@ -12035,6 +12035,15 @@ puts the new #4867 block at inferencex-e2e/perf-changelog.yaml:12038-12045. That is mid-file. compare_entries (validate_perf_changelog.py:129-138) compares base entries to head entries index by index. So changelog validation fails in the sweep workflow.
| AIPERF_HTTP_TCP_USER_TIMEOUT: '900000' | ||
| AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'vllm:' | ||
|
|
||
| # max-num-seqs stays below CONC from c4 to c32, and equals it at c1 and c48. The capture list is |
There was a problem hiding this comment.
🟡 nit (optional): The block comment above the overrides is a section header describing them, which AGENTS.md:56 forbids. Fix: drop it, or keep only a short line-level note at a specific value, such as why max-num-seqs is below CONC.
Why this was flagged
agentic.yaml:99-102 is a four-line comment above override_gpu_kv_c1. It describes every override as a group: max-num-seqs relative to CONC, how the capture list is built, and golden-AL replacement in throughput runs. AGENTS.md:56 says no header comments describing a file, topology, config, recipe, override or section. The comment mostly restates values the overrides already spell out. This is a convention issue with no runtime effect.
Verification: In inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/kimik3.1/vllm/mi355x-fp4-mtp/agentic.yaml, lines 96-99 are a four-line block comment that sits above override_gpu_kv_c1: and is not attached to any single value. AGENTS.md:56 says: "No header comments describing a file, topology, config, recipe, override, or section." The file is new in this PR, so the base does not have this violation.
Builds on #4806 (branched from its head 740ee08) and changes only the Kimi-K3.1-Pro MI355X vLLM AgentX point. The diff against main therefore also shows #4806's changes until that PR merges; the changes made here are the last two commits (f7d2b0e, 2e8d1d4).
Changes (conc 4–48; conc 1 unchanged)
cpu_bytes_to_use_per_rankdram-utilization→TOTAL_CPU_DRAM_GBTHP_MEM_ALLOC_ENABLEmax-num-batched-tokensmax_cudagraph_capture_size/ capture listnum_speculative_tokensat conc 32–48Apart from the token count at conc 32–48, the speculative settings are unchanged. CI still replaces block rejection with the golden AL for each point's token count.
Why
The Simple CPU offload registers its host pool with
cudaHostRegister. That costs VRAM for page tables on every GPU, outside thegpu-memory-utilizationbudget, and the cost grows with the pool size. A replay of the conc-48 trace through a model of the offload loses about 0.5 pp of overall prefix-cache hit rate at 2700 blocks per rank, compared with the 1799 GB pool. The model is optimistic, and the hit rate falls steeply below about 2150 blocks.Validation
python -m infx.matrix.generate test-config --config-keys kimik3.1-fp4-mi355x-vllm-agentic-dsparkresolvestotal-cpu-dram-gbto 898 at conc 4–48 and to 0 at conc 1.pytest infx/tests/srt_slurm infx/tests/matrixgives the same result as [K3.1-Pro] Day-0 support vLLM on MI355X / [K3.1-Pro] MI355X 上 vLLM 的 Day-0 支持 #4806's head. The 27 failures and 3 errors on both come fromsrtctlnot being installed locally.🤖 Generated with Claude Code