Repository navigation
[K3.1-Pro] Day-0 support SGLang on MI355X - #4808
Conversation
TP8 on the decode monokernel at c1 and c4 without KV offload, and TP8 DCP8 with HiCache from c4 to c60, both with the RadixArk DFLASH draft, following the SGLang cookbook's MI355X AgentX recipes.
|
Thanks for the contribution!
中文感谢你的贡献！
|
cquil11
added
the
non-canary-full-sweep-enabled
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| AIPERF_HTTP_TCP_USER_TIMEOUT: '900000' | ||
| AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'sglang:' | ||
|
|
||
| # The decode monokernel takes no KV offload; admission and decode graphs follow the agent count. |
There was a problem hiding this comment.
🟡 nit (optional): The section comment above override_mono_c1 says admission and decode graphs follow the agent count. That is not true for override_hicache_c60, which sets 64 at CONC 60. AGENTS.md also bans header comments that describe an override or section. Fix: drop the clause "admission and decode graphs follow the agent count". If the 64 at c60 is intentional, add a one-line why on override_hicache_c60's max-running-requests (for example, that it matches the cookbook's 64 at 56).
Why this was flagged
The comment at agentic.yaml:70 sits above the overrides and says "admission and decode graphs follow the agent count". Every override matches that except override_hicache_c60. There, max-running-requests and cuda-graph-max-bs-decode are 64 while benchmark.env CONC is '60'. A maintainer who trusts the comment may "fix" c60 down to 60, or copy the wrong rule into later recipes. AGENTS.md:56 forbids header comments that describe a recipe, override or section, and AGENTS.md:58 allows only short why-comments on specific lines. This is documentation only, so runtime behaviour is unaffected.
Verification: agentic.yaml:70 is a section header placed above override_mono_c1. override_hicache_c60 sets max-running-requests: 64 and cuda-graph-max-bs-decode: 64 with CONC: '60', while c1, c4, c10, c16, c24, c32, c48 and c52 all match their CONC. The convention breach holds either way. Nothing breaks at runtime: c60 at 64 just leaves a little admission headroom.
…glang # Conflicts: # inferencex-e2e/perf-changelog.yaml
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=41977056700 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=41977056700 |
Take Kevin Mi's Kimi-K3.1-Pro MI355X SGLang DFLASH recipe and points: TP8 monokernel at c1 and c4, and TP8/DCP8 with the HiCache host tier at c8, c16, c32 and c48. Points above c48 are dropped because the node's GTT limit makes their host pools overflow. K3.1 is not staged in the shared HF cache on mi355x-amds, so the model keeps loading from the node-local cache, without HF_HUB_OFFLINE. Co-authored-by: Kevin Mi <mikevin920@yahoo.com>
|
/use 41977056700 |
|
@cquil11 staged run 41977056700. This run remains available across future |
|
/reuse-sweep-run 41977056700 |
# Conflicts: # inferencex-e2e/perf-changelog.yaml
|
Ingest report — ✅ Results ingested into the Ingest workflow · Merge Ingest run · Benchmark source run
benchmarkFiles 6, rawBenchmarkRows 6, coveredByPerJobFiles 0, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 2638, gpuMetricSamplesNew 247208, powerPublicationPoints 6, changelogEntriesWritten 1 Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun. 中文Ingest 报告 — ✅ 结果已写入 上表仅统计本次运行写入的数据，各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。 |
Adds Kimi K3.1 Pro day-0 SGLang AgentX on MI355X with the RadixArk DFLASH draft, following the SGLang K3.1 cookbook's MI355X AgentX cells (Low-Latency monokernel and High-Throughput TP8 · DCP8 HiCache).
Recipe and points ported from Kevin Mi's (AMD) #4823, which this PR supersedes.
kimik3.1-fp4-mi355x-sglang-agentic-dflashlmsysorg/sglang:dev-kimi-k31-pro-mi35x@sha256:6af91676…. The cookbook measured its MI355X numbers on a custom ROCm 10 image with the K3.1 ROCm port, and says this public image "has not been checked against these commands".RadixArk/Kimi-K3.1-Dflash2-Previewpinned atd0d8bc2, the revision the golden curve was measured on. The draft attention backend istritonon ROCm. Block verification is on to match the curvekimi-k3.1-pro__dflash__kimi-k3.1-dflash2-preview__default-block. Throughput runs inject that curve's AL at 5 drafts (3.43). Nothing is hard-coded.dram-utilization: 0.513(1,538 GB) covers the HiCache host pools. That is within the 3 TB per-server limit at 8 of 8 GPUs./it-share/hf-hub-cache, so it loads from the node-local HF cache (/var/lib/hf-hub-cache). A node's first point downloads the ~1.56 TB checkpoint there. [AMD][AgentX] MI355X Kimi-K3.1-Pro FP4 SGLang: DFLASH single-node points / [AMD][AgentX] MI355X Kimi-K3.1-Pro FP4 SGLang：DFLASH 单节点配置点 #4823's shared-cache lane andHF_HUB_OFFLINE=1are left out until the checkpoint is staged.RUSTUP_TOOLCHAIN=stableon the HiCache points keeps every rank on the Rust TreeCore. This works around an image bug where write-back asserts on a mix of Rust and Python TreeCores (upstream fix [Feat] [AMD] model: Add Kimi-K3.1 MI35x support sgl-project/sglang#48427).K3.1 precision rules
--quantizationoverride.SGLANG_ROCM_DISABLE_LINEARQUANT=0(the image default) only applies the checkpoint's own config, which keeps those layers BF16.AITER_SITUV2_A8W4=1runs FP4 weights with FP8 activations, which is above the A4W4 floor.AITER_OPUS_STAGE2_FP8=0keeps the MoE stage-2 partials BF16.ROCM_QUICK_REDUCE_QUANTIZATION: NONEis set explicitly, as the all-reduce policy requires for SGLang ROCm.--enable-linear-replayssm-spec, SGLang resolves--mamba-ssm-dtypeto float32. That is also its default when the checkpoint sets none.kv-cache-dtype: fp8_e4m3. The draft KV is FP8 as well, consistent with the target, as the cookbook's DFLASH overlay sets it.