Repository navigation
Conversation
…ightly Move dsv41flash-fp4-mi355x-atom-agentic-dspark to rocm/atom-dev:nightly_202610131505 (ATOM 661ef7661, with ROCm/ATOM#2376 V4.1 DPA and prefill TBO). Run TP2 c32/64/128/256 and TP4 c32 with DP attention, prefill TBO, DP session affinity and the prefill delayer; drop TP2 c6. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/dsv41flash/atom/mi355x-fp4-mtp/agentic.yaml # inferencex-e2e/configs/amd-master.yaml # inferencex-e2e/perf-changelog.yaml
…1013-dpa # Conflicts: # inferencex-e2e/perf-changelog.yaml
rocm/atom-dev:nightly_202610221638 (sha256:32c81e56e468...; ATOM 0.1.9.dev11+gce46f6877, ROCm 7.2.4). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… without TBO Merge origin/main, move the image to rocm/atom-dev:nightly_202610231458 (sha256:05322b86...) and drop --enable-tbo from the DP-attention points: TBO does not support SWA bounded replay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Use the ATOM 2c772a5 release image (ROCm/ATOM run 39986380362, sha256:01a19e8b...) instead of nightly_202610231458. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
max-num-seqs 128 queued half of the 256 sessions (TTFT 490 s); raise it to 256 and capture a 256 decode graph on that point. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…seqs as B300 vLLM Move to the ATOM 11a2d56 release image (ROCm/ATOM run 40072622970, sha256:42716923...) and set max-num-seqs per point (per DP rank) like the B300 vLLM AgentX recipe: 1 at c1, 2x concurrency for c2-c16, concurrency from c32 up; capture sizes stop at each point's max-num-seqs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1013-dpa # Conflicts: # inferencex-e2e/perf-changelog.yaml
max-num-seqs is twice the concurrency at c1 too (TP2 and TP4). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop every max-num-seqs setting; decode graphs go back to the base list (sizes through 32 plus 48, 64, 128), with 256 added on the c256 points. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…OM AgentX on nightly_202610261558 Image rocm/atom-dev:nightly_202610261558 (ATOM 713eb06, digest sha256:57477e7a...) adds ROCm/ATOM#2696/#2697, which fix the hang every DP-attention point hit on nightly_202610240802, and #2719 (trace labels). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ith FP8 KV, TP and DEP2
…t c64, DEP2 at c128/c256 Plain TP2 / TP4 at concurrency 1-32: c1 keeps the ROCm/ATOM#2659 recipe, c2-c32 turn on the prefill delayer (decode interval 192 at c2-c8, 96 at c16, 30 / 32 at c32). DTP2 (DP attention without TBO) takes c64 and DEP2 c128 and c256; plain TP at 64 and above is dropped. The server keeps idle connections for 900 s, past uvicorn's 5 s default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1013-dpa # Conflicts: # inferencex-e2e/perf-changelog.yaml
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # inferencex-e2e/perf-changelog.yaml
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # inferencex-e2e/perf-changelog.yaml
…val 40 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1013-dpa # Conflicts: # inferencex-e2e/perf-changelog.yaml
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1013-dpa # Conflicts: # inferencex-e2e/perf-changelog.yaml
…e diff - Update the image from rocm/atom-dev:nightly_202610121516 to nightly_202611101510 (ATOM e5b74bc86, ROCm 7.2.4) in the master config, and the AToMesh frontend container_image from sha256:a60811a8... to sha256:160a4ec4..., the same image. - Replace plain TP2 at concurrency [1, 2, 4, 6, 8, 16, 32, 64] and plain TP4 at [1, 8, 32] with plain TP2 at [1, 2, 16, 32], DTP2 at 64, DEP2 at [128, 256] and plain TP4 at [1, 2, 4, 8]. TP2 c4, c6, c8 and c64 and TP4 c32 are dropped; TP4 c2 and c4 are new. - Switch the KV cache from BF16 to MXFP8 at every point; the index cache stays FP4. - Drop max-num-seqs 128. Concurrency 1 runs max-num-seqs 2, decode graphs [1, 2] and ATOM_PREFILL_DECODE_INTERVAL=1000000, the C1 recipe of ATOM's recipes/Kimi-K3.md (ROCm/ATOM#2659). The c256 point also captures 256. - Set ATOM_PREFILL_DECODE_INTERVAL per point: 192 at TP2 c2 and TP4 c2-c8, 96 at TP2 c16, 40 at TP2 c32 and 10 at the DP points. - The DP points add dp-load-balance least_tokens, ATOM_DP_SESSION_AFFINITY=1, ATOM_DP_LB_REQ_EQUIV=512 and GPU_MAX_HW_QUEUES=5, with AIPerf session ids as X-Dynamo-Session-Id. - Every point adds timeout-keep-alive 900. - The new ATOM brings V4.1 DP attention (#2376), the V4.1 MXFP8 KV cache with DSpark (#2725), DP prefill balancing and graph replay (#2551, #2696, #2697) and the default HIP Mega-MHC kernel (#2568). - No change to the DSpark draft's weight precision; its KV cache follows the target from BF16 to MXFP8. Also replace the recipe and master-config header comments with line comments (AGENTS.md "Comments"); no setting changes. - 镜像从 rocm/atom-dev:nightly_202610121516 更新为 nightly_202611101510(ATOM e5b74bc86,ROCm 7.2.4),AToMesh 前端 container_image 同步更新为同一镜像。 - 纯 TP2 [1, 2, 4, 6, 8, 16, 32, 64] 和纯 TP4 [1, 8, 32] 改为纯 TP2 [1, 2, 16, 32]、DTP2 c64、DEP2 [128, 256] 和纯 TP4 [1, 2, 4, 8]。删除 TP2 c4、c6、c8、c64 和 TP4 c32;新增 TP4 c2 和 c4。 - 所有点的 KV cache 从 BF16 改为 MXFP8;index cache 仍为 FP4。 - 去掉 max-num-seqs 128。并发 1 使用 max-num-seqs 2、decode 图 [1, 2] 和 ATOM_PREFILL_DECODE_INTERVAL=1000000,即 ATOM recipes/Kimi-K3.md 的 C1 recipe(ROCm/ATOM#2659)。c256 额外捕获 256。 - 各点设置 ATOM_PREFILL_DECODE_INTERVAL:TP2 c2 和 TP4 c2-c8 为 192,TP2 c16 为 96,TP2 c32 为 40,DP 点为 10。 - DP 点增加 dp-load-balance least_tokens、ATOM_DP_SESSION_AFFINITY=1、ATOM_DP_LB_REQ_EQUIV=512 和 GPU_MAX_HW_QUEUES=5,AIPerf 以 X-Dynamo-Session-Id 发送 session id。 - 所有点增加 timeout-keep-alive 900。 - 新 ATOM 带来 V4.1 DP attention(#2376)、V4.1 MXFP8 KV cache 与 DSpark(#2725)、DP prefill 均衡与图回放(#2551、#2696、#2697)以及默认 HIP Mega-MHC kernel(#2568)。 - DSpark draft 权重精度不变;其 KV cache 随 target 从 BF16 变为 MXFP8。 另将 recipe 和 master config 的头部注释改为行注释(AGENTS.md "Comments"),不改变任何设置。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
Thanks for the contribution!
中文感谢你的贡献!
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=43387638995 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=43387638995 |
…1111-tp4c8 # Conflicts: # inferencex-e2e/perf-changelog.yaml
…g main Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1111-tp4c8 # Conflicts: # inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/dsv41flash/atom/mi355x-fp4-mtp/agentic.yaml # inferencex-e2e/configs/amd-master.yaml # inferencex-e2e/perf-changelog.yaml
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft. Single-point check of DeepSeek-V4.1-Flash on ATOM AgentX (MI355X) with the 1111 nightly: plain TP4 at concurrency 1.
只测一个点:DSv4.1-Flash ATOM AgentX(MI355X),1111 nightly,plain TP4 c1。
rocm/atom-dev:nightly_202611111518(ATOM 3659b9865), frontendcontainer_imagepinned to the same digestsha256:167ce262…7491d.amd-master.yamlis narrowed to TP4 c1.kv_cache_dtype: bf16) so ATOM's V4.1 mono hi decode path is enabled; that path only serves BF16, and the earlier run (43058239532, MXFP8 KV) loggedV4.1 mono hi off.🤖 Generated with Claude Code