Skip to content

Commit 8f29d8d

Browse files
authored
perf(amd): DSv4.1-Flash ATOM AgentX on the 1110 nightly with MXFP8 KV, tuned delayer TP points, DTP2 at c64 and DEP2 at c128/c256 / DSv4.1-Flash ATOM AgentX 更新至 1110 nightly,MXFP8 KV,TP 点调 delayer,c64 DTP2,c128/c256 DEP2 (#3965)
1 parent 7b64020 commit 8f29d8d

3 files changed

Lines changed: 113 additions & 41 deletions

File tree

‎inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/dsv41flash/atom/mi355x-fp4-mtp/agentic.yaml‎

Lines changed: 92 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ base:
66
gpus_per_node: 8
77
frontend:
88
type: atomesh
9-
container_image: rocm/atom-dev@sha256:a60811a8467b43e3e50e9e9eb8cec34f68b7bca03f8c6dbd6a0ff813dde49d4c
9+
container_image: rocm/atom-dev@sha256:160a4ec49379e7f1980a4e783254366b1d5c643b012abca57a21c0bcbbfe83a7
1010
enable_multiple_frontends: false
1111
observability:
1212
enabled: false
@@ -20,10 +20,9 @@ base:
2020
args:
2121
served-model-name: deepseek-ai/DeepSeek-V4.1-Flash
2222
trust-remote-code: true
23-
kv_cache_dtype: bf16
23+
kv_cache_dtype: fp8
2424
index-cache-dtype: fp4
2525
gpu-memory-utilization: 0.9
26-
max-num-seqs: 128
2726
max-num-batched-tokens: 16384
2827
attn-prefill-chunk-size: 16384
2928
enable_prefix_caching: true
@@ -38,6 +37,8 @@ base:
3837
method: dspark
3938
num-speculative-tokens: 5
4039
tool-call-parser: dsml_v41
40+
# uvicorn's 5 s default resets AIPerf's pooled connections at high concurrency.
41+
timeout-keep-alive: 900
4142
env:
4243
OMP_NUM_THREADS: '4'
4344
ATOM_NUMA_BIND: '0'
@@ -50,13 +51,18 @@ base:
5051
AIPERF_TIMING_CANCEL_DRAIN_TIMEOUT: '300'
5152
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'atom:'
5253

53-
# One variant per point. Every point admits 128 sequences and captures every
54-
# decode size through 32 plus 48, 64 and 128.
55-
5654
override_tp2_c1:
5755
roles:
5856
agg:
5957
gpus: 2
58+
args: &c1_args
59+
# DSpark's block-drafter warmup needs two sequences.
60+
max-num-seqs: 2
61+
cudagraph-capture-sizes: '[1,2]'
62+
env: &c1_env
63+
ATOM_ENABLE_PREFILL_DELAYER: '1'
64+
# No step reaches this interval, so a new turn's prefill waits for the running decode.
65+
ATOM_PREFILL_DECODE_INTERVAL: '1000000'
6066
benchmark:
6167
env:
6268
CONC: '1'
@@ -65,86 +71,135 @@ override_tp2_c2:
6571
roles:
6672
agg:
6773
gpus: 2
74+
env:
75+
ATOM_ENABLE_PREFILL_DELAYER: '1'
76+
ATOM_PREFILL_DECODE_INTERVAL: '192'
6877
benchmark:
6978
env:
7079
CONC: '2'
7180

72-
override_tp2_c4:
81+
override_tp2_c16:
7382
roles:
7483
agg:
7584
gpus: 2
85+
env:
86+
ATOM_ENABLE_PREFILL_DELAYER: '1'
87+
ATOM_PREFILL_DECODE_INTERVAL: '96'
7688
benchmark:
7789
env:
78-
CONC: '4'
90+
CONC: '16'
7991

80-
override_tp2_c6:
92+
override_tp2_c32:
8193
roles:
8294
agg:
8395
gpus: 2
96+
env:
97+
ATOM_ENABLE_PREFILL_DELAYER: '1'
98+
ATOM_PREFILL_DECODE_INTERVAL: '40'
8499
benchmark:
85100
env:
86-
CONC: '6'
101+
CONC: '32'
87102

88-
override_tp2_c8:
103+
override_tp4_c1:
89104
roles:
90105
agg:
91-
gpus: 2
106+
gpus: 4
107+
args: *c1_args
108+
env: *c1_env
92109
benchmark:
93110
env:
94-
CONC: '8'
111+
CONC: '1'
95112

96-
override_tp2_c16:
113+
override_tp4_c2:
97114
roles:
98115
agg:
99-
gpus: 2
116+
gpus: 4
117+
env:
118+
ATOM_ENABLE_PREFILL_DELAYER: '1'
119+
ATOM_PREFILL_DECODE_INTERVAL: '192'
100120
benchmark:
101121
env:
102-
CONC: '16'
122+
CONC: '2'
103123

104-
override_tp2_c32:
124+
override_tp4_c4:
105125
roles:
106126
agg:
107-
gpus: 2
127+
gpus: 4
128+
env:
129+
ATOM_ENABLE_PREFILL_DELAYER: '1'
130+
ATOM_PREFILL_DECODE_INTERVAL: '192'
108131
benchmark:
109132
env:
110-
CONC: '32'
133+
CONC: '4'
111134

112-
override_tp2_c64:
135+
override_tp4_c8:
113136
roles:
114137
agg:
115-
gpus: 2
116-
# Run at least 4 decode passes after each prefill and batch the prefills
117-
# that arrive meanwhile; at this load the batching also raises throughput.
138+
gpus: 4
118139
env:
119-
ATOM_PREFILL_DECODE_INTERVAL: '4'
140+
ATOM_ENABLE_PREFILL_DELAYER: '1'
141+
ATOM_PREFILL_DECODE_INTERVAL: '192'
120142
benchmark:
121143
env:
122-
CONC: '64'
144+
CONC: '8'
123145

124-
override_tp4_c1:
146+
override_tp4_c32:
125147
roles:
126148
agg:
127149
gpus: 4
150+
env:
151+
ATOM_ENABLE_PREFILL_DELAYER: '1'
152+
ATOM_PREFILL_DECODE_INTERVAL: '16'
128153
benchmark:
129154
env:
130-
CONC: '1'
155+
CONC: '32'
131156

132-
override_tp4_c8:
157+
override_tp2_c64_dtp:
133158
roles:
134159
agg:
135-
gpus: 4
160+
gpus: 2
161+
args:
162+
# TBO stays off: it does not support SWA bounded replay.
163+
enable-dp-attention: true
164+
# ATOM_DP_LB_REQ_EQUIV weights in-flight requests under least_tokens.
165+
dp-load-balance: least_tokens
166+
env: &dp_env
167+
GPU_MAX_HW_QUEUES: '5'
168+
ATOM_DP_SESSION_AFFINITY: '1'
169+
ATOM_DP_LB_REQ_EQUIV: '512'
170+
ATOM_ENABLE_PREFILL_DELAYER: '1'
171+
ATOM_PREFILL_DECODE_INTERVAL: '10'
136172
benchmark:
137173
env:
138-
CONC: '8'
174+
CONC: '64'
175+
# Session affinity keys on the AIPerf session id.
176+
AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: '1'
139177

140-
override_tp4_c32:
178+
override_tp2_c128_dep:
141179
roles:
142180
agg:
143-
gpus: 4
144-
# Run at least 16 decode passes after each prefill and batch the prefills
145-
# that arrive meanwhile, so new turns stall decode less often.
146-
env:
147-
ATOM_PREFILL_DECODE_INTERVAL: '16'
181+
gpus: 2
182+
args:
183+
enable-dp-attention: true
184+
enable-expert-parallel: true
185+
dp-load-balance: least_tokens
186+
env: *dp_env
148187
benchmark:
149188
env:
150-
CONC: '32'
189+
CONC: '128'
190+
AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: '1'
191+
192+
override_tp2_c256_dep:
193+
roles:
194+
agg:
195+
gpus: 2
196+
args:
197+
enable-dp-attention: true
198+
enable-expert-parallel: true
199+
dp-load-balance: least_tokens
200+
cudagraph-capture-sizes: '[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,48,64,128,256]'
201+
env: *dp_env
202+
benchmark:
203+
env:
204+
CONC: '256'
205+
AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: '1'

‎inferencex-e2e/configs/amd-master.yaml‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1017,10 +1017,9 @@ minimaxm31flash-fp4-mi355x-sglang-agentic-dspark:
10171017
# High-Throughput: TP4 + HiCache, 180 GB of host KV per rank (720 of 735 GB).
10181018
- { tp: 4, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [48, 64], srt-recipe: agentic.yaml }
10191019

1020-
# DeepSeek-V4.1-Flash on ATOM, following the upstream TP2 / TP4 AgentX recipe.
10211020
# Five-token DSpark uses golden AL 3.51 for throughput and real acceptance for eval.
10221021
dsv41flash-fp4-mi355x-atom-agentic-dspark:
1023-
image: rocm/atom-dev:nightly_202610121516
1022+
image: rocm/atom-dev:nightly_202611101510
10241023
model: deepseek-ai/DeepSeek-V4.1-Flash
10251024
model-prefix: dsv41flash
10261025
runner: cluster:mi355x-amds
@@ -1031,8 +1030,10 @@ dsv41flash-fp4-mi355x-atom-agentic-dspark:
10311030
scenarios:
10321031
agentic-coding:
10331032
- search-space:
1034-
- { tp: 2, ep: 1, dp-attn: false, kv-offloading: none, spec-decoding: draft_model, conc-list: [1, 2, 4, 6, 8, 16, 32, 64], srt-recipe: agentic.yaml }
1035-
- { tp: 4, ep: 1, dp-attn: false, kv-offloading: none, spec-decoding: draft_model, conc-list: [1, 8, 32], srt-recipe: agentic.yaml }
1033+
- { tp: 2, ep: 1, dp-attn: false, kv-offloading: none, spec-decoding: draft_model, conc-list: [1, 2, 16, 32], srt-recipe: agentic.yaml }
1034+
- { tp: 2, ep: 1, dp-attn: true, kv-offloading: none, spec-decoding: draft_model, conc-list: [64], srt-recipe: agentic.yaml }
1035+
- { tp: 2, ep: 2, dp-attn: true, kv-offloading: none, spec-decoding: draft_model, conc-list: [128, 256], srt-recipe: agentic.yaml }
1036+
- { tp: 4, ep: 1, dp-attn: false, kv-offloading: none, spec-decoding: draft_model, conc-list: [1, 2, 4, 8, 32], srt-recipe: agentic.yaml }
10361037

10371038
# The golden acceptance curve is the GLM-5.3 NVFP4 SGLang curve in
10381039
# infx/golden_al_distribution/glm-5.3-nvfp4__mtp__native__default-default.yaml.

‎inferencex-e2e/perf-changelog.yaml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12827,3 +12827,19 @@
1282712827
- "Drop 3 prefill workers at c700 and c1228 (DEP16 decode): the DEP32 c3018 point has both higher interactivity and higher throughput per GPU than each of them. Drop 5 prefill workers at c2884 (DEP16 decode): it lies under the line between DEP32 c3018 and 7P c5540."
1282812828
- "Set stream-interval 32 on the agg role and on the disagg prefill role, which used the SGLang default of 1; disagg decode already used 32. Agg TP4 c1, c4 and c16 change otherwise not at all; disagg 7P c5540 and 9P c7480 change otherwise only by the image. The agg key is listed so the whole Dynamo + SGLang curve, agg points included, comes from this sweep."
1282912829
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4997
12830+
12831+
- config-keys:
12832+
- dsv41flash-fp4-mi355x-atom-agentic-dspark
12833+
scenario-type:
12834+
- agentic-coding
12835+
description:
12836+
- "Update the image from rocm/atom-dev:nightly_202610121516 to nightly_202611101510 (ATOM e5b74bc86, ROCm 7.2.4) in the master config, and the AToMesh frontend container_image from sha256:a60811a8467b43e3e50e9e9eb8cec34f68b7bca03f8c6dbd6a0ff813dde49d4c to sha256:160a4ec49379e7f1980a4e783254366b1d5c643b012abca57a21c0bcbbfe83a7, the same image."
12837+
- "Replace plain TP2 at concurrency [1, 2, 4, 6, 8, 16, 32, 64] and plain TP4 at [1, 8, 32] with plain TP2 at [1, 2, 16, 32], DTP2 (DP attention) at 64, DEP2 (DP attention plus expert parallelism) at [128, 256] and plain TP4 at [1, 2, 4, 8, 32]. TP2 c4, c6, c8 and c64 are dropped; TP4 c2 and c4 are new."
12838+
- "Switch the KV cache from BF16 to MXFP8 (kv_cache_dtype fp8) at every point; the index cache stays FP4."
12839+
- "Drop max-num-seqs 128, so every point except concurrency 1 runs ATOM's default admission. Concurrency 1 (TP2 and TP4) runs max-num-seqs 2, decode graphs [1, 2] and ATOM_PREFILL_DECODE_INTERVAL=1000000, the C1 recipe of ATOM's recipes/Kimi-K3.md (ROCm/ATOM#2659). The c256 point also captures graph size 256."
12840+
- "Set ATOM_PREFILL_DECODE_INTERVAL per point: 192 at TP2 c2 and TP4 c2, c4 and c8, 96 at TP2 c16, 40 at TP2 c32, 16 at TP4 c32 (unchanged) and 10 at the DP points (previously unset except 4 at TP2 c64 and 16 at TP4 c32)."
12841+
- "The DP points add dp-load-balance least_tokens, ATOM_DP_SESSION_AFFINITY=1, ATOM_DP_LB_REQ_EQUIV=512 and GPU_MAX_HW_QUEUES=5, and AIPerf sends the correlation ID as the X-Dynamo-Session-Id header for session affinity."
12842+
- "Every point adds timeout-keep-alive 900 so the server keeps AIPerf's pooled connections open past uvicorn's 5 s default."
12843+
- "The new ATOM brings DeepSeek-V4.1 DP attention (ROCm/ATOM#2376), the V4.1 MXFP8 KV cache with DSpark on an FP8 cache (ROCm/ATOM#2725), DP-attention prefill balancing and graph replay (ROCm/ATOM#2551, #2696, #2697) and the default HIP Mega-MHC kernel on gfx950 (ROCm/ATOM#2568)."
12844+
- "No change to the DSpark draft's weight precision: ATOM loads the checkpoint's mtp.0-2 layers as shipped (FP8 E4M3 attention, shared experts and main_proj with 32x32 UE8M0 scales, MXFP4 routed experts, BF16 router, norms and heads) at both images. The draft shares the target's KV pool, so its KV cache follows the target from BF16 to MXFP8."
12845+
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3965

0 commit comments

Comments
 (0)