Repository navigation
Conversation
…i35x-20261109 Co-authored-by: Cursor <cursoragent@cursor.com>
|
Thanks for the contribution!
中文感谢你的贡献！
|
Co-authored-by: Cursor <cursoragent@cursor.com>
|
Claude finished @chuyeh's task in 29s —— View job Review
1 blocking issue found. The image bump itself is fine. The problem is in the recipe this config runs. The bump reruns every AgentX point, and |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
I reviewed this PR and didn't find any bugs. Because inferencex-e2e/configs/amd-master.yaml has designated AMD code owners in CODEOWNERS, one of them should still take a look.
What was reviewed:
- The only change in
amd-master.yamlis theimage:line forqwen3.5-fp8-mi355x-sglang-agentic-mtp. The model, runner, recipe dir, HiCache band and concurrency grid are unchanged, and the comment above the key still describes the current config accurately. - The
perf-changelog.yamlblock is appended at the physical end of the file, and no earlier bytes changed. Itsconfig-keys,scenario-type: [agentic-coding]and PR #4975 link match the changed key and the existing entries for that key. - Not done: I didn't check whether the image tag exists or what its digest is, and no sweep results were available. The required
/use <run_id>sweep is still outstanding.
Extended reasoning...
This is an 8-line config change. One image: tag in inferencex-e2e/configs/amd-master.yaml is bumped for a single AMD MI355X AgentX key, and one append-only perf-changelog.yaml entry is added for that key. It touches no security-sensitive code: no auth, injection or data-exposure surface. The change follows existing patterns and looks correct. I deferred instead of approving because CODEOWNERS assigns amd-master.yaml to specific AMD owners, and because the PR is a draft with no benchmark sweep run yet.
Co-authored-by: Cursor <cursoragent@cursor.com>
chuyeh
changed the title
chuyeh
added
the
full-sweep-fail-fast
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=43037834183 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=43037834183 |
…-agentx-v0.5.23-20261109 Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # inferencex-e2e/perf-changelog.yaml
…55X AgentX Set SGLANG_AITER_HONOR_EXPLICIT_MEM_FRACTION=1 so mem-fraction-static 0.8 is used as-is instead of being scaled to 0.68 by aiter above 8k context, and set ROCM_QUICK_REDUCE_QUANTIZATION=NONE explicitly now that the image no longer defaults to INT8. Restore stream-interval to 50, which the current AgentX policy allows (at most 60, no lower bound). Co-authored-by: Cursor <cursoragent@cursor.com>
chuyeh
changed the title
Summary
mainrunsqwen3.5-fp8-mi355x-sglang-agentic-mtpon the 2026-10-20 imagelmsysorg/sglang-rocm:v0.5.22-rocm720-mi35x-20261020. This bumps it to the latest MI355X image,lmsysorg/sglang-rocm:v0.5.23-rocm720-mi35x-20261109(Docker Hub: pushed 2026-11-09T14:19:26Z, amd64 digestsha256:e728a52213cce18493a46ac8ac62d5c9eaee66591941871acd213fe989e55957), and sets two recipe env variables so the new image does not change behavior silently:ROCM_QUICK_REDUCE_QUANTIZATION=NONE: the previous image defaulted to INT8 all-reduce and the new one does not (sgl-project/sglang#47163). This pins a full-precision all-reduce instead of relying on an image default.SGLANG_AITER_HONOR_EXPLICIT_MEM_FRACTION=1: with the aiter attention backend and context above 8192, SGLang multiplies an explicitmem-fraction-staticby 0.85, so the recipe's 0.8 ran as 0.68. With the variable set, 0.8 is used as-is and the KV pool grows from 7.83M to 11.32M tokens (+45%). That removes KV-pool queueing at the top of the curve (c72 to c80).stream-intervalstays at 50, which satisfies the current AgentX limit of at most 60 (checklist). The HiCachewrite_throughpolicy, the TP4 concurrency grid and every other serving flag are unchanged, and other MI355X arms stay on their current tags.The image, the all-reduce setting and the effective memory fraction change together, so differences against the
v0.5.22results cannot be attributed to the image alone.Validation:
run-sweep.ymlrun 43037834183 (full-sweep-fail-fast, head76be03775) is in progress. Local A/B evidence for the memory fraction is in the details below; it is not a CI run.Risk:
hicache-ratio(1.5), so it grows with the KV pool. Those points were not exercised locally. c32 and above use pinned sizes.NONEhere. It is set to avoid integer-quantized all-reduce. INT8 is not restored, so the prefill all-reduce speedup it gave on the previous image is not recovered.Validation details
Local exclusive A/B (not a CI run). TP4 FP8, 3600 s AgentX replays with
benchmarks/srt_agentic.sh, one run per arm, 0 request errors. Both arms use the previous head80a58d278plusROCM_QUICK_REDUCE_QUANTIZATION=NONE; they differ only inSGLANG_AITER_HONOR_EXPLICIT_MEM_FRACTION. The runs used the local imagerocm/sgl-dev:v0.5.23-rocm724-mi35x-20261030(no INT8 default either) andstream-interval: 20, because the PR image was not on the host. Absolute numbers are below CI (c80: 46k to 48k against about 60k total tok/s/GPU), so only the relative change is claimed.OOM stress test. c80 recipe flags with 0.8 honored. 96 unique prompts of 40k to 250k tokens (13.8M tokens), 80 in flight, then the same prompts again with 2k new tokens. 192 of 192 requests completed with no OOM, the KV pool reached 100%, and peak memory was 253 to 255 GB of about 288 GB per GPU (about 33 GB free).
Earlier sweep on this branch (run 42785772087, head
80a58d278, new image without these two variables):v0.5.22results. The analysis attributes this to slower prefill (about 10% above 8k uncached tokens) after the INT8 default disappeared. This PR does not recover it.mia1-p01-g11, which looks degraded (decode 7% slower at batch size 1, against 4% to 9% faster on other nodes). This is a suspicion, not a confirmed cause, and those points have not been rerun.AI model disclosure
perf-changelog.yamlentry, the merge withmainand this description. The local A/B experiment and its analysis note were provided by the PR author; this model did not run them. No delegated agents.PR checklist and change type
Type of Change
Checklist
python -m infx.matrix.changed --base origin/mainlists affected config keys, I have appended one new entry covering them to the physical end ofinferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entriesinferencex-e2e/perf-changelog.yamlentry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs):full-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied.OWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.中文
main上qwen3.5-fp8-mi355x-sglang-agentic-mtp仍使用 2026-10-20 的lmsysorg/sglang-rocm:v0.5.22-rocm720-mi35x-20261020镜像。本 PR 将其升级到最新的 MI355X 镜像lmsysorg/sglang-rocm:v0.5.23-rocm720-mi35x-20261109（Docker Hub 推送时间 2026-11-09T14:19:26Z，amd64 摘要（digest）为sha256:e728a52213cce18493a46ac8ac62d5c9eaee66591941871acd213fe989e55957），并在配方中设置两个环境变量，避免新镜像悄悄改变行为：ROCM_QUICK_REDUCE_QUANTIZATION=NONE：旧镜像默认使用 INT8 all-reduce，新镜像不再默认如此（sgl-project/sglang#47163）。显式设置可固定使用全精度 all-reduce，而不是依赖镜像默认值。SGLANG_AITER_HONOR_EXPLICIT_MEM_FRACTION=1：使用 aiter 注意力后端且上下文超过 8192 时，SGLang 会将显式设置的mem-fraction-static乘以 0.85，因此配方中的 0.8 实际按 0.68 运行。设置该变量后按 0.8 原值使用，KV 池由 783 万 token 增至 1132 万 token（+45%），从而消除曲线高并发端（c72 至 c80）的 KV 池排队。stream-interval保持为 50，满足当前 AgentX 不超过 60 的限制（检查清单）。HiCachewrite_through策略、TP4 并发网格及其余服务参数均不变，其他 MI355X 配置保持原镜像。镜像、all-reduce 设置和实际显存占比同时变化，因此与
v0.5.22结果的差异不能仅归因于镜像。验证：
run-sweep.yml运行 43037834183（full-sweep-fail-fast，head76be03775）正在进行中。显存占比的本地 A/B 证据见上方折叠详情，它不是 CI 运行。风险：
hicache-ratio（1.5）确定大小，会随 KV 池一起增大，这些点未在本地测试。c32 及以上使用固定大小。NONE；设置它是为了避免整数量化 all-reduce。INT8 不会恢复，因此旧镜像上它带来的预填充 all-reduce 加速不会找回。本地 A/B（非 CI 运行）： TP4 FP8，3600 秒 AgentX 回放，每组一次，无请求错误。两组均使用此前的 head
80a58d278加ROCM_QUICK_REDUCE_QUANTIZATION=NONE，只在SGLANG_AITER_HONOR_EXPLICIT_MEM_FRACTION上不同。运行使用本地镜像rocm/sgl-dev:v0.5.23-rocm724-mi35x-20261030和stream-interval: 20（PR 镜像不在该主机上）。绝对数值低于 CI，因此只采信相对变化。c80：总 tok/s/GPU +3.3%，P90 交互性 +6.3%，P90 TTFT -31%，平均排队请求由 3.58 降至 0；c64：无变化（在 0.3% 以内）。OOM 压力测试：192/192 请求完成，无 OOM，峰值显存约 253 至 255 GB（每 GPU 约 288 GB），约 33 GB 空闲。此前扫描（运行 42785772087）：c56 至 c80 吞吐量低 2% 至 3%，分析认为是 INT8 默认值消失后预填充变慢所致，本 PR 不会找回。c1、c22、c48 低 4% 至 8%，三者都运行在节点
mia1-p01-g11上，该节点看起来已降级。这只是怀疑而非已确认原因，这些点尚未重跑。AI 模型使用说明：
perf-changelog.yaml条目、与main的合并及本说明。本地 A/B 实验及其分析笔记由 PR 作者提供，该模型未运行这些实验。未委派其他 agent。改动类型： 配置变更。