Repository navigation
[AMD][MI355X] Add GLM-5.2 MXFP4 AgentX P/D behind the Infera router (infera-sglang) / [AMD][MI355X] 新增 GLM-5.2 MXFP4 AgentX PD 分离部署（Infera router，infera-sglang） - #4629
Conversation
infera-sglang runs the stock SGLang image with Infera installed at the master config's router version from Infera's GitHub release (configs/infera-sglang.sh, bound through INSTALLERS); synthetic acceptance follows SGLang. Signed-off-by: liyingli <liyingli@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
glm5.2-fp4-mi355x-infera-sglang-agentic-disagg: stock SGLang nightly behind the Infera router (frontend.type: infera, carried as runners/srt-slurm/patches/630-infera-frontend.patch until the pin includes NVIDIA/srt-slurm#630). Prefill TP1 x PP4 with write-back HiCache at 250 GB per stage, decode TP + DP attention, EAGLE 5/1/6, Mooncake KV. Shapes: 1P1D with an 8-GPU decode (conc 32-88), 1P1D with a 4-GPU decode (64, 80), 2P1D (176). Signed-off-by: liyingli <liyingli@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
| "${pip_install[@]}" --quiet --no-cache-dir --no-deps \ | ||
| "nats-py==2.16.0" \ | ||
| "amd-infera==${ROUTER_VERSION}" \ | ||
| --find-links "https://github.com/AMD-AGI/Infera/releases/expanded_assets/v${ROUTER_VERSION}" |
There was a problem hiding this comment.
Highly recommended suggestion: put this wheel on pypi plz
…ith waiver infera-sglang.sh now runs infera-sglang-patches/apply.sh after installing Infera: B1 (sglang#33059), B2 (#46830), B6 (#46820), B7 (#47184), B8 (issue #46829), each anchored, idempotent and marker-checked; a fix that does not land fails the job. docs/waiver/4629.md covers them. The router check resolves the binary the launcher will exec (the wheel's bundled infera-router) instead of looking on PATH. The recipe registers VRAM the peer way (MOONCAKE_DISABLE_HIP_DMABUF=1, as ATOM P/D on this cluster) and mounts the host's Ionic verbs provider (as Kimi-K3 vLLM). Signed-off-by: liyingli <liyingli@amd.com>
srt-slurm ae3d6877's start_srun_process takes neither reuse_container nor container_prepare_timeout_seconds (SemiAnalysisAI#630 is written against a newer main), so the router never started: AttributeError on RuntimeContext. Regenerated against ae3d6877, the router call matches the pin's static routers. Signed-off-by: liyingli <liyingli@amd.com>
Signed-off-by: liyingli <liyingli@amd.com> # Conflicts: # inferencex-e2e/perf-changelog.yaml
…s 59920ea1 This reverts commit a898651. main moved the srt-slurm submodule to v2.65.2 (59920ea1, SemiAnalysisAI#4601), the base NVIDIA/srt-slurm#630 is written against, where start_srun_process takes reuse_container and container_prepare_timeout_seconds. 630-infera-frontend.patch is again SemiAnalysisAI#630's source change; it applies to 59920ea1 after 573-participating-gpus.patch. Signed-off-by: liyingli <liyingli@amd.com>
Infera v0.3.3 (https://github.com/AMD-AGI/Infera/releases/tag/v0.3.3) is the first release whose Linux x86_64 wheel bundles infera-router (SemiAnalysisAI#223) and whose worker advertises the srt-slurm-allocated KV-events port (SemiAnalysisAI#222). Signed-off-by: liyingli <liyingli@amd.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package. 中文大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要，改动通常也很小。请让你的 agent 清理 PR，使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。 |
llying-001
added
the
full-sweep-fail-fast
|
Dispatched trusted sweep run for approved external revision 已为获批的外部提交 |
functionstackx
removed
the
full-sweep-fail-fast
There was a problem hiding this comment.
no patches to engines plz, AMD has merge perms on SGLang upstream
@chunfangamd is happy to explain to u about the ground rules on when to sweep and PR to be respectful of everyone else on this cluster
Hi, @functionstackx. Sorry about that. I did the sweep before I fully understood the ground rules. Going forward, I’ll make sure to get the patch into SGLang upstream first before using the cluster. |
amd-infera 0.3.3 is on PyPI with the same Linux x86_64 wheel as the GitHub release (bundled infera-router). Install it by version from PyPI instead of --find-links on the release page; the bundled-router check stays, so a fallback to the pure-Python wheel still fails at setup. Checked in the pinned 20261018 image and the 20261101 nightly: pip picks the manylinux_2_28 wheel and the router resolves. Signed-off-by: liyingli <liyingli@amd.com>
Signed-off-by: liyingli <liyingli@amd.com> # Conflicts: # inferencex-e2e/perf-changelog.yaml
|
I will temporarily close this PR. @llying-001, could you please remove the patches, complete the local dev experiment, and push upstream? Then, you're warmly welcome to reopen this PR. We are monitoring our CI queue time. Thanks for understanding! |
Adds
glm5.2-fp4-mi355x-infera-sglang-agentic-disagg: GLM-5.2 MXFP4 AgentX on MI355X with stock SGLang prefill/decode disaggregation behind AMD's Infera KV-aware router, as a new frameworkinfera-sglang. Infera is a deployment layer in the sense ofdynamo-sglang: each worker is SGLang's own server, started throughpython3 -m infera.engine.sglang, which forwards the SGLang flags unchanged and registers the worker in etcd; the router discovers the workers there, routes prefill on the cached prefix each worker reports through SGLang's ZMQ KV events, keeps a session on one prefill worker, and balances decode DP ranks.Status
infera-router(Infera#223) and the worker advertises the KV-events port srtctl allocates (Infera#222).infera-sglang.shinstalls it from PyPI at the master's router version and fails unless the bundled router resolves.frontend.type: infera— NVIDIA/srt-slurm#630, carried asrunners/srt-slurm/patches/630-infera-frontend.patch: Update README with new image and remove picture tag #630's source change, which applies to the pinned v2.66.1 (96f1111a; Update README with new image and remove picture tag #630 is written against v2.65.259920ea1, and the four commits between them do not touch the frontend) after573-participating-gpus.patch. Update README with new image and remove picture tag #630 is not merged yet, so this is a backport of an open upstream PR.configs/infera-sglang-patches/apply.shand covered bydocs/waiver/4629.md: #33059, #46830, #46820, #47184 (rebase of #21194) and issue #46829. Each fix is anchored, idempotent and marker-checked; a fix that does not land fails the job. They go when a pinned nightly carries them.Validation on MI355X (AMD AUS cluster, InferenceX CI path)
Run through the InferenceX launcher on the smci355 cluster record (AMD's internal mirror of this repo's MI355X CI): the stock image above, Infera installed and the five fixes applied by
infera-sglang.shat job start, srtctl v2.65.2 with the #630 patch, golden AL 3.61 injected by the driver. AgentX results are the launcher's own result JSON; GSM8K is the multi-node agentic eval job (real acceptance, 1319 questions).The KV-events path was checked in the router logs: each prefill publishes on its allocated port (the two co-located 2P1D prefills on 3000 and 3001), the router subscribes to exactly those endpoints, and 59–66% of prefill picks hit a cached prefix.
Known: in two of eleven bring-ups, a decode worker's RDMA registration returned ENOMEM on part of its KV pool; Infera's start-up PD probe caught it and the job failed at start-up rather than serving. The retry passed. Root cause not identified yet (the MR count is far below the device limit).
Configuration
Image
lmsysorg/sglang-rocm:v0.5.22-rocm720-mi35x-20261018pinned by digest;router: { name: infera-router, version: "0.3.3" };kv-p2p-transfer: mooncake;dram-utilization: 0.85.Prefill: layers 20/20/20/18, overlap scheduler off (PP), 8k chunks, shortest-prefill-first, write-back HiCache at 250 GB per stage. Both roles: EAGLE 5/1/6, FP8 KV, Triton DSA backends,
index_share_for_mtp_iteration: false,--disable-custom-all-reduce. Mooncake registers VRAM the peer way (MOONCAKE_DISABLE_HIP_DMABUF=1, 1 GiB regions, as ATOM P/D on this cluster) and the host's Ionic verbs provider is mounted read-only (as Kimi-K3 vLLM). Router: kv-aware, prefill session affinity onX-Session-ID(AIPERF_HTTP_X_SESSION_ID_FROM_CORRELATION_ID), decode input demand.Changes
infx/srt_slurm/synthetic_acceptance.py:infera-sglangmaps to the SGLang engine.benchmarks/multi_node/srt-slurm-recipes/configs/infera-sglang.sh, registered inINSTALLERSasrouter: infera-router: installsamd-inferaat the master router version from PyPI with--no-deps(onlynats-pyis missing from the image), checks that the bundledinfera-routerresolves, then runsinfera-sglang-patches/apply.sh.benchmarks/multi_node/srt-slurm-recipes/configs/infera-sglang-patches/: the five fixes andapply.sh.docs/waiver/4629.md: the engine-patch waiver.configs/amd-master.yaml: the new key.benchmarks/multi_node/srt-slurm-recipes/glm5.2/infera-sglang/mi355x-fp4/agentx/disagg-variants.yaml: base plusoverride_1p1d,override_1p1d_d4,override_2p1d.runners/srt-slurm/patches/630-infera-frontend.patchand its README row.perf-changelog.yaml: one block for the key.infx/testssuite has no failures beyond those ofmainon the same node.The dashboard label is SemiAnalysisAI/InferenceX-app#1542.
AI model disclosure
Prepared with Claude Opus 5.5, running in Claude Code (earlier revisions in Cursor). It wrote the code, recipe, configuration, ran the validation above and wrote this description; the submitter reviewed them.
中文
新增
glm5.2-fp4-mi355x-infera-sglang-agentic-disagg：MI355X 上的 GLM-5.2 MXFP4 AgentX，原样 SGLang 的 prefill/decode 分离部署，前面是 AMD Infera 的 KV-aware router，新 framework 名为infera-sglang。Infera 与dynamo-sglang一样是部署层：每个 worker 都是 SGLang 自己的服务，由python3 -m infera.engine.sglang拉起并登记到 etcd；router 从 etcd 发现 worker，按 SGLang ZMQ KV 事件上报的已缓存前缀为 prefill 选路，会话固定在一个 prefill worker 上，并均衡 decode 的各 DP rank。状态
infera-router（Infera#223），worker 登记 srtctl 分配的 KV 事件端口（Infera#222）。infera-sglang.sh按 master 的 router 版本从 PyPI 安装，并检查自带的 router 能找到。frontend.type: infera（feat(frontends): add an infera frontend for etcd-registered SGLang workers NVIDIA/srt-slurm#630）以补丁携带，就是 Update README with new image and remove picture tag #630 的源码改动，按顺序打在固定的 v2.66.1（96f1111a）上；Update README with new image and remove picture tag #630 尚未合入。infera-sglang-patches/apply.sh在作业启动时打上，见docs/waiver/4629.md；任一修复打不上，作业即失败。MI355X 实测（AMD AUS 集群，走 InferenceX launcher 的 CI 流程，原样镜像，作业启动时安装 Infera 并打补丁）：