Repository navigation
[AMD][MI355X] Add GLM-5.2 MXFP4 AgentX P/D behind the Infera router (infera-sglang) / [AMD][MI355X] 新增 GLM-5.2 MXFP4 AgentX PD 分离部署(Infera router,infera-sglang) #4629
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
+823
−0
Closed
Changes from all commits
Commits
Show all changes
17 commits
Select commit
Hold shift + click to select a range
e6a34cf
feat(infx): register the infera-sglang framework and its installer
llying-001 c956034
feat(glm5.2): add the MI355X infera-sglang AgentX P/D curve
llying-001 996ea04
feat(infera-sglang): apply the SGLang fixes the pinned image lacks, w…
llying-001 a898651
fix(srt-slurm patch): backport the infera frontend to the pinned srtctl
llying-001 54b5911
Merge remote-tracking branch 'upstream/main' into llying/infera-sglang
llying-001 4a40001
revert(srt-slurm patch): drop the ae3d6877 backport now that main pin…
llying-001 a7aed81
Merge upstream/main; infera-router 0.3.3
llying-001 2c1fcb7
feat(infera-sglang): install amd-infera from PyPI
llying-001 69be557
Merge remote-tracking branch 'upstream/main' into llying/infera-sglang
llying-001 5263461
Merge remote-tracking branch 'upstream/main' into llying/infera-sglang
llying-001 2414dbe
chore(srt-slurm patch): refresh the infera frontend to NVIDIA/srt-slu…
llying-001 eb0bec2
feat(infera-sglang): drop the SGLang fixes that are now upstream
llying-001 20faef9
feat(infera-sglang): serve the stock SGLang image unpatched
llying-001 3bbcdd2
feat(glm5.2): pin the MI355X infera-sglang curve to the v0.5.23 ROCm …
llying-001 8b1c9c3
Merge remote-tracking branch 'upstream/main' into llying/infera-sglang
llying-001 fabd6a8
fix(infera-sglang): lift RLIMIT_MEMLOCK in the worker's shell before …
llying-001 89594d7
chore(srt-slurm patch): refresh the infera frontend to NVIDIA/srt-slu…
llying-001 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
33 changes: 33 additions & 0 deletions
33
inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/infera-sglang.sh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| #!/usr/bin/env bash | ||
| # Install Infera, at the master config's router version, into the stock SGLang image for the | ||
| # infera-sglang points. amd-infera's Linux x86_64 wheel on PyPI bundles the infera-router | ||
| # binary next to the infera.engine.sglang worker wrapper, so the master's router version is the | ||
| # Infera version. Only nats-py is missing from the lmsysorg ROCm image; --no-deps keeps every | ||
| # package the image ships (torch, sglang, transformers, ...) at its pinned version. | ||
| set -eo pipefail | ||
| source /infmax-workspace/benchmarks/check_env.sh | ||
| check_env_vars ROUTER_VERSION | ||
| pip_install=(python3 -m pip install) | ||
| if python3 -m pip install --help 2>/dev/null | grep -q -- --break-system-packages; then | ||
| pip_install+=(--break-system-packages) | ||
| fi | ||
| "${pip_install[@]}" --quiet --no-cache-dir --no-deps \ | ||
| "nats-py==2.16.0" \ | ||
| "amd-infera==${ROUTER_VERSION}" | ||
| # The router the launcher will exec: the wheel's bundled binary, not one on PATH. Fails if pip | ||
| # fell back to the pure-Python wheel (a platform outside manylinux_2_28 x86_64). | ||
| python3 -c "import infera.engine.sglang; from infera.server.launch_rust import _find_binary; _find_binary()" | ||
| # Lift RLIMIT_MEMLOCK's soft limit to the hard limit in the shell that starts the worker | ||
| # ($PPID: srtctl runs this script, then the worker, from one shell). The RDMA driver charges | ||
| # every KV registration against the soft limit before falling back to peer memory, and the | ||
| # remapped container root has no CAP_IPC_LOCK, so the 8 GiB limit Slurm propagates fails | ||
| # Mooncake's concurrent 1 GiB registrations with ENOMEM. Infera does this itself from | ||
| # AMD-AGI/Infera#239; drop this once the router version carries it. | ||
| python3 - "$PPID" <<'PY' | ||
| import resource, sys | ||
| pid = int(sys.argv[1]) | ||
| soft, hard = resource.prlimit(pid, resource.RLIMIT_MEMLOCK) | ||
| if soft != hard: | ||
| resource.prlimit(pid, resource.RLIMIT_MEMLOCK, (hard, hard)) | ||
| print(f"RLIMIT_MEMLOCK of pid {pid}: soft {soft} -> {hard}") | ||
| PY |
170 changes: 170 additions & 0 deletions
170
.../multi_node/srt-slurm-recipes/glm5.2/infera-sglang/mi355x-fp4/agentx/disagg-variants.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,170 @@ | ||
| # GLM-5.2-MXFP4 AgentX on MI355X: stock SGLang P/D behind the Infera router. | ||
| # | ||
| # Every worker is `python3 -m infera.engine.sglang`, which starts SGLang's own server with | ||
| # the flags below and registers it in the job's etcd; `infera-router` discovers the | ||
| # workers there (srt-slurm `frontend.type: infera`) and routes kv-aware, with prefill | ||
| # session affinity and decode input demand. infera-sglang.sh installs Infera at the | ||
| # master config's router version. | ||
| # | ||
| # Prefill: TP1 x PP4 on four GPUs, 20/20/20/18 layers (the last stage also holds the LM | ||
| # head and the NextN layer), overlap scheduler off as PP requires, 8k chunks, | ||
| # shortest-prefill-first, write-back HiCache at 250 GB per stage. Prefill carries the | ||
| # decode's EAGLE flags so the KV transfer includes the draft layer. Decode: TP + DP | ||
| # attention over its GPUs, EAGLE MTP 5/1/6. KV moves over Mooncake on the eight rails. | ||
| schema: 2 | ||
| base: | ||
| name: mi355x-glm52-infera-sglang-agentx-disagg | ||
| slurm: | ||
| time_limit: "08:00:00" | ||
| resources: | ||
| gpu_type: mi355x | ||
| gpus_per_node: 8 | ||
| setup_script: infera-sglang.sh | ||
| frontend: | ||
| type: infera | ||
| enable_multiple_frontends: false | ||
| args: | ||
| router-policy: kv-aware | ||
| kv-prefill-overlap-weight: 20 | ||
| kv-decode-overlap-weight: 2 | ||
| session-affinity: prefill | ||
| pd-prefill-guard-release: completion | ||
| decode-input-demand: "on" | ||
| engine: sglang | ||
| roles: | ||
| prefill: | ||
| nodes: 1 | ||
| workers: 1 | ||
| gpus: 4 | ||
| kv_events: true | ||
| env: | ||
| <<: &environment | ||
| PYTHONUNBUFFERED: "1" | ||
| PYTHONHASHSEED: "0" | ||
| # srtctl's pre-download sees only HF_HOME, so the node-local cache is mounted at $HF_HOME/hub. | ||
| HF_HOME: /mnt/hf_home | ||
| HF_HUB_CACHE: /mnt/hf_home/hub | ||
| GLOO_SOCKET_IFNAME: "@fabric.socket-ifname" | ||
| NCCL_SOCKET_IFNAME: "@fabric.socket-ifname" | ||
| NCCL_IB_DISABLE: "1" | ||
| NCCL_IGNORE_CPU_AFFINITY: "1" | ||
| HSA_NO_SCRATCH_RECLAIM: "1" | ||
| HIP_FORCE_DEV_KERNARG: "1" | ||
| AITER_USE_FLYDSL_MOE_SORTING: "1" | ||
| SAFETENSORS_FAST_GPU: "1" | ||
| SGLANG_DP_USE_GATHERV: "1" | ||
| SGLANG_OPT_USE_TOPK_V2: "true" | ||
| SGLANG_OPT_USE_JIT_KERNEL_GROUPED_TOPK: "0" | ||
| SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT: "1800" | ||
| SGLANG_DISAGGREGATION_WAITING_TIMEOUT: "1800" | ||
| SGLANG_TIMEOUT_KEEP_ALIVE: "900" | ||
| INFERA_ENGINE_READY_TIMEOUT: "3600" | ||
| # Same-host HIP IPC cannot reach a peer node; cross-node KV goes over RDMA. | ||
| MC_DISABLE_HIP: "1" | ||
| MC_GID_INDEX: "@fabric.rdma-gid-index" | ||
| # Rail-isolated fabric: each transfer uses the NIC of the same name on both ends. | ||
| MC_ENABLE_DEST_DEVICE_AFFINITY: "1" | ||
| # Register VRAM the peer way, not through dma-buf, as ATOM P/D does on this cluster; | ||
| # 1 GiB regions keep every registration small. | ||
| MOONCAKE_DISABLE_HIP_DMABUF: "1" | ||
| MC_MAX_MR_SIZE: "1073741824" | ||
| RDMAV_FORK_SAFE: "1" | ||
| SGLANG_HICACHE_HOST_REGISTER_CHUNK_GB: "512" | ||
| SGLANG_PP_LAYER_PARTITION: "20,20,20,18" | ||
| args: | ||
| <<: &server | ||
| served-model-name: amd/GLM-5.2-MXFP4 | ||
| trust-remote-code: true | ||
| disaggregation-transfer-backend: mooncake | ||
| disaggregation-ib-device: rdma0,rdma1,rdma2,rdma3,rdma4,rdma5,rdma6,rdma7 | ||
| ep-size: 1 | ||
| # AITER custom all-reduce garbles decode output under DP attention with CUDA graphs. | ||
| disable-custom-all-reduce: true | ||
| kv-cache-dtype: fp8_e4m3 | ||
| dsa-prefill-backend: triton | ||
| dsa-decode-backend: triton | ||
| max-running-requests: 256 | ||
| json-model-override-args: '{"index_share_for_mtp_iteration":false}' | ||
| enable-fused-qk-norm-rope: true | ||
| enable-aiter-allreduce-fusion: true | ||
| reasoning-parser: glm45 | ||
| tool-call-parser: glm47 | ||
| enable-metrics: true | ||
| enable-cache-report: true | ||
| watchdog-timeout: 3600 | ||
| speculative-algorithm: EAGLE | ||
| speculative-num-steps: 5 | ||
| speculative-eagle-topk: 1 | ||
| speculative-num-draft-tokens: 6 | ||
| tp-size: 1 | ||
| pp-size: 4 | ||
| disable-overlap-schedule: true | ||
| chunked-prefill-size: 8192 | ||
| max-prefill-tokens: 8192 | ||
| mem-fraction-static: 0.88 | ||
| schedule-policy: shortest-prefill-first | ||
| enable-hierarchical-cache: true | ||
| hicache-size: 250 | ||
| hicache-ratio: 1.5 | ||
| hicache-write-policy: write_back | ||
| hicache-io-backend: kernel | ||
| hicache-mem-layout: page_first | ||
| decode: | ||
| nodes: 1 | ||
| workers: 1 | ||
| gpus: 8 | ||
| env: *environment | ||
| args: | ||
| <<: *server | ||
| tp-size: 8 | ||
| dp-size: 8 | ||
| enable-dp-attention: true | ||
| chunked-prefill-size: 32768 | ||
| mem-fraction-static: 0.92 | ||
| cuda-graph-max-bs-decode: 256 | ||
| tokenizer-worker-num: 4 | ||
| sbatch_directives: | ||
| cpus-per-task: "128" | ||
| mem: "0" | ||
| srun_options: | ||
| mem: "0" | ||
| container-writable: "" | ||
| container-remap-root: "" | ||
| extra_mount: | ||
| # The node-local hub cache the single-node GLM-5.2 points already fill. | ||
| - /var/lib/hf-hub-cache:/mnt/hf_home/hub | ||
| # Match the host's Ionic provider without replacing the image's verbs core. | ||
| - /usr/lib/x86_64-linux-gnu/libibverbs/libionic-rdmav34.so:/usr/lib/x86_64-linux-gnu/libibverbs/libionic-rdmav34.so:ro | ||
| health_check: | ||
| # Cold start: weight load, the 1 TB pinned HiCache pool per prefill and graph capture. | ||
| max_attempts: 1080 | ||
| interval_seconds: 5 | ||
| benchmark: | ||
| env: | ||
| AIPERF_REQUIRED_SERVER_METRIC_PREFIX: "sglang:" | ||
| # infera-router keys session affinity on X-Session-ID. | ||
| AIPERF_HTTP_X_SESSION_ID_FROM_CORRELATION_ID: "1" | ||
| TOKENIZERS_PARALLELISM: "false" | ||
| TRANSFORMERS_VERBOSITY: error | ||
|
|
||
| # One PP4 prefill and an 8-GPU decode (12 GPUs). Decode balances requests held per rank: | ||
| # under DP attention a step waits for the busiest rank. | ||
| override_1p1d: | ||
| frontend: | ||
| args: | ||
| decode-request-weight: 1000000 | ||
|
|
||
| # One PP4 prefill and a 4-GPU decode (8 GPUs). KV-bound, so token-only decode demand. | ||
| override_1p1d_d4: | ||
| roles: | ||
| decode: | ||
| gpus: 4 | ||
| args: | ||
| tp-size: 4 | ||
| dp-size: 4 | ||
|
|
||
| # Two PP4 prefills on one node and an 8-GPU decode (16 GPUs). | ||
| override_2p1d: | ||
| roles: | ||
| prefill: | ||
| workers: 2 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you mean lmsysorg/sglang-rocm:v0.5.23-rocm720-mi35x-20261117, right?