Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
e6a34cf
feat(infx): register the infera-sglang framework and its installer
llying-001 Oct 28, 2026
c956034
feat(glm5.2): add the MI355X infera-sglang AgentX P/D curve
llying-001 Oct 28, 2026
996ea04
feat(infera-sglang): apply the SGLang fixes the pinned image lacks, w…
llying-001 Oct 29, 2026
a898651
fix(srt-slurm patch): backport the infera frontend to the pinned srtctl
llying-001 Oct 30, 2026
54b5911
Merge remote-tracking branch 'upstream/main' into llying/infera-sglang
llying-001 Oct 30, 2026
4a40001
revert(srt-slurm patch): drop the ae3d6877 backport now that main pin…
llying-001 Oct 30, 2026
a7aed81
Merge upstream/main; infera-router 0.3.3
llying-001 Nov 2, 2026
2c1fcb7
feat(infera-sglang): install amd-infera from PyPI
llying-001 Nov 3, 2026
69be557
Merge remote-tracking branch 'upstream/main' into llying/infera-sglang
llying-001 Nov 3, 2026
5263461
Merge remote-tracking branch 'upstream/main' into llying/infera-sglang
llying-001 Nov 13, 2026
2414dbe
chore(srt-slurm patch): refresh the infera frontend to NVIDIA/srt-slu…
llying-001 Nov 13, 2026
eb0bec2
feat(infera-sglang): drop the SGLang fixes that are now upstream
llying-001 Nov 13, 2026
20faef9
feat(infera-sglang): serve the stock SGLang image unpatched
llying-001 Nov 13, 2026
3bbcdd2
feat(glm5.2): pin the MI355X infera-sglang curve to the v0.5.23 ROCm …
llying-001 Nov 14, 2026
8b1c9c3
Merge remote-tracking branch 'upstream/main' into llying/infera-sglang
llying-001 Nov 17, 2026
fabd6a8
fix(infera-sglang): lift RLIMIT_MEMLOCK in the worker's shell before …
llying-001 Nov 17, 2026
89594d7
chore(srt-slurm patch): refresh the infera frontend to NVIDIA/srt-slu…
llying-001 Nov 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
#!/usr/bin/env bash
# Install Infera, at the master config's router version, into the stock SGLang image for the
# infera-sglang points. amd-infera's Linux x86_64 wheel on PyPI bundles the infera-router
# binary next to the infera.engine.sglang worker wrapper, so the master's router version is the
# Infera version. Only nats-py is missing from the lmsysorg ROCm image; --no-deps keeps every
# package the image ships (torch, sglang, transformers, ...) at its pinned version.
set -eo pipefail
source /infmax-workspace/benchmarks/check_env.sh
check_env_vars ROUTER_VERSION
pip_install=(python3 -m pip install)
if python3 -m pip install --help 2>/dev/null | grep -q -- --break-system-packages; then
pip_install+=(--break-system-packages)
fi
"${pip_install[@]}" --quiet --no-cache-dir --no-deps \
"nats-py==2.16.0" \
"amd-infera==${ROUTER_VERSION}"
# The router the launcher will exec: the wheel's bundled binary, not one on PATH. Fails if pip
# fell back to the pure-Python wheel (a platform outside manylinux_2_28 x86_64).
python3 -c "import infera.engine.sglang; from infera.server.launch_rust import _find_binary; _find_binary()"
# Lift RLIMIT_MEMLOCK's soft limit to the hard limit in the shell that starts the worker
# ($PPID: srtctl runs this script, then the worker, from one shell). The RDMA driver charges
# every KV registration against the soft limit before falling back to peer memory, and the
# remapped container root has no CAP_IPC_LOCK, so the 8 GiB limit Slurm propagates fails
# Mooncake's concurrent 1 GiB registrations with ENOMEM. Infera does this itself from
# AMD-AGI/Infera#239; drop this once the router version carries it.
python3 - "$PPID" <<'PY'
import resource, sys
pid = int(sys.argv[1])
soft, hard = resource.prlimit(pid, resource.RLIMIT_MEMLOCK)
if soft != hard:
resource.prlimit(pid, resource.RLIMIT_MEMLOCK, (hard, hard))
print(f"RLIMIT_MEMLOCK of pid {pid}: soft {soft} -> {hard}")
PY
Original file line number Diff line number Diff line change
@@ -0,0 +1,170 @@
# GLM-5.2-MXFP4 AgentX on MI355X: stock SGLang P/D behind the Infera router.
#
# Every worker is `python3 -m infera.engine.sglang`, which starts SGLang's own server with
# the flags below and registers it in the job's etcd; `infera-router` discovers the
# workers there (srt-slurm `frontend.type: infera`) and routes kv-aware, with prefill
# session affinity and decode input demand. infera-sglang.sh installs Infera at the
# master config's router version.
#
# Prefill: TP1 x PP4 on four GPUs, 20/20/20/18 layers (the last stage also holds the LM
# head and the NextN layer), overlap scheduler off as PP requires, 8k chunks,
# shortest-prefill-first, write-back HiCache at 250 GB per stage. Prefill carries the
# decode's EAGLE flags so the KV transfer includes the draft layer. Decode: TP + DP
# attention over its GPUs, EAGLE MTP 5/1/6. KV moves over Mooncake on the eight rails.
schema: 2
base:
name: mi355x-glm52-infera-sglang-agentx-disagg
slurm:
time_limit: "08:00:00"
resources:
gpu_type: mi355x
gpus_per_node: 8
setup_script: infera-sglang.sh
frontend:
type: infera
enable_multiple_frontends: false
args:
router-policy: kv-aware
kv-prefill-overlap-weight: 20
kv-decode-overlap-weight: 2
session-affinity: prefill
pd-prefill-guard-release: completion
decode-input-demand: "on"
engine: sglang
roles:
prefill:
nodes: 1
workers: 1
gpus: 4
kv_events: true
env:
<<: &environment
PYTHONUNBUFFERED: "1"
PYTHONHASHSEED: "0"
# srtctl's pre-download sees only HF_HOME, so the node-local cache is mounted at $HF_HOME/hub.
HF_HOME: /mnt/hf_home
HF_HUB_CACHE: /mnt/hf_home/hub
GLOO_SOCKET_IFNAME: "@fabric.socket-ifname"
NCCL_SOCKET_IFNAME: "@fabric.socket-ifname"
NCCL_IB_DISABLE: "1"
NCCL_IGNORE_CPU_AFFINITY: "1"
HSA_NO_SCRATCH_RECLAIM: "1"
HIP_FORCE_DEV_KERNARG: "1"
AITER_USE_FLYDSL_MOE_SORTING: "1"
SAFETENSORS_FAST_GPU: "1"
SGLANG_DP_USE_GATHERV: "1"
SGLANG_OPT_USE_TOPK_V2: "true"
SGLANG_OPT_USE_JIT_KERNEL_GROUPED_TOPK: "0"
SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT: "1800"
SGLANG_DISAGGREGATION_WAITING_TIMEOUT: "1800"
SGLANG_TIMEOUT_KEEP_ALIVE: "900"
INFERA_ENGINE_READY_TIMEOUT: "3600"
# Same-host HIP IPC cannot reach a peer node; cross-node KV goes over RDMA.
MC_DISABLE_HIP: "1"
MC_GID_INDEX: "@fabric.rdma-gid-index"
# Rail-isolated fabric: each transfer uses the NIC of the same name on both ends.
MC_ENABLE_DEST_DEVICE_AFFINITY: "1"
# Register VRAM the peer way, not through dma-buf, as ATOM P/D does on this cluster;
# 1 GiB regions keep every registration small.
MOONCAKE_DISABLE_HIP_DMABUF: "1"
MC_MAX_MR_SIZE: "1073741824"
RDMAV_FORK_SAFE: "1"
SGLANG_HICACHE_HOST_REGISTER_CHUNK_GB: "512"
SGLANG_PP_LAYER_PARTITION: "20,20,20,18"
args:
<<: &server
served-model-name: amd/GLM-5.2-MXFP4
trust-remote-code: true
disaggregation-transfer-backend: mooncake
disaggregation-ib-device: rdma0,rdma1,rdma2,rdma3,rdma4,rdma5,rdma6,rdma7
ep-size: 1
# AITER custom all-reduce garbles decode output under DP attention with CUDA graphs.
disable-custom-all-reduce: true
kv-cache-dtype: fp8_e4m3
dsa-prefill-backend: triton
dsa-decode-backend: triton
max-running-requests: 256
json-model-override-args: '{"index_share_for_mtp_iteration":false}'
enable-fused-qk-norm-rope: true
enable-aiter-allreduce-fusion: true
reasoning-parser: glm45
tool-call-parser: glm47
enable-metrics: true
enable-cache-report: true
watchdog-timeout: 3600
speculative-algorithm: EAGLE
speculative-num-steps: 5
speculative-eagle-topk: 1
speculative-num-draft-tokens: 6
tp-size: 1
pp-size: 4
disable-overlap-schedule: true
chunked-prefill-size: 8192
max-prefill-tokens: 8192
mem-fraction-static: 0.88
schedule-policy: shortest-prefill-first
enable-hierarchical-cache: true
hicache-size: 250
hicache-ratio: 1.5
hicache-write-policy: write_back
hicache-io-backend: kernel
hicache-mem-layout: page_first
decode:
nodes: 1
workers: 1
gpus: 8
env: *environment
args:
<<: *server
tp-size: 8
dp-size: 8
enable-dp-attention: true
chunked-prefill-size: 32768
mem-fraction-static: 0.92
cuda-graph-max-bs-decode: 256
tokenizer-worker-num: 4
sbatch_directives:
cpus-per-task: "128"
mem: "0"
srun_options:
mem: "0"
container-writable: ""
container-remap-root: ""
extra_mount:
# The node-local hub cache the single-node GLM-5.2 points already fill.
- /var/lib/hf-hub-cache:/mnt/hf_home/hub
# Match the host's Ionic provider without replacing the image's verbs core.
- /usr/lib/x86_64-linux-gnu/libibverbs/libionic-rdmav34.so:/usr/lib/x86_64-linux-gnu/libibverbs/libionic-rdmav34.so:ro
health_check:
# Cold start: weight load, the 1 TB pinned HiCache pool per prefill and graph capture.
max_attempts: 1080
interval_seconds: 5
benchmark:
env:
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: "sglang:"
# infera-router keys session affinity on X-Session-ID.
AIPERF_HTTP_X_SESSION_ID_FROM_CORRELATION_ID: "1"
TOKENIZERS_PARALLELISM: "false"
TRANSFORMERS_VERBOSITY: error

# One PP4 prefill and an 8-GPU decode (12 GPUs). Decode balances requests held per rank:
# under DP attention a step waits for the busiest rank.
override_1p1d:
frontend:
args:
decode-request-weight: 1000000

# One PP4 prefill and a 4-GPU decode (8 GPUs). KV-bound, so token-only decode demand.
override_1p1d_d4:
roles:
decode:
gpus: 4
args:
tp-size: 4
dp-size: 4

# Two PP4 prefills on one node and an 8-GPU decode (16 GPUs).
override_2p1d:
roles:
prefill:
workers: 2
75 changes: 75 additions & 0 deletions inferencex-e2e/configs/amd-master.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -777,6 +777,81 @@ glm5.2-fp4-mi355x-sglang-agentic-mtp:
- { tp: 4, ep: 4, dcp-size: 4, kv-offloading: none, conc-list: [16, 24, 40, 48], spec-decoding: mtp, srt-recipe: agentic.yaml }
- { tp: 8, ep: 1, kv-offloading: none, conc-list: [1, 2, 4], spec-decoding: mtp, srt-recipe: agentic.yaml }

# GLM-5.2 MXFP4 AgentX on MI355X: stock SGLang prefill/decode disaggregation behind the
# Infera router (infera-sglang). Workers start through infera.engine.sglang and register in
# etcd; the router routes kv-aware with prefill session affinity and decode input demand.
# Prefill is TP1 x PP4 on four GPUs with write-back HiCache (250 GB per stage); decode is
# TP + DP attention; EAGLE MTP 5/1/6 on both roles; KV over Mooncake. Three shapes:
# (1) 1P1D, 8-GPU decode (12 GPUs), decode request weight 1,000,000, conc [32, 48, 64, 88];
# (2) 1P1D, 4-GPU decode (8 GPUs), conc [64, 80];
# (3) 2P1D, two prefills on one node and an 8-GPU decode (16 GPUs), conc [176].
glm5.2-fp4-mi355x-infera-sglang-agentic-disagg:
image: lmsysorg/sglang-rocm:v0.5.23-rocm10-mi35x-20261112@sha256:d62fd730ec3f962073bd0e303942f207c178a0d953ee7c9ef9760624d5d532f0

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you mean lmsysorg/sglang-rocm:v0.5.23-rocm720-mi35x-20261117, right?

model: amd/GLM-5.2-MXFP4
model-prefix: glm5.2
runner: cluster:mi355x-amds
precision: fp4
framework: infera-sglang
router: { name: infera-router, version: "0.3.3" }
kv-p2p-transfer: mooncake
multinode: true
disagg: true
srt-recipe-dir: glm5.2/infera-sglang/mi355x-fp4/agentx
scenarios:
agentic-coding:
# A PP4 prefill pins 4 x 250 GB of KV plus the DSA indexer's host pool, about 1.23 TB
# of the 4-GPU share.
- dram-utilization: 0.85
search-space:
- spec-decoding: mtp
conc-list: [32, 48, 64, 88]
kv-offloading: dram
kv-offload-backend: { name: hicache }
srt-recipe: disagg-variants.yaml:override_1p1d
prefill:
num-worker: 1
tp: 1
pp: 4
ep: 1
dp-attn: false
decode:
num-worker: 1
tp: 8
ep: 1
dp-attn: true
- spec-decoding: mtp
conc-list: [64, 80]
kv-offloading: dram
kv-offload-backend: { name: hicache }
srt-recipe: disagg-variants.yaml:override_1p1d_d4
prefill:
num-worker: 1
tp: 1
pp: 4
ep: 1
dp-attn: false
decode:
num-worker: 1
tp: 4
ep: 1
dp-attn: true
- spec-decoding: mtp
conc-list: [176]
kv-offloading: dram
kv-offload-backend: { name: hicache }
srt-recipe: disagg-variants.yaml:override_2p1d
prefill:
num-worker: 2
tp: 1
pp: 4
ep: 1
dp-attn: false
decode:
num-worker: 1
tp: 8
ep: 1
dp-attn: true

# GLM-5.2 FP8 agentic-coding benchmark on MI355X via SGLang with MTP speculative
# decoding: the FP8 precision sibling of glm5.2-fp4-mi355x-sglang-agentic-mtp,
# serving the upstream zai-org/GLM-5.2-FP8 checkpoint (~756 GB block-FP8).
Expand Down
1 change: 1 addition & 0 deletions inferencex-e2e/infx/srt_slurm/synthetic_acceptance.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
"dynamo-vllm": "vllm",
"llmd-vllm": "vllm",
"dynamo-sglang": "sglang",
"infera-sglang": "sglang",
"trt": "trtllm",
"dynamo-trt": "trtllm",
"atom": "atom",
Expand Down
1 change: 1 addition & 0 deletions inferencex-e2e/infx/srt_slurm/workload.py
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,7 @@
"tokenspeed-nightly.sh": ("router", "smg"),
"glm5.3-tilert-rocm.sh": ("router", "tilert-pd-router"),
"dsv41flash-tilert-nv.sh": ("router", "tilert-pd-router"),
"infera-sglang.sh": ("router", "infera-router"),
"vllm-mooncake.sh": ("kv-offload-backend", "mooncake"),
"vllm-router.sh": ("router", "vllm-router"),
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ def test_an_installer_fails_without_the_version_the_binder_writes(tmp_path, scri
("vllm-mooncake.sh", {"KV_OFFLOAD_BACKEND_VERSION": "9.8.7"},
{"mooncake-transfer-engine-cuda13==9.8.7"}),
("glm5.3-tilert-rocm.sh", {"ROUTER_VERSION": "9.8.7", "TILERT_ROLE": "prefill"}, {"tilert==9.8.7"}),
("infera-sglang.sh", {"ROUTER_VERSION": "9.8.7"}, {"--no-deps", "amd-infera==9.8.7"}),
]) # fmt: skip
def test_an_installer_installs_exactly_the_bound_version(tmp_path, script, env, expected):
result, installs = run(tmp_path, script, env)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -406,6 +406,16 @@ def test_kimi_curve_requires_an_explicit_supported_sampler(
"SGLANG_SIMULATE_ACC_TOKEN_MODE": "real-draft-token",
},
),
(
"infera-sglang",
{"speculative-algorithm": "EAGLE", "speculative-num-steps": 3},
{},
{
"SGLANG_SIMULATE_ACC_LEN": "2.4",
"SGLANG_SIMULATE_ACC_METHOD": "match-expected",
"SGLANG_SIMULATE_ACC_TOKEN_MODE": "real-draft-token",
},
),
(
"trt",
{
Expand Down
9 changes: 9 additions & 0 deletions inferencex-e2e/perf-changelog.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12941,3 +12941,12 @@
- "Place the B300 Dynamo + vLLM AgentX frontend on the allocation head for eval-only jobs."
- "Capture DSpark step sizes at multiples of five and six through 40 for aggregated TP4 concurrency 1."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4585

- config-keys:
- glm5.2-fp4-mi355x-infera-sglang-agentic-disagg
scenario-type:
- agentic-coding
description:
- "Add GLM-5.2 MXFP4 AgentX on MI355X with stock, unpatched SGLang (lmsysorg/sglang-rocm:v0.5.23-rocm10-mi35x-20261112, pinned by digest) prefill/decode disaggregation behind the Infera router (framework infera-sglang, infera-router 0.3.3 installed by infera-sglang.sh). Workers start through infera.engine.sglang and register in etcd; the router routes kv-aware with prefill session affinity and decode input demand."
- "Prefill TP1 x PP4 (layers 20/20/20/18) with write-back HiCache at 250 GB per stage; decode TP + DP attention; EAGLE MTP 5/1/6 on both roles; Mooncake KV transfer. Shapes: 1P1D with an 8-GPU decode at concurrency 32, 48, 64, 88 (decode request weight 1,000,000); 1P1D with a 4-GPU decode at 64, 80; 2P1D at 176."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4629
Loading
Loading