diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/infera-sglang.sh b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/infera-sglang.sh new file mode 100755 index 0000000000..f974ca6991 --- /dev/null +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/infera-sglang.sh @@ -0,0 +1,33 @@ +#!/usr/bin/env bash +# Install Infera, at the master config's router version, into the stock SGLang image for the +# infera-sglang points. amd-infera's Linux x86_64 wheel on PyPI bundles the infera-router +# binary next to the infera.engine.sglang worker wrapper, so the master's router version is the +# Infera version. Only nats-py is missing from the lmsysorg ROCm image; --no-deps keeps every +# package the image ships (torch, sglang, transformers, ...) at its pinned version. +set -eo pipefail +source /infmax-workspace/benchmarks/check_env.sh +check_env_vars ROUTER_VERSION +pip_install=(python3 -m pip install) +if python3 -m pip install --help 2>/dev/null | grep -q -- --break-system-packages; then + pip_install+=(--break-system-packages) +fi +"${pip_install[@]}" --quiet --no-cache-dir --no-deps \ + "nats-py==2.16.0" \ + "amd-infera==${ROUTER_VERSION}" +# The router the launcher will exec: the wheel's bundled binary, not one on PATH. Fails if pip +# fell back to the pure-Python wheel (a platform outside manylinux_2_28 x86_64). +python3 -c "import infera.engine.sglang; from infera.server.launch_rust import _find_binary; _find_binary()" +# Lift RLIMIT_MEMLOCK's soft limit to the hard limit in the shell that starts the worker +# ($PPID: srtctl runs this script, then the worker, from one shell). The RDMA driver charges +# every KV registration against the soft limit before falling back to peer memory, and the +# remapped container root has no CAP_IPC_LOCK, so the 8 GiB limit Slurm propagates fails +# Mooncake's concurrent 1 GiB registrations with ENOMEM. Infera does this itself from +# AMD-AGI/Infera#239; drop this once the router version carries it. +python3 - "$PPID" <<'PY' +import resource, sys +pid = int(sys.argv[1]) +soft, hard = resource.prlimit(pid, resource.RLIMIT_MEMLOCK) +if soft != hard: + resource.prlimit(pid, resource.RLIMIT_MEMLOCK, (hard, hard)) + print(f"RLIMIT_MEMLOCK of pid {pid}: soft {soft} -> {hard}") +PY diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/glm5.2/infera-sglang/mi355x-fp4/agentx/disagg-variants.yaml b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/glm5.2/infera-sglang/mi355x-fp4/agentx/disagg-variants.yaml new file mode 100644 index 0000000000..8d5c3dd7f2 --- /dev/null +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/glm5.2/infera-sglang/mi355x-fp4/agentx/disagg-variants.yaml @@ -0,0 +1,170 @@ +# GLM-5.2-MXFP4 AgentX on MI355X: stock SGLang P/D behind the Infera router. +# +# Every worker is `python3 -m infera.engine.sglang`, which starts SGLang's own server with +# the flags below and registers it in the job's etcd; `infera-router` discovers the +# workers there (srt-slurm `frontend.type: infera`) and routes kv-aware, with prefill +# session affinity and decode input demand. infera-sglang.sh installs Infera at the +# master config's router version. +# +# Prefill: TP1 x PP4 on four GPUs, 20/20/20/18 layers (the last stage also holds the LM +# head and the NextN layer), overlap scheduler off as PP requires, 8k chunks, +# shortest-prefill-first, write-back HiCache at 250 GB per stage. Prefill carries the +# decode's EAGLE flags so the KV transfer includes the draft layer. Decode: TP + DP +# attention over its GPUs, EAGLE MTP 5/1/6. KV moves over Mooncake on the eight rails. +schema: 2 +base: + name: mi355x-glm52-infera-sglang-agentx-disagg + slurm: + time_limit: "08:00:00" + resources: + gpu_type: mi355x + gpus_per_node: 8 + setup_script: infera-sglang.sh + frontend: + type: infera + enable_multiple_frontends: false + args: + router-policy: kv-aware + kv-prefill-overlap-weight: 20 + kv-decode-overlap-weight: 2 + session-affinity: prefill + pd-prefill-guard-release: completion + decode-input-demand: "on" + engine: sglang + roles: + prefill: + nodes: 1 + workers: 1 + gpus: 4 + kv_events: true + env: + <<: &environment + PYTHONUNBUFFERED: "1" + PYTHONHASHSEED: "0" + # srtctl's pre-download sees only HF_HOME, so the node-local cache is mounted at $HF_HOME/hub. + HF_HOME: /mnt/hf_home + HF_HUB_CACHE: /mnt/hf_home/hub + GLOO_SOCKET_IFNAME: "@fabric.socket-ifname" + NCCL_SOCKET_IFNAME: "@fabric.socket-ifname" + NCCL_IB_DISABLE: "1" + NCCL_IGNORE_CPU_AFFINITY: "1" + HSA_NO_SCRATCH_RECLAIM: "1" + HIP_FORCE_DEV_KERNARG: "1" + AITER_USE_FLYDSL_MOE_SORTING: "1" + SAFETENSORS_FAST_GPU: "1" + SGLANG_DP_USE_GATHERV: "1" + SGLANG_OPT_USE_TOPK_V2: "true" + SGLANG_OPT_USE_JIT_KERNEL_GROUPED_TOPK: "0" + SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT: "1800" + SGLANG_DISAGGREGATION_WAITING_TIMEOUT: "1800" + SGLANG_TIMEOUT_KEEP_ALIVE: "900" + INFERA_ENGINE_READY_TIMEOUT: "3600" + # Same-host HIP IPC cannot reach a peer node; cross-node KV goes over RDMA. + MC_DISABLE_HIP: "1" + MC_GID_INDEX: "@fabric.rdma-gid-index" + # Rail-isolated fabric: each transfer uses the NIC of the same name on both ends. + MC_ENABLE_DEST_DEVICE_AFFINITY: "1" + # Register VRAM the peer way, not through dma-buf, as ATOM P/D does on this cluster; + # 1 GiB regions keep every registration small. + MOONCAKE_DISABLE_HIP_DMABUF: "1" + MC_MAX_MR_SIZE: "1073741824" + RDMAV_FORK_SAFE: "1" + SGLANG_HICACHE_HOST_REGISTER_CHUNK_GB: "512" + SGLANG_PP_LAYER_PARTITION: "20,20,20,18" + args: + <<: &server + served-model-name: amd/GLM-5.2-MXFP4 + trust-remote-code: true + disaggregation-transfer-backend: mooncake + disaggregation-ib-device: rdma0,rdma1,rdma2,rdma3,rdma4,rdma5,rdma6,rdma7 + ep-size: 1 + # AITER custom all-reduce garbles decode output under DP attention with CUDA graphs. + disable-custom-all-reduce: true + kv-cache-dtype: fp8_e4m3 + dsa-prefill-backend: triton + dsa-decode-backend: triton + max-running-requests: 256 + json-model-override-args: '{"index_share_for_mtp_iteration":false}' + enable-fused-qk-norm-rope: true + enable-aiter-allreduce-fusion: true + reasoning-parser: glm45 + tool-call-parser: glm47 + enable-metrics: true + enable-cache-report: true + watchdog-timeout: 3600 + speculative-algorithm: EAGLE + speculative-num-steps: 5 + speculative-eagle-topk: 1 + speculative-num-draft-tokens: 6 + tp-size: 1 + pp-size: 4 + disable-overlap-schedule: true + chunked-prefill-size: 8192 + max-prefill-tokens: 8192 + mem-fraction-static: 0.88 + schedule-policy: shortest-prefill-first + enable-hierarchical-cache: true + hicache-size: 250 + hicache-ratio: 1.5 + hicache-write-policy: write_back + hicache-io-backend: kernel + hicache-mem-layout: page_first + decode: + nodes: 1 + workers: 1 + gpus: 8 + env: *environment + args: + <<: *server + tp-size: 8 + dp-size: 8 + enable-dp-attention: true + chunked-prefill-size: 32768 + mem-fraction-static: 0.92 + cuda-graph-max-bs-decode: 256 + tokenizer-worker-num: 4 + sbatch_directives: + cpus-per-task: "128" + mem: "0" + srun_options: + mem: "0" + container-writable: "" + container-remap-root: "" + extra_mount: + # The node-local hub cache the single-node GLM-5.2 points already fill. + - /var/lib/hf-hub-cache:/mnt/hf_home/hub + # Match the host's Ionic provider without replacing the image's verbs core. + - /usr/lib/x86_64-linux-gnu/libibverbs/libionic-rdmav34.so:/usr/lib/x86_64-linux-gnu/libibverbs/libionic-rdmav34.so:ro + health_check: + # Cold start: weight load, the 1 TB pinned HiCache pool per prefill and graph capture. + max_attempts: 1080 + interval_seconds: 5 + benchmark: + env: + AIPERF_REQUIRED_SERVER_METRIC_PREFIX: "sglang:" + # infera-router keys session affinity on X-Session-ID. + AIPERF_HTTP_X_SESSION_ID_FROM_CORRELATION_ID: "1" + TOKENIZERS_PARALLELISM: "false" + TRANSFORMERS_VERBOSITY: error + +# One PP4 prefill and an 8-GPU decode (12 GPUs). Decode balances requests held per rank: +# under DP attention a step waits for the busiest rank. +override_1p1d: + frontend: + args: + decode-request-weight: 1000000 + +# One PP4 prefill and a 4-GPU decode (8 GPUs). KV-bound, so token-only decode demand. +override_1p1d_d4: + roles: + decode: + gpus: 4 + args: + tp-size: 4 + dp-size: 4 + +# Two PP4 prefills on one node and an 8-GPU decode (16 GPUs). +override_2p1d: + roles: + prefill: + workers: 2 diff --git a/inferencex-e2e/configs/amd-master.yaml b/inferencex-e2e/configs/amd-master.yaml index b4d526d837..1eadc4fd16 100644 --- a/inferencex-e2e/configs/amd-master.yaml +++ b/inferencex-e2e/configs/amd-master.yaml @@ -777,6 +777,81 @@ glm5.2-fp4-mi355x-sglang-agentic-mtp: - { tp: 4, ep: 4, dcp-size: 4, kv-offloading: none, conc-list: [16, 24, 40, 48], spec-decoding: mtp, srt-recipe: agentic.yaml } - { tp: 8, ep: 1, kv-offloading: none, conc-list: [1, 2, 4], spec-decoding: mtp, srt-recipe: agentic.yaml } +# GLM-5.2 MXFP4 AgentX on MI355X: stock SGLang prefill/decode disaggregation behind the +# Infera router (infera-sglang). Workers start through infera.engine.sglang and register in +# etcd; the router routes kv-aware with prefill session affinity and decode input demand. +# Prefill is TP1 x PP4 on four GPUs with write-back HiCache (250 GB per stage); decode is +# TP + DP attention; EAGLE MTP 5/1/6 on both roles; KV over Mooncake. Three shapes: +# (1) 1P1D, 8-GPU decode (12 GPUs), decode request weight 1,000,000, conc [32, 48, 64, 88]; +# (2) 1P1D, 4-GPU decode (8 GPUs), conc [64, 80]; +# (3) 2P1D, two prefills on one node and an 8-GPU decode (16 GPUs), conc [176]. +glm5.2-fp4-mi355x-infera-sglang-agentic-disagg: + image: lmsysorg/sglang-rocm:v0.5.23-rocm10-mi35x-20261112@sha256:d62fd730ec3f962073bd0e303942f207c178a0d953ee7c9ef9760624d5d532f0 + model: amd/GLM-5.2-MXFP4 + model-prefix: glm5.2 + runner: cluster:mi355x-amds + precision: fp4 + framework: infera-sglang + router: { name: infera-router, version: "0.3.3" } + kv-p2p-transfer: mooncake + multinode: true + disagg: true + srt-recipe-dir: glm5.2/infera-sglang/mi355x-fp4/agentx + scenarios: + agentic-coding: + # A PP4 prefill pins 4 x 250 GB of KV plus the DSA indexer's host pool, about 1.23 TB + # of the 4-GPU share. + - dram-utilization: 0.85 + search-space: + - spec-decoding: mtp + conc-list: [32, 48, 64, 88] + kv-offloading: dram + kv-offload-backend: { name: hicache } + srt-recipe: disagg-variants.yaml:override_1p1d + prefill: + num-worker: 1 + tp: 1 + pp: 4 + ep: 1 + dp-attn: false + decode: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: true + - spec-decoding: mtp + conc-list: [64, 80] + kv-offloading: dram + kv-offload-backend: { name: hicache } + srt-recipe: disagg-variants.yaml:override_1p1d_d4 + prefill: + num-worker: 1 + tp: 1 + pp: 4 + ep: 1 + dp-attn: false + decode: + num-worker: 1 + tp: 4 + ep: 1 + dp-attn: true + - spec-decoding: mtp + conc-list: [176] + kv-offloading: dram + kv-offload-backend: { name: hicache } + srt-recipe: disagg-variants.yaml:override_2p1d + prefill: + num-worker: 2 + tp: 1 + pp: 4 + ep: 1 + dp-attn: false + decode: + num-worker: 1 + tp: 8 + ep: 1 + dp-attn: true + # GLM-5.2 FP8 agentic-coding benchmark on MI355X via SGLang with MTP speculative # decoding: the FP8 precision sibling of glm5.2-fp4-mi355x-sglang-agentic-mtp, # serving the upstream zai-org/GLM-5.2-FP8 checkpoint (~756 GB block-FP8). diff --git a/inferencex-e2e/infx/srt_slurm/synthetic_acceptance.py b/inferencex-e2e/infx/srt_slurm/synthetic_acceptance.py index 75c3bf1f2c..a6356e1b3d 100644 --- a/inferencex-e2e/infx/srt_slurm/synthetic_acceptance.py +++ b/inferencex-e2e/infx/srt_slurm/synthetic_acceptance.py @@ -29,6 +29,7 @@ "dynamo-vllm": "vllm", "llmd-vllm": "vllm", "dynamo-sglang": "sglang", + "infera-sglang": "sglang", "trt": "trtllm", "dynamo-trt": "trtllm", "atom": "atom", diff --git a/inferencex-e2e/infx/srt_slurm/workload.py b/inferencex-e2e/infx/srt_slurm/workload.py index 14c19ef106..cd6f8062f5 100644 --- a/inferencex-e2e/infx/srt_slurm/workload.py +++ b/inferencex-e2e/infx/srt_slurm/workload.py @@ -102,6 +102,7 @@ "tokenspeed-nightly.sh": ("router", "smg"), "glm5.3-tilert-rocm.sh": ("router", "tilert-pd-router"), "dsv41flash-tilert-nv.sh": ("router", "tilert-pd-router"), + "infera-sglang.sh": ("router", "infera-router"), "vllm-mooncake.sh": ("kv-offload-backend", "mooncake"), "vllm-router.sh": ("router", "vllm-router"), } diff --git a/inferencex-e2e/infx/tests/srt_slurm/test_setup_scripts.py b/inferencex-e2e/infx/tests/srt_slurm/test_setup_scripts.py index 729c664b35..0e59a6d27f 100644 --- a/inferencex-e2e/infx/tests/srt_slurm/test_setup_scripts.py +++ b/inferencex-e2e/infx/tests/srt_slurm/test_setup_scripts.py @@ -60,6 +60,7 @@ def test_an_installer_fails_without_the_version_the_binder_writes(tmp_path, scri ("vllm-mooncake.sh", {"KV_OFFLOAD_BACKEND_VERSION": "9.8.7"}, {"mooncake-transfer-engine-cuda13==9.8.7"}), ("glm5.3-tilert-rocm.sh", {"ROUTER_VERSION": "9.8.7", "TILERT_ROLE": "prefill"}, {"tilert==9.8.7"}), + ("infera-sglang.sh", {"ROUTER_VERSION": "9.8.7"}, {"--no-deps", "amd-infera==9.8.7"}), ]) # fmt: skip def test_an_installer_installs_exactly_the_bound_version(tmp_path, script, env, expected): result, installs = run(tmp_path, script, env) diff --git a/inferencex-e2e/infx/tests/srt_slurm/test_synthetic_acceptance.py b/inferencex-e2e/infx/tests/srt_slurm/test_synthetic_acceptance.py index ea700316b6..723de8c118 100644 --- a/inferencex-e2e/infx/tests/srt_slurm/test_synthetic_acceptance.py +++ b/inferencex-e2e/infx/tests/srt_slurm/test_synthetic_acceptance.py @@ -406,6 +406,16 @@ def test_kimi_curve_requires_an_explicit_supported_sampler( "SGLANG_SIMULATE_ACC_TOKEN_MODE": "real-draft-token", }, ), + ( + "infera-sglang", + {"speculative-algorithm": "EAGLE", "speculative-num-steps": 3}, + {}, + { + "SGLANG_SIMULATE_ACC_LEN": "2.4", + "SGLANG_SIMULATE_ACC_METHOD": "match-expected", + "SGLANG_SIMULATE_ACC_TOKEN_MODE": "real-draft-token", + }, + ), ( "trt", { diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index d22c264b7d..2be36e6eb1 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -12941,3 +12941,12 @@ - "Place the B300 Dynamo + vLLM AgentX frontend on the allocation head for eval-only jobs." - "Capture DSpark step sizes at multiples of five and six through 40 for aggregated TP4 concurrency 1." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4585 + +- config-keys: + - glm5.2-fp4-mi355x-infera-sglang-agentic-disagg + scenario-type: + - agentic-coding + description: + - "Add GLM-5.2 MXFP4 AgentX on MI355X with stock, unpatched SGLang (lmsysorg/sglang-rocm:v0.5.23-rocm10-mi35x-20261112, pinned by digest) prefill/decode disaggregation behind the Infera router (framework infera-sglang, infera-router 0.3.3 installed by infera-sglang.sh). Workers start through infera.engine.sglang and register in etcd; the router routes kv-aware with prefill session affinity and decode input demand." + - "Prefill TP1 x PP4 (layers 20/20/20/18) with write-back HiCache at 250 GB per stage; decode TP + DP attention; EAGLE MTP 5/1/6 on both roles; Mooncake KV transfer. Shapes: 1P1D with an 8-GPU decode at concurrency 32, 48, 64, 88 (decode request weight 1,000,000); 1P1D with a 4-GPU decode at 64, 80; 2P1D at 176." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4629 diff --git a/inferencex-e2e/runners/srt-slurm/patches/630-infera-frontend.patch b/inferencex-e2e/runners/srt-slurm/patches/630-infera-frontend.patch new file mode 100644 index 0000000000..58a9ecba0b --- /dev/null +++ b/inferencex-e2e/runners/srt-slurm/patches/630-infera-frontend.patch @@ -0,0 +1,522 @@ +diff --git a/src/srtctl/backends/base.py b/src/srtctl/backends/base.py +index 625c858..9d9d996 100644 +--- a/src/srtctl/backends/base.py ++++ b/src/srtctl/backends/base.py +@@ -294,6 +294,14 @@ class Backend(ABC): + """ + return {} + ++ def get_kv_events_config_for_mode(self, mode: WorkerMode) -> dict[str, Any] | None: ++ """``roles..kv_events`` for a worker mode over the engine's defaults; None when it publishes none. ++ ++ A worker gets an allocated KV-events port whether or not its role ++ publishes, so this, not ``Process.kv_events_port``, says whether it does. ++ """ ++ return None ++ + def get_mooncake_worker_env( + self, infra_node_ip: str, local_hostname: str, mode: WorkerMode | None = None + ) -> dict[str, str]: +diff --git a/src/srtctl/backends/sglang.py b/src/srtctl/backends/sglang.py +index 4b48b9f..766565b 100644 +--- a/src/srtctl/backends/sglang.py ++++ b/src/srtctl/backends/sglang.py +@@ -357,8 +357,9 @@ class SGLangBackend(Backend): + leader_ip = get_hostname_ip(endpoint_nodes[0], runtime.network_interface) + + # Direct frontends run the native server; Dynamo frontends run the registering worker. ++ # A frontend may name a module that wraps the native server (Infera registers it in etcd). + use_sglang = frontend.worker_launch == "direct" +- python_module = "sglang.launch_server" if use_sglang else "dynamo.sglang" ++ python_module = frontend.worker_module(self.type) or ("sglang.launch_server" if use_sglang else "dynamo.sglang") + + # Get served model name from config + served_model_name = self.get_served_model_name(runtime.model_path.name) +@@ -368,11 +369,9 @@ class SGLangBackend(Backend): + # Start with nsys prefix if provided + cmd: list[str] = list(nsys_prefix) if nsys_prefix else [] + ++ cmd.extend(["python3", "-m", python_module, *frontend.worker_launcher_args(self, process, runtime)]) + cmd.extend( + [ +- "python3", +- "-m", +- python_module, + "--model-path", + model_arg, + "--served-model-name", +diff --git a/src/srtctl/cli/mixins/worker_stage.py b/src/srtctl/cli/mixins/worker_stage.py +index 80c8fb4..a24ab17 100644 +--- a/src/srtctl/cli/mixins/worker_stage.py ++++ b/src/srtctl/cli/mixins/worker_stage.py +@@ -221,8 +221,9 @@ class WorkerStageMixin: + return {self.runtime.visible_devices_env: process.cuda_visible_devices} + + def _worker_environment_defaults(self, process: "Process") -> dict[str, str]: +- env = {"HEAD_NODE_IP": self.runtime.head_node_ip} +- if get_frontend(self.config.frontend.type).worker_launch == "dynamo": ++ frontend = get_frontend(self.config.frontend.type) ++ env = {"HEAD_NODE_IP": self.runtime.head_node_ip, **frontend.worker_environment(process, self.runtime)} ++ if frontend.worker_launch == "dynamo": + env.update(discovery_env(self.config, self.runtime)) + env.update(lifecycle_otel_env(self.config.services, self.runtime.head_node_ip, process.endpoint_mode)) + env.update( +diff --git a/src/srtctl/core/schema.py b/src/srtctl/core/schema.py +index 7b3b50a..45d1ff9 100755 +--- a/src/srtctl/core/schema.py ++++ b/src/srtctl/core/schema.py +@@ -2640,7 +2640,8 @@ class FrontendConfig: + """Frontend/router configuration. + + Attributes: +- type: Frontend type - "dynamo" (default); "sglang-router" (SGLang Model ++ type: Frontend type - "dynamo" (default) and "infera" (Infera router, ++ SGLang workers register in etcd); "sglang-router" (SGLang Model + Gateway), "vllm-router", "smg" (Shepherd Model Gateway, any backend), "llm-d" + (llm-d Endpoint Picker behind Envoy, vLLM), "atomesh", and + "tilert-router" (static routers); "sglang", "vllm", and +diff --git a/src/srtctl/frontends/AGENTS.md b/src/srtctl/frontends/AGENTS.md +index 421d45c..dc4fb00 100644 +--- a/src/srtctl/frontends/AGENTS.md ++++ b/src/srtctl/frontends/AGENTS.md +@@ -4,11 +4,11 @@ Rules for `src/srtctl/frontends/`. The Design Rules in the root `CLAUDE.md` appl + + ## Frontends and readiness + +-The frontend is the process that owns the public OpenAI port (`FRONTEND_PUBLIC_PORT`) and decides when the job is ready. Implementations live under `src/srtctl/frontends/`, register with `@register_frontend("")`, and are imported from the package `__init__`; `frontend.type` resolves through that registry (`get_frontend`, `list_frontend_types`) and nowhere else. Registered today: `dynamo` (etcd/NATS discovery), `sglang-router`, `vllm-router` and `smg` (static URL routers built on `StaticRouterFrontend`; `smg` pairs with any backend), `llm-d` (the llm-d EPP behind Envoy, on `StaticRouterFrontend`: it writes the workers into an endpoints file instead of a command line, and reaches decode workers through the P/D sidecar service it implies), `trtllm_serve` (direct aggregate or the disaggregated orchestrator), `sglang` and `vllm` (one direct worker owns the port); `none` is the services-only job with no implementation and no gate. ++The frontend is the process that owns the public OpenAI port (`FRONTEND_PUBLIC_PORT`) and decides when the job is ready. Implementations live under `src/srtctl/frontends/`, register with `@register_frontend("")`, and are imported from the package `__init__`; `frontend.type` resolves through that registry (`get_frontend`, `list_frontend_types`) and nowhere else. Registered today: `dynamo` (etcd/NATS discovery), `infera` (etcd discovery of SGLang workers started by Infera's registering launcher), `sglang-router`, `vllm-router` and `smg` (static URL routers built on `StaticRouterFrontend`; `smg` pairs with any backend), `llm-d` (the llm-d EPP behind Envoy, on `StaticRouterFrontend`: it writes the workers into an endpoints file instead of a command line, and reaches decode workers through the P/D sidecar service it implies), `trtllm_serve` (direct aggregate or the disaggregated orchestrator), `sglang` and `vllm` (one direct worker owns the port); `none` is the services-only job with no implementation and no gate. + + `Frontend` hooks: `required_backend` and `validate(config)` (recipe rules, run by `SrtConfig._validate_frontend` at load so dry-run catches them; raise `ValueError` with the user-facing message), `health_endpoint` and `parse_health` (readiness), `get_backend_health_urls` (second gate: every advertised worker URL must answer 200 before traffic), `start_frontends` (launch on `topology.frontend_nodes`, one `ManagedProcess` per node with a `step_name`), `get_frontend_args_list` (`frontend.args` to CLI). The schema knows no individual frontend: pairing and per-type rules come from these two members. + +-Two base classes cover the two ways a frontend learns about its workers. `DynamicFrontend` (`frontends/dynamic_frontend.py`) is for frontends whose workers register themselves over a discovery plane: it fronts any engine, needs no per-worker URL gate, and no worker is the public endpoint; Dynamo is its only implementation. A router binary that can also take static URLs (vLLM Router's ZMQ discovery mode) is a mode of a static router, not a dynamic frontend. `StaticRouterFrontend` (`frontends/static_router.py`) is the base for routers that take worker URLs on the command line. Subclasses set `executable`, `pd_flag`, `process_name` and override only what differs: `worker_scheme` (the backend's `direct_worker_scheme`, normally HTTP/gRPC), `worker_bootstrap_port` (the P/D port advertised next to a prefill URL), `resolve_worker_host`, `get_managed_frontend_args` (arguments derived from the allocated topology and runtime paths; a conflicting user value raises instead of being overwritten), `build_bash_preamble`, `build_router_command`, `start_process` (test seam). `collect_workers` asks `Backend.router_worker_address` for the listener node and port. Its default uses a positive `Process.http_port` on the worker node, including vLLM node-local DP pools; a headless engine can instead advertise an allocated listener on the frontend node. Keep `Process.node` physical and use `runtime.shared_model_arg` for gateway model/tokenizer paths even when workers stage weights locally. ++Two base classes cover the two ways a frontend learns about its workers. `DynamicFrontend` (`frontends/dynamic_frontend.py`) is for frontends whose workers register themselves over a discovery plane: it fronts any engine, needs no per-worker URL gate, and no worker is the public endpoint; Dynamo and Infera implement it. Infera's workers are still `direct` (SGLang's own server): `worker_module` names the registering wrapper the backend starts as `python3 -m `, `worker_launcher_args` its flags, and the backend appends the same engine flags as for any direct worker; per-process variables such as the wrapper's readiness port go through `worker_environment`, which the worker stage merges for every frontend beneath the recipe's environment, never as an `env` prefix in the command. A router binary that can also take static URLs (vLLM Router's ZMQ discovery mode) is a mode of a static router, not a dynamic frontend. `StaticRouterFrontend` (`frontends/static_router.py`) is the base for routers that take worker URLs on the command line. Subclasses set `executable`, `pd_flag`, `process_name` and override only what differs: `worker_scheme` (the backend's `direct_worker_scheme`, normally HTTP/gRPC), `worker_bootstrap_port` (the P/D port advertised next to a prefill URL), `resolve_worker_host`, `get_managed_frontend_args` (arguments derived from the allocated topology and runtime paths; a conflicting user value raises instead of being overwritten), `build_bash_preamble`, `build_router_command`, `start_process` (test seam). `collect_workers` asks `Backend.router_worker_address` for the listener node and port. Its default uses a positive `Process.http_port` on the worker node, including vLLM node-local DP pools; a headless engine can instead advertise an allocated listener on the frontend node. Keep `Process.node` physical and use `runtime.shared_model_arg` for gateway model/tokenizer paths even when workers stage weights locally. + + Readiness runs in `BenchmarkStageMixin._wait_for_service_ready`: `wait_for_model` owns timing, abort, and progress logging, and calls the frontend's `probe_ready(host, port, n_prefill, n_decode, config)` once per poll with counts from the frontend's `health_expectations(config, processes)`; then `get_backend_health_urls` are polled with `wait_for_http_endpoints`. A probe raises `requests.RequestException` while the endpoint is down and otherwise returns a `WorkerHealthResult`. `core/health.py` has the three probe shapes to build on: `probe_json_health` (a JSON worker count through `parse_health`), `probe_http_ok` (a bare 200), `probe_direct_server` (`/health` then `/v1/models`). No frontend name appears in `wait_for_model` or `_get_health_expectations`. + +diff --git a/src/srtctl/frontends/__init__.py b/src/srtctl/frontends/__init__.py +index 611e15b..cb0639d 100644 +--- a/src/srtctl/frontends/__init__.py ++++ b/src/srtctl/frontends/__init__.py +@@ -10,6 +10,7 @@ importing it from here is what makes ``frontend.type: `` resolvable. + + Supported frontend types: + - dynamo: Dynamo frontend with NATS/etcd communication ++- infera: Infera router discovering self-registering SGLang workers through etcd + - llm-d: llm-d Endpoint Picker behind Envoy, with the P/D sidecar on decode workers + - sglang: Direct sglang.launch_server for a single aggregate worker (no router) + - sglang-router: SGLang Model Gateway router in front of static workers +@@ -30,6 +31,7 @@ from srtctl.frontends.base import ( + ) + from srtctl.frontends.dynamic_frontend import DynamicFrontend + from srtctl.frontends.dynamo import DynamoFrontend ++from srtctl.frontends.infera import InferaFrontend + from srtctl.frontends.llm_d import LLMDFrontend + from srtctl.frontends.sglang import SGLangRouterFrontend + from srtctl.frontends.sglang_direct import SGLangFrontend +@@ -45,6 +47,7 @@ __all__ = [ + "DynamicFrontend", + "DynamoFrontend", + "Frontend", ++ "InferaFrontend", + "LLMDFrontend", + "SGLangFrontend", + "SGLangRouterFrontend", +diff --git a/src/srtctl/frontends/base.py b/src/srtctl/frontends/base.py +index 124a38c..1b2c879 100644 +--- a/src/srtctl/frontends/base.py ++++ b/src/srtctl/frontends/base.py +@@ -195,6 +195,25 @@ class Frontend(ABC): + """ + return frozenset() + ++ def worker_module(self, backend_type: str) -> str | None: ++ """Python module that starts a worker in place of the engine's server module, or ``None``. ++ ++ The module runs the engine's own OpenAI server on the allocated port and ++ registers it with this frontend's discovery plane (Infera's ++ ``infera.engine.sglang``). The backend still builds ``python3 -m ``, ++ then ``worker_launcher_args``, then the engine flags it would pass to its ++ own server module, unchanged. ++ """ ++ return None ++ ++ def worker_launcher_args(self, backend: Any, process: "Process", runtime: "RuntimeContext") -> list[str]: ++ """Arguments of ``worker_module`` for ``process``, ahead of the engine flags; empty without one.""" ++ return [] ++ ++ def worker_environment(self, process: "Process", runtime: "RuntimeContext") -> dict[str, str]: ++ """Per-process environment this frontend's workers need, beneath the recipe's own environment.""" ++ return {} ++ + def kv_events_subscriber( + self, process: "Process", runtime: "RuntimeContext", model_name: str + ) -> tuple[str, int, str] | None: +diff --git a/src/srtctl/frontends/dynamic_frontend.py b/src/srtctl/frontends/dynamic_frontend.py +index 22e749f..d47697c 100644 +--- a/src/srtctl/frontends/dynamic_frontend.py ++++ b/src/srtctl/frontends/dynamic_frontend.py +@@ -15,10 +15,12 @@ itself the public endpoint. + A router binary that can also take static URLs (vLLM Router's ZMQ discovery + mode) is a mode of a static router, not a dynamic frontend. + +-Dynamo is the only implementation today. A subclass sets ``type`` and ++Dynamo and Infera implement it. A subclass sets ``type`` and + ``worker_launch``, registers with ``@register_frontend``, parses its own + registration count in ``parse_health``, decides which rank serves which port, +-and launches the process in ``start_frontends``. ++and launches the process in ``start_frontends``. Infera's workers are direct ++(the engine's own server, started by the wrapper module ``worker_module`` ++names, which registers it), so it overrides the Dynamo-shaped metrics path. + """ + + from __future__ import annotations +diff --git a/src/srtctl/frontends/infera.py b/src/srtctl/frontends/infera.py +new file mode 100644 +index 0000000..c75c1f9 +--- /dev/null ++++ b/src/srtctl/frontends/infera.py +@@ -0,0 +1,324 @@ ++# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. ++# SPDX-License-Identifier: Apache-2.0 ++ ++"""Infera's router in front of SGLang workers that register themselves in etcd. ++ ++Each worker is ``python3 -m infera.engine.sglang``: it starts SGLang's own ++server (``sglang.launch_server``) on the allocated HTTP port with the flags the ++SGLang backend builds, then registers the worker's URL, role, bootstrap address, ++DP size and KV-events endpoint under an etcd prefix. ``infera-router`` (the ++Rust data plane, started through ``python3 -m infera.server --router-backend ++rust``) watches that prefix and routes over HTTP, reading KV events straight ++from SGLang's ZMQ publishers. The worker is therefore a direct worker (the ++engine's own server); the frontend is dynamic because the router learns its ++workers from etcd. ++""" ++ ++from __future__ import annotations ++ ++import logging ++import shlex ++import threading ++from typing import TYPE_CHECKING, Any, ClassVar, Literal ++ ++from srtctl.core.health import WorkerHealthResult ++from srtctl.core.slurm import get_hostname_ip, start_srun_process ++from srtctl.frontends.base import Frontend, numactl_prefix, register_frontend ++from srtctl.frontends.dynamic_frontend import DynamicFrontend ++from srtctl.frontends.static_router import setup_script_preamble ++from srtctl.ports import ETCD_CLIENT_PORT ++from srtctl.services.config import ServiceConfig ++from srtctl.services.implicit import ETCD_SERVICE_NAME, EffectiveService, infra_placement ++ ++if TYPE_CHECKING: ++ from srtctl.backends.base import WorkerMode ++ from srtctl.core.processes import ManagedProcess ++ from srtctl.core.runtime import RuntimeContext ++ from srtctl.core.topology import Process ++ ++logger = logging.getLogger(__name__) ++ ++# Both sides speak HTTP and ZMQ only, so no NATS service is implied. ++ROUTER_MANAGED_ARGS = frozenset( ++ { ++ "router-backend", ++ "host", ++ "port", ++ "discovery-backend", ++ "etcd-endpoint", ++ "request-transport", ++ "kv-event-transport", ++ } ++) ++_ROLE_BY_DISAGG_MODE = {"prefill": "prefill", "decode": "decode", "mixed": "decode"} ++# Infera's registering launcher for each engine this frontend pairs with. Only SGLang: ++# Infera also wraps vLLM (infera.engine.vllm), but a row for it would also need the vLLM ++# backend to call the worker hooks and this frontend's port answers checked against it. ++WORKER_LAUNCHERS = {"sglang": "infera.engine.sglang"} ++# The launcher's readiness listener port; unset, it binds a fixed default. ++READINESS_PORT_ENV = "INFERA_READINESS_PORT" ++# Unless this is "0", the launcher sets SGLang's KV cache to fp8_e4m3 whenever the ++# command line has no --kv-cache-dtype. ++DEFAULT_KV_FP8_ENV = "INFERA_DEFAULT_KV_FP8" ++# The release setup_script installs (configs/install-infera.sh); the router and the ++# workers must run the same one. ++VERSION_ENV = "INFERA_VERSION" ++# `python3 -m infera.server` routes kv-aware unless frontend.args says otherwise. ++DEFAULT_ROUTER_POLICY = "kv-aware" ++# kv-aware's decode balancing by in-flight input tokens, which needs no KV events; the ++# router reads this env when frontend.args leaves decode-input-demand unset. ++DECODE_INPUT_DEMAND_ENV = "INFERA_DECODE_INPUT_DEMAND" ++ ++ ++def etcd_endpoint(runtime: RuntimeContext) -> str: ++ """The job's etcd on the infra node; workers and the router must name the same one.""" ++ return f"http://{runtime.infra_node_ip}:{ETCD_CLIENT_PORT}" ++ ++ ++def decode_input_demand(config: Any) -> str: ++ """The router's decode-input-demand mode: frontend.args, else the router's env, else off.""" ++ env = {**(config.environment or {}), **(config.frontend.env or {})} ++ return str((config.frontend.args or {}).get("decode-input-demand", env.get(DECODE_INPUT_DEMAND_ENV, "off"))) ++ ++ ++@register_frontend("infera") ++class InferaFrontend(DynamicFrontend): ++ """``infera-router`` discovering ``infera.engine.sglang`` workers through etcd.""" ++ ++ type: ClassVar[str] = "infera" ++ required_backend: ClassVar[str | None] = "sglang" ++ # The worker is SGLang's own OpenAI server; the launcher only adds registration. ++ worker_launch: ClassVar[Literal["dynamo", "direct"]] = "direct" ++ ++ @property ++ def health_endpoint(self) -> str: ++ """The router lists every registered worker with its role and status here.""" ++ return "/v1/workers" ++ ++ def validate(self, config: Any) -> None: ++ clashes = sorted(ROUTER_MANAGED_ARGS & set(config.frontend.args or {})) ++ if clashes: ++ raise ValueError( ++ f"frontend.args {clashes} are set by srtctl for frontend.type: infera " ++ "(etcd discovery, HTTP requests, ZMQ KV events); remove them" ++ ) ++ for service in config.services or (): ++ if service.name == ETCD_SERVICE_NAME and service.enabled and service.external: ++ raise ValueError( ++ "frontend.type: infera registers workers in the job's own etcd; " ++ "an external etcd service is not supported" ++ ) ++ for role, backend in config.active_role_backends(): ++ if backend.is_grpc_mode(role): ++ raise ValueError(f"frontend.type: infera routes HTTP workers; roles.{role} serves gRPC") ++ policy = (config.frontend.args or {}).get("router-policy", DEFAULT_ROUTER_POLICY) ++ publishing = [ ++ role for role, backend in config.active_role_backends() if backend.get_kv_events_config_for_mode(role) ++ ] ++ if policy == "kv-aware" and not publishing: ++ if decode_input_demand(config) in ("on", "shadow"): ++ logger.warning( ++ "frontend.type: infera: no role publishes KV events, so kv-aware picks prefill workers " ++ "on load (and session affinity) alone; decode-input-demand still balances decode" ++ ) ++ else: ++ raise ValueError( ++ "frontend.type: infera routes kv-aware (infera.server's default router-policy) on KV events, " ++ "but no role publishes them, so the router would route on load alone; set kv_events: true " ++ "on the prefill (or agg) role, set frontend.args.router-policy, or turn on " ++ "decode-input-demand if kv-aware is there for decode balancing" ++ ) ++ versions = { ++ source: env[VERSION_ENV] ++ for source, env in ( ++ ("environment", config.environment), ++ ("frontend.env", config.frontend.env), ++ *((f"roles.{name}.env", role.env) for name, role in (config.roles or {}).items()), ++ ) ++ if env and VERSION_ENV in env ++ } ++ if len(set(versions.values())) > 1: ++ raise ValueError( ++ f"the Infera router and workers must run one release, but {VERSION_ENV} differs: " ++ + ", ".join(f"{source}={version}" for source, version in versions.items()) ++ ) ++ ++ def worker_metrics_path(self, backend: Any, mode: WorkerMode) -> str | None: ++ """Workers are SGLang's own servers, so metrics are the engine's, not a runtime's.""" ++ return Frontend.worker_metrics_path(self, backend, mode) ++ ++ def worker_metrics_port(self, process: Process, runtime: RuntimeContext) -> int | None: ++ """The leader rank's SGLang HTTP server carries /metrics; followers serve nothing.""" ++ if process.is_leader and process.http_port > 0: ++ return process.http_port ++ return None ++ ++ def worker_endpoint_port(self, process: Process, config: Any, runtime: RuntimeContext) -> int | None: ++ if process.is_leader and process.http_port > 0: ++ return process.http_port ++ return None ++ ++ def profiling_control_port(self, process: Process, config: Any, runtime: RuntimeContext) -> int | None: ++ return process.http_port if process.http_port > 0 else None ++ ++ def worker_ready_port(self, process: Process) -> int: ++ return process.http_port ++ ++ def implied_services(self, config: Any) -> list[EffectiveService]: ++ """etcd for worker registration; requests and KV events do not ride on NATS.""" ++ return [ ++ EffectiveService( ++ ServiceConfig(name=ETCD_SERVICE_NAME, type="etcd", placement=infra_placement(config)), ++ implicit=True, ++ reason="frontend.type infera", ++ ) ++ ] ++ ++ def worker_module(self, backend_type: str) -> str: ++ """Infera's launcher, which starts the engine's own server and registers it in etcd.""" ++ try: ++ return WORKER_LAUNCHERS[backend_type] ++ except KeyError: ++ raise ValueError(f"frontend.type: infera has no worker launcher for engine {backend_type!r}") from None ++ ++ def worker_launcher_args(self, backend: Any, process: Process, runtime: RuntimeContext) -> list[str]: ++ """Registration in the job's etcd over HTTP and ZMQ. ++ ++ Every listener the launcher opens is the allocator's. A role that publishes ++ KV events gets ``--kv-events-config`` with its allocated port from the ++ backend, and the launcher advertises that endpoint. Infera's own KV probe ++ plane, which the router does not read, stays off. ++ """ ++ args = [ ++ "--discovery-backend", ++ "etcd", ++ "--etcd-endpoint", ++ etcd_endpoint(runtime), ++ "--advertise-host", ++ get_hostname_ip(process.node, runtime.network_interface), ++ "--request-transport", ++ "http", ++ "--kv-event-transport", ++ "zmq", ++ "--kv-events", ++ "off", ++ ] ++ # Without --kv-events-config the launcher publishes anyway, on a port it picks itself. ++ if backend.get_kv_events_config_for_mode(process.endpoint_mode) is None: ++ args.append("--no-enable-kv-events") ++ return args ++ ++ def worker_environment(self, process: Process, runtime: RuntimeContext) -> dict[str, str]: ++ """The launcher's readiness port, and its fp8 KV-cache default turned off. ++ ++ The readiness listener the launcher opens once registered is the process's ++ system port: srtctl reads readiness from the router, and the system port is ++ otherwise unused without Dynamo, so co-located workers do not contend for ++ Infera's default. With the fp8 default off, the KV-cache dtype is the recipe's ++ (``kv-cache-dtype`` in the role's args), as under any other frontend. ++ """ ++ return {READINESS_PORT_ENV: str(process.sys_port), DEFAULT_KV_FP8_ENV: "0"} ++ ++ def parse_health(self, response_json: dict, expected_prefill: int, expected_decode: int) -> WorkerHealthResult: ++ """Count active registrations by role; an aggregate (``mixed``) worker counts as decode. ++ ++ An entry without ``status`` or ``disagg_mode`` does not count, so a schema ++ change in the router fails readiness instead of passing it. ++ """ ++ counts = {"prefill": 0, "decode": 0} ++ for worker in response_json.get("workers") or []: ++ role = _ROLE_BY_DISAGG_MODE.get(str(worker.get("disagg_mode"))) ++ if role is not None and worker.get("status") == "active": ++ counts[role] += 1 ++ ready = counts["prefill"] >= expected_prefill and counts["decode"] >= expected_decode ++ return WorkerHealthResult( ++ ready=ready, ++ message=( ++ f"infera-router: {counts['prefill']}/{expected_prefill} prefill, " ++ f"{counts['decode']}/{expected_decode} decode registered" ++ ), ++ prefill_ready=counts["prefill"], ++ prefill_expected=expected_prefill, ++ decode_ready=counts["decode"], ++ decode_expected=expected_decode, ++ ) ++ ++ def build_router_command(self, port: int, runtime: RuntimeContext, config: Any) -> list[str]: ++ """Infera's launcher execs the Rust data plane after resolving a Hugging Face id to a local tokenizer. ++ ++ kv-aware routing hashes prompts with the served model's tokenizer, so it ++ defaults to the checkpoint the workers load. ++ """ ++ args = dict(config.frontend.args or {}) ++ args.setdefault("router-tokenizer-path", runtime.worker_model_arg) ++ return [ ++ "python3", ++ "-m", ++ "infera.server", ++ "--router-backend", ++ "rust", ++ "--host", ++ "0.0.0.0", ++ "--port", ++ str(port), ++ "--discovery-backend", ++ "etcd", ++ "--etcd-endpoint", ++ etcd_endpoint(runtime), ++ "--request-transport", ++ "http", ++ "--kv-event-transport", ++ "zmq", ++ *self.get_frontend_args_list(args), ++ ] ++ ++ def start_process(self, **kwargs: Any) -> Any: ++ """Launch one router process; split out for tests.""" ++ return start_srun_process(**kwargs) ++ ++ def start_frontends( ++ self, ++ topology: Any, ++ runtime: RuntimeContext, ++ config: Any, ++ backend: Any, ++ backend_processes: list[Process], ++ stop_event: threading.Event | None = None, # unused: returns immediately ++ ) -> list[ManagedProcess]: ++ from srtctl.core.processes import FRONTEND_TERMINATE_TIMEOUT_SECONDS, ManagedProcess ++ ++ processes: list[ManagedProcess] = [] ++ for idx, node in enumerate(topology.frontend_nodes): ++ log_file = runtime.log_dir / f"{node}_infera_router_{idx}.out" ++ cmd = numactl_prefix(config) + self.build_router_command(topology.frontend_port, runtime, config) ++ logger.info("Starting infera router %d on %s: %s", idx, node, shlex.join(cmd)) ++ ++ env = dict(runtime.environment) ++ env.update(config.frontend.env or {}) ++ step_name = f"infera_router_{idx}" ++ proc = self.start_process( ++ command=cmd, ++ nodelist=[node], ++ output=str(log_file), ++ container_image=config.frontend.container_image or str(runtime.container_image), ++ reuse_container=runtime.reuse_container, ++ container_prepare_timeout_seconds=runtime.container_prepare_timeout_seconds, ++ container_mounts=runtime.container_mounts, ++ env_to_set=env or None, ++ bash_preamble=setup_script_preamble(config), ++ het_group=runtime.nodes.het_group_for(node), ++ step_name=step_name, ++ srun_options=runtime.srun_options, ++ ) ++ processes.append( ++ ManagedProcess( ++ name=step_name, ++ popen=proc, ++ log_file=log_file, ++ node=node, ++ critical=True, ++ terminate_timeout=FRONTEND_TERMINATE_TIMEOUT_SECONDS, ++ step_name=step_name, ++ ) ++ ) ++ return processes +diff --git a/src/srtctl/mock.py b/src/srtctl/mock.py +index 1458b28..0b8e9eb 100644 +--- a/src/srtctl/mock.py ++++ b/src/srtctl/mock.py +@@ -295,6 +295,7 @@ def mock_infrastructure(*, options: MockOptions, output_dir: Path): + ("srtctl.cli.mixins.postprocess_stage.get_batch_host", _fake_batch_host), + ("srtctl.cli.mixins.postprocess_stage.shrink_job_allocation", _fake_shrink), + ("srtctl.frontends.dynamo.start_srun_process", _fake_srun), ++ ("srtctl.frontends.infera.start_srun_process", _fake_srun), + ("srtctl.frontends.sglang.start_srun_process", _fake_srun), + ("srtctl.frontends.static_router.start_srun_process", _fake_srun), + ("srtctl.frontends.trtllm_serve.start_srun_process", _fake_srun), +@@ -307,6 +308,7 @@ def mock_infrastructure(*, options: MockOptions, output_dir: Path): + ("srtctl.cli.mixins.frontend_stage.get_hostname_ip", _fake_hostname_ip), + ("srtctl.cli.mixins.benchmark_stage.get_hostname_ip", _fake_hostname_ip), + ("srtctl.cli.mixins.service_stage.get_hostname_ip", _fake_hostname_ip), ++ ("srtctl.frontends.infera.get_hostname_ip", _fake_hostname_ip), + ("srtctl.frontends.sglang.get_hostname_ip", _fake_hostname_ip), + ("srtctl.frontends.static_router.get_hostname_ip", _fake_hostname_ip), + ("srtctl.frontends.trtllm_serve.get_hostname_ip", _fake_hostname_ip), diff --git a/inferencex-e2e/runners/srt-slurm/patches/README.md b/inferencex-e2e/runners/srt-slurm/patches/README.md index 61937581cf..e8bc442914 100644 --- a/inferencex-e2e/runners/srt-slurm/patches/README.md +++ b/inferencex-e2e/runners/srt-slurm/patches/README.md @@ -9,4 +9,5 @@ Each patch backports an upstream fix that the shared submodule pin does not incl | Patch | Upstream PR | Fix | |-------|-------------|-----| | `573-participating-gpus.patch` | [NVIDIA/srt-slurm#573](https://github.com/NVIDIA/srt-slurm/pull/573) | Record only the GPUs workers occupy on a worker node; an exclusive node's exporter (DCGM or AMD) reports every GPU, which the collector otherwise flags as `unexpected_device`. The pin includes #573, but upstream merged a different design: it keeps every row and excuses idle worker-node GPUs only after a DCGM worker GPU UUID probe. [`infx/results/power/multinode.py`](../../../infx/results/power/multinode.py) does not apply that excusal yet, so keep this filter until it does. | +| `630-infera-frontend.patch` | [NVIDIA/srt-slurm#630](https://github.com/NVIDIA/srt-slurm/pull/630) (open) | `frontend.type: infera`: SGLang workers start through Infera's `infera.engine.sglang`, register in the job's etcd and are discovered by `infera-router` (the `infera-sglang` framework). Adds the `Frontend.worker_module` / `worker_launcher_args` / `worker_environment` hooks and `Backend.get_kv_events_config_for_mode`; the patch is the PR's `src/srtctl` diff at `290bc0de`. | | `900-pool-expected-devices.patch` | pending | Plan every service pool node's `gpus_per_node` GPUs as expected power-telemetry devices under the service's name (`worker_role=`, `worker_index=`), so a services-only recipe (`frontend.type: none`, one `services[].nodes` pool, `benchmark.type: custom`) reaches collector readiness and publishes instead of failing with `exporter_startup_timeout` + `expected_device_missing` + `unexpected_device`. A services-only job records `worker_identity_proof.error = "no engine workers to probe"` instead of a probe traceback, the identity gate only scopes engine-role devices, and the offline validator accepts pool roles. Engine-only manifests are unchanged. With `573-participating-gpus.patch` an exclusive pool node's unplanned exporter rows are dropped at scrape time, so a recipe whose `gpus_per_node` is below the node's physical GPU count still validates; upstream alone reports them as `unexpected_device`. |