Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
100 commits
Select commit Hold shift + click to select a range
2366dc4
feat: integrate Grace CPU-side power from the multinode cpu/ package
edwingao28 Sep 18, 2026
f21ddd0
fix: keep cpu_power_valid independent of the GPU producer pin
edwingao28 Sep 18, 2026
cc62f0e
feat: enable srt-slurm cpu power leg on Qwen3.5 FP8 GB200 and GB300 l…
edwingao28 Sep 18, 2026
8832296
chore: merge origin/main and keep the cpu power changelog entry at th…
edwingao28 Sep 18, 2026
d365901
chore: link the cpu power leg changelog entry to PR #3296
edwingao28 Sep 19, 2026
7b3ba49
chore: merge origin/main and keep the cpu power changelog entry at th…
edwingao28 Sep 19, 2026
79fc6b2
chore: merge origin/main and keep the cpu power changelog entry at th…
edwingao28 Sep 21, 2026
b8f85f4
fix: pin srt-slurm to the hwmon name discovery fix on the gap-budget …
edwingao28 Sep 21, 2026
fd34e0c
chore: merge origin/main and keep the cpu power changelog entry at th…
edwingao28 Sep 21, 2026
b0fb074
feat: prepare Kimi K3 NVL72 CPU telemetry and host power
edwingao28 Sep 21, 2026
6f50f83
chore: sync NVL72 CPU telemetry with current main
edwingao28 Sep 30, 2026
0f5081e
fix: pin NVL72 CPU exporter after noprof integration
edwingao28 Oct 1, 2026
15dbee4
fix: enforce NVL72 CPU exporter preflight
edwingao28 Oct 1, 2026
bf9487e
chore: sync NVL72 power collection with main
edwingao28 Oct 1, 2026
de3f6a4
chore: sync NVL72 power collection with main
edwingao28 Oct 2, 2026
8bd735e
fix: require complete CPU power for NVL72 sweeps
edwingao28 Oct 2, 2026
94a7f53
fix: retain AgentX CPU power audit
edwingao28 Oct 2, 2026
7969696
chore: merge main into NVL72 power collection
edwingao28 Oct 2, 2026
fb7dfee
chore: sync NVL72 CPU power with main
edwingao28 Oct 5, 2026
508a892
chore: consolidate NVL72 power changelog entry
edwingao28 Oct 8, 2026
236eea6
chore: merge main into NVL72 power collection
edwingao28 Oct 8, 2026
a02c8ae
fix: migrate Qwen NVL72 head placement to srt schema 2
edwingao28 Oct 8, 2026
bbf1ad7
chore: drop upstreamed srt-slurm 553 patch and pin exporter v2.47.2
edwingao28 Oct 8, 2026
93950d3
feat: publish Grace CPU-only and SysIO rail power
edwingao28 Oct 8, 2026
2a02d1d
test: prune redundant NVL72 power tests
edwingao28 Oct 8, 2026
65c6d40
test: restore distinct CPU power regression coverage
edwingao28 Oct 8, 2026
4954627
docs: define the Grace CPU power sensors and their published keys
edwingao28 Oct 9, 2026
5eaac56
chore: merge main into NVL72 power collection
edwingao28 Oct 9, 2026
25f465a
docs: note that GB200 exposes module power through NVML, not ACPI
edwingao28 Oct 9, 2026
6d87ea2
docs: correct NVL72 CPU power policy, rail keys, and changelog
edwingao28 Oct 9, 2026
2ccc1bb
chore: merge main into NVL72 power collection
edwingao28 Oct 9, 2026
3c71183
chore: merge main into NVL72 power collection
edwingao28 Oct 13, 2026
1e1aa3c
chore: pin cpu-power-exporter v2.51.1 to match the srt-slurm source pin
edwingao28 Oct 13, 2026
2d5b63c
chore: merge main into NVL72 power collection
edwingao28 Oct 13, 2026
8988d91
chore: merge main into NVL72 power collection
edwingao28 Oct 13, 2026
6bb3c91
chore: merge main into NVL72 CPU power collection
edwingao28 Oct 14, 2026
94f01bb
fix: drop launcher-bound keys from NVL72 CPU power recipes
edwingao28 Oct 14, 2026
02b05fe
chore: merge main into NVL72 CPU power collection
edwingao28 Oct 14, 2026
85dc251
feat(power): declare the Grace CPU exporter per cluster and select it…
edwingao28 Oct 14, 2026
9115045
refactor(power): read only the wide Grace socket samples in the CPU leg
edwingao28 Oct 14, 2026
847227b
refactor(power): expect the CPU leg from cpu-power instead of re-read…
edwingao28 Oct 14, 2026
b775432
refactor(power): share the sample-gap limit and drop the dead window-…
edwingao28 Oct 14, 2026
c0a9c79
docs(power): describe the cpu-power switch and Grace sensor semantics
edwingao28 Oct 14, 2026
0005144
fix(power): count sample_loss_exceeded as GPU coverage for the CPU wi…
edwingao28 Oct 14, 2026
ec74d5d
chore: merge main into NVL72 CPU power collection
edwingao28 Oct 14, 2026
6ce28a2
test(power): restore the v3 temperature rejection test
edwingao28 Oct 14, 2026
728d4bf
refactor(power): drop the launcher cpu-power guard the matrix validat…
edwingao28 Oct 14, 2026
e9e50d3
refactor(power): accept only the ACPI Grace CPU exporter source
edwingao28 Oct 14, 2026
fc3faaf
refactor(power): stop gating the CPU leg on cpu_manifest.json
edwingao28 Oct 14, 2026
dd130b7
chore(recipes): drop the Qwen3.5 8P1D benchmark placement that restat…
edwingao28 Oct 14, 2026
eabb245
refactor(power): drop the multinode CLI flag no caller passes
edwingao28 Oct 14, 2026
faffae2
refactor(matrix): read cpu-power fields directly like the sibling val…
edwingao28 Oct 14, 2026
73971a0
test(power): assert the setup pin through CPU_POWER_EXPORTER_RELEASE
edwingao28 Oct 14, 2026
a581254
chore: merge main into NVL72 CPU power collection
edwingao28 Oct 14, 2026
c8e19aa
chore: merge main into NVL72 CPU power collection
edwingao28 Oct 16, 2026
323371b
feat(power): publish NVML module power from CPU samples v3
edwingao28 Oct 16, 2026
995a280
chore: merge main into NVL72 CPU power collection
edwingao28 Oct 19, 2026
e49df57
chore: merge main into NVL72 CPU power collection
edwingao28 Oct 20, 2026
4cc9b17
chore: bump srt-slurm and cpu-power-exporter to v2.59.3
edwingao28 Oct 20, 2026
cfe27c4
chore: merge main into NVL72 CPU power collection
edwingao28 Oct 22, 2026
c44aec9
chore: merge main into NVL72 CPU power collection
edwingao28 Oct 22, 2026
611ce71
chore: merge main into NVL72 CPU power collection
edwingao28 Oct 22, 2026
986021c
chore: merge main into NVL72 CPU power collection
edwingao28 Oct 22, 2026
4035603
chore: merge main into NVL72 CPU power collection
edwingao28 Oct 23, 2026
e47abed
chore: merge main into NVL72 CPU power collection
edwingao28 Oct 23, 2026
a727e1d
feat: collect Grace CPU power on the Kimi K3 GB200 disaggregated Agen…
edwingao28 Oct 23, 2026
cbd52a9
chore: merge main into NVL72 CPU power collection
edwingao28 Oct 23, 2026
eb52a00
chore: merge main into NVL72 CPU power collection
edwingao28 Oct 28, 2026
daeeaec
chore: merge main into NVL72 CPU power collection
edwingao28 Oct 29, 2026
c0aa8a9
chore: limit CPU power to Qwen3.5 8k1k and Kimi K3 GB300 agg
edwingao28 Oct 29, 2026
619f65a
chore: merge main into NVL72 CPU power collection
edwingao28 Oct 29, 2026
cfe2ad2
chore: drop Qwen3.5 8k1k from NVL72 CPU power collection
edwingao28 Nov 1, 2026
982d2df
chore: merge main into NVL72 CPU power collection
edwingao28 Nov 1, 2026
9a1cfff
chore: pin cpu-power-exporter v2.65.2 to match srt-slurm
edwingao28 Nov 1, 2026
8c99e85
chore: merge main into NVL72 CPU power collection
edwingao28 Nov 3, 2026
089d341
chore: bump cpu-power-exporter to v2.66.1 with the submodule
edwingao28 Nov 3, 2026
3b45892
chore: drop the dead cpu_power_exporter fingerprint exclusion
edwingao28 Nov 3, 2026
e945f0e
chore: remove section header comments from the power leg
edwingao28 Nov 3, 2026
acc6846
docs: describe the cpu-power row key, exporter record and workers ent…
edwingao28 Nov 3, 2026
e1a6573
chore: merge main into NVL72 CPU power collection
edwingao28 Nov 3, 2026
93fbf71
feat: integrate AMD RAPL package power in the CPU power leg
edwingao28 Nov 3, 2026
94f3116
feat: bind the cluster's RAPL CPU exporter on single-node cpu-power p…
edwingao28 Nov 3, 2026
76a3295
chore: collect CPU power on the AMD 8k1k sglang rows
edwingao28 Nov 3, 2026
a0de238
fix: warn instead of failing the job when the RAPL grant fails
edwingao28 Nov 3, 2026
93deb9e
refactor: define the CPU power source enum once
edwingao28 Nov 3, 2026
7249327
docs: note where a failed RAPL grant or restore shows up
edwingao28 Nov 3, 2026
d831e3a
test: check CPU socket meters against srt-slurm's sensor suffixes
edwingao28 Nov 3, 2026
57d50b9
fix: index workers[] power entries by the manifest worker
edwingao28 Nov 4, 2026
1932aab
chore: merge main into NVL72 CPU power collection
edwingao28 Nov 4, 2026
7c09638
docs: shorten the NVL72 CPU power changelog entry
edwingao28 Nov 4, 2026
74b72b5
chore: merge final NVL72 CPU power head into AMD RAPL power
edwingao28 Nov 4, 2026
757c5f2
chore: merge main into AMD RAPL power
edwingao28 Nov 4, 2026
284c92e
test: log fake sudo arguments with printf so dash keeps -n
edwingao28 Nov 4, 2026
b95ce4e
chore: merge main and resolve changelog conflict
edwingao28 Nov 4, 2026
30b9b3b
chore: merge main and resolve changelog conflict
edwingao28 Nov 4, 2026
c4ead87
chore: merge main and resolve test conflict
edwingao28 Nov 4, 2026
873e788
style: rewrap bound_variant docstring to 100 columns
edwingao28 Nov 4, 2026
49d4034
fix: refuse RAPL CPU power on shared nodes
edwingao28 Nov 4, 2026
c5e140e
feat: measure CPU power by default on clusters with a CPU exporter
edwingao28 Nov 4, 2026
2ae902d
Merge remote-tracking branch 'origin/main' into pr-4154-reuse-15112
edwingao28 Nov 5, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/AGENT_OPERATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,8 @@ A multinode master row with `cpu-power: true` beside `power: true` adds srt-slur

The package then carries `power/cpu/samples.csv` in srt-slurm's wide format (one row per scrape, host, and Grace socket; two sockets per worker host in the manifest topology), and the aggregate adds `cpu_power_valid`, `avg_cpu_socket_power_w`, `avg_total_cpu_power_w`, and `total_cpu_energy_j` from the `CPU<n>:cpuSidePowerUsageW` series, plus `avg_total_cpu_rail_power_w` / `total_cpu_rail_energy_j` (ACPI `CPU Power Socket N`, cores only) and `avg_total_cpu_sysio_power_w` / `total_cpu_sysio_energy_j` (ACPI `SysIO Power Socket N`), each pair only when every socket reports that rail without gaps. The meters nest ([NVIDIA Grace power guide](https://docs.nvidia.com/dccpu/grace-perf-tuning-guide/power-thermals.html)): `Grace Power Socket N` is the whole socket (CPU rail, SysIO rail, LPDDR5X, and regulator loss) and the two rails are a breakdown inside it, so never add them to or subtract them from the total; a GB300 socket reads about 97 W while its rails sum to about 55 W. The DRAM rail is blank on the CI NVL72 firmware. srt-slurm v2.59.0+ writes NVML superchip module power as `module_w` (CPU samples schema v3) when its exporter can load NVML (the aarch64 glibc build, srt-slurm #609), and the aggregate then publishes `avg_total_module_power_w` / `total_module_energy_j` under the same every-socket, gap-free rule; the module includes both GPUs, so never add it to the CPU or GPU totals. Every series is integrated over the bound formal GPU window with the GPU leg's trapezoid, boundary interpolation, and 3.0 s gap limit; the sidecar's `cpu` block, mirrored as `power_audit.cpu`, records `sensor_kind: grace_socket`, `source`, socket and row counts, and reason codes. A valid GPU leg also publishes `workers[]`: one entry per worker per host (`role`, the manifest's `worker_idx`, `hosts` holding that one host, `num_gpus`, `avg_power_w`), so a worker spanning trays repeats its index once per host and can be modeled per tray; it is omitted when a GPU belongs to two workers.

A native single-node fixed-sequence row with `cpu-power: true` adds the same leg. Single-node rows have no `power` field because their native points always measure GPU power; eval-only and acceptance-gate jobs bind no CPU exporter. RAPL clusters (`mi300x-amd`, `mi325x-amd`, `mi355x-amds`) read each AMD EPYC package as `CPU<n>:packagePowerUsageW` and publish the three headline CPU keys with `sensor_kind: rapl_package` and no rail keys. RAPL `energy_uj` counters are root-only, so a job that binds a RAPL exporter adds a host-setup command that gives the job user each top-level counter and a teardown that returns them to root. A failed grant only warns, so the CPU leg is then judged by the verdict below; a cancelled job skips teardown, and a failed restore shows up only in the job's `logs/host_teardown_<node>.out`.

The CPU verdict is independent of `power_valid`: a failed leg records `cpu_power_valid: 0` with no CPU keys and leaves every GPU field unchanged. It fails the job only when power is required (`REQUIRE_POWER=1`, or always on AgentX lanes whose launcher validates power) and the row set `cpu-power`; missing CPU artifacts are then a verdict (`cpu_artifacts_missing`) rather than silence. Other reason codes: `cpu_samples_missing`, `cpu_samples_header_mismatch`, `cpu_samples_malformed`, `cpu_socket_count_mismatch` (also when a host reports only the DCGM CPU rail), `cpu_sample_gap_exceeded`, `cpu_window_not_bracketed`, `cpu_window_unavailable`, and the shared `non_finite_power_metric`.

Every power result, valid or invalid, carries `power_metric_schema_version`. Version 2 defines each unprefixed `joules_per_*` field as whole-deployment GPU-board energy over the named denominator; role-scoped energy uses the explicit `prefill_*` / `decode_*` keys. Rows without the field predate the whole-deployment switch and their unprefixed joules are not comparable across topologies.
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/benchmark-tmpl.yml
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,7 @@ env:
SALLOC_TIME_LIMIT: '480'
IS_MULTINODE: 'false'
REQUIRE_POWER: ${{ inputs.require-power && '1' || '0' }}
CPU_POWER: ${{ fromJSON(inputs.config)['cpu-power'] && '1' || '0' }}
RANDOM_RANGE_RATIO: 0.8
HF_TOKEN: ${{ secrets.INFERENCEX_OFFICIAL_RO_HF_TOKEN }}
HF_HUB_CACHE: '/mnt/hf_hub_cache/'
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ recipe and master settings; the recipe still owns the native configuration.

Active recipes, fixed-sequence and AgentX, single- and multi-node, are fragments: native srt-slurm YAML holding only recipe-specific settings. At launch the fragment is composed and bound:

1. The lane's shared block in [`configs/srt-recipes/`](../../../configs/srt-recipes) (`fixed-sequence-{single,multi}.yaml` or `agentic.yaml`) is merged under the fragment (under `base` for bundles). A multi-node row with `power: true` also gets [`telemetry-dcgm.yaml`](../../../configs/srt-recipes/telemetry-dcgm.yaml), its exporter on the cluster's `srt-slurm.power-exporter-port`; the lane must allow power (`POWER_LANES`). The fragment wins: mappings merge and lists replace, so a fragment may keep telemetry tunables such as `collector_join_timeout_seconds`. Native single-node benchmarks also get [`telemetry-native.yaml`](../../../configs/srt-recipes/telemetry-native.yaml) before variant selection. That block enables best-effort power collection with the cluster's `default_gpu_exporter`; eval-only points omit collection. The binder sets `telemetry.required` from `REQUIRE_POWER`, so a single-node fragment may not set it, and gives AgentX clients matching power switches.
1. The lane's shared block in [`configs/srt-recipes/`](../../../configs/srt-recipes) (`fixed-sequence-{single,multi}.yaml` or `agentic.yaml`) is merged under the fragment (under `base` for bundles). A multi-node row with `power: true` also gets [`telemetry-dcgm.yaml`](../../../configs/srt-recipes/telemetry-dcgm.yaml), its exporter on the cluster's `srt-slurm.power-exporter-port`; the lane must allow power (`POWER_LANES`). The fragment wins: mappings merge and lists replace, so a fragment may keep telemetry tunables such as `collector_join_timeout_seconds`. Native single-node benchmarks also get [`telemetry-native.yaml`](../../../configs/srt-recipes/telemetry-native.yaml) before variant selection. That block enables best-effort power collection with the cluster's `default_gpu_exporter`; eval-only points omit collection. A native single-node point on a cluster that declares `srt-slurm.cpu-power-exporter`, and a `cpu-power: true` row, also gets `telemetry.cpu_power_exporter` with that source. The binder sets `telemetry.required` from `REQUIRE_POWER`, so a single-node fragment may not set it, and gives AgentX clients matching power switches.
2. The row's selector picks the variant. A single-node variant may name its `benchmark.env.CONC`, which keeps that concurrency paired with its tuning.
3. The binder ([`workload.py`](../../../infx/srt_slurm/workload.py)) writes the matrix point into the selected recipe: `model.path: hf:<model>`, `model.container: <image>`, `model.precision`, `identity.container.image` (as a registry reference) and `identity.model.repo` when the fragment declares `identity.container`/`identity.model`, and `benchmark.concurrencies` when telemetry is enabled. Fixed-sequence recipes also get `benchmark.env.ISL`/`OSL`; single-node recipes get `MODEL` and `CONC`, and fixed-sequence ones `RANDOM_RANGE_RATIO` and `USE_CHAT_TEMPLATE` (`true` exactly when the recipe speculates). Multi-node AgentX recipes get the client's `RESULT_DIR`, `AIPERF_DATASET_MMAP_CACHE_DIR` and `HF_HUB_CACHE` from the launcher, which derives them from the volumes it mounts (`volume-mounts`, `agentic-volume-mounts`, lane mounts); a fragment that sets `HF_HOME` keeps its own cache. They also get `IS_MULTINODE: "true"` and `ENABLE_AGENTX_POWER`, `"1"` exactly when the launcher collects the point's AgentX power (a `power: true` row whose lane rule collects AgentX or adapter power) and `"0"` otherwise, and `REQUIRE_POWER`, `"1"` when the workflow requires power or the lane rule marks the point `require`, so the client fails a point whose power is invalid. The multi-node client reads `CONC_LIST`, `CONC`, `MODEL` and the other matrix inputs, prefill and decode topology included, from the job environment, so fragments do not copy them (`TP`, `PP_SIZE`, `PCP_SIZE` and `EP_SIZE` describe single-node points only); single-node AgentX points get them as runtime arguments.
4. A component a repository setup script installs gets the master config's version. When `setup_script`, or a service `preamble` that runs `/configs/<script>`, is one of the installers in the binder's `INSTALLERS`, the binder writes the point's `router` or `kv-offload-backend` version as `ROUTER_VERSION` or `KV_OFFLOAD_BACKEND_VERSION` into the top-level `environment` (workers and static routers) or into that service's `env` (services do not inherit `environment`). A point that does not declare the version fails, and a `vllm-router==` pin in `SETUP_PIP_PACKAGES` must equal the master router version.
Expand All @@ -102,7 +102,7 @@ Inspect what the launcher submits without a cluster:
uv run --extra recipes infx generate --config-key 'dsr1-fp8-h200-*' --output-dir /tmp/recipes
```

It writes one bound recipe per fixed-sequence or AgentX point, validated by the pinned srtctl, plus a `manifest.json` that maps each file to its matrix point and to the one cluster its runner label schedules on (`null` when the label spans several). Multi-node points take their cluster's binder inputs (DCGM exporter port, AgentX client env); a point that needs them fails unless its runner label names one cluster and the lane accepts the row's `power`. Native single-node points validate against each eligible cluster's exporter defaults while retaining cluster-independent recipe output. Fabric references take that cluster's facts; when the label spans several clusters they stay references. Launch-time edits such as the job name, health-check floor and runtime `--set` values are not applied.
It writes one bound recipe per fixed-sequence or AgentX point, validated by the pinned srtctl, plus a `manifest.json` that maps each file to its matrix point and to the one cluster its runner label schedules on (`null` when the label spans several). Multi-node points take their cluster's binder inputs (DCGM exporter port, AgentX client env); a point that needs them fails unless its runner label names one cluster and the lane accepts the row's `power`. Native single-node points validate against each eligible cluster's exporter defaults while retaining cluster-independent recipe output, except that a point whose runner label names one cluster takes that cluster's CPU exporter source, and a `cpu-power` point must name one. Fabric references take that cluster's facts; when the label spans several clusters they stay references. Launch-time edits such as the job name, health-check floor and runtime `--set` values are not applied.

## Recipe fingerprints

Expand Down
9 changes: 5 additions & 4 deletions inferencex-e2e/configs/CONFIGS.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@ input.
Notes:
- The fields above are the common ones, not the full schema. The Pydantic models in [`infx/matrix/validation.py`](../infx/matrix/validation.py) are the authoritative contract; they also accept fields such as `spec-decoding`, `num-nodes`, `require-power` and `power`, and they reject any field they do not define, which fails matrix generation.
- A multinode search-space entry with `power: true` measures DCGM power: its rows carry `power: true`, the launcher merges the shared telemetry block into the recipe and rejects lanes without a power rule.
- A multinode entry with `cpu-power: true` beside `power: true` also measures CPU power: its rows carry `cpu-power: true` and the launcher binds the cluster's `cpu-power-exporter` record beside the DCGM block. Single-node entries cannot carry it.
- A multinode entry with `cpu-power: true` beside `power: true` also measures CPU power: its rows carry `cpu-power: true` and the launcher binds the cluster's `cpu-power-exporter` record beside the DCGM block. Native single-node points measure CPU power by default on a cluster that declares a `cpu-power-exporter`; a single-node fixed-sequence entry sets `cpu-power: true` alone to count a missing CPU leg as invalid.
- srt-slurm entries require `srt-recipe-dir`. Every row requires `srt-recipe: <file>[:<selector>]` under it (selector: `base`, `override_<name>` or `zip_override_<name>[<i>]`). Multinode rows may add `eval-srt-recipe`. Validation rejects missing files or variants, and `CONFIG_FILE`/`EVAL_CONFIG_FILE` settings. Only `llmd-vllm` keeps `CONFIG_FILE`. Generated rows carry the path from `inferencex-e2e/`.
- Setting the fields above only guarantees that their values are passed as environment variables to benchmark scripts. Single-node jobs receive `PP_SIZE`, `DCP_SIZE`, and `PCP_SIZE`. Multinode jobs receive `PREFILL_PP_SIZE`, `PREFILL_DCP_SIZE`, `PREFILL_PCP_SIZE`, `DECODE_PP_SIZE`, `DECODE_DCP_SIZE`, and `DECODE_PCP_SIZE`. Actually using those variables is an implementation detail of the benchmark Bash script.

Expand Down Expand Up @@ -266,9 +266,10 @@ schema; unknown keys fail.
volumes every job mounts to container paths, `mounts` does the same for host
paths outside the volumes (devices), `agentic-volume-mounts` adds world-writable
volumes for multi-node AgentX jobs, `power-exporter-port` is the DCGM exporter port
of power points, `cpu-power-exporter` (`source`) is the CPU power exporter a
`cpu-power` row binds beside it (`make setup` installs it from the pinned srt-slurm
release), `env` is the environment of the srt-slurm
of power points, `cpu-power-exporter` (`source: acpi` or `rapl`) is the CPU power exporter native
single-node points and `cpu-power` rows bind beside it (`make setup` installs it from the pinned srt-slurm
release; a RAPL job's host setup gives the job user the RAPL counters until teardown),
`env` is the environment of the srt-slurm
launch, `container-aliases` and `nginx-aliases` name the recipe containers that
resolve to the main image and to the staged nginx, and `outputs`,
`shared-run-root` and `uv-cache-root` are the directories the srt-slurm launcher
Expand Down
3 changes: 3 additions & 0 deletions inferencex-e2e/configs/amd-master.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,7 @@ dsr1-fp8-mi300x-sglang:
conc-start: 4
conc-end: 64
srt-recipe: 8k1k.yaml
cpu-power: true

dsr1-fp8-mi325x-sglang:
image: lmsysorg/sglang:v0.5.22-rocm720-mi30x@sha256:58b6bc333f527c88fe5ff1b4bd8d17e82214eabcb3044549117159878967c930
Expand All @@ -126,6 +127,7 @@ dsr1-fp8-mi325x-sglang:
conc-start: 4
conc-end: 64
srt-recipe: 8k1k.yaml
cpu-power: true

dsr1-fp8-mi355x-sglang:
image: lmsysorg/sglang:v0.5.22-rocm720-mi35x@sha256:c4b61360b251d62bb17da63f175d571ab22479eb46a144548d79f3ed3998ea11
Expand Down Expand Up @@ -170,6 +172,7 @@ dsr1-fp8-mi355x-sglang-mtp:
conc-end: 64
spec-decoding: mtp
srt-recipe: 8k1k.yaml
cpu-power: true

# MI355X FP8 AgentX, TP4 only. The grid follows the TP4 band of the B200 FP8
# AgentX arm so the two SKUs are directly comparable point for point, with the
Expand Down
3 changes: 3 additions & 0 deletions inferencex-e2e/configs/runners.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -709,6 +709,7 @@ clusters:
single-node-time-limit: 180
gpus-per-node-directive: true
segment-directive: false
cpu-power-exporter: {source: rapl}
host-setup:
script: runners/srt-slurm/hooks/mi300x-amd/setup.sh
mounts:
Expand Down Expand Up @@ -758,6 +759,7 @@ clusters:
single-node-time-limit: 480
gpus-per-node-directive: true
segment-directive: false
cpu-power-exporter: {source: rapl}
mounts:
/dev/kfd: /dev/kfd
/dev/dri: /dev/dri
Expand Down Expand Up @@ -807,6 +809,7 @@ clusters:
gpus-per-node-directive: true
segment-directive: false
outputs: /it-share/gharunners2/srt-slurm/outputs
cpu-power-exporter: {source: rapl}
host-setup:
script: runners/srt-slurm/hooks/mi355x-amds/setup.sh
env:
Expand Down
9 changes: 5 additions & 4 deletions inferencex-e2e/infx/clusters/slurm.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
from pydantic import AfterValidator, Field, PlainSerializer, field_validator, model_validator

from infx.clusters.base import Record, SchedulerSettings, Visibility, Volume
from infx.srt_slurm.workload import resolve_fabric
from infx.srt_slurm.workload import CpuPowerSource, resolve_fabric

if TYPE_CHECKING:
from infx.clusters import Cluster
Expand Down Expand Up @@ -174,13 +174,14 @@ def rendered(self) -> dict[str, str | None]:


class CpuPowerExporter(Record):
"""The Grace CPU power exporter srt-slurm scrapes on this cluster's compute trays.
"""The CPU power exporter srt-slurm scrapes on this cluster's nodes.

A point that selects ``cpu-power`` binds ``telemetry.cpu_power_exporter`` with this source.
Only ACPI serves the Grace socket meter the CPU leg integrates.
ACPI serves the Grace socket meter and RAPL the AMD EPYC package counter, the meters the
CPU leg integrates.
"""

source: Literal["acpi"]
source: CpuPowerSource


class SrtSlurmSettings(Record):
Expand Down
13 changes: 9 additions & 4 deletions inferencex-e2e/infx/launch/drivers/srt/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -69,12 +69,16 @@ def run_single_node(launch: Launch) -> int:
step_mib, job_mib = None, 0
if (options := config.srun_options(run.backend.settings, step_mib)) is not None:
run.env["SRT_SRUN_OPTIONS"] = options
if rc := submit.bind_point(run, checkout, root / "arguments"):
native_power = not request.eval_only and not request.al_gate
cpu_source = config.cpu_power_source(
run.cluster, run.srt, native_power and request.cpu_power, default=native_power
)
if rc := submit.bind_point(run, checkout, root / "arguments", cpu_source):
return rc
selected, bound, runtime_args = submit.bound_arguments(root / "arguments")
event.set(recipe=selected.removeprefix(f"{run.workspace}/"))
containers = {}
if not request.eval_only and not request.al_gate:
if native_power:
image, reference = config.stage_gpu_exporter(run, single_node=True)
containers[image] = reference
job_config = config.SrtJob(
Expand All @@ -90,10 +94,11 @@ def run_single_node(launch: Launch) -> int:
single_node=True,
account=run.account,
mem_mib=job_mib,
cpu_power_source=cpu_source,
)
config.create_volume_mounts(run)
config.write(checkout.root / "srtslurm.yaml", config.render(run.cluster, job_config))
if rc := run_setup(run, checkout):
if rc := run_setup(run, checkout, cpu_power_exporter=cpu_source is not None):
return rc

fetched = root / "fetched-outputs"
Expand All @@ -114,7 +119,7 @@ def run_single_node(launch: Launch) -> int:
status = run.backend.state(job)
with event.stage("collect"):
logs = run.backend.fetch_outputs(job, fetched) / "logs"
if not request.eval_only and not request.al_gate:
if native_power:
(logs / "power").mkdir(parents=True, exist_ok=True)
(logs / "power/native-job-status.txt").write_text(f"{job.id}|{status.raw}\n")
rc = collect.finalize_single_node_results(run, logs, checkout.commit)
Expand Down
Loading
Loading