Skip to content

Commit 115b00b

Browse files
authored
[PowerX] collect AMD x86 CPU package power through RAPL / [PowerX] 通过 RAPL 采集 AMD x86 CPU 封装功耗 (#4154)
* feat: integrate Grace CPU-side power from the multinode cpu/ package Read srt-slurm's power/cpu/{samples.csv,cpu_manifest.json} (v1 long and v2 wide) next to the GPU DCGM package, integrate each (host, socket) Grace-side and module series over the bound formal window with the shared trapezoid and gap rule, and emit cpu_power_valid plus avg_cpu_socket_power_w, avg_total_cpu_power_w, total_cpu_energy_j and the module keys. The verdict is independent of power_valid and REQUIRE_POWER; a package without cpu/ produces byte-identical output (GB200 fixture replay). 中文:在多节点功耗校验器中接入 srt-slurm 的 power/cpu/ 子产物包(v1 长格式与 v2 宽格式),按 (主机, socket) 在同一正式测量窗口内用共享的梯形法和采样间隔规则积分 Grace 侧及模块功耗,输出 cpu_power_valid 与 avg_cpu_socket_power_w、 avg_total_cpu_power_w、total_cpu_energy_j 及模块字段;结论独立于 power_valid 和 REQUIRE_POWER,无 cpu/ 的产物包输出逐字节不变(GB200 fixture 回放验证)。 * fix: keep cpu_power_valid independent of the GPU producer pin validate_and_integrate() forced cpu_power_valid=0 with cpu_producer_unverified whenever the GPU leg's producer pin failed, so a run with valid Grace samples lost its CPU energy for a GPU-side reason, against the ticket contract ("independent of power_valid") and the spec's independent-leg rule. Drop that gate and the unused blocking_reasons hook: the CPU leg borrows only the bound formal window and the host topology as inputs, and no GPU verdict reaches it. Fold the pin-mismatch and pin-missing cases into the existing invalid-GPU-leg test. Docs (EN/ZH) drop the contradictory sentence and list the codes the leg actually emits, including the shared non_finite_power_metric and aggregate_result_* ones. GB200 fixture replay stays byte-identical. 中文:validate_and_integrate() 原本在 GPU 侧 producer 固定版本校验失败时,用 cpu_producer_unverified 强制 cpu_power_valid=0,导致 Grace 采样有效的运行因 GPU 侧 原因丢失 CPU 能耗,与工单契约(“独立于 power_valid”)和 spec 的独立测量环节规则相悖。 移除该门控及未使用的 blocking_reasons 参数:CPU 侧只借用已绑定的正式窗口和主机拓扑 作为输入,GPU 侧的任何结论都不会传导过来。把固定版本不匹配与缺失两种情形并入现有的 GPU 侧无效测试。中英文文档删除自相矛盾的表述,并列出实际输出的原因码,包括共享的 non_finite_power_metric 与 aggregate_result_*。GB200 fixture 回放输出逐字节不变。 * feat: enable srt-slurm cpu power leg on Qwen3.5 FP8 GB200 and GB300 lanes Add telemetry.cpu_power_exporter (port 9405, source acpi) to the 1P1D TP4/TP4 STP recipes of qwen3.5-fp8-gb200-dynamo-sglang and qwen3.5-fp8-gb300-dynamo-sglang, the smallest active non-Kimi-K3 NVL72 power lanes; nothing else in the recipes changes. srt-slurm v2.2.1 runs the exporter on the bare worker host and writes power/cpu/samples.csv plus cpu_manifest.json beside the GPU DCGM package for the multinode validator's CPU leg. Append the perf-changelog entry. DSV4 8k1k lanes are not used because that scenario was deprecated after 2026-09-08. 中文:在 qwen3.5-fp8-gb200-dynamo-sglang 与 qwen3.5-fp8-gb300-dynamo-sglang 的 1P1D TP4/TP4 STP recipe 中加入 telemetry.cpu_power_exporter(port 9405, source acpi),这是当前活跃、非 Kimi-K3 的最小 NVL72 功耗 lane;recipe 其余内容不变。srt-slurm v2.2.1 在 worker 裸主机上运行 exporter,并在 GPU DCGM 产物包旁写入 power/cpu/samples.csv 与 cpu_manifest.json,供多节点 校验器的 CPU 侧测量环节使用。追加 perf-changelog 条目。未采用 DSV4 8k1k lane,因为该场景已于 2026-09-08 之后弃用。 * chore: link the cpu power leg changelog entry to PR #3296 Replace the pr-link placeholder in the appended perf-changelog.yaml entry with the opened PR URL; no other bytes change. 中文:将新增 perf-changelog.yaml 条目中的 pr-link 占位符替换为已开启的 PR 链接,其余内容不变。 * fix: pin srt-slurm to the hwmon name discovery fix on the gap-budget branch The first sweep showed the pinned cpu-power-exporter exiting with "no ACPI power_meter hwmon sensors found" on every GB200 node: on 6.8.0-nvidia-64k the acpi_power_meter hwmon node has no `name` attribute, only `device/name`. Pin utils/srt-slurm to SemiAnalysisAI/srt-slurm#23 head 473ecaa8 (984180e5 plus the gap-budget fix and the discovery fix) so the CPU leg can publish; the wire contract stays v1 long, so no consumer change. 中文:首次 sweep 显示固定版本的 cpu-power-exporter 在所有 GB200 节点上以 "no ACPI power_meter hwmon sensors found" 退出:6.8.0-nvidia-64k 内核下 acpi_power_meter 的 hwmon 节点没有 `name` 属性,只有 `device/name`。将 utils/srt-slurm 固定到 SemiAnalysisAI/srt-slurm#23 的 head 473ecaa8(984180e5 + 采样间隙预算修复 + 传感器发现修复),使 CPU 侧采集可以发布;CSV 契约仍为 v1 长表,消费端无需改动。 * feat: prepare Kimi K3 NVL72 CPU telemetry and host power Enable best-effort CPU telemetry on 13 existing GPU-power recipes and retain validated physical-host GPU power. Keep GPU validation and serving settings intact. 为 13 份已有 GPU 功耗配方准备独立 CPU 遥测,并保留通过验证的逐物理主机 GPU 功耗。保持 GPU 校验与服务设置不变。 * fix: enforce NVL72 CPU exporter preflight 应用上游 #553 的提交检查修复,令启用 CPU 采集的配方在 exporter 缺失、不可执行或已知架构不匹配时于 sbatch 前失败,并明确源码采集链路与二进制版本的区别。 * fix: require complete CPU power for NVL72 sweeps 要求 NVL72 扫描提供完整 CPU 功耗,并补齐 Qwen 采集与时钟主机配置。 * fix: retain AgentX CPU power audit 中文:在 AgentX 结果中保留 CPU 功耗审计信息。 * chore: consolidate NVL72 power changelog entry 将 NVL72 功耗变更记录合并为一个条目。 * fix: migrate Qwen NVL72 head placement to srt schema 2 将两份 Qwen NVL72 配方的前端与基准客户端位置改为当前 schema。 * chore: drop upstreamed srt-slurm 553 patch and pin exporter v2.47.2 中文:移除已合入上游的 srt-slurm 553 补丁,exporter 固定为 v2.47.2。 * feat: publish Grace CPU-only and SysIO rail power 中文:发布 Grace CPU 核心与 SysIO 供电轨功耗。 * test: prune redundant NVL72 power tests 中文:精简 NVL72 功耗冗余测试。 * test: restore distinct CPU power regression coverage 恢复 CPU 电源来源选择、错误隔离和时间窗口的回归测试。 * docs: define the Grace CPU power sensors and their published keys 中文:定义 Grace CPU 各功耗传感器的范围及对应发布字段。 * docs: note that GB200 exposes module power through NVML, not ACPI 中文:说明 GB200 的模块功耗由 NVML 提供而非 ACPI 功耗计,srt-slurm 尚未读取。 * docs: correct NVL72 CPU power policy, rail keys, and changelog 中文:修正 NVL72 CPU 功耗失败策略说明,补充供电轨字段与实测数值,并删除 changelog 中不属于本 PR 的 Kimi K3 采样间隔描述。 * chore: pin cpu-power-exporter v2.51.1 to match the srt-slurm source pin 中文:将 cpu-power-exporter 固定到 v2.51.1,与 srt-slurm 源码 pin 保持一致 * fix: drop launcher-bound keys from NVL72 CPU power recipes * feat(power): declare the Grace CPU exporter per cluster and select it with cpu-power The binder merges telemetry.cpu_power_exporter from the cluster record when a master row sets cpu-power beside power, so recipes and per-point settings no longer carry it; make setup pins the exporter release only for those jobs. 中文:Grace CPU 功耗 exporter 改由 cluster 记录声明,master 行用 cpu-power 选择;recipe 与逐点 additional-settings 不再携带,make setup 仅对这些任务固定 exporter 版本。 * refactor(power): read only the wide Grace socket samples in the CPU leg srt-slurm v2.51.1 cpu_samples.py:110 writes only CPU_SAMPLES_HEADER and binds no module meter, so the v1 long format, the OEM-label regexes and the module and DCGM headline paths never ran. The headline comes from the CPU<n>:cpuSidePowerUsageW rows alone and the two module keys leave CPU_METRIC_KEYS. 中文:CPU leg 仅读取宽格式 Grace socket 采样;删除从未执行的 v1 长格式、OEM 标签正则以及 module/DCGM headline 路径,并从 CPU_METRIC_KEYS 移除两个 module 指标。 * refactor(power): expect the CPU leg from cpu-power instead of re-reading the bound recipe The point's cpu-power switch already decides whether the exporter is bound (PowerDecision.cpu), so the launcher no longer re-reads the recipe or writes POWER_EXPECTED_CPU_SOURCE to GITHUB_ENV, and the processors take --expected-cpu-power or the workflow's CPU_POWER. The source check goes with it: the cluster record requires a source (clusters/slurm.py CpuPowerExporter), the binder always writes it, and the Grace leg accepts only its own sensor. 中文:CPU leg 是否必需改由 cpu-power 开关决定;launcher 不再回读 recipe 或写 GITHUB_ENV,处理器改用 --expected-cpu-power 或 workflow 的 CPU_POWER。cluster 记录必须声明 source 且 binder 总会写入,故删除多余的 source 核对。 * refactor(power): share the sample-gap limit and drop the dead window-trust field The CPU leg now asks _window_contract_holds directly on the serialized window instead of carrying formal_window_trusted across the audit, MAX_SAMPLE_GAP_SECONDS has one definition in common.py, and patch_power_metrics goes because _patch_agg became its only caller's replacement on this branch. 中文:CPU leg 直接用序列化窗口调用 _window_contract_holds,不再在 audit 上携带 formal_window_trusted;MAX_SAMPLE_GAP_SECONDS 只在 common.py 定义一处;删除在本分支上已无调用方的 patch_power_metrics。 * docs(power): describe the cpu-power switch and Grace sensor semantics AGENT_OPERATIONS.md carries the Grace sensor table Oren asked for (whole-socket meter, CPU and SysIO rails, no module series) in place of the link to the deleted results-and-ingestion.md, and the #3296 changelog entry describes the final diff. 中文:AGENT_OPERATIONS.md 补充 Oren 要求的 Grace 传感器说明(整 socket 读数、CPU/SysIO rail、无 module 序列),替换指向已删除文档的链接;#3296 的 changelog 条目改为描述最终 diff。 * fix(power): count sample_loss_exceeded as GPU coverage for the CPU window srt-slurm power/windows.py _check_coverage has emitted sample_loss_exceeded beside the three other coverage codes since v2.39.3 (#537), so a window that only lost GPU samples keeps binding the CPU leg instead of reporting cpu_window_unavailable. 中文:srt-slurm 自 v2.39.3 起在同一覆盖检查中输出 sample_loss_exceeded;仅丢失 GPU 采样的窗口仍可绑定 CPU leg,而不再报告 cpu_window_unavailable。 * test(power): restore the v3 temperature rejection test 9115045 deleted this GPU sample test with the CPU parser cleanup; read_samples still enforces MAX_TEMPERATURE_C. 中文:恢复被 9115045 随 CPU 解析清理误删的 v3 温度校验测试;read_samples 仍执行 MAX_TEMPERATURE_C 检查。 * refactor(power): drop the launcher cpu-power guard the matrix validator owns validation.py _cpu_power_rides_on_power rejects cpu-power without power on every search-space entry, and CPU_POWER reaches the launcher only from that validated matrix row. 中文:删除启动器中重复的 cpu-power 检查;validation.py 已在所有 search-space 条目上拒绝缺少 power 的 cpu-power,启动器的 CPU_POWER 只来自已校验的矩阵行。 * refactor(power): accept only the ACPI Grace CPU exporter source cpu_side.py integrates only CPU<n>:cpuSidePowerUsageW rows, which srt-slurm emits from ACPI; a dcgm source always ended in cpu_socket_count_mismatch. 中文:CPU exporter 来源只接受 ACPI;cpu_side.py 只积分 ACPI 输出的 CPU<n>:cpuSidePowerUsageW 行,dcgm 来源必然以 cpu_socket_count_mismatch 失败。 * refactor(power): stop gating the CPU leg on cpu_manifest.json srt-slurm documents cpu_manifest.json as non-authoritative debugging metadata and only logs a failed teardown write (cpu_session.py:218), so a broken manifest no longer invalidates a leg whose samples cover the window. 中文:CPU leg 不再依据 cpu_manifest.json 判定;srt-slurm 将其定义为非权威调试元数据且写入失败只记日志,样本已覆盖窗口时不再因 manifest 损坏而判为无效。 * chore(recipes): drop the Qwen3.5 8P1D benchmark placement that restates the head default srt-slurm schema.py:1162 already places the benchmark client on head, the dedicated frontend node. 中文:删除 Qwen3.5 8P1D recipe 中重复默认值的 benchmark placement;srt-slurm schema.py:1162 默认即在 head(即 dedicated frontend 节点)运行 benchmark client。 * refactor(power): drop the multinode CLI flag no caller passes fixed_sequence.py and the AgentX adapter call run() with expected_cpu directly; nothing invokes multinode.py as a command. 中文:删除 multinode CLI 中无人传入的 --expected-cpu-power;fixed_sequence.py 与 AgentX adapter 直接以 expected_cpu 调用 run(),没有任何地方以命令行方式调用 multinode.py。 * refactor(matrix): read cpu-power fields directly like the sibling validators Both schemas declare cpu_power and power, so the getattr defaults implied fields that could be absent. 中文:cpu-power 校验与同文件其他校验器一致,直接读取字段;两个 schema 都声明 cpu_power 与 power,getattr 默认值暗示字段可能缺失。 * test(power): assert the setup pin through CPU_POWER_EXPORTER_RELEASE The release bump then edits one constant instead of a test literal. 中文:测试改用 CPU_POWER_EXPORTER_RELEASE 常量断言 setup 版本,升级版本时只需修改一处常量。 * feat(power): publish NVML module power from CPU samples v3 v2 files are still accepted, and module power is a reference rail that is never summed into the CPU totals. 中文:从 CPU 采样 v3 中发布 NVML module 功耗。仍兼容 v2 文件;module 功耗仅作为参考 rail,不计入 CPU 总功耗。 * chore: bump srt-slurm and cpu-power-exporter to v2.59.3 v2.59.3's aarch64 exporter links glibc, so the NVML module reader loads and module_w is filled. 中文:将 srt-slurm 子模块与 cpu-power-exporter 升级到 v2.59.3;其 aarch64 exporter 改为 glibc 链接,可加载 NVML,module_w 不再为空。 * feat: collect Grace CPU power on the Kimi K3 GB200 disaggregated AgentX recipe 中文:在 Kimi K3 GB200 disaggregated AgentX 配置上采集 Grace CPU 功耗。 * chore: limit CPU power to Qwen3.5 8k1k and Kimi K3 GB300 agg Each published AgentX curve is replaced by the newest run, and the GB300 agg key is the only Kimi key that owns its curve alone. 中文:仅在 Qwen3.5 8k1k 与 Kimi K3 GB300 agg 配置上采集 CPU 功耗。 * chore: drop Qwen3.5 8k1k from NVL72 CPU power collection 中文:从 NVL72 CPU 功耗采集中移除 Qwen3.5 8k1k * chore: pin cpu-power-exporter v2.65.2 to match srt-slurm 中文:将 cpu-power-exporter 固定到 v2.65.2,与 srt-slurm 子模块一致 * chore: bump cpu-power-exporter to v2.66.1 with the submodule * chore: drop the dead cpu_power_exporter fingerprint exclusion * chore: remove section header comments from the power leg * docs: describe the cpu-power row key, exporter record and workers entries * feat: integrate AMD RAPL package power in the CPU power leg The leg picks each source's socket meter, so a RAPL package no longer reads as a Grace socket count mismatch. * feat: bind the cluster's RAPL CPU exporter on single-node cpu-power points Only native power points bind it; a job that binds a RAPL exporter grants the job user the RAPL counters and restores root at teardown. * chore: collect CPU power on the AMD 8k1k sglang rows * fix: warn instead of failing the job when the RAPL grant fails A missing counter costs the CPU leg, whose verdict enforces required power; teardown is appended, not replaced. * refactor: define the CPU power source enum once * docs: note where a failed RAPL grant or restore shows up * test: check CPU socket meters against srt-slurm's sensor suffixes * fix: index workers[] power entries by the manifest worker * docs: shorten the NVL72 CPU power changelog entry * test: log fake sudo arguments with printf so dash keeps -n * style: rewrap bound_variant docstring to 100 columns * fix: refuse RAPL CPU power on shared nodes [skip-sweep] * feat: measure CPU power by default on clusters with a CPU exporter [skip-sweep]
1 parent 62d2e10 commit 115b00b

25 files changed

Lines changed: 456 additions & 103 deletions

File tree

‎.github/AGENT_OPERATIONS.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,8 @@ A multinode master row with `cpu-power: true` beside `power: true` adds srt-slur
7878

7979
The package then carries `power/cpu/samples.csv` in srt-slurm's wide format (one row per scrape, host, and Grace socket; two sockets per worker host in the manifest topology), and the aggregate adds `cpu_power_valid`, `avg_cpu_socket_power_w`, `avg_total_cpu_power_w`, and `total_cpu_energy_j` from the `CPU<n>:cpuSidePowerUsageW` series, plus `avg_total_cpu_rail_power_w` / `total_cpu_rail_energy_j` (ACPI `CPU Power Socket N`, cores only) and `avg_total_cpu_sysio_power_w` / `total_cpu_sysio_energy_j` (ACPI `SysIO Power Socket N`), each pair only when every socket reports that rail without gaps. The meters nest ([NVIDIA Grace power guide](https://docs.nvidia.com/dccpu/grace-perf-tuning-guide/power-thermals.html)): `Grace Power Socket N` is the whole socket (CPU rail, SysIO rail, LPDDR5X, and regulator loss) and the two rails are a breakdown inside it, so never add them to or subtract them from the total; a GB300 socket reads about 97 W while its rails sum to about 55 W. The DRAM rail is blank on the CI NVL72 firmware. srt-slurm v2.59.0+ writes NVML superchip module power as `module_w` (CPU samples schema v3) when its exporter can load NVML (the aarch64 glibc build, srt-slurm #609), and the aggregate then publishes `avg_total_module_power_w` / `total_module_energy_j` under the same every-socket, gap-free rule; the module includes both GPUs, so never add it to the CPU or GPU totals. Every series is integrated over the bound formal GPU window with the GPU leg's trapezoid, boundary interpolation, and 3.0 s gap limit; the sidecar's `cpu` block, mirrored as `power_audit.cpu`, records `sensor_kind: grace_socket`, `source`, socket and row counts, and reason codes. A valid GPU leg also publishes `workers[]`: one entry per worker per host (`role`, the manifest's `worker_idx`, `hosts` holding that one host, `num_gpus`, `avg_power_w`), so a worker spanning trays repeats its index once per host and can be modeled per tray; it is omitted when a GPU belongs to two workers.
8080

81+
A native single-node fixed-sequence row with `cpu-power: true` adds the same leg. Single-node rows have no `power` field because their native points always measure GPU power; eval-only and acceptance-gate jobs bind no CPU exporter. RAPL clusters (`mi300x-amd`, `mi325x-amd`, `mi355x-amds`) read each AMD EPYC package as `CPU<n>:packagePowerUsageW` and publish the three headline CPU keys with `sensor_kind: rapl_package` and no rail keys. RAPL `energy_uj` counters are root-only, so a job that binds a RAPL exporter adds a host-setup command that gives the job user each top-level counter and a teardown that returns them to root. A failed grant only warns, so the CPU leg is then judged by the verdict below; a cancelled job skips teardown, and a failed restore shows up only in the job's `logs/host_teardown_<node>.out`.
82+
8183
The CPU verdict is independent of `power_valid`: a failed leg records `cpu_power_valid: 0` with no CPU keys and leaves every GPU field unchanged. It fails the job only when power is required (`REQUIRE_POWER=1`, or always on AgentX lanes whose launcher validates power) and the row set `cpu-power`; missing CPU artifacts are then a verdict (`cpu_artifacts_missing`) rather than silence. Other reason codes: `cpu_samples_missing`, `cpu_samples_header_mismatch`, `cpu_samples_malformed`, `cpu_socket_count_mismatch` (also when a host reports only the DCGM CPU rail), `cpu_sample_gap_exceeded`, `cpu_window_not_bracketed`, `cpu_window_unavailable`, and the shared `non_finite_power_metric`.
8284

8385
Every power result, valid or invalid, carries `power_metric_schema_version`. Version 2 defines each unprefixed `joules_per_*` field as whole-deployment GPU-board energy over the named denominator; role-scoped energy uses the explicit `prefill_*` / `decode_*` keys. Rows without the field predate the whole-deployment switch and their unprefixed joules are not comparable across topologies.

‎.github/workflows/benchmark-tmpl.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,7 @@ env:
101101
SALLOC_TIME_LIMIT: '480'
102102
IS_MULTINODE: 'false'
103103
REQUIRE_POWER: ${{ inputs.require-power && '1' || '0' }}
104+
CPU_POWER: ${{ fromJSON(inputs.config)['cpu-power'] && '1' || '0' }}
104105
RANDOM_RANGE_RATIO: 0.8
105106
HF_TOKEN: ${{ secrets.INFERENCEX_OFFICIAL_RO_HF_TOKEN }}
106107
HF_HUB_CACHE: '/mnt/hf_hub_cache/'

‎inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/RECIPES.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -87,7 +87,7 @@ recipe and master settings; the recipe still owns the native configuration.
8787

8888
Active recipes, fixed-sequence and AgentX, single- and multi-node, are fragments: native srt-slurm YAML holding only recipe-specific settings. At launch the fragment is composed and bound:
8989

90-
1. The lane's shared block in [`configs/srt-recipes/`](../../../configs/srt-recipes) (`fixed-sequence-{single,multi}.yaml` or `agentic.yaml`) is merged under the fragment (under `base` for bundles). A multi-node row with `power: true` also gets [`telemetry-dcgm.yaml`](../../../configs/srt-recipes/telemetry-dcgm.yaml), its exporter on the cluster's `srt-slurm.power-exporter-port`; the lane must allow power (`POWER_LANES`). The fragment wins: mappings merge and lists replace, so a fragment may keep telemetry tunables such as `collector_join_timeout_seconds`. Native single-node benchmarks also get [`telemetry-native.yaml`](../../../configs/srt-recipes/telemetry-native.yaml) before variant selection. That block enables best-effort power collection with the cluster's `default_gpu_exporter`; eval-only points omit collection. The binder sets `telemetry.required` from `REQUIRE_POWER`, so a single-node fragment may not set it, and gives AgentX clients matching power switches.
90+
1. The lane's shared block in [`configs/srt-recipes/`](../../../configs/srt-recipes) (`fixed-sequence-{single,multi}.yaml` or `agentic.yaml`) is merged under the fragment (under `base` for bundles). A multi-node row with `power: true` also gets [`telemetry-dcgm.yaml`](../../../configs/srt-recipes/telemetry-dcgm.yaml), its exporter on the cluster's `srt-slurm.power-exporter-port`; the lane must allow power (`POWER_LANES`). The fragment wins: mappings merge and lists replace, so a fragment may keep telemetry tunables such as `collector_join_timeout_seconds`. Native single-node benchmarks also get [`telemetry-native.yaml`](../../../configs/srt-recipes/telemetry-native.yaml) before variant selection. That block enables best-effort power collection with the cluster's `default_gpu_exporter`; eval-only points omit collection. A native single-node point on a cluster that declares `srt-slurm.cpu-power-exporter`, and a `cpu-power: true` row, also gets `telemetry.cpu_power_exporter` with that source. The binder sets `telemetry.required` from `REQUIRE_POWER`, so a single-node fragment may not set it, and gives AgentX clients matching power switches.
9191
2. The row's selector picks the variant. A single-node variant may name its `benchmark.env.CONC`, which keeps that concurrency paired with its tuning.
9292
3. The binder ([`workload.py`](../../../infx/srt_slurm/workload.py)) writes the matrix point into the selected recipe: `model.path: hf:<model>`, `model.container: <image>`, `model.precision`, `identity.container.image` (as a registry reference) and `identity.model.repo` when the fragment declares `identity.container`/`identity.model`, and `benchmark.concurrencies` when telemetry is enabled. Fixed-sequence recipes also get `benchmark.env.ISL`/`OSL`; single-node recipes get `MODEL` and `CONC`, and fixed-sequence ones `RANDOM_RANGE_RATIO` and `USE_CHAT_TEMPLATE` (`true` exactly when the recipe speculates). Multi-node AgentX recipes get the client's `RESULT_DIR`, `AIPERF_DATASET_MMAP_CACHE_DIR` and `HF_HUB_CACHE` from the launcher, which derives them from the volumes it mounts (`volume-mounts`, `agentic-volume-mounts`, lane mounts); a fragment that sets `HF_HOME` keeps its own cache. They also get `IS_MULTINODE: "true"` and `ENABLE_AGENTX_POWER`, `"1"` exactly when the launcher collects the point's AgentX power (a `power: true` row whose lane rule collects AgentX or adapter power) and `"0"` otherwise, and `REQUIRE_POWER`, `"1"` when the workflow requires power or the lane rule marks the point `require`, so the client fails a point whose power is invalid. The multi-node client reads `CONC_LIST`, `CONC`, `MODEL` and the other matrix inputs, prefill and decode topology included, from the job environment, so fragments do not copy them (`TP`, `PP_SIZE`, `PCP_SIZE` and `EP_SIZE` describe single-node points only); single-node AgentX points get them as runtime arguments.
9393
4. A component a repository setup script installs gets the master config's version. When `setup_script`, or a service `preamble` that runs `/configs/<script>`, is one of the installers in the binder's `INSTALLERS`, the binder writes the point's `router` or `kv-offload-backend` version as `ROUTER_VERSION` or `KV_OFFLOAD_BACKEND_VERSION` into the top-level `environment` (workers and static routers) or into that service's `env` (services do not inherit `environment`). A point that does not declare the version fails, and a `vllm-router==` pin in `SETUP_PIP_PACKAGES` must equal the master router version.
@@ -102,7 +102,7 @@ Inspect what the launcher submits without a cluster:
102102
uv run --extra recipes infx generate --config-key 'dsr1-fp8-h200-*' --output-dir /tmp/recipes
103103
```
104104

105-
It writes one bound recipe per fixed-sequence or AgentX point, validated by the pinned srtctl, plus a `manifest.json` that maps each file to its matrix point and to the one cluster its runner label schedules on (`null` when the label spans several). Multi-node points take their cluster's binder inputs (DCGM exporter port, AgentX client env); a point that needs them fails unless its runner label names one cluster and the lane accepts the row's `power`. Native single-node points validate against each eligible cluster's exporter defaults while retaining cluster-independent recipe output. Fabric references take that cluster's facts; when the label spans several clusters they stay references. Launch-time edits such as the job name, health-check floor and runtime `--set` values are not applied.
105+
It writes one bound recipe per fixed-sequence or AgentX point, validated by the pinned srtctl, plus a `manifest.json` that maps each file to its matrix point and to the one cluster its runner label schedules on (`null` when the label spans several). Multi-node points take their cluster's binder inputs (DCGM exporter port, AgentX client env); a point that needs them fails unless its runner label names one cluster and the lane accepts the row's `power`. Native single-node points validate against each eligible cluster's exporter defaults while retaining cluster-independent recipe output, except that a point whose runner label names one cluster takes that cluster's CPU exporter source, and a `cpu-power` point must name one. Fabric references take that cluster's facts; when the label spans several clusters they stay references. Launch-time edits such as the job name, health-check floor and runtime `--set` values are not applied.
106106

107107
## Recipe fingerprints
108108

‎inferencex-e2e/configs/CONFIGS.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -154,7 +154,7 @@ input.
154154
Notes:
155155
- The fields above are the common ones, not the full schema. The Pydantic models in [`infx/matrix/validation.py`](../infx/matrix/validation.py) are the authoritative contract; they also accept fields such as `spec-decoding`, `num-nodes`, `require-power` and `power`, and they reject any field they do not define, which fails matrix generation.
156156
- A multinode search-space entry with `power: true` measures DCGM power: its rows carry `power: true`, the launcher merges the shared telemetry block into the recipe and rejects lanes without a power rule.
157-
- A multinode entry with `cpu-power: true` beside `power: true` also measures CPU power: its rows carry `cpu-power: true` and the launcher binds the cluster's `cpu-power-exporter` record beside the DCGM block. Single-node entries cannot carry it.
157+
- A multinode entry with `cpu-power: true` beside `power: true` also measures CPU power: its rows carry `cpu-power: true` and the launcher binds the cluster's `cpu-power-exporter` record beside the DCGM block. Native single-node points measure CPU power by default on a cluster that declares a `cpu-power-exporter`; a single-node fixed-sequence entry sets `cpu-power: true` alone to count a missing CPU leg as invalid.
158158
- srt-slurm entries require `srt-recipe-dir`. Every row requires `srt-recipe: <file>[:<selector>]` under it (selector: `base`, `override_<name>` or `zip_override_<name>[<i>]`). Multinode rows may add `eval-srt-recipe`. Validation rejects missing files or variants, and `CONFIG_FILE`/`EVAL_CONFIG_FILE` settings. Only `llmd-vllm` keeps `CONFIG_FILE`. Generated rows carry the path from `inferencex-e2e/`.
159159
- Setting the fields above only guarantees that their values are passed as environment variables to benchmark scripts. Single-node jobs receive `PP_SIZE`, `DCP_SIZE`, and `PCP_SIZE`. Multinode jobs receive `PREFILL_PP_SIZE`, `PREFILL_DCP_SIZE`, `PREFILL_PCP_SIZE`, `DECODE_PP_SIZE`, `DECODE_DCP_SIZE`, and `DECODE_PCP_SIZE`. Actually using those variables is an implementation detail of the benchmark Bash script.
160160

@@ -266,9 +266,10 @@ schema; unknown keys fail.
266266
volumes every job mounts to container paths, `mounts` does the same for host
267267
paths outside the volumes (devices), `agentic-volume-mounts` adds world-writable
268268
volumes for multi-node AgentX jobs, `power-exporter-port` is the DCGM exporter port
269-
of power points, `cpu-power-exporter` (`source`) is the CPU power exporter a
270-
`cpu-power` row binds beside it (`make setup` installs it from the pinned srt-slurm
271-
release), `env` is the environment of the srt-slurm
269+
of power points, `cpu-power-exporter` (`source: acpi` or `rapl`) is the CPU power exporter native
270+
single-node points and `cpu-power` rows bind beside it (`make setup` installs it from the pinned srt-slurm
271+
release; a RAPL job's host setup gives the job user the RAPL counters until teardown),
272+
`env` is the environment of the srt-slurm
272273
launch, `container-aliases` and `nginx-aliases` name the recipe containers that
273274
resolve to the main image and to the staged nginx, and `outputs`,
274275
`shared-run-root` and `uv-cache-root` are the directories the srt-slurm launcher

‎inferencex-e2e/configs/amd-master.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -107,6 +107,7 @@ dsr1-fp8-mi300x-sglang:
107107
conc-start: 4
108108
conc-end: 64
109109
srt-recipe: 8k1k.yaml
110+
cpu-power: true
110111

111112
dsr1-fp8-mi325x-sglang:
112113
image: lmsysorg/sglang:v0.5.22-rocm720-mi30x@sha256:58b6bc333f527c88fe5ff1b4bd8d17e82214eabcb3044549117159878967c930
@@ -126,6 +127,7 @@ dsr1-fp8-mi325x-sglang:
126127
conc-start: 4
127128
conc-end: 64
128129
srt-recipe: 8k1k.yaml
130+
cpu-power: true
129131

130132
dsr1-fp8-mi355x-sglang:
131133
image: lmsysorg/sglang:v0.5.22-rocm720-mi35x@sha256:c4b61360b251d62bb17da63f175d571ab22479eb46a144548d79f3ed3998ea11
@@ -170,6 +172,7 @@ dsr1-fp8-mi355x-sglang-mtp:
170172
conc-end: 64
171173
spec-decoding: mtp
172174
srt-recipe: 8k1k.yaml
175+
cpu-power: true
173176

174177
# MI355X FP8 AgentX, TP4 only. The grid follows the TP4 band of the B200 FP8
175178
# AgentX arm so the two SKUs are directly comparable point for point, with the

‎inferencex-e2e/configs/runners.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -709,6 +709,7 @@ clusters:
709709
single-node-time-limit: 180
710710
gpus-per-node-directive: true
711711
segment-directive: false
712+
cpu-power-exporter: {source: rapl}
712713
host-setup:
713714
script: runners/srt-slurm/hooks/mi300x-amd/setup.sh
714715
mounts:
@@ -758,6 +759,7 @@ clusters:
758759
single-node-time-limit: 480
759760
gpus-per-node-directive: true
760761
segment-directive: false
762+
cpu-power-exporter: {source: rapl}
761763
mounts:
762764
/dev/kfd: /dev/kfd
763765
/dev/dri: /dev/dri
@@ -807,6 +809,7 @@ clusters:
807809
gpus-per-node-directive: true
808810
segment-directive: false
809811
outputs: /it-share/gharunners2/srt-slurm/outputs
812+
cpu-power-exporter: {source: rapl}
810813
host-setup:
811814
script: runners/srt-slurm/hooks/mi355x-amds/setup.sh
812815
env:

‎inferencex-e2e/infx/clusters/slurm.py‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@
1313
from pydantic import AfterValidator, Field, PlainSerializer, field_validator, model_validator
1414

1515
from infx.clusters.base import Record, SchedulerSettings, Visibility, Volume
16-
from infx.srt_slurm.workload import resolve_fabric
16+
from infx.srt_slurm.workload import CpuPowerSource, resolve_fabric
1717

1818
if TYPE_CHECKING:
1919
from infx.clusters import Cluster
@@ -174,13 +174,14 @@ def rendered(self) -> dict[str, str | None]:
174174

175175

176176
class CpuPowerExporter(Record):
177-
"""The Grace CPU power exporter srt-slurm scrapes on this cluster's compute trays.
177+
"""The CPU power exporter srt-slurm scrapes on this cluster's nodes.
178178
179179
A point that selects ``cpu-power`` binds ``telemetry.cpu_power_exporter`` with this source.
180-
Only ACPI serves the Grace socket meter the CPU leg integrates.
180+
ACPI serves the Grace socket meter and RAPL the AMD EPYC package counter, the meters the
181+
CPU leg integrates.
181182
"""
182183

183-
source: Literal["acpi"]
184+
source: CpuPowerSource
184185

185186

186187
class SrtSlurmSettings(Record):

‎inferencex-e2e/infx/launch/drivers/srt/__init__.py‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -69,12 +69,16 @@ def run_single_node(launch: Launch) -> int:
6969
step_mib, job_mib = None, 0
7070
if (options := config.srun_options(run.backend.settings, step_mib)) is not None:
7171
run.env["SRT_SRUN_OPTIONS"] = options
72-
if rc := submit.bind_point(run, checkout, root / "arguments"):
72+
native_power = not request.eval_only and not request.al_gate
73+
cpu_source = config.cpu_power_source(
74+
run.cluster, run.srt, native_power and request.cpu_power, default=native_power
75+
)
76+
if rc := submit.bind_point(run, checkout, root / "arguments", cpu_source):
7377
return rc
7478
selected, bound, runtime_args = submit.bound_arguments(root / "arguments")
7579
event.set(recipe=selected.removeprefix(f"{run.workspace}/"))
7680
containers = {}
77-
if not request.eval_only and not request.al_gate:
81+
if native_power:
7882
image, reference = config.stage_gpu_exporter(run, single_node=True)
7983
containers[image] = reference
8084
job_config = config.SrtJob(
@@ -90,10 +94,11 @@ def run_single_node(launch: Launch) -> int:
9094
single_node=True,
9195
account=run.account,
9296
mem_mib=job_mib,
97+
cpu_power_source=cpu_source,
9398
)
9499
config.create_volume_mounts(run)
95100
config.write(checkout.root / "srtslurm.yaml", config.render(run.cluster, job_config))
96-
if rc := run_setup(run, checkout):
101+
if rc := run_setup(run, checkout, cpu_power_exporter=cpu_source is not None):
97102
return rc
98103

99104
fetched = root / "fetched-outputs"
@@ -114,7 +119,7 @@ def run_single_node(launch: Launch) -> int:
114119
status = run.backend.state(job)
115120
with event.stage("collect"):
116121
logs = run.backend.fetch_outputs(job, fetched) / "logs"
117-
if not request.eval_only and not request.al_gate:
122+
if native_power:
118123
(logs / "power").mkdir(parents=True, exist_ok=True)
119124
(logs / "power/native-job-status.txt").write_text(f"{job.id}|{status.raw}\n")
120125
rc = collect.finalize_single_node_results(run, logs, checkout.commit)

0 commit comments

Comments
 (0)