diff --git a/.github/AGENT_OPERATIONS.md b/.github/AGENT_OPERATIONS.md index 12678f4976..e624d59699 100644 --- a/.github/AGENT_OPERATIONS.md +++ b/.github/AGENT_OPERATIONS.md @@ -78,6 +78,8 @@ A multinode master row with `cpu-power: true` beside `power: true` adds srt-slur The package then carries `power/cpu/samples.csv` in srt-slurm's wide format (one row per scrape, host, and Grace socket; two sockets per worker host in the manifest topology), and the aggregate adds `cpu_power_valid`, `avg_cpu_socket_power_w`, `avg_total_cpu_power_w`, and `total_cpu_energy_j` from the `CPU:cpuSidePowerUsageW` series, plus `avg_total_cpu_rail_power_w` / `total_cpu_rail_energy_j` (ACPI `CPU Power Socket N`, cores only) and `avg_total_cpu_sysio_power_w` / `total_cpu_sysio_energy_j` (ACPI `SysIO Power Socket N`), each pair only when every socket reports that rail without gaps. The meters nest ([NVIDIA Grace power guide](https://docs.nvidia.com/dccpu/grace-perf-tuning-guide/power-thermals.html)): `Grace Power Socket N` is the whole socket (CPU rail, SysIO rail, LPDDR5X, and regulator loss) and the two rails are a breakdown inside it, so never add them to or subtract them from the total; a GB300 socket reads about 97 W while its rails sum to about 55 W. The DRAM rail is blank on the CI NVL72 firmware. srt-slurm v2.59.0+ writes NVML superchip module power as `module_w` (CPU samples schema v3) when its exporter can load NVML (the aarch64 glibc build, srt-slurm #609), and the aggregate then publishes `avg_total_module_power_w` / `total_module_energy_j` under the same every-socket, gap-free rule; the module includes both GPUs, so never add it to the CPU or GPU totals. Every series is integrated over the bound formal GPU window with the GPU leg's trapezoid, boundary interpolation, and 3.0 s gap limit; the sidecar's `cpu` block, mirrored as `power_audit.cpu`, records `sensor_kind: grace_socket`, `source`, socket and row counts, and reason codes. A valid GPU leg also publishes `workers[]`: one entry per worker per host (`role`, the manifest's `worker_idx`, `hosts` holding that one host, `num_gpus`, `avg_power_w`), so a worker spanning trays repeats its index once per host and can be modeled per tray; it is omitted when a GPU belongs to two workers. +A native single-node fixed-sequence row with `cpu-power: true` adds the same leg. Single-node rows have no `power` field because their native points always measure GPU power; eval-only and acceptance-gate jobs bind no CPU exporter. RAPL clusters (`mi300x-amd`, `mi325x-amd`, `mi355x-amds`) read each AMD EPYC package as `CPU:packagePowerUsageW` and publish the three headline CPU keys with `sensor_kind: rapl_package` and no rail keys. RAPL `energy_uj` counters are root-only, so a job that binds a RAPL exporter adds a host-setup command that gives the job user each top-level counter and a teardown that returns them to root. A failed grant only warns, so the CPU leg is then judged by the verdict below; a cancelled job skips teardown, and a failed restore shows up only in the job's `logs/host_teardown_.out`. + The CPU verdict is independent of `power_valid`: a failed leg records `cpu_power_valid: 0` with no CPU keys and leaves every GPU field unchanged. It fails the job only when power is required (`REQUIRE_POWER=1`, or always on AgentX lanes whose launcher validates power) and the row set `cpu-power`; missing CPU artifacts are then a verdict (`cpu_artifacts_missing`) rather than silence. Other reason codes: `cpu_samples_missing`, `cpu_samples_header_mismatch`, `cpu_samples_malformed`, `cpu_socket_count_mismatch` (also when a host reports only the DCGM CPU rail), `cpu_sample_gap_exceeded`, `cpu_window_not_bracketed`, `cpu_window_unavailable`, and the shared `non_finite_power_metric`. Every power result, valid or invalid, carries `power_metric_schema_version`. Version 2 defines each unprefixed `joules_per_*` field as whole-deployment GPU-board energy over the named denominator; role-scoped energy uses the explicit `prefill_*` / `decode_*` keys. Rows without the field predate the whole-deployment switch and their unprefixed joules are not comparable across topologies. diff --git a/.github/workflows/benchmark-tmpl.yml b/.github/workflows/benchmark-tmpl.yml index 2dd76c0137..cc85c88638 100644 --- a/.github/workflows/benchmark-tmpl.yml +++ b/.github/workflows/benchmark-tmpl.yml @@ -101,6 +101,7 @@ env: SALLOC_TIME_LIMIT: '480' IS_MULTINODE: 'false' REQUIRE_POWER: ${{ inputs.require-power && '1' || '0' }} + CPU_POWER: ${{ fromJSON(inputs.config)['cpu-power'] && '1' || '0' }} RANDOM_RANGE_RATIO: 0.8 HF_TOKEN: ${{ secrets.INFERENCEX_OFFICIAL_RO_HF_TOKEN }} HF_HUB_CACHE: '/mnt/hf_hub_cache/' diff --git a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/RECIPES.md b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/RECIPES.md index b041efa853..8b7be64380 100644 --- a/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/RECIPES.md +++ b/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/RECIPES.md @@ -87,7 +87,7 @@ recipe and master settings; the recipe still owns the native configuration. Active recipes, fixed-sequence and AgentX, single- and multi-node, are fragments: native srt-slurm YAML holding only recipe-specific settings. At launch the fragment is composed and bound: -1. The lane's shared block in [`configs/srt-recipes/`](../../../configs/srt-recipes) (`fixed-sequence-{single,multi}.yaml` or `agentic.yaml`) is merged under the fragment (under `base` for bundles). A multi-node row with `power: true` also gets [`telemetry-dcgm.yaml`](../../../configs/srt-recipes/telemetry-dcgm.yaml), its exporter on the cluster's `srt-slurm.power-exporter-port`; the lane must allow power (`POWER_LANES`). The fragment wins: mappings merge and lists replace, so a fragment may keep telemetry tunables such as `collector_join_timeout_seconds`. Native single-node benchmarks also get [`telemetry-native.yaml`](../../../configs/srt-recipes/telemetry-native.yaml) before variant selection. That block enables best-effort power collection with the cluster's `default_gpu_exporter`; eval-only points omit collection. The binder sets `telemetry.required` from `REQUIRE_POWER`, so a single-node fragment may not set it, and gives AgentX clients matching power switches. +1. The lane's shared block in [`configs/srt-recipes/`](../../../configs/srt-recipes) (`fixed-sequence-{single,multi}.yaml` or `agentic.yaml`) is merged under the fragment (under `base` for bundles). A multi-node row with `power: true` also gets [`telemetry-dcgm.yaml`](../../../configs/srt-recipes/telemetry-dcgm.yaml), its exporter on the cluster's `srt-slurm.power-exporter-port`; the lane must allow power (`POWER_LANES`). The fragment wins: mappings merge and lists replace, so a fragment may keep telemetry tunables such as `collector_join_timeout_seconds`. Native single-node benchmarks also get [`telemetry-native.yaml`](../../../configs/srt-recipes/telemetry-native.yaml) before variant selection. That block enables best-effort power collection with the cluster's `default_gpu_exporter`; eval-only points omit collection. A native single-node point on a cluster that declares `srt-slurm.cpu-power-exporter`, and a `cpu-power: true` row, also gets `telemetry.cpu_power_exporter` with that source. The binder sets `telemetry.required` from `REQUIRE_POWER`, so a single-node fragment may not set it, and gives AgentX clients matching power switches. 2. The row's selector picks the variant. A single-node variant may name its `benchmark.env.CONC`, which keeps that concurrency paired with its tuning. 3. The binder ([`workload.py`](../../../infx/srt_slurm/workload.py)) writes the matrix point into the selected recipe: `model.path: hf:`, `model.container: `, `model.precision`, `identity.container.image` (as a registry reference) and `identity.model.repo` when the fragment declares `identity.container`/`identity.model`, and `benchmark.concurrencies` when telemetry is enabled. Fixed-sequence recipes also get `benchmark.env.ISL`/`OSL`; single-node recipes get `MODEL` and `CONC`, and fixed-sequence ones `RANDOM_RANGE_RATIO` and `USE_CHAT_TEMPLATE` (`true` exactly when the recipe speculates). Multi-node AgentX recipes get the client's `RESULT_DIR`, `AIPERF_DATASET_MMAP_CACHE_DIR` and `HF_HUB_CACHE` from the launcher, which derives them from the volumes it mounts (`volume-mounts`, `agentic-volume-mounts`, lane mounts); a fragment that sets `HF_HOME` keeps its own cache. They also get `IS_MULTINODE: "true"` and `ENABLE_AGENTX_POWER`, `"1"` exactly when the launcher collects the point's AgentX power (a `power: true` row whose lane rule collects AgentX or adapter power) and `"0"` otherwise, and `REQUIRE_POWER`, `"1"` when the workflow requires power or the lane rule marks the point `require`, so the client fails a point whose power is invalid. The multi-node client reads `CONC_LIST`, `CONC`, `MODEL` and the other matrix inputs, prefill and decode topology included, from the job environment, so fragments do not copy them (`TP`, `PP_SIZE`, `PCP_SIZE` and `EP_SIZE` describe single-node points only); single-node AgentX points get them as runtime arguments. 4. A component a repository setup script installs gets the master config's version. When `setup_script`, or a service `preamble` that runs `/configs/