Skip to content

Commit 1e9feb6

Browse files
committed
feat: collect NVL72 power by default
1 parent 6481779 commit 1e9feb6

14 files changed

Lines changed: 244 additions & 70 deletions

File tree

‎.github/AGENT_OPERATIONS.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -74,15 +74,15 @@ Multinode srt-slurm results, and single-node fixed-sequence and AgentX results w
7474

7575
Multinode disaggregated results add `prefill_gpu_energy_j`, `decode_gpu_energy_j`, `prefill_avg_power_w`, `decode_avg_power_w`, `prefill_joules_per_input_token`, and `decode_joules_per_output_token`. Role energy covers the full formal benchmark window, not kernel-level phases, and the role watts are that energy divided by the same window and by the role's GPU count.
7676

77-
A multinode master row with `cpu-power: true` beside `power: true` adds srt-slurm's Grace CPU leg. The binder merges `telemetry.cpu_power_exporter` into the bound recipe from the cluster's `srt-slurm.cpu-power-exporter` record in `inferencex-e2e/configs/runners.yaml` (ACPI on `gb200-nv` and `gb300-nv`), the same way it merges the DCGM block, so recipe fragments and `additional-settings` never carry it; `make setup` pins the exporter release from `CPU_POWER_EXPORTER_RELEASE` in `infx/launch/drivers/srt/checkout.py` for those jobs only, and the workflow exports `CPU_POWER=1` once for both the launcher and the result processor.
77+
A multinode master row with `cpu-power: true` beside `power: true` adds srt-slurm's Grace CPU leg; on `gb200-nv` and `gb300-nv` every multinode benchmark point binds the leg, and `cpu-power: true` makes it expected. The binder merges `telemetry.cpu_power_exporter` into the bound recipe from the cluster's `srt-slurm.cpu-power-exporter` record in `inferencex-e2e/configs/runners.yaml` (ACPI on `gb200-nv` and `gb300-nv`), the same way it merges the DCGM block, so recipe fragments and `additional-settings` never carry it; `make setup` pins the exporter release from `CPU_POWER_EXPORTER_RELEASE` in `infx/launch/drivers/srt/checkout.py` for jobs that bind it, and the workflow exports `CPU_POWER=1` once for both the launcher and the result processor.
7878

7979
The package then carries `power/cpu/samples.csv` in srt-slurm's wide format (one row per scrape, host, and Grace socket; two sockets per worker host in the manifest topology), and the aggregate adds `cpu_power_valid`, `avg_cpu_socket_power_w`, `avg_total_cpu_power_w`, and `total_cpu_energy_j` from the `CPU<n>:cpuSidePowerUsageW` series, plus `avg_total_cpu_rail_power_w` / `total_cpu_rail_energy_j` (ACPI `CPU Power Socket N`, cores only) and `avg_total_cpu_sysio_power_w` / `total_cpu_sysio_energy_j` (ACPI `SysIO Power Socket N`), each pair only when every socket reports that rail without gaps. The meters nest ([NVIDIA Grace power guide](https://docs.nvidia.com/dccpu/grace-perf-tuning-guide/power-thermals.html)): `Grace Power Socket N` is the whole socket (CPU rail, SysIO rail, LPDDR5X, and regulator loss) and the two rails are a breakdown inside it, so never add them to or subtract them from the total; a GB300 socket reads about 97 W while its rails sum to about 55 W. The DRAM rail is blank on the CI NVL72 firmware. srt-slurm v2.59.0+ writes NVML superchip module power as `module_w` (CPU samples schema v3) when its exporter can load NVML (the aarch64 glibc build, srt-slurm #609), and the aggregate then publishes `avg_total_module_power_w` / `total_module_energy_j` under the same every-socket, gap-free rule; the module includes both GPUs, so never add it to the CPU or GPU totals. Every series is integrated over the bound formal GPU window with the GPU leg's trapezoid, boundary interpolation, and 3.0 s gap limit; the sidecar's `cpu` block, mirrored as `power_audit.cpu`, records `sensor_kind: grace_socket`, `source`, socket and row counts, and reason codes. A valid GPU leg also publishes `workers[]`: one entry per worker per host (`role`, the manifest's `worker_idx`, `hosts` holding that one host, `num_gpus`, `avg_power_w`), so a worker spanning trays repeats its index once per host and can be modeled per tray; it is omitted when a GPU belongs to two workers.
8080

81-
The CPU verdict is independent of `power_valid`: a failed leg records `cpu_power_valid: 0` with no CPU keys and leaves every GPU field unchanged. It fails the job only when power is required (`REQUIRE_POWER=1`, or always on AgentX lanes whose launcher validates power) and the row set `cpu-power`; missing CPU artifacts are then a verdict (`cpu_artifacts_missing`) rather than silence. Other reason codes: `cpu_samples_missing`, `cpu_samples_header_mismatch`, `cpu_samples_malformed`, `cpu_socket_count_mismatch` (also when a host reports only the DCGM CPU rail), `cpu_sample_gap_exceeded`, `cpu_window_not_bracketed`, `cpu_window_unavailable`, and the shared `non_finite_power_metric`.
81+
The CPU verdict is independent of `power_valid`: a failed leg records `cpu_power_valid: 0` with no CPU keys and leaves every GPU field unchanged. It fails the job only when power is required (`REQUIRE_POWER=1`, or on AgentX lanes whose launcher validates power, except NVL72 points whose row omits `power`) and the row set `cpu-power`; missing CPU artifacts are then a verdict (`cpu_artifacts_missing`) rather than silence. Other reason codes: `cpu_samples_missing`, `cpu_samples_header_mismatch`, `cpu_samples_malformed`, `cpu_socket_count_mismatch` (also when a host reports only the DCGM CPU rail), `cpu_sample_gap_exceeded`, `cpu_window_not_bracketed`, `cpu_window_unavailable`, and the shared `non_finite_power_metric`.
8282

8383
Every power result, valid or invalid, carries `power_metric_schema_version`. Version 2 defines each unprefixed `joules_per_*` field as whole-deployment GPU-board energy over the named denominator; role-scoped energy uses the explicit `prefill_*` / `decode_*` keys. Rows without the field predate the whole-deployment switch and their unprefixed joules are not comparable across topologies.
8484

85-
A multinode master row with `power: true` enables official energy collection through the shared telemetry fragment; native single-node throughput and AgentX jobs enable telemetry through their bound recipe. Native single-node jobs inherit the exporter image, labels and metrics from the cluster's `default_gpu_exporter` in `inferencex-e2e/configs/runners.yaml`: DCGM on NVIDIA, the `kind: custom` AMD device-metrics-exporter on AMD. Multi-node power rows keep their shared DCGM telemetry block. Eval-only jobs omit collection. The Git submodule pointer at `inferencex-e2e/utils/srt-slurm` is the source of truth for every srt-slurm job, including TileRT. CI derives `POWER_PRODUCER_SHA` from the launcher stamp. The aggregate-power and AgentX power tests validate telemetry and provenance. These local tests do not prove hardware power collection. Eligible `dynamo-sglang` dcgm-power lanes are validated.
85+
A multinode master row with `power: true` enables official energy collection through the shared telemetry fragment. On `gb200-nv` and `gb300-nv` every multinode benchmark point measures GPU and Grace CPU power whether or not its row sets `power`; invalid telemetry then records `power_valid: 0` and keeps the result, while `power: true` keeps invalid AgentX power fatal. Native single-node throughput and AgentX jobs enable telemetry through their bound recipe. Native single-node jobs inherit the exporter image, labels and metrics from the cluster's `default_gpu_exporter` in `inferencex-e2e/configs/runners.yaml`: DCGM on NVIDIA, the `kind: custom` AMD device-metrics-exporter on AMD. Multi-node power rows keep their shared DCGM telemetry block. Eval-only jobs omit collection. The Git submodule pointer at `inferencex-e2e/utils/srt-slurm` is the source of truth for every srt-slurm job, including TileRT. CI derives `POWER_PRODUCER_SHA` from the launcher stamp. The aggregate-power and AgentX power tests validate telemetry and provenance. These local tests do not prove hardware power collection. Eligible `dynamo-sglang` dcgm-power lanes are validated.
8686

8787
Power audit artifacts are named `power_audit_<result>` and contain the `power_validation*.json` sidecars plus, for native telemetry jobs, `LOGS/power/` (`samples.csv` v3 with `temperature_c`, `manifest.json`, `windows/`). They are uploaded even when validation fails.
8888

‎inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/RECIPES.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -53,9 +53,9 @@ All referenced recipes must be checked in: srt-slurm 2 ships curated examples in
5353

5454
Active recipes, fixed-sequence and AgentX, single- and multi-node, are fragments: native srt-slurm YAML holding only recipe-specific settings. At launch the fragment is composed and bound:
5555

56-
1. The lane's shared block in [`configs/srt-recipes/`](../../../configs/srt-recipes) (`fixed-sequence-{single,multi}.yaml` or `agentic.yaml`) is merged under the fragment (under `base` for bundles). A multi-node row with `power: true` also gets [`telemetry-dcgm.yaml`](../../../configs/srt-recipes/telemetry-dcgm.yaml), its exporter on the cluster's `srt-slurm.power-exporter-port`; the lane must allow power (`POWER_LANES`). The fragment wins: mappings merge and lists replace, so a fragment may keep telemetry tunables such as `collector_join_timeout_seconds`. Native single-node benchmarks also get [`telemetry-native.yaml`](../../../configs/srt-recipes/telemetry-native.yaml) before variant selection. That block enables best-effort power collection with the cluster's `default_gpu_exporter`; eval-only points omit collection. The binder sets `telemetry.required` from `REQUIRE_POWER`, so a single-node fragment may not set it, and gives AgentX clients matching power switches.
56+
1. The lane's shared block in [`configs/srt-recipes/`](../../../configs/srt-recipes) (`fixed-sequence-{single,multi}.yaml` or `agentic.yaml`) is merged under the fragment (under `base` for bundles). A multi-node row with `power: true` also gets [`telemetry-dcgm.yaml`](../../../configs/srt-recipes/telemetry-dcgm.yaml), its exporter on the cluster's `srt-slurm.power-exporter-port`; the lane must allow power (`POWER_LANES`). On `gb200-nv` and `gb300-nv` every multi-node benchmark point gets it with the Grace CPU exporter, unless its recipe runs etcd and NATS on a dedicated node. The fragment wins: mappings merge and lists replace, so a fragment may keep telemetry tunables such as `collector_join_timeout_seconds`. Native single-node benchmarks also get [`telemetry-native.yaml`](../../../configs/srt-recipes/telemetry-native.yaml) before variant selection. That block enables best-effort power collection with the cluster's `default_gpu_exporter`; eval-only points omit collection. The binder sets `telemetry.required` from `REQUIRE_POWER`, so a single-node fragment may not set it, and gives AgentX clients matching power switches.
5757
2. The row's selector picks the variant. A single-node variant may name its `benchmark.env.CONC`, which keeps that concurrency paired with its tuning.
58-
3. The binder ([`workload.py`](../../../infx/srt_slurm/workload.py)) writes the matrix point into the selected recipe: `model.path: hf:<model>`, `model.container: <image>`, `model.precision`, `identity.container.image` (as a registry reference) and `identity.model.repo` when the fragment declares `identity.container`/`identity.model`, and `benchmark.concurrencies` when telemetry is enabled. Fixed-sequence recipes also get `benchmark.env.ISL`/`OSL`; single-node recipes get `MODEL` and `CONC`, and fixed-sequence ones `RANDOM_RANGE_RATIO` and `USE_CHAT_TEMPLATE` (`true` exactly when the recipe speculates). Multi-node AgentX recipes get the client's `RESULT_DIR`, `AIPERF_DATASET_MMAP_CACHE_DIR` and `HF_HUB_CACHE` from the launcher, which derives them from the volumes it mounts (`volume-mounts`, `agentic-volume-mounts`, lane mounts); a fragment that sets `HF_HOME` keeps its own cache. They also get `IS_MULTINODE: "true"` and `ENABLE_AGENTX_POWER`, `"1"` exactly when the launcher collects the point's AgentX power (a `power: true` row whose lane rule collects AgentX or adapter power) and `"0"` otherwise, and `REQUIRE_POWER`, `"1"` when the workflow requires power or the lane rule marks the point `require`, so the client fails a point whose power is invalid. The multi-node client reads `CONC_LIST`, `CONC`, `MODEL` and the other matrix inputs, prefill and decode topology included, from the job environment, so fragments do not copy them (`TP`, `PP_SIZE`, `PCP_SIZE` and `EP_SIZE` describe single-node points only); single-node AgentX points get them as runtime arguments.
58+
3. The binder ([`workload.py`](../../../infx/srt_slurm/workload.py)) writes the matrix point into the selected recipe: `model.path: hf:<model>`, `model.container: <image>`, `model.precision`, `identity.container.image` (as a registry reference) and `identity.model.repo` when the fragment declares `identity.container`/`identity.model`, and `benchmark.concurrencies` when telemetry is enabled. Fixed-sequence recipes also get `benchmark.env.ISL`/`OSL`; single-node recipes get `MODEL` and `CONC`, and fixed-sequence ones `RANDOM_RANGE_RATIO` and `USE_CHAT_TEMPLATE` (`true` exactly when the recipe speculates). Multi-node AgentX recipes get the client's `RESULT_DIR`, `AIPERF_DATASET_MMAP_CACHE_DIR` and `HF_HUB_CACHE` from the launcher, which derives them from the volumes it mounts (`volume-mounts`, `agentic-volume-mounts`, lane mounts); a fragment that sets `HF_HOME` keeps its own cache. They also get `IS_MULTINODE: "true"` and `ENABLE_AGENTX_POWER`, `"1"` exactly when the launcher collects the point's AgentX power (a `power: true` row whose lane rule collects AgentX or adapter power, or any GB200 or GB300 AgentX point the launcher measures by default) and `"0"` otherwise, and `REQUIRE_POWER`, `"1"` when the workflow requires power or the lane rule marks the point `require`, so the client fails a point whose power is invalid. The multi-node client reads `CONC_LIST`, `CONC`, `MODEL` and the other matrix inputs, prefill and decode topology included, from the job environment, so fragments do not copy them (`TP`, `PP_SIZE`, `PCP_SIZE` and `EP_SIZE` describe single-node points only); single-node AgentX points get them as runtime arguments.
5959
4. A component a repository setup script installs gets the master config's version. When `setup_script`, or a service `preamble` that runs `/configs/<script>`, is one of the installers in the binder's `INSTALLERS`, the binder writes the point's `router` or `kv-offload-backend` version as `ROUTER_VERSION` or `KV_OFFLOAD_BACKEND_VERSION` into the top-level `environment` (workers and static routers) or into that service's `env` (services do not inherit `environment`). A point that does not declare the version fails, and a `vllm-router==` pin in `SETUP_PIP_PACKAGES` must equal the master router version.
6060

6161
A fragment that sets a key the binder or launcher writes, or that the workflow exports from the matrix point (`_POINT_KEYS`, `BOUND_ENV` in `workload.py`, and `ROUTER_VERSION` or `KV_OFFLOAD_BACKEND_VERSION` in any env; single-node variants may still name `CONC`), fails before submission, even with the bound value. Values `benchmarks/runtime_settings.sh` and the multi-node `FRAMEWORK` settings export reach every process through the job environment, so a fragment sets one only to override it (an AgentX client `TOKENIZER` names a local checkpoint while `MODEL` stays the published Hugging Face id). `hf:<model>` resolves to the cluster's staged checkpoint (`models.entries`) unless a `models.OVERRIDES` row serves the Hub snapshot, and the master image to its staged container. A multi-node fixed-sequence point with neither checkpoint nor Hub row fails before submission.
@@ -72,7 +72,7 @@ It writes one bound recipe per fixed-sequence or AgentX point, validated by the
7272

7373
## Recipe fingerprints
7474

75-
The planner gives every benchmark row a `recipe-fingerprint`: a SHA-256 of its matrix fields (except `conc` and `exp-name`) and the bound variant the launcher submits, composed with its shared block and the native single-node or multi-node `power: true` telemetry block. Concurrency values, the job `name` and everything the launcher adds for a cluster stay out, so a recipe keeps one fingerprint across concurrencies and clusters. An `eval-srt-recipe` contributes only its path; rows without an srt-slurm recipe hash their matrix fields alone.
75+
The planner gives every benchmark row a `recipe-fingerprint`: a SHA-256 of its matrix fields (except `conc` and `exp-name`) and the bound variant the launcher submits, composed with its shared block and the native single-node or multi-node `power: true` telemetry block; the GB200 and GB300 default telemetry stays out. Concurrency values, the job `name` and everything the launcher adds for a cluster stay out, so a recipe keeps one fingerprint across concurrencies and clusters. An `eval-srt-recipe` contributes only its path; rows without an srt-slurm recipe hash their matrix fields alone.
7676

7777
The planner selects variants without srtctl ([`variants.py`](../../../infx/srt_slurm/variants.py)), the way the launcher does, so a row that no variant serves fails planning instead of its launch. Before writing a `perf-changelog.yaml` entry, list the config keys whose results the change invalidates:
7878

‎inferencex-e2e/configs/CONFIGS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,7 +153,7 @@ input.
153153

154154
Notes:
155155
- The fields above are the common ones, not the full schema. The Pydantic models in [`infx/matrix/validation.py`](../infx/matrix/validation.py) are the authoritative contract; they also accept fields such as `spec-decoding`, `num-nodes`, `require-power` and `power`, and they reject any field they do not define, which fails matrix generation.
156-
- A multinode search-space entry with `power: true` measures DCGM power: its rows carry `power: true`, the launcher merges the shared telemetry block into the recipe and rejects lanes without a power rule.
156+
- A multinode search-space entry with `power: true` measures DCGM power: its rows carry `power: true`, the launcher merges the shared telemetry block into the recipe and rejects lanes without a power rule. On `gb200-nv` and `gb300-nv` every multinode benchmark point measures GPU and Grace CPU power without it, and keeps its result when telemetry fails; there `power: true` makes invalid AgentX power fail the point.
157157
- A multinode entry with `cpu-power: true` beside `power: true` also measures CPU power: its rows carry `cpu-power: true` and the launcher binds the cluster's `cpu-power-exporter` record beside the DCGM block. Single-node entries cannot carry it.
158158
- srt-slurm entries require `srt-recipe-dir`. Every row requires `srt-recipe: <file>[:<selector>]` under it (selector: `base`, `override_<name>` or `zip_override_<name>[<i>]`). Multinode rows may add `eval-srt-recipe`. Validation rejects missing files or variants, and `CONFIG_FILE`/`EVAL_CONFIG_FILE` settings. Only `llmd-vllm` keeps `CONFIG_FILE`. Generated rows carry the path from `inferencex-e2e/`.
159159
- Setting the fields above only guarantees that their values are passed as environment variables to benchmark scripts. Single-node jobs receive `PP_SIZE`, `DCP_SIZE`, and `PCP_SIZE`. Multinode jobs receive `PREFILL_PP_SIZE`, `PREFILL_DCP_SIZE`, `PREFILL_PCP_SIZE`, `DECODE_PP_SIZE`, `DECODE_DCP_SIZE`, and `DECODE_PCP_SIZE`. Actually using those variables is an implementation detail of the benchmark Bash script.

‎inferencex-e2e/infx/launch/artifacts.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -174,6 +174,7 @@ def collect_agentic_power_results(
174174
concurrencies: Sequence[int],
175175
*,
176176
results_python: str | None,
177+
require_power: bool = True,
177178
expected_cpu: bool = False,
178179
) -> int:
179180
"""Stage the AgentX power audit inputs and validate each concurrency's power window.
@@ -197,7 +198,7 @@ def collect_agentic_power_results(
197198

198199
if step := validate_agentic_power(
199200
logs_dir, workspace, result_filename, producer_sha, concurrencies,
200-
results_python=results_python, require_power=True, expected_cpu=expected_cpu,
201+
results_python=results_python, require_power=require_power, expected_cpu=expected_cpu,
201202
): # fmt: skip
202203
rc = step
203204
return rc

0 commit comments

Comments
 (0)