Skip to content

Add p6-b300 and p6e-gb200 GPU health-check instance profiles - #1201

Open
Zhenye-Na wants to merge 1 commit into
awslabs:mainfrom
Zhenye-Na:feat/healthcheck-p6-b300-gb200-profiles
Open

Add p6-b300 and p6e-gb200 GPU health-check instance profiles#1201
Zhenye-Na wants to merge 1 commit into
awslabs:mainfrom
Zhenye-Na:feat/healthcheck-p6-b300-gb200-profiles

Conversation

@Zhenye-Na

@Zhenye-Na Zhenye-Na commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Purpose

The GPU health check compares each node's real hardware against a per-instance profile in instance-profiles.conf. If an instance type isn't in that file, the loader falls back to a GPU/EFA count of 0, and every check guarded by > 0 (GPU count, EFA enumeration, topology) just skips. So the suite happily reports PASS on an unrecognized node while barely checking anything.

The file currently stops at p6-b200, which means the newer Blackwell nodes fall into exactly that gap.

Changes

Added profiles for two Blackwell-generation instances:

  • p6-b300.48xlarge8|16|true|efa — 8 GPUs, NVLink. The AWS EFA docs list 17 network cards: NCI 0 is ENA-only and NCI 1–16 are EFA-only, so 16 EFA devices.
  • p6e-gb200.36xlarge4|4|true|efa — 4 GPUs (2× GB200 superchips), NVLink (NVL72 domain). GB200's attached EFA count is provisioning-dependent; 4 is the AWS-recommended max-EFA layout (4 × 400 Gbps). Called out in an inline comment.

I also added the AWS doc links to the file header so the counts are traceable.

This is a data-only change — no code touched, existing rows untouched, so nothing changes for already-supported instances.

Source: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-acc-inst-types.html

Test Plan

It's a config change, so I checked that the rows are well-formed and that the loader can actually find them.

# Every data row has the expected 5 pipe-delimited fields
awk -F'|' '/^[a-z]/ {if (NF!=5) print "BAD: "$0}' \
  4.validation_and_observability/2.gpu-cluster-healthcheck/instance-profiles.conf

# The new types resolve via the same exact-match lookup load_instance_profile() uses
for t in p6-b300.48xlarge p6e-gb200.36xlarge; do
  awk -F'|' -v inst="$t" '$1==inst {print}' \
    4.validation_and_observability/2.gpu-cluster-healthcheck/instance-profiles.conf
done

Test Results

Both new types resolve to a non-zero profile, so the GPU/EFA/topology checks now run on these nodes instead of skipping:

p6-b300.48xlarge|8|16|true|efa
p6e-gb200.36xlarge|4|4|true|efa

I don't have p6-b300 / p6e-gb200 hardware on hand, so a live run of the suite on real nodes would be a good follow-up for anyone with fleet access.

Directory Structure

N/A — updates an existing config file, not a test case.

Checklist

  • I have read the [contributing guidelines](https://github.com/awslabs/awsome-distributed-training/blob/main/CONTRIBUTING.md).
  • I am working against the latest main branch.
  • I have searched existing open and recently merged PRs to confirm this is not a duplicate.
  • The contribution is self-contained with documentation and scripts.
  • External dependencies are pinned to a specific version or tag (no latest). — N/A, no dependencies
  • A README is included or updated with prerequisites, instructions, and known issues. — N/A, config-only
  • New test cases follow the expected directory structure. — N/A, not a test case

The GPU cluster health check validates detected hardware against a
per-instance profile in instance-profiles.conf. When an instance type
has no matching row, the loader falls back to a zero GPU/EFA count and
the GPU-count, EFA-enumeration, and topology checks silently skip, so a
run on an unrecognized node reports PASS while validating almost nothing.

Add profiles for the Blackwell-generation instances:
- p6-b300.48xlarge: 8 GPU, 16 EFA, NVLink (17 network cards, NCI 0
  ENA-only and NCI 1-16 EFA-only).
- p6e-gb200.36xlarge: 4 GPU, 4 EFA, NVLink (2 GB200 superchips;
  recommended max-EFA layout of 4 x 400 Gbps interfaces).

Also add AWS documentation links for the GPU and EFA/network-card counts
to the file header. Values verified against:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-acc-inst-types.html

Signed-off-by: Nathan Na <nzhenye@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant