Skip to content

perf(nemotron): add Nemotron 3.5 Lightning GB300 recipes - #5693

Merged
malay-nagda merged 4 commits into
mainfrom
malay/nemotron3_5_gb300
Aug 21, 2026
Merged

malay-nagda merged 4 commits into
mainfrom
malay/nemotron3_5_gb300

Conversation

@malay-nagda

@malay-nagda malay-nagda commented Aug 19, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds canonical Nemotron 3.5 Lightning throughput-benchmark recipes for an 8-GPU GB300 system.

  • Adds BF16, MXFP8, NVFP4, and MXFP8 Megatron FSDP variants.
  • Exports the recipes through megatron.bridge.perf_recipes.nemotronh and registers them in the performance-config inventory.
  • Extends the existing Nemotron 3.5 recipe tests to cover GB300 model finalization, MTP settings, topology, precision-specific tuning, and FSDP defaults.
  • Corrects the copied FSDP comment: the GB300 FSDP microbatch is 3, while the non-FSDP GB300 recipes use 4.

These are benchmark recipes for upper-bound throughput measurement, not production convergence recipes.

Testing

  • uv run --with pre-commit --no-project pre-commit run --all-files — passed.
  • Targeted pytest was attempted but could not run in the local checkout: project sync fails while building fast-hadamard-transform because its isolated build environment does not declare torch, and the existing .venv does not contain pytest or torch.
  • Static inventory validation confirmed that all four GB300 inventory entries resolve to defined recipe factories.

Signed-off-by: Malay Nagda <malayn@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Aug 19, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

Signed-off-by: Malay Nagda <malayn@nvidia.com>
@malay-nagda malay-nagda changed the title nemotron 3.5 gb300 perf(nemotron): add Nemotron 3.5 Lightning GB300 recipes Aug 19, 2026
@malay-nagda
malay-nagda marked this pull request as ready for review August 19, 2026 11:59
@malay-nagda
malay-nagda requested review from a team, cuichenx and yaoyu-33 August 19, 2026 12:00
@claude

claude Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

LGTM

Clean mirror of the existing GB200 Nemotron 3.5 Lightning perf recipes onto GB300, with matching unit-test coverage. Verified the four new config functions differ from their GB200 twins only in the intended env-var/topology set, and that the test registries (_GB300_RECIPES, _GB_FSDP_RECIPES, base-recipe pairs, finalize factories, and the MTP / topology / FSDP-defaults / CutDSL assertions) all reference the new functions. The dump_perf_configs.py entries also include the fsdp variant.

Minor (non-blocking): in _build_nemotron_3_5_lightning_gb300_mxfp8, moe_hybridep_num_sms = 16 is redundant. The GB300 nano MXFP8 builder already sets it to 16 (unlike the GB200 nano MXFP8 builder, which sets 32, where the override is meaningful). Harmless, just a copy artifact.

Suggested test cases:

  • nemotron_3_5_lightning_pretrain_8gpu_gb300_bf16_config
  • nemotron_3_5_lightning_pretrain_8gpu_gb300_fp8mx_config
  • nemotron_3_5_lightning_pretrain_8gpu_gb300_fp8mx_fsdp_config
  • nemotron_3_5_lightning_pretrain_8gpu_gb300_nvfp4_config
  • nemotron_3_nano_pretrain_8gpu_gb300_bf16_config
  • nemotron_3_nano_pretrain_8gpu_gb300_fp8mx_config
  • nemotron_3_nano_pretrain_8gpu_gb300_nvfp4_config

@malay-nagda malay-nagda self-assigned this Aug 19, 2026
@malay-nagda malay-nagda added performance/release Performance items related with NeMo release area:recipe Training recipes and launch configs area:perf Performance optimizations and benchmarking needs-review PR is ready for code review and waiting on a reviewer r0.6.0 Auto-cherrypick to release branch. Apply before merge; cherrypick happens after merge. 26.08.01 labels Aug 19, 2026
@yaoyu-33 yaoyu-33 added feature New capabilities, enhancements, or enablement work needs-more-tests Requires additional L0 and L1 test coverage before merge labels Aug 19, 2026
Comment thread src/megatron/bridge/perf_recipes/nemotronh/gb300/nemotronh.py Outdated
Signed-off-by: Malay Nagda <malayn@nvidia.com>
Signed-off-by: Malay Nagda <malayn@nvidia.com>

@cuichenx cuichenx left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@malay-nagda
malay-nagda enabled auto-merge (squash) August 21, 2026 16:16
@malay-nagda
malay-nagda merged commit a575666 into main Aug 21, 2026
128 checks passed
@malay-nagda
malay-nagda deleted the malay/nemotron3_5_gb300 branch August 21, 2026 16:39

This branch was previously deployed

2 inactive deployments
public — 1d0cc2c9 Deployed Aug 21, 2026 by copy-pr-bot[bot] via release / finalize / publish-docs #4869
test — 1d0cc2c9 Deployed Aug 21, 2026 by copy-pr-bot[bot] via cicd-wait-in-queue #20845
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

26.08.01 area:perf Performance optimizations and benchmarking area:recipe Training recipes and launch configs feature New capabilities, enhancements, or enablement work needs-more-tests Requires additional L0 and L1 test coverage before merge needs-review PR is ready for code review and waiting on a reviewer performance/release Performance items related with NeMo release r0.6.0 Auto-cherrypick to release branch. Apply before merge; cherrypick happens after merge.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants