perf(recipe): default GB200 MoE perf recipes to NCCL EP and grouped tensors

[malay-nagda]

What does this PR do ?

This PR moves selected GB200 MoE perf recipes from the HybridEP flex dispatcher to NCCL EP, with moe_use_grouped_tensor=True. It follows #6171 (GB300) and #6261 (GPT-OSS 120B GB200 MXFP8).

Changelog

• The following 15 GB200 recipes now call _enable_ncclep and set moe_use_grouped_tensor=True. Their inline environment sets NCCL_EP_HT_EM_PULL_PUSH=1 and drops the HybridEP topology names. Parallelism, precision, recompute, CUDA graph mode and comm overlap are unchanged.

  Family	Recipes
  DeepSeek V3, 256 GPUs	BF16, MXFP8, NVFP4, MXFP8 large-scale
  DeepSeek V4 Flash, 128 GPUs	MXFP8
  Nemotron 3 Super, 64 GPUs	BF16, MXFP8, NVFP4
  Nemotron 3 Ultra, 256 GPUs	MXFP8
  Nemotron 3.5 Lightning, 8 GPUs	BF16, MXFP8
  Qwen3 235B-A22B, 256 GPUs	BF16, MXFP8, NVFP4, MXFP8 large-scale

• Each switched recipe applies the NCCL EP overlay itself; none inherits it from a HybridEP sibling. Where a switched recipe and a HybridEP recipe derive from the same parent:

  • DeepSeek V3 NVFP4 builds on a new shared HybridEP base, _build_deepseek_v3_gb200_bf16.
  • DeepSeek V4 Flash: the GB300 recipe now builds on the new HybridEP base _build_deepseek_v4_flash_gb200_fp8mx instead of the GB200 recipe. GB300, and the VR200 and B300 recipes derived from it, keep HybridEP unchanged.
  • Nemotron 3.5 Lightning BF16 still derives from the Nemotron 3 Nano BF16 recipe, which stays on HybridEP, and switches explicitly.
  • The MXFP8 large-scale proxies inherit the switch from their 256-GPU MXFP8 recipes; only their environment changes.

• Nemotron 3 Ultra MXFP8 drops a HybridEP rank variable that is no longer used, and its docstring is updated.

• The new test_gb200_ncclep_perf_recipes.py checks two things:

  • The full NCCL EP stack on the 15 switched recipes.
  • That 11 GB200 recipes kept on HybridEP don't pick it up. Most of these share a base or parent recipe with a switched one.

• Existing tests that hardcoded HybridEP for the switched recipes are updated:

  • The Nemotron 3 Super BF16 perf-vs-library test now treats NCCL EP as a benchmark-only difference. The library recipe stays on HybridEP.
  • The Lightning-vs-Nano policy test now allows for the dispatcher environment differing between the two.
  • The DeepSeek V4 Flash GB300 test now also asserts that GB300 stays on HybridEP.

Scope

• nemo-ci benchmarks 22 GB200 MoE recipes on HybridEP, and no job overrides the dispatcher. This PR switches 15 of them; 14 match GB300 recipes that already run NCCL EP after perf(recipe): default GB300 MoE perf recipes to NCCL EP and grouped tensors #6171.
• DeepSeek V4 Flash MXFP8 is switched without GB300 precedent. perf(recipe): default GB300 MoE perf recipes to NCCL EP and grouped tensors #6171 removed its GB300 switch as unverified. See the known failure below.
• Unchanged in this PR, although their GB300 counterparts run NCCL EP:
  • Kimi K2 MXFP8
  • Nemotron 3 Nano BF16/MXFP8
  • Qwen3 30B-A3B 8-GPU BF16/MXFP8 and 32-GPU BF16
• Unchanged, matching GB300:
  • Nemotron 3 Ultra NVFP4 stays on HybridEP.
  • GPT-OSS 120B BF16 stays on alltoall.
• GB200 recipes without a nemo-ci benchmark are not touched.
• Deliberately kept on HybridEP: deepseek_v3_pretrain_256gpu_gb200_fp8mx_partial_cg_dev_config. It offloads expert_fc1 with EP overlap, and MCore rejects that combination under ncclep.
• Known failure: NVTE_ALLOW_NONDETERMINISTIC_ALGO=0 on the DeepSeek recipes.
  • In nemo-ci, DeepSeek V4 Flash GB200 on NCCL EP failed in the first backward pass with TE's Deterministic execution was requested (NVTE_ALLOW_NONDETERMINISTIC_ALGO=0 …), but the scale gradient (dprob) is accumulated with nondeterministic atomics error.
  • The same recipe on HybridEP runs with this setting, according to its nemo-ci golden.
  • The GB200 DeepSeek V3 MXFP8/NVFP4/large-scale recipes keep the same setting and the same op-fuser grouped MLP, so they are expected to fail the same way. That is unverified, because those runs have not reached training yet.
  • perf(recipe): default GB300 MoE perf recipes to NCCL EP and grouped tensors #6171 dropped this setting for the GB300 DeepSeek V3 recipes; doing the same here is the open fix.

Validation

• pre-commit run --all-files passes.
• Local CPU unit tests were run with CPU torch and Megatron-Core at main's pin a98aed55, using import-only stubs for Triton and TE.
  • The new GB200 NCCL EP test passes 26/26.
  • Across the related recipe test files, 448 passed. 35 failed for environmental reasons only: they build recipes without the offline fixture, and the HF config download is blocked locally. They include recipes this PR does not touch.
  • The functional test test_perf_config_integration.py was updated but could not run locally, because it also needs HF configs.
• nemo-ci GB200 runs so far. These used the container's Megatron-LM, not Bridge's pin.
  • Nemotron 3.5 Lightning, 8 GPUs, ran on NCCL EP.
    • BF16: 841.4 vs 791 TFLOP/s/GPU, +6.4%. That exceeds the ±5% gate, so the golden needs a refresh.
    • MXFP8: 896.7 vs 877.3, +2.2%.
  • No data yet for the 64–256-GPU runs: DeepSeek V3 BF16/MXFP8/NVFP4, Qwen3 235B BF16/MXFP8/NVFP4 and Nemotron 3 Super MXFP8/NVFP4.
    • They failed before training, at the TCPStore rendezvous. One or two nodes were still in nemo-ci's per-node ntrace install, which compiles a CUPTI backend and waits up to 900 s, longer than the 600 s store timeout.
    • The missing ranks never reached Bridge code, so this is unrelated to the change.
  • DeepSeek V4 Flash: the determinism error above.
  • Dropped from this PR after measurement on NCCL EP:
    • GPT-OSS 120B BF16 measured 543.6 vs 581 TFLOP/s/GPU (−6.3%) against its alltoall golden.
    • Qwen3 30B-A3B 8-GPU measured +2.4% (BF16) and +1.9% (MXFP8).

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?
• Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  • Reviewer: Does the PR have correct import guards for all optional libraries?

Still to do:

• Resolve the DeepSeek determinism failure, for example by dropping NVTE_ALLOW_NONDETERMINISTIC_ALGO=0 from the GB200 DeepSeek NCCL EP recipes as perf(recipe): default GB300 MoE perf recipes to NCCL EP and grouped tensors #6171 did, then rerun DeepSeek V4 Flash.
• Rerun the 64–256-GPU benchmarks once the ntrace startup timeout is avoided, and compare against HybridEP on step time, peak and device memory, and loss.
  • GB200 has much less HBM than GB300, where NCCL EP added up to about 13 GB of device memory for Nemotron 3 Super.
  • The llmb scale-out jobs reuse these base recipes.
• Refresh the nemo-ci goldens afterwards, starting with Lightning BF16. Re-check the KNOWN_ISSUE flags (MB-1707, MB-1203) and allow_failure on DeepSeek V4 Flash.
• Model verification cards still record the earlier HybridEP measurements for some of these recipes. Refresh them once new numbers exist.
• Expect a small conflict with refactor(recipe): split DeepSeek full-iteration helper into independent toggles #6372 in perf_recipes/deepseek/gb200/deepseek_v3.py.

Additional Information

• Follows perf(recipe): default GB300 MoE perf recipes to NCCL EP and grouped tensors #6171 and perf(recipe): use NCCL EP for GPT-OSS GB200 and remove temporary examples #6261.

🤖 Generated with Claude Code

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.