feat(recipe): add deterministic Nemotron 3 Ultra, Qwen3.5-9B and Moonlight-16B pretrain recipes by ZhiyuLi-Nvidia · Pull Request #6440 · NVIDIA-NeMo/Megatron-Bridge

ZhiyuLi-Nvidia · 2026-10-13T18:01:03Z

Summary

Add nemotron_3_ultra_pretrain_256gpu_gb200_bf16_ep16_deterministic_config, wrapping the 256-GPU GB200 EP16 recipe with apply_determinism_overrides.
The recipe assigns a static cfg.env_vars dict: COMMON_RECIPE_ENV_VARS, the new DETERMINISM_ENV_VARS (CUBLAS workspace, NCCL Ring, NCCL_EP_HT_EM_AG_SCAN_MODE, TE non-deterministic algo off, Mamba / causal-conv1d deterministic), and the existing Ultra/HybridEP values inline.
Export the recipe from both recipes/nemotronh __init__.py files and add a unit test.

Test plan

tests/unit_tests/recipes/nemotronh/test_nemotron_3_ultra.py: 17 passed (run in the NeMo nightly container on the r0.7.0 base before this was cherry-picked onto main; CI covers main)
ruff check / ruff format clean on the changed files
run: a deterministic A/B training run

🤖 Generated with Claude Code

Add nemotron_3_ultra_pretrain_256gpu_gb200_bf16_ep16_deterministic_config, which applies apply_determinism_overrides and a static env_vars dict that spreads the new DETERMINISM_ENV_VARS constant next to COMMON_RECIPE_ENV_VARS. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Zhiyu Li <zhiyul@nvidia.com>

copy-pr-bot · 2026-10-13T18:01:07Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

… recipes Wrap qwen35_text_9b_pretrain_8gpu_gb200_bf16_config and moonlight_16b_pretrain_16gpu_h100_bf16_config with apply_determinism_overrides and a static env_vars dict that spreads DETERMINISM_ENV_VARS. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Zhiyu Li <zhiyul@nvidia.com>

…nd for Qwen3.5-9B - Spread the shared DETERMINISM_ENV_VARS (string values, matching apply_determinism_overrides) in the DSv3 GB300 deterministic recipe. - Qwen3.5 uses Gated DeltaNet; Megatron-Core requires gdn_kernel_backend='torch' under deterministic_mode, so set it as a standalone override in the deterministic Qwen3.5-9B recipe. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Zhiyu Li <zhiyul@nvidia.com>

Wrap nemotron_3_nano_pretrain_8gpu_gb200_bf16_config with apply_determinism_overrides and the shared DETERMINISM_ENV_VARS, keeping the GB200 HybridEP environment visible on the recipe. A 2-node, 20-step A/B run on the NeMo 26.10.rc5 image matched bitwise on every logged step. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Zhiyu Li <zhiyul@nvidia.com>

Bridge's initialize_megatron never called megatron.core.tuning.install, which Megatron-Core's own initializer calls under --deterministic-mode. Timed Triton autotuning could therefore pick different kernel configs between ranks and runs, changing the floating-point reduction order. Call tuning.install(deterministic=True) before any kernel runs when model.deterministic_mode is set. Runs without it keep timed autotuning. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Zhiyu Li <zhiyul@nvidia.com>

Replace the side_effect assertion with a call-order check on a parent mock so a failure reads as an ordering failure. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Zhiyu Li <zhiyul@nvidia.com>

ZhiyuLi-Nvidia · 2026-10-14T00:28:36Z

/ok to test ca1ceb2

github-actions · 2026-10-14T00:28:54Z

Automatic Claude reviews have been retired. To request a pull-request review, post a comment containing:

/review

Add model=claude to use a Claude reviewer (the default is model=codex). mode=light|strict selects the review depth; for example, /review model=claude mode=strict. Comment /review help for all options.

The recipe-environment tests parse each builder's env_vars dict with ast and only accept the COMMON_*_ENV_VARS spread plus literal entries. Spreading DETERMINISM_ENV_VARS in the Moonlight H100 and DSv3 GB300 perf recipes failed test_explicit_recipe_environment_invariants and test_every_flat_recipe_builder_declares_its_environment_inline. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Zhiyu Li <zhiyul@nvidia.com>

Restore the DETERMINISM_ENV_VARS spread in the Moonlight H100 and DSv3 GB300 recipes. The two recipe-environment tests now accept that spread next to the common-default constants and merge its values when checking invariants, and docs/recipe-usage.md documents it as a third shared constant. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Zhiyu Li <zhiyul@nvidia.com>

ZhiyuLi-Nvidia · 2026-10-14T03:59:00Z

/ok to test 601a6fe

copy-pr-bot · 2026-10-14T03:59:04Z

/ok to test 601a6fe

@ZhiyuLi-Nvidia, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

ZhiyuLi-Nvidia · 2026-10-14T03:59:36Z

/ok to test 601a6fe

github-actions · 2026-10-14T04:00:45Z

🌿 Preview your docs: https://nvidia-preview-pr-6440-93b297ebc512.docs.buildwithfern.com/nemo/megatron-bridge

…-main Signed-off-by: adityavavreNVDA <avavre@nvidia.com>

…-main

adityavavreNVDA · 2026-10-15T20:56:25Z

/ok to test 93b297e

cuichenx · 2026-10-15T23:02:49Z

+
+    # Standalone Qwen3.5 override, not part of apply_determinism_overrides: Megatron-Core rejects
+    # deterministic_mode with Gated DeltaNet unless the torch kernel backend is used.
+    cfg.model.gdn_kernel_backend = "torch"


will this impact perf?

Yes.

There seem no determinism fused gdn kernel yet, which raise an error without this.
Plan to dive into it after the release.

…light-16B pretrain recipes (NVIDIA-NeMo#6440) Signed-off-by: Zhiyu Li <zhiyul@nvidia.com> Signed-off-by: adityavavreNVDA <avavre@nvidia.com> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> Co-authored-by: adityavavreNVDA <avavre@nvidia.com> (cherry picked from commit 7c2c9ba) Signed-off-by: Zhiyu Li <zhiyul@nvidia.com>

ZhiyuLi-Nvidia added feature New capabilities, enhancements, or enablement work area:recipe Training recipes and launch configs needs-review PR is ready for code review and waiting on a reviewer labels Oct 13, 2026

ZhiyuLi-Nvidia changed the title ~~feat(recipe): add deterministic Nemotron 3 Ultra GB200 pretrain recipe~~ feat(recipe): add deterministic Nemotron 3 Ultra, Qwen3.5-9B and Moonlight-16B pretrain recipes Oct 13, 2026

ZhiyuLi-Nvidia requested review from dingqingy-nv and malay-nagda as code owners October 13, 2026 18:40

ZhiyuLi-Nvidia and others added 3 commits October 13, 2026 15:19

test(training): check tuning.install order with a parent mock

ca1ceb2

Replace the side_effect assertion with a call-order check on a parent mock so a failure reads as an ordering failure. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Zhiyu Li <zhiyul@nvidia.com>

ZhiyuLi-Nvidia added the 26.10 label Oct 13, 2026

copy-pr-bot Bot deployed to test October 14, 2026 00:29 Active

ZhiyuLi-Nvidia and others added 2 commits October 13, 2026 20:42

copy-pr-bot Bot deployed to test October 14, 2026 04:00 Active

adityavavreNVDA added 2 commits October 15, 2026 13:55

Merge branch 'main' into zhiyul/nemotron-3-ultra-deterministic-recipe…

df8df47

…-main Signed-off-by: adityavavreNVDA <avavre@nvidia.com>

Merge branch 'main' into zhiyul/nemotron-3-ultra-deterministic-recipe…

93b297e

…-main

copy-pr-bot Bot deployed to test October 15, 2026 20:57 Active

cuichenx added the r0.7.0 Auto-cherrypick to release branch. Apply before merge; cherrypick happens after merge. label Oct 15, 2026

cuichenx approved these changes Oct 15, 2026

View reviewed changes

cuichenx merged commit 7c2c9ba into NVIDIA-NeMo:main Oct 15, 2026
87 of 88 checks passed

ZhiyuLi-Nvidia mentioned this pull request Oct 15, 2026

cp: feat(recipe): add deterministic Nemotron 3 Ultra, Qwen3.5-9B and Moonlight-16B pretrain recipes (6440) into r0.7.0 #6497

Merged

3 of 5 tasks

kamran-nvidia mentioned this pull request Oct 16, 2026

ci: stop norecursedirs from hiding 52 unit test files #6500

Open

ZhiyuLi-Nvidia mentioned this pull request Oct 17, 2026

cp: deterministic recipes (6272, 6440) into r0.7.0 #6527

Closed

Navigation Menu

Repository navigation

feat(recipe): add deterministic Nemotron 3 Ultra, Qwen3.5-9B and Moonlight-16B pretrain recipes - #6440

feat(recipe): add deterministic Nemotron 3 Ultra, Qwen3.5-9B and Moonlight-16B pretrain recipes#6440
cuichenx merged 10 commits into
NVIDIA-NeMo:mainfrom
ZhiyuLi-Nvidia:zhiyul/nemotron-3-ultra-deterministic-recipe-main

ZhiyuLi-Nvidia commented Oct 13, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Oct 13, 2026

Uh oh!

ZhiyuLi-Nvidia commented Oct 14, 2026

Uh oh!

github-actions Bot commented Oct 14, 2026

Uh oh!

ZhiyuLi-Nvidia commented Oct 14, 2026

Uh oh!

copy-pr-bot Bot commented Oct 14, 2026

Uh oh!

ZhiyuLi-Nvidia commented Oct 14, 2026

Uh oh!

github-actions Bot commented Oct 14, 2026 •

edited

Loading

Uh oh!

adityavavreNVDA commented Oct 15, 2026

Uh oh!

cuichenx Oct 15, 2026

Uh oh!

ZhiyuLi-Nvidia Oct 15, 2026

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ZhiyuLi-Nvidia commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

copy-pr-bot Bot commented Oct 13, 2026

Uh oh!

ZhiyuLi-Nvidia commented Oct 14, 2026

Uh oh!

github-actions Bot commented Oct 14, 2026

Uh oh!

ZhiyuLi-Nvidia commented Oct 14, 2026

Uh oh!

copy-pr-bot Bot commented Oct 14, 2026

Uh oh!

ZhiyuLi-Nvidia commented Oct 14, 2026

Uh oh!

github-actions Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adityavavreNVDA commented Oct 15, 2026

Uh oh!

cuichenx Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

ZhiyuLi-Nvidia Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ZhiyuLi-Nvidia commented Oct 13, 2026 •

edited

Loading

github-actions Bot commented Oct 14, 2026 •

edited

Loading