Skip to content

fix(training): reuse Megatron-Core determinism setup under --deterministic - #6271

Merged
cuichenx merged 4 commits into
mainfrom
zhiyul/det-reuse-mcore-setup
Oct 13, 2026
Merged

fix(training): reuse Megatron-Core determinism setup under --deterministic#6271
cuichenx merged 4 commits into
mainfrom
zhiyul/det-reuse-mcore-setup

Conversation

@ZhiyuLi-Nvidia

@ZhiyuLi-Nvidia ZhiyuLi-Nvidia commented Sep 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What does this PR do ?

Make Bridge's --deterministic delegate to Megatron-Core's determinism setup (megatron/training/determinism.py) instead of a hand-maintained copy that had drifted from it.

Changelog

  • ConfigContainer._validate_and_apply_deterministic_mode calls apply_determinism_to_args: required option values (cross-entropy loss fusion, TP comm overlap, MoE aux-loss fusion), env-var validation and defaults, and torch deterministic algorithms.
  • apply_determinism_overrides disables only the MoE aux-loss fusion (fused TopK routing stays on, per fix(determinism): Separate the MoE aux-loss fusion flag so --deterministic-mode guards only the op that breaks it NVIDIA/Megatron-LM#6967), exports Megatron-Core's DETERMINISM_ENV_VAR_DEFAULTS at launch, and pins MAMBA_DETERMINISTIC / CAUSAL_CONV1D_DETERMINISTIC.
  • Tests: new test_determinism_utils.py; test_config.py covers the rejected options, env validation/defaults and the torch state; test_recipe_environment.py expects Megatron-Core's string value "0" for NVTE_ALLOW_NONDETERMINISTIC_ALGO.

Behavior changes (inherited from Megatron-Core's rules):

before after
NCCL_ALGO=Tree accepted rejected (Megatron-Core accepts Ring, CollnetDirect, CollnetChain, ^NVLS)
unset NCCL_ALGO assertion error defaults to Ring
NVTE_ALLOW_NONDETERMINISTIC_ALGO set to a non-deterministic value not checked rejected
MoE aux-loss fusion not checked must be off (set by --deterministic)
torch.utils.deterministic.fill_uninitialized_memory on (torch default) off

No Megatron-LM bump needed: the pinned Megatron-LM already provides megatron/training/determinism.py and moe_router_aux_loss_fusion.

Context: this came out of making DeepSeek-V3 MXFP8 on 256 GB300 GPUs bitwise reproducible; a 50-step A/B with this setup (plus kernel-level fixes outside this repo) matched on all 550 logged metric values. A follow-up PR adds the deterministic DSv3 GB300 recipe on top of this change.

GitHub Actions CI

See the CI section in the Contributing doc for how to trigger the CI. A Nvidia developer will need to approve and trigger the CI for external contributors.

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?
  • Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
    • Reviewer: Does the PR have correct import guards for all optional libraries?

Additional Information

  • Local run: affected unit tests pass (test_determinism_utils.py, test_recipe_environment.py, test_config.py::TestConfigContainerValidation), except the GPU-only test_native_energon_packing_allows_other_ep_dispatchers_with_fixed_width[flex] which needs a GPU and fails on a CPU node with Found no NVIDIA driver (unrelated to this change).

🤖 Generated with Claude Code

@ZhiyuLi-Nvidia ZhiyuLi-Nvidia added bug Something isn't working area:training Training loop, callbacks, and runtime integration labels Sep 30, 2026
@copy-pr-bot

copy-pr-bot Bot commented Sep 30, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@ZhiyuLi-Nvidia ZhiyuLi-Nvidia mentioned this pull request Sep 30, 2026
Merged
2 of 5 tasks
@cuichenx cuichenx added needs-review PR is ready for code review and waiting on a reviewer breaking-change Public behavior or API compatibility changes labels Oct 1, 2026
@NVSkySC NVSkySC added the 26.10 label Oct 8, 2026
@malay-nagda
malay-nagda requested a review from cuichenx October 9, 2026 04:27
malay-nagda
malay-nagda previously approved these changes Oct 9, 2026
View reviewed changes
@malay-nagda

malay-nagda commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

/ok to test 2cefe2a

@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Automatic Claude reviews have been retired. To request a pull-request review, post a comment containing:

/review

Add model=claude to use a Claude reviewer (the default is model=codex). mode=light|strict selects the review depth; for example, /review model=claude mode=strict. Comment /review help for all options.

cuichenx
cuichenx previously approved these changes Oct 9, 2026
View reviewed changes
@ZhiyuLi-Nvidia
ZhiyuLi-Nvidia dismissed stale reviews from cuichenx and malay-nagda via c879c5e October 9, 2026 19:52
@ZhiyuLi-Nvidia

ZhiyuLi-Nvidia commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test c879c5e

@ZhiyuLi-Nvidia

ZhiyuLi-Nvidia commented Oct 11, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test b60f603

@ZhiyuLi-Nvidia
ZhiyuLi-Nvidia force-pushed the zhiyul/det-reuse-mcore-setup branch from b60f603 to ec13f2e Compare October 12, 2026 08:49
@ZhiyuLi-Nvidia

ZhiyuLi-Nvidia commented Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test ec13f2e

@cuichenx

cuichenx commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

/ok to test 3ef4d9b

@ZhiyuLi-Nvidia @claude
fix(training): reuse Megatron-Core determinism setup under --determin…
1101fb0 
…istic

Bridge's --deterministic hand-maintained its own copy of Megatron-Core's
determinism rules, and the two had drifted. Delegate to Megatron-Core
(megatron/training/determinism.py) so both frameworks enforce the same setup:

- ConfigContainer calls apply_determinism_to_args: required option values
  (cross-entropy fusion, TP comm overlap, MoE aux-loss fusion), env-var
  validation + defaults, torch deterministic algorithms.
- apply_determinism_overrides disables only the MoE aux-loss fusion (fused
  TopK routing stays on) and exports DETERMINISM_ENV_VAR_DEFAULTS at launch,
  plus MAMBA_DETERMINISTIC / CAUSAL_CONV1D_DETERMINISTIC.

Behavior now follows Megatron-Core: NCCL_ALGO=Tree is rejected, an unset
NCCL_ALGO defaults to Ring, a non-deterministic NVTE_ALLOW_NONDETERMINISTIC_ALGO
is rejected, and fill_uninitialized_memory is left off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Zhiyu Li <zhiyul@nvidia.com>
ZhiyuLi-Nvidia and others added 3 commits October 12, 2026 09:57
@ZhiyuLi-Nvidia @claude
feat(recipe): add DSv3 256-GPU GB300 MXFP8 deterministic perf recipe (#…
03c19c6 
…6272)

Signed-off-by: Zhiyu Li <zhiyul@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@ZhiyuLi-Nvidia @claude
fix(recipe): declare DSv3 GB300 deterministic recipe environment inline
95ddad0 
Flat perf recipes must assign one inline cfg.env_vars literal
(docs/recipe-usage.md, enforced by test_perf_recipe_environment.py).
Replace the pop/update on the parent mapping with the full literal,
grouped into determinism and other settings. The resulting environment
is unchanged.

Allow *_deterministic_config recipes to pin
NVTE_ALLOW_NONDETERMINISTIC_ALGO=0 in the environment invariant test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Zhiyu Li <zhiyul@nvidia.com>
@ZhiyuLi-Nvidia @claude
fix(recipe): add NCCL EP scan mode to DSv3 GB300 deterministic env
6a219ba 
Megatron-Core's DETERMINISM_ENV_VAR_DEFAULTS on main now includes
NCCL_EP_HT_EM_AG_SCAN_MODE=1. Mirror it in the inline determinism
section so the recipe environment matches apply_determinism_overrides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Zhiyu Li <zhiyul@nvidia.com>
@ZhiyuLi-Nvidia
ZhiyuLi-Nvidia force-pushed the zhiyul/det-reuse-mcore-setup branch from 3ef4d9b to 6a219ba Compare October 12, 2026 16:57
@ZhiyuLi-Nvidia

ZhiyuLi-Nvidia commented Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 6a219ba

@ZhiyuLi-Nvidia
ZhiyuLi-Nvidia enabled auto-merge (squash) October 13, 2026 04:32
@cuichenx cuichenx added the r0.7.0 Auto-cherrypick to release branch. Apply before merge; cherrypick happens after merge. label Oct 13, 2026
@cuichenx
cuichenx disabled auto-merge October 13, 2026 16:40
@cuichenx
cuichenx enabled auto-merge (squash) October 13, 2026 16:40
@cuichenx
cuichenx merged commit 0968a32 into main Oct 13, 2026
96 checks passed
@cuichenx
cuichenx deleted the zhiyul/det-reuse-mcore-setup branch October 13, 2026 17:04
@ZhiyuLi-Nvidia ZhiyuLi-Nvidia mentioned this pull request Oct 16, 2026
Merged
3 of 5 tasks

This branch was successfully deployed

1 active deployment
test — 6a219bac Deployed Oct 12, 2026 by copy-pr-bot[bot] via cicd-wait-in-queue #22786
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cuichenx cuichenx cuichenx approved these changes
@malay-nagda malay-nagda malay-nagda approved these changes
@dingqingy-nv dingqingy-nv Awaiting requested review from dingqingy-nv dingqingy-nv is a code owner

Assignees

@ZhiyuLi-Nvidia ZhiyuLi-Nvidia

Labels

26.10 area:training Training loop, callbacks, and runtime integration breaking-change Public behavior or API compatibility changes bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer r0.7.0 Auto-cherrypick to release branch. Apply before merge; cherrypick happens after merge.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@ZhiyuLi-Nvidia @malay-nagda @cuichenx @NVSkySC