Skip to content

fix(model): backport verification card and DiffusionGemma parity fixes to r0.7.0 - #6512

Merged
cuichenx merged 7 commits into
NVIDIA-NeMo:r0.7.0from
cuichenx:chcui/nvbug-2/backport-pruth-fixes-r0.7.0
Oct 17, 2026
Merged

cuichenx merged 7 commits into
NVIDIA-NeMo:r0.7.0from
cuichenx:chcui/nvbug-2/backport-pruth-fixes-r0.7.0

Conversation

@cuichenx

@cuichenx cuichenx commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Complete the combined Pruth backport to r0.7.0 with the four fixes still missing from release:

#6425 is now on the release branch through merged #6442. This branch merges that release update and retains the Lightning fix without a duplicate change in the PR diff. Each backport retains its source author and separate signed cherry-pick commit. A small follow-up fixes the DeepSeek test fixture to import selected recipes before patching their external dependencies, so the layout check also works when selected alone.

The card measurements remain historical results at their recorded revisions, not new release-performance measurements. DeepSeek GB300 and Nemotron Super GB300 follow-ups remain outside this change. #6474 is included unchanged from its merged source. Its author reports a next-token match at token 2879 and cosine similarity 0.999997 using a clamp-aware HF reference. This is attributed historical source QA, not independently repeated release validation. The command already required a separately produced HF logits file on the release base; the reference producer remains outside the public repository, so this PR does not make that comparison self-contained. The relevant model provider and comparison code match the recorded public revision on release. The separate inference revision remains excluded from the reachability test pending #6294.

The GLM results are also attributed historical source QA, not fresh release workload validation. #6499 preserves the pending pretrain/resume and long-context exclusions: source QA reports a GLM-5 resume OOM, an unpublished 128K dataset generator, and a missing MTP-preserving import for GLM-5.2 pretrain/resume. Those unresolved items are not repaired or validated by this PR.

The release merge resolves the overlapping revision-test file while retaining the Lightning, K-EXAONE-2 and GLM checks. Documentation tests, card-validator tests, catalog consistency and pre-commit checks were rerun after the merge. The numerical measurements below were obtained before this release merge; the DiffusionGemma implementation is unchanged, but a fresh GPU run on the updated base was not performed.

Validation on the release checkout:

  • 76 documentation consistency tests and 42 card-validator tests passed; catalog regeneration and pre-commit checks passed.
  • The isolated DeepSeek layout test failed before the fixture correction and passed afterward; all 13 focused recipe/precision tests passed. A negative control confirms that stale EP64 text is rejected against the real EP32 recipe.
  • The Lightning, K-EXAONE-2 and GLM-5/5.2 revision checks passed locally with all checked candidate commits available. CI shallow checkouts can skip these history-dependent checks.
  • On H100 with the release Bridge/Megatron-Core sources, the unchanged baseline fails the toy encoder threshold; the backport passes. Text/image max absolute errors improve from 0.01068 / 0.00570 to 0.000162 / 0.000147 (limit 0.0003), and all 82 exported tensors match exactly. TP1, TP2 and EP2 loss/gradient parity also pass with complete gradient coverage; TP1 completes an 80-step overfit check and TP2/EP2 each perform two finite updates. Fresh-process TF32, BF16 and import-behavior probes pass. All four related functional tests pass, including step-3 save/resume through step 6 with logged losses, gradients and sample counts equal to the uninterrupted run.
  • The shared precision helper passes strict mypy. Strict checking across all touched upstream examples/tests remains non-clean due to existing typing and dependency-stub diagnostics; no typing cleanup is bundled.

The real 26B model and GB200 numerical checks were not rerun for this backport.

NVBugs: 6626551, 6900116, 6631267, 6875251, 6932589, 6938912.

pruprakash and others added 4 commits October 16, 2026 12:21
…ntaining revision (NVIDIA-NeMo#6425)

Signed-off-by: Pruthviraj Prakash <pruprakash@nvidia.com>

(cherry picked from commit 08ab678)
Signed-off-by: Chen Cui <chcui@nvidia.com>
…rent recipe (NVIDIA-NeMo#6467)

Signed-off-by: Pruthviraj Prakash <pruprakash@nvidia.com>

(cherry picked from commit 0869454)
Signed-off-by: Chen Cui <chcui@nvidia.com>
…ipts (NVIDIA-NeMo#6486)

Signed-off-by: Pruthviraj Prakash <pruprakash@nvidia.com>

(cherry picked from commit 25a994b)
Signed-off-by: Chen Cui <chcui@nvidia.com>
…checks

Signed-off-by: Chen Cui <chcui@nvidia.com>
@cuichenx cuichenx added bug Something isn't working cherry-pick area:model Model implementations and HF bridge logic needs-review PR is ready for code review and waiting on a reviewer labels Oct 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Automatic Claude reviews have been retired. To request a pull-request review, post a comment containing:

/review

Add model=claude to use a Claude reviewer (the default is model=codex). mode=light|strict selects the review depth; for example, /review model=claude mode=strict. Comment /review help for all options.

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

…NVIDIA-NeMo#6474)

Signed-off-by: Pruthviraj Prakash <pruprakash@nvidia.com>

(cherry picked from commit 52458eb)
Signed-off-by: Chen Cui <chcui@nvidia.com>
…ons (NVIDIA-NeMo#6499)

Signed-off-by: Pruthviraj Prakash <pruprakash@nvidia.com>

(cherry picked from commit 2856c2e)
Signed-off-by: Chen Cui <chcui@nvidia.com>
@cuichenx

Copy link
Copy Markdown
Contributor Author

/review

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No verified in-scope defects found. Reviewed the card and generated-documentation updates, TF32 helper and call ordering, and affected test contracts.

This was a read-only source review; tests were not executed, and historical GPU measurements were not independently reproduced. No perf tests impacted.

Suggested test cases

  • test_disable_tf32_also_disables_cublas_tf32
  • test_torch_flags_alone_do_not_count_as_tf32_disabled
  • test_verified_performance_layout_matches_the_published_recipe[deepseek-v4-flash-GB200-megatron.bridge.perf_recipes.deepseek] — run in isolation.
  • test_lightning_card_recipes_exist_at_their_recorded_revision — use a checkout containing the recorded commits.
  • test_k_exaone_2_verified_items_record_reachable_revisions
  • test_glm5_verified_items_pin_reachable_revisions[glm5]
  • test_glm5_verified_items_pin_reachable_revisions[glm5-2]
  • test_generated_outputs_and_navigation_are_current

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex

Signed-off-by: Chen Cui <chcui@nvidia.com>
@cuichenx
cuichenx merged commit a670bcc into NVIDIA-NeMo:r0.7.0 Oct 17, 2026
91 of 94 checks passed

This branch was successfully deployed

1 active deployment
test — cfc52fe5 Deployed Oct 16, 2026 by copy-pr-bot[bot] via cicd-wait-in-queue #22977
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:model Model implementations and HF bridge logic bug Something isn't working cherry-pick needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants