Repository navigation
fix(model): backport verification card and DiffusionGemma parity fixes to r0.7.0 - #6512
Conversation
…ntaining revision (NVIDIA-NeMo#6425) Signed-off-by: Pruthviraj Prakash <pruprakash@nvidia.com> (cherry picked from commit 08ab678) Signed-off-by: Chen Cui <chcui@nvidia.com>
…rent recipe (NVIDIA-NeMo#6467) Signed-off-by: Pruthviraj Prakash <pruprakash@nvidia.com> (cherry picked from commit 0869454) Signed-off-by: Chen Cui <chcui@nvidia.com>
…ipts (NVIDIA-NeMo#6486) Signed-off-by: Pruthviraj Prakash <pruprakash@nvidia.com> (cherry picked from commit 25a994b) Signed-off-by: Chen Cui <chcui@nvidia.com>
…checks Signed-off-by: Chen Cui <chcui@nvidia.com>
|
Automatic Claude reviews have been retired. To request a pull-request review, post a comment containing: Add |
…NVIDIA-NeMo#6474) Signed-off-by: Pruthviraj Prakash <pruprakash@nvidia.com> (cherry picked from commit 52458eb) Signed-off-by: Chen Cui <chcui@nvidia.com>
…ons (NVIDIA-NeMo#6499) Signed-off-by: Pruthviraj Prakash <pruprakash@nvidia.com> (cherry picked from commit 2856c2e) Signed-off-by: Chen Cui <chcui@nvidia.com>
|
/review |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
No verified in-scope defects found. Reviewed the card and generated-documentation updates, TF32 helper and call ordering, and affected test contracts.
This was a read-only source review; tests were not executed, and historical GPU measurements were not independently reproduced. No perf tests impacted.
Suggested test cases
test_disable_tf32_also_disables_cublas_tf32test_torch_flags_alone_do_not_count_as_tf32_disabledtest_verified_performance_layout_matches_the_published_recipe[deepseek-v4-flash-GB200-megatron.bridge.perf_recipes.deepseek]— run in isolation.test_lightning_card_recipes_exist_at_their_recorded_revision— use a checkout containing the recorded commits.test_k_exaone_2_verified_items_record_reachable_revisionstest_glm5_verified_items_pin_reachable_revisions[glm5]test_glm5_verified_items_pin_reachable_revisions[glm5-2]test_generated_outputs_and_navigation_are_current
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex
Signed-off-by: Chen Cui <chcui@nvidia.com>
Complete the combined Pruth backport to
r0.7.0with the four fixes still missing from release:#6425 is now on the release branch through merged #6442. This branch merges that release update and retains the Lightning fix without a duplicate change in the PR diff. Each backport retains its source author and separate signed cherry-pick commit. A small follow-up fixes the DeepSeek test fixture to import selected recipes before patching their external dependencies, so the layout check also works when selected alone.
The card measurements remain historical results at their recorded revisions, not new release-performance measurements. DeepSeek GB300 and Nemotron Super GB300 follow-ups remain outside this change. #6474 is included unchanged from its merged source. Its author reports a next-token match at token 2879 and cosine similarity 0.999997 using a clamp-aware HF reference. This is attributed historical source QA, not independently repeated release validation. The command already required a separately produced HF logits file on the release base; the reference producer remains outside the public repository, so this PR does not make that comparison self-contained. The relevant model provider and comparison code match the recorded public revision on release. The separate inference revision remains excluded from the reachability test pending #6294.
The GLM results are also attributed historical source QA, not fresh release workload validation. #6499 preserves the pending pretrain/resume and long-context exclusions: source QA reports a GLM-5 resume OOM, an unpublished 128K dataset generator, and a missing MTP-preserving import for GLM-5.2 pretrain/resume. Those unresolved items are not repaired or validated by this PR.
The release merge resolves the overlapping revision-test file while retaining the Lightning, K-EXAONE-2 and GLM checks. Documentation tests, card-validator tests, catalog consistency and pre-commit checks were rerun after the merge. The numerical measurements below were obtained before this release merge; the DiffusionGemma implementation is unchanged, but a fresh GPU run on the updated base was not performed.
Validation on the release checkout:
0.01068 / 0.00570to0.000162 / 0.000147(limit0.0003), and all 82 exported tensors match exactly. TP1, TP2 and EP2 loss/gradient parity also pass with complete gradient coverage; TP1 completes an 80-step overfit check and TP2/EP2 each perform two finite updates. Fresh-process TF32, BF16 and import-behavior probes pass. All four related functional tests pass, including step-3 save/resume through step 6 with logged losses, gradients and sample counts equal to the uninterrupted run.The real 26B model and GB200 numerical checks were not rerun for this backport.
NVBugs: 6626551, 6900116, 6631267, 6875251, 6932589, 6938912.