Repository navigation
docs(model): re-verify Nemotron 3.5 Lightning training at a recipe-containing revision - #6425
Merged
Merged
Conversation
…ntaining revision The Lightning card's training leaves were pinned to bridge_commits that predate the recipe rename in #5303, so the recipes named in their published commands did not exist at the recorded revision. Move the card default bridge_commit to bcd2d13 (#5303), keep the conversion and inference items at 619a2df with explicit pins, drop the per-leaf training pins, and replace the re-run training leaves' metrics and expected results with values measured in nvcr.io/nvidia/nemo:26.08 at bcd2d13 on 2x8 H100 and 2x4 GB200. The H100 long-context SFT leaf is re-run in the following commit. Add a doc-consistency test that each verified Lightning recipe exists at the revision its leaf records. Signed-off-by: Pruthviraj Prakash <pruprakash@nvidia.com>
… nodes The published two-node H100 sft_long_context command runs out of memory in iteration 2 on H100 80GB at the card's recipe-containing revision, so the leaf was never reproducible. Four nodes halve the distributed optimizer shard without changing the convergence contract (recipe GBS/MBS, sequence length, TP/CP/EP, recompute and precision are unchanged). Re-verify the leaf on nvcr.io/nvidia/nemo:26.08 at bcd2d13, record the measured metrics, update the audited token-slot fixture to 32 GPUs, and state in the card summary that every verified training leaf was measured at the card's bridge_commit. Extend the recipe-revision test to require each verified leaf's last_verified date to be no earlier than its bridge_commit's commit date. Signed-off-by: Pruthviraj Prakash <pruprakash@nvidia.com>
pruprakash
added
docs
cuichenx
added
the
r0.7.0
cuichenx
approved these changes
Oct 13, 2026
Contributor
|
/ok to test 659355b |
cuichenx
enabled auto-merge (squash)
October 13, 2026 00:58
cuichenx
added
ready-to-merge
Contributor
|
Automatic Claude reviews have been retired. To request a pull-request review, post a comment containing: Add |
Contributor
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Fixes two problems in the Nemotron 3.5 Lightning verification card's training items.
sft_long_contextcommand runs out of memory in iteration 2 on H100 80GB. The leaf now runs on four eight-GPU nodes. Recipe GBS/MBS, sequence length, TP/CP/EP, recompute and precision are unchanged. It was re-verified on 4x8 H100 at bcd2d13 (100/100 steps, LM loss 0.4268879 -> 0.2653436), and the audited token-slot fixture is updated to 32 GPUs.Catalog, nightly MDX and model guide are regenerated from the card.
Changelog
examples/model_verification_cards/nemotron-3.5-lightning/card.yaml: defaultbridge_commit->bcd2d134c; QA-measured metrics,last_verifiedand expected results for every training leaf on H100 and GB200; H100sft_long_contexton 4 nodes; summary updated.docs/_static/model-verification/catalog-v1.json,docs/models/nemotron/nemotron3.5-lightning.md,docs/fern/.../nemotron3.5-lightning.mdx: regenerated.tests/unit_tests/doc_consistency/test_verification_card_recipe_revisions.py(new): each verified Lightning leaf's recipe is defined undersrc/at its effective bridge_commit, and itslast_verifiedis no earlier than that commit's date.tests/unit_tests/skills/create_model_verification_card/test_validate_card.py: H100sft_long_contextaudited GPU count 16 -> 32.Additional Information
QA runs:
nvcr.io/nvidia/nemo:26.08, Bridgebcd2d134c, MCore6513e3e2; H100 on EOS (jobs 6223620-6223626, 6223752, 6239236), GB200 on 2x4 nodes (jobs 3289307-3289313, 3289316, 3289411, 3289412, 3290217, 3318807). All leaves 0 skipped / 0 NaN, checkpoints complete.Metrics: card skill definitions, parsed from complete optimizer-step rows; last-10-step means.
This new test skips in CI. It needs the recorded commits in git history, and CI's depth-1 checkout does not have them; it skips only in a shallow checkout, and a missing commit fails in a full clone. Run it locally with full history. It fails on the current main card and passes here.
peak_*_memory_gibvalues are as reported by Megatron, which uses GB (1e9 bytes) despite the field name; left unchanged here.Checks:
validate_card.pyvalid;generate_model_verification_catalog.py --checkclean;tests/unit_tests/doc_consistency+test_validate_card.pypass; pre-commit clean.NVBug: 6626551, 6900116.