Repository navigation
fix(ckpt): preserve MoE fine-tuning state and refresh Super VL verification - #6220
Merged
cuichenx merged 20 commits intoSep 28, 2026
Merged
Conversation
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Allow exact ChatML/reasoning delimiters in public inference prompts and output, and loopback literals for single-node coordinators. Preserve checks on enclosed content, other addresses and paths, credentials, URLs, and deny terms. Add focused positive and negative regressions without changing verification scripts or numerical thresholds. Signed-off-by: Chen Cui <chcui@nvidia.com>
Publish the refreshed cards, companion guide, generated catalog and navigation, with focused consistency and throughput fixtures. Preserve incomplete GB200 pretrain/resume and deferred workflows as unverified. Leave the public HF revision blank as requested; its four validator errors remain acknowledged rather than weakening revision checks. Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Contributor
Author
|
/ok to test 52d80a3 |
Signed-off-by: Chen Cui <chcui@nvidia.com>
Contributor
Author
|
/ok to test 4d7ab0b |
3 of 4 tasks
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Fix shared MoE checkpoint/inference issues found during Nemotron Super VL verification, and publish refreshed full-VL and text-only verification cards.
Changelog
expert_biasbuffers in native PEFT save/load and merge. MCore updates these buffers during gradient finalization even when router parameters are frozen.Noneoverrides clear saved capacity limits rather than silently retaining them.nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16in public examples and recipes, refresh the full-VL card, and add a text-only companion card and generated documentation.GitHub Actions CI
Request standard CI plus L1 (
needs-more-tests). The addedL1_Launch_peft_router_checkpointruns the native one-GPU router checkpoint regression under a distributed launch; ordinary unit execution skips that distributed path. No L2/full-suite label is requested.Validation
nvcr.io/nvidia/nemo:26.08.01.Known verification limits
Compatibility and scope
Older bias-enabled PEFT checkpoints may omit trained router-bias state. Native adapter inference/merge rejects missing state instead of silently replacing it with base-model values. A permissive training-resume policy may retain base bias, but that is not exact recovery; missing trained bias cannot be reconstructed from adapter tensors.
Standalone HF adapter export still omits router-bias buffers. The preservation path verified here is native adapter reload, merge into the base, then full-model HF export.
The validator correction is reusable infrastructure behavior: literal ChatML/reasoning tokens are prompt/output data, and loopback is a portable single-node binding. Tests retain rejection of other placeholders, non-loopback addresses, private paths, and explicit deny terms. Numerical verification thresholds are unchanged.
No dependency pins, CI workflow definitions, or MCore code are changed by this PR. Dropless overrides affect inference, not training. Existing copyright years are unchanged.
Before your PR is "Ready for review"
Pre checks:
Additional Information
Follow-up to #6100 and #6130, not a new architecture implementation. Primary area: checkpointing. Verification coverage is partial and source-specific as detailed above.