Repository navigation
fix(ckpt): resolve tied embeddings for builder-backed models at PP > 1 - #6460
Merged
Merged
Conversation
Conversion helpers read share_embeddings_and_output_weights from the model's .config. A builder-backed model's .config is only the nested TransformerConfig; the flag lives on the outer ModelConfig, so the helpers defaulted to untied. At PP > 1 the last stage owns a separate output_layer.weight, which conversion then tried to map to the absent lm_head.weight (or found no mapping for), and the post-import shared-embedding broadcast was skipped. _share_embeddings_and_output_weights() now takes the unwrapped model: it keeps reading model.config when the config declares the flag (provider-built models, VLM wrappers), and otherwise falls back to the flag Megatron Core stores on the built language model. Tests: helper resolution cases, build_conversion_tasks with the flag on the model, and a PP=2 builder-path functional check (tied Qwen3) that compares the last-stage output layer with the HF embeddings and the export with the HF weights. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Kamran Jafari <kjafarisadeg@nvidia.com>
Contributor
Author
|
/ok to test 3eb69a2 |
2 of 5 tasks
2 of 5 tasks
kamran-nvidia
marked this pull request as ready for review
October 14, 2026 16:23
cuichenx
approved these changes
Oct 16, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Fixes conversion of tied-embedding models built through the builder path (
get_model_config()→get_model()) at pipeline parallelism > 1. Raised by @cuichenx on #6379 and #6401; the same bug affects Qwen3 onmainsince #6365.The bug: the conversion helpers read
share_embeddings_and_output_weightsfrom the model's.config. On the builder path that is only the nestedTransformerConfig; the flag lives on the outerModelConfig, so the helpers fell back to "untied". At PP > 1 the last stage owns a separateoutput_layer.weight, which conversion then tried to convert. The post-import broadcast that copies the input embeddings into that output layer was also skipped. On a tied Qwen3 checkpoint at PP=2 (currentmain):Affected:
scripts/conversionGPU import/export with--pp > 1, andAutoBridge.get_model()/load_hf_weights()at PP > 1 for families withUSE_MODEL_CONFIG_FOR_CONVERSION = True(Qwen3 onmain, plus the families migrating in #6379, #6383, #6392 and #6401). PP=1 andAutoBridge.import_ckpt()(single-process) never allocate the duplicate output parameter, and the provider path keeps the flag on.config, so the existing L0 PP=2 roundtrip (provider path) did not catch it.Changelog
MegatronModelBridge._share_embeddings_and_output_weights()now takes the unwrapped model instead of its config:model.configdeclares the flag (provider-built models and VLM wrapper configs), it is used as before, so the provider path is unchanged. Some VLM wrappers set their own module attribute toFalsewhile the config says tied; the config still wins.language_modelfirst because it owns the output layer.model_bridge.pyandquant_bridge.pypass the unwrapped model. The helper is private, and the existing one-argument test stubs keep working.build_conversion_taskswith the flag on the model.test_qwen3_tied_embeddings_builder_ppbuilds the tied toy Qwen3 at PP=2 through the builder path. It checks that the last-stageoutput_layer.weightequals the HFembed_tokens, and that the export reproduces every HF tensor with nolm_head.weight.Verification
GB200, NeMo 26.10 container, Megatron Core a98aed553, tiny random-weight tied checkpoints, TP=1 / PP=2:
main)ValueError(missinglm_head.weight)scripts/conversionGPU import + exportValueErrorValueError(missinglm_head.weight)ValueError(no mapping foroutput_layer.weight)test_model_bridge.py,test_model_bridge_lora.py,test_fp8_param_export.py,quantization/test_quant_bridge.py,test_gemma3_vl_bridge.py,test_auto_bridge.py,test_gpt_model_config.pyandmodels/qwen.ruff check/ruff format --check: clean (ruff 0.16.7;pre-commititself not run).Before your PR is "Ready for review"
Pre checks:
Additional Information
🤖 Generated with Claude Code