Repository navigation
refactor(model): add Qwen3 builder-backed model config with YaRN support - #6365
Merged
Merged
Conversation
Move dense Qwen3 onto the ModelConfig/ModelBuilder path, following the Llama migration (#5132): - Qwen3Bridge maps Hugging Face configs in hf_config_to_model_config_kwargs(). The deprecated provider path reuses the same kwargs, so provider_bridge() is no longer overridden. - Declare the YaRN fields on the Bridge TransformerConfig (moved from GPTModelProvider) so builder-backed GPT configs can express YaRN, and move the MCore concentration-factor patch next to them. - Fill the Hugging Face YaRN defaults (beta_fast=32, beta_slow=1, truncate=True) for Qwen3 configs that omit them. These previously reached Megatron Core as None and failed at model construction on both paths. - Switch all 22 Qwen3 H100 recipes to get_model_config(). Signed-off-by: Kamran Jafari <kjafarisadeg@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
/ok to test afbc585 |
Contributor
2 of 5 tasks
Dense Qwen3 recipes now call AutoBridge.get_model_config(), so the shared _FakeBridge in tests/unit_tests/recipes/test_qwen_recipes.py must provide it alongside to_megatron_provider(). Signed-off-by: Kamran Jafari <kjafarisadeg@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Set USE_MODEL_CONFIG_FOR_CONVERSION on Qwen3Bridge, following Muse Glimmer, so AutoBridge.import_ckpt and the conversion/compare tools construct Qwen3 from its builder-backed config instead of the deprecated provider. Imported checkpoints now record BridgeGPTModelConfig + GPTModelBuilder in run_config.yaml; provider-era Qwen3 checkpoints still load and export. Signed-off-by: Kamran Jafari <kjafarisadeg@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
/ok to test 9a92fb4 |
cuichenx
reviewed
Oct 9, 2026
| `GPTModelBuilder`. The provider API remains available for compatibility while | ||
| other model families migrate. Calling the legacy provider API for a | ||
| builder-backed family emits a deprecation warning. | ||
| Llama and dense Qwen3 also support the builder-backed configuration path. This |
Contributor
There was a problem hiding this comment.
i think we can remove naming specific models in this doc. there are more newly added models that follow the builder-config pattern
Contributor
Author
There was a problem hiding this comment.
Good point — reworded the section to describe the builder-backed path generically (no per-model list), and it now names the family's ModelBuilder (GPTModelBuilder / HybridModelBuilder) rather than only the GPT builder. Kept the Llama snippet as a runnable example; happy to make it model-neutral too if you prefer. Fixed in c58fe0a.
…milies More model families now follow the builder-config pattern, so the guide describes the path generically instead of naming individual models, and names the family's ModelBuilder rather than only GPTModelBuilder. Signed-off-by: Kamran Jafari <kjafarisadeg@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
/ok to test c58fe0a |
kamran-nvidia
marked this pull request as ready for review
October 9, 2026 16:53
2 of 5 tasks
This was referenced Oct 9, 2026
cuichenx
approved these changes
Oct 9, 2026
This was referenced Oct 9, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Moves dense Qwen3 onto the builder-backed
ModelConfig/ModelBuilderpath for recipes and checkpoint conversion (following Muse Glimmer, #5479), and adds YaRN support to that path.Why: every GPT bridge inherits
MODEL_CONFIG_CLASS = BridgeGPTModelConfig, soAutoBridge.get_model_config()already returns a config for Qwen3, but it ignores everything Qwen3'sprovider_bridge()set. Onmain, that config useslearned_absoluteposition embeddings, LayerNorm, no gated MLP, no QK layernorm, and linear biases. The provider path's deprecation warning points users to this path.Changelog
Qwen3Bridge: model settings move fromprovider_bridge()intohf_config_to_model_config_kwargs(), andhf_config_to_provider_kwargs()reuses them, so the deprecated provider path is unchanged (same pattern asLlamaBridge). It also states the provider-implied defaults explicitly (masked_softmax_fusion,rope_scaling=False,rope_scaling_factor=1.0, andposition_embedding_typeviasetdefault("rope"), which keeps"yarn"for YaRN checkpoints).yarn_*fields move fromGPTModelProviderto Bridge'sTransformerConfig. Megatron Core'sGPTModelreads them from the transformer config, which the builder path passes asmodel_config.transformer. The MCore concentration-factor patch moves with them, because MCore'shasattr()check otherwise treats aNone-valued field as YaRN.GPTModelProviderinherits the fields, so provider configs are unchanged.beta_fast/beta_slow/truncate, Bridge fills in Hugging Face's defaults (32 / 1 /True). Onmainthese reach MCore asNone, and model construction fails with aTypeErroron both paths for standard Qwen3 YaRN checkpoints. The YaRN recipe only works because it sets them by hand. Values the checkpoint sets explicitly are preserved. A shared fix for all GPT families is in fix(model): apply Hugging Face YaRN defaults for omitted rope_scaling keys #6367; whichever PR merges second should drop the Qwen3-local copy.get_model_config().Qwen3Bridge.USE_MODEL_CONFIG_FOR_CONVERSION = True, as inMuseGlimmerBridge.AutoBridge.import_ckpt,scripts/conversion(GPU import/export/round-trip) andcompare_hf_and_megatron/compare.pynow build Qwen3 fromget_model_config(). New checkpoints recordBridgeGPTModelConfig+GPTModelBuilderinrun_config.yaml; provider-era Qwen3 checkpoints still load and export.docs/bridge-guide.mddescribes the path generically (no per-model list) and names the familyModelBuilder(GPTModelBuilder/HybridModelBuilder)._FakeBridgeintests/unit_tests/recipes/test_qwen_recipes.pystubsget_model_config()for the migrated recipes.Verification
mainvs this branch (all fields; tied-bf16, untied-fp16, YaRN)beta_fast/beta_slow/round_to_int:None→ HF defaults (the crash fix)get_model_config()vs provider (shared fields)main)TransformerConfiggpu_backendimport TP2 → export, andAutoBridge.import_ckpt→export_ckptmaincompare.pybuilder branch): this branch vsmain, and this branch on amain-imported checkpointgpu_backend.roundtrip_checkpointTP2models,training,recipes,conversion,doc_consistency)mainruff check/ruff format --checkpre-commititself not run)Impact on other models: configs not derived from
GPTModelProvider(hybrid, T5, BERT, MLA, diffusion, vision sub-configs) now carryyarn_*fields set toNone. Every reader handles that:None. This matches the previous behaviour, where those fields didn't exist andhasattr()was False.GPTModel/HybridModelread the fields only whenposition_embedding_type == "yarn".getattr(..., None).hasattrcheck (nemotron-labs diffusion) only receivesMinistral3ModelProvider-derived configs, which already declared these fields.Checkpoints with the new keys load on older Bridge versions: unknown provider kwargs are dropped with a warning, and builder configs filter to declared fields.
Before your PR is "Ready for review"
Pre checks:
Additional Information
Qwen2Bridge), then Qwen3-MoE. Qwen3-Next and Qwen3.5 need a hybrid model config and come later.yarn_correction_range_round_to_int = False, while Hugging Face defaults totruncate=True. It is left unchanged here.🤖 Generated with Claude Code