Skip to content

refactor(model): add Qwen3 builder-backed model config with YaRN support - #6365

Merged
cuichenx merged 4 commits into
mainfrom
kjafarisadeg/qwen3-model-config
Oct 9, 2026
Merged

cuichenx merged 4 commits into
mainfrom
kjafarisadeg/qwen3-model-config

Conversation

@kamran-nvidia

@kamran-nvidia kamran-nvidia commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do ?

Moves dense Qwen3 onto the builder-backed ModelConfig / ModelBuilder path for recipes and checkpoint conversion (following Muse Glimmer, #5479), and adds YaRN support to that path.

Why: every GPT bridge inherits MODEL_CONFIG_CLASS = BridgeGPTModelConfig, so AutoBridge.get_model_config() already returns a config for Qwen3, but it ignores everything Qwen3's provider_bridge() set. On main, that config uses learned_absolute position embeddings, LayerNorm, no gated MLP, no QK layernorm, and linear biases. The provider path's deprecation warning points users to this path.

Changelog

  • Qwen3Bridge: model settings move from provider_bridge() into hf_config_to_model_config_kwargs(), and hf_config_to_provider_kwargs() reuses them, so the deprecated provider path is unchanged (same pattern as LlamaBridge). It also states the provider-implied defaults explicitly (masked_softmax_fusion, rope_scaling=False, rope_scaling_factor=1.0, and position_embedding_type via setdefault("rope"), which keeps "yarn" for YaRN checkpoints).
  • YaRN on the builder path: the seven yarn_* fields move from GPTModelProvider to Bridge's TransformerConfig. Megatron Core's GPTModel reads them from the transformer config, which the builder path passes as model_config.transformer. The MCore concentration-factor patch moves with them, because MCore's hasattr() check otherwise treats a None-valued field as YaRN. GPTModelProvider inherits the fields, so provider configs are unchanged.
  • Qwen3 YaRN defaults: when a Qwen3 YaRN config omits beta_fast / beta_slow / truncate, Bridge fills in Hugging Face's defaults (32 / 1 / True). On main these reach MCore as None, and model construction fails with a TypeError on both paths for standard Qwen3 YaRN checkpoints. The YaRN recipe only works because it sets them by hand. Values the checkpoint sets explicitly are preserved. A shared fix for all GPT families is in fix(model): apply Hugging Face YaRN defaults for omitted rope_scaling keys #6367; whichever PR merges second should drop the Qwen3-local copy.
  • Recipes: all 22 Qwen3 H100 recipes, including the YaRN 128K SFT recipe, now use get_model_config().
  • Conversion: Qwen3Bridge.USE_MODEL_CONFIG_FOR_CONVERSION = True, as in MuseGlimmerBridge. AutoBridge.import_ckpt, scripts/conversion (GPU import/export/round-trip) and compare_hf_and_megatron/compare.py now build Qwen3 from get_model_config(). New checkpoints record BridgeGPTModelConfig + GPTModelBuilder in run_config.yaml; provider-era Qwen3 checkpoints still load and export.
  • Docs: the builder-backed section of docs/bridge-guide.md describes the path generically (no per-model list) and names the family ModelBuilder (GPTModelBuilder / HybridModelBuilder).
  • Tests: the shared _FakeBridge in tests/unit_tests/recipes/test_qwen_recipes.py stubs get_model_config() for the migrated recipes.

Verification

Check Result
Provider path, main vs this branch (all fields; tied-bf16, untied-fp16, YaRN) identical, except YaRN beta_fast/beta_slow/round_to_int: None → HF defaults (the crash fix)
get_model_config() vs provider (shared fields) 0 differences (9 on main)
GPU forward, tiny Qwen3, provider vs builder (same HF weights) RoPE and YaRN: bitwise-identical logits
GPU forward, builder vs HF RoPE cosine 0.999990; YaRN cosine 0.999989, argmax 100%
Unit tests: Qwen3 bridge, Qwen3 recipes, TransformerConfig 101 passed
Regression: GPT provider, GPT model config, Llama, DeepSeek, Kimi K2.5-VL, Ministral3, StepFun, AutoBridge 282 passed, 2 skipped
Recipe factories (Qwen3 + Llama) 54 passed
Conversion, tiny Qwen3 (RoPE, YaRN with betas, YaRN without betas): gpu_backend import TP2 → export, and AutoBridge.import_ckpt → export_ckpt exported weights bit-exact vs the source HF checkpoint (47/47 tensors)
Backward compatibility: export a provider-era checkpoint imported on main bit-exact
Load imported checkpoint + forward (compare.py builder branch): this branch vs main, and this branch on a main-imported checkpoint bitwise-identical logits
gpu_backend.roundtrip_checkpoint TP2 pass
Unit tests: Qwen recipe fakes, conversion launcher, compare / VLM-generation flag dispatch 198 + 194 passed (1 skipped)
Unit-test directories reached by the shared change (models, training, recipes, conversion, doc_consistency) all pass except one GB200-only energon dispatcher case that fails identically on main
ruff check / ruff format --check clean (ruff 0.16.7; pre-commit itself not run)

Impact on other models: configs not derived from GPTModelProvider (hybrid, T5, BERT, MLA, diffusion, vision sub-configs) now carry yarn_* fields set to None. Every reader handles that:

  • The MCore concentration-factor helper is patched to return 1.0 for None. This matches the previous behaviour, where those fields didn't exist and hasattr() was False.
  • GPTModel / HybridModel read the fields only when position_embedding_type == "yarn".
  • HF export uses getattr(..., None).
  • The one Bridge hasattr check (nemotron-labs diffusion) only receives Ministral3ModelProvider-derived configs, which already declared these fields.

Checkpoints with the new keys load on older Bridge versions: unknown provider kwargs are dropped with a warning, and builder configs filter to declared fields.

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?
  • Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
    • Reviewer: Does the PR have correct import guards for all optional libraries?

Additional Information

  • Part of an incremental provider → builder migration for Qwen. Next: Qwen2 + MiMo (MiMo subclasses Qwen2Bridge), then Qwen3-MoE. Qwen3-Next and Qwen3.5 need a hybrid model config and come later.
  • The YaRN 128K recipe sets yarn_correction_range_round_to_int = False, while Hugging Face defaults to truncate=True. It is left unchanged here.

🤖 Generated with Claude Code

Move dense Qwen3 onto the ModelConfig/ModelBuilder path, following the
Llama migration (#5132):

- Qwen3Bridge maps Hugging Face configs in hf_config_to_model_config_kwargs().
  The deprecated provider path reuses the same kwargs, so provider_bridge()
  is no longer overridden.
- Declare the YaRN fields on the Bridge TransformerConfig (moved from
  GPTModelProvider) so builder-backed GPT configs can express YaRN, and move
  the MCore concentration-factor patch next to them.
- Fill the Hugging Face YaRN defaults (beta_fast=32, beta_slow=1,
  truncate=True) for Qwen3 configs that omit them. These previously reached
  Megatron Core as None and failed at model construction on both paths.
- Switch all 22 Qwen3 H100 recipes to get_model_config().

Signed-off-by: Kamran Jafari <kjafarisadeg@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kamran-nvidia kamran-nvidia added feature New capabilities, enhancements, or enablement work area:model Model implementations and HF bridge logic labels Oct 8, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 8, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@kamran-nvidia

Copy link
Copy Markdown
Contributor Author

/ok to test afbc585

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

kamran-nvidia and others added 2 commits October 8, 2026 18:59
Dense Qwen3 recipes now call AutoBridge.get_model_config(), so the shared
_FakeBridge in tests/unit_tests/recipes/test_qwen_recipes.py must provide it
alongside to_megatron_provider().

Signed-off-by: Kamran Jafari <kjafarisadeg@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Set USE_MODEL_CONFIG_FOR_CONVERSION on Qwen3Bridge, following Muse Glimmer,
so AutoBridge.import_ckpt and the conversion/compare tools construct Qwen3
from its builder-backed config instead of the deprecated provider. Imported
checkpoints now record BridgeGPTModelConfig + GPTModelBuilder in
run_config.yaml; provider-era Qwen3 checkpoints still load and export.

Signed-off-by: Kamran Jafari <kjafarisadeg@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kamran-nvidia

Copy link
Copy Markdown
Contributor Author

/ok to test 9a92fb4

Comment thread docs/bridge-guide.md Outdated
`GPTModelBuilder`. The provider API remains available for compatibility while
other model families migrate. Calling the legacy provider API for a
builder-backed family emits a deprecation warning.
Llama and dense Qwen3 also support the builder-backed configuration path. This

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think we can remove naming specific models in this doc. there are more newly added models that follow the builder-config pattern

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point — reworded the section to describe the builder-backed path generically (no per-model list), and it now names the family's ModelBuilder (GPTModelBuilder / HybridModelBuilder) rather than only the GPT builder. Kept the Llama snippet as a runnable example; happy to make it model-neutral too if you prefer. Fixed in c58fe0a.

…milies

More model families now follow the builder-config pattern, so the guide
describes the path generically instead of naming individual models, and
names the family's ModelBuilder rather than only GPTModelBuilder.

Signed-off-by: Kamran Jafari <kjafarisadeg@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kamran-nvidia

Copy link
Copy Markdown
Contributor Author

/ok to test c58fe0a

This branch was successfully deployed

1 active deployment
test — c58fe0ad Deployed Oct 9, 2026 by copy-pr-bot[bot] via cicd-wait-in-queue #22678
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:model Model implementations and HF bridge logic feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants