Repository navigation
fix(model): apply Hugging Face YaRN defaults for omitted rope_scaling keys - #6367
Merged
Merged
Conversation
Contributor
Author
|
/ok to test c9d6412 |
3 of 5 tasks
2 of 5 tasks
… keys The shared HF -> Megatron mapping copied only the YaRN keys a checkpoint sets. Standard Qwen2.5/Qwen3 YaRN configs omit beta_fast/beta_slow, which then reached Megatron Core as None and failed in YarnRotaryEmbedding at model construction. Fill the transformers defaults (beta_fast=32, beta_slow=1, truncate=True) for omitted keys on the GPT YaRN path. Explicit checkpoint values still win; MLA models and bridges that build providers by hand are unaffected. Signed-off-by: Kamran Jafari <kjafarisadeg@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ared mapping Qwen3 filled the Hugging Face YaRN defaults (beta_fast=32, beta_slow=1, truncate=True) locally because the shared mapping passed None for omitted keys. The shared GPT YaRN mapping now applies the same defaults, so the local copy is redundant. Qwen3's existing YaRN tests cover the shared path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Kamran Jafari <kjafarisadeg@nvidia.com>
kamran-nvidia
force-pushed
the
kjafarisadeg/yarn-hf-defaults
branch
from
October 10, 2026 02:36
c9d6412 to
04c9ff7
Compare
Contributor
Author
|
/ok to test 04c9ff7 |
kamran-nvidia
marked this pull request as ready for review
October 13, 2026 15:54
cuichenx
approved these changes
Oct 14, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Applies the Hugging Face YaRN defaults (
beta_fast=32,beta_slow=1,truncate=True) in the shared HF → Megatron config mapping when a YaRNrope_scalingomits them, so GPT-family checkpoints with YaRN no longer crash at model construction.The bug: the shared mapping only copies YaRN keys that are present. A config such as
{"rope_type": "yarn", "factor": 4.0, "original_max_position_embeddings": 32768}is the standard form for Qwen2.5 / Qwen3 long context. With it,yarn_beta_fast/yarn_beta_slowstayNone. Megatron Core'sGPTModelpasses these fields toYarnRotaryEmbeddingexplicitly, which overrides its defaults, so model construction fails:Changelog
MegatronModelBridge.YARN_HF_DEFAULTS: the defaults transformers'_compute_yarn_parametersuses for omitted keys.setdefault. Values the checkpoint sets explicitly always win.Qwen3Bridge: drops its local copy of the same three defaults, added by refactor(model): add Qwen3 builder-backed model config with YaRN support #6365, now that the shared mapping fills them. Qwen3's existing YaRN tests, which assert 32 / 1 /Truefor omitted keys and keep explicit values, now cover the shared path.Scope and behaviour change
MLA_ROPE_SCALING_MAPPINGbranch).truncatepreviously gotyarn_correction_range_round_to_int=None(no rounding). It now getsTrue, matching Hugging Face's default. Agreement with HF logits improves (table below).Verification
Provider path (
AutoBridge.to_megatron_provider()), tiny random-weight checkpoints, single GPU, logits compared with Hugging Face on the same input:rope_scalingmainTypeErrorat constructionTypeErrorat constructiontruncatetests/unit_tests/models/test_model_bridge.py: 57 passed.ruff check/ruff format --check: clean (ruff 0.16.7;pre-commititself not run).main(Megatron Core 8949def81) and removing the Qwen3 copy: 564 passed, 1 skipped acrosstest_model_bridge.py,models/qwen,recipes/qwen,test_transformer_config.py,test_gpt_provider.py, and the DeepSeek, Ministral3, Kimi K2.5-VL, StepFun and GPT-OSS model tests.Before your PR is "Ready for review"
Pre checks:
Additional Information
Qwen3Bridge; this PR, merging second, removes that copy.🤖 Generated with Claude Code