Skip to content

fix(model): apply Hugging Face YaRN defaults for omitted rope_scaling keys - #6367

Merged
cuichenx merged 2 commits into
mainfrom
kjafarisadeg/yarn-hf-defaults
Oct 14, 2026
Merged

cuichenx merged 2 commits into
mainfrom
kjafarisadeg/yarn-hf-defaults

Conversation

@kamran-nvidia

@kamran-nvidia kamran-nvidia commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do ?

Applies the Hugging Face YaRN defaults (beta_fast=32, beta_slow=1, truncate=True) in the shared HF → Megatron config mapping when a YaRN rope_scaling omits them, so GPT-family checkpoints with YaRN no longer crash at model construction.

The bug: the shared mapping only copies YaRN keys that are present. A config such as {"rope_type": "yarn", "factor": 4.0, "original_max_position_embeddings": 32768} is the standard form for Qwen2.5 / Qwen3 long context. With it, yarn_beta_fast / yarn_beta_slow stay None. Megatron Core's GPTModel passes these fields to YarnRotaryEmbedding explicitly, which overrides its defaults, so model construction fails:

TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'   (yarn_rotary_pos_embedding.py:244)

Changelog

  • MegatronModelBridge.YARN_HF_DEFAULTS: the defaults transformers' _compute_yarn_parameters uses for omitted keys.
  • The GPT YaRN branch of the shared mapping fills those defaults with setdefault. Values the checkpoint sets explicitly always win.
  • Unit tests: defaults are filled for omitted keys, explicit keys are preserved, and non-YaRN configs are unaffected.
  • Qwen3Bridge: drops its local copy of the same three defaults, added by refactor(model): add Qwen3 builder-backed model config with YaRN support #6365, now that the shared mapping fills them. Qwen3's existing YaRN tests, which assert 32 / 1 / True for omitted keys and keep explicit values, now cover the shared path.

Scope and behaviour change

  • Affected: generic GPT bridges whose YaRN config goes through the shared mapping (for example Qwen2/2.5, Qwen3, Qwen3-MoE).
  • Not affected:
    • MLA models (separate MLA_ROPE_SCALING_MAPPING branch).
    • Mistral and Ministral3, which build their providers by hand.
    • GPT-OSS, which sets every YaRN field explicitly after the mapping.
  • Behaviour change for configs that already worked: a YaRN config that sets the betas but omits truncate previously got yarn_correction_range_round_to_int=None (no rounding). It now gets True, matching Hugging Face's default. Agreement with HF logits improves (table below).

Verification

Provider path (AutoBridge.to_megatron_provider()), tiny random-weight checkpoints, single GPU, logits compared with Hugging Face on the same input:

HF rope_scaling main this PR
Qwen2, YaRN, no betas TypeError at construction builds; cosine 0.999993, argmax 100%
Qwen3, YaRN, no betas TypeError at construction builds; cosine 0.999989, argmax 100%
Qwen2, YaRN, betas set, no truncate cosine 0.999980, argmax 99.2% cosine 0.999993, argmax 100%
  • tests/unit_tests/models/test_model_bridge.py: 57 passed.
  • Regression run (GPT provider, Ministral3, GPT-OSS, StepFun, DeepSeek, Qwen, Llama, AutoBridge): 555 passed, 1 skipped.
  • ruff check / ruff format --check: clean (ruff 0.16.7; pre-commit itself not run).
  • After rebasing onto main (Megatron Core 8949def81) and removing the Qwen3 copy: 564 passed, 1 skipped across test_model_bridge.py, models/qwen, recipes/qwen, test_transformer_config.py, test_gpt_provider.py, and the DeepSeek, Ministral3, Kimi K2.5-VL, StepFun and GPT-OSS model tests.

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?
  • Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
    • Reviewer: Does the PR have correct import guards for all optional libraries?

Additional Information

🤖 Generated with Claude Code

@kamran-nvidia kamran-nvidia added bug Something isn't working area:model Model implementations and HF bridge logic labels Oct 8, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 8, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@kamran-nvidia

Copy link
Copy Markdown
Contributor Author

/ok to test c9d6412

kamran-nvidia and others added 2 commits October 9, 2026 18:24
… keys

The shared HF -> Megatron mapping copied only the YaRN keys a checkpoint
sets. Standard Qwen2.5/Qwen3 YaRN configs omit beta_fast/beta_slow, which
then reached Megatron Core as None and failed in YarnRotaryEmbedding at
model construction.

Fill the transformers defaults (beta_fast=32, beta_slow=1, truncate=True)
for omitted keys on the GPT YaRN path. Explicit checkpoint values still
win; MLA models and bridges that build providers by hand are unaffected.

Signed-off-by: Kamran Jafari <kjafarisadeg@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ared mapping

Qwen3 filled the Hugging Face YaRN defaults (beta_fast=32, beta_slow=1,
truncate=True) locally because the shared mapping passed None for omitted
keys. The shared GPT YaRN mapping now applies the same defaults, so the
local copy is redundant. Qwen3's existing YaRN tests cover the shared path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Kamran Jafari <kjafarisadeg@nvidia.com>
@kamran-nvidia
kamran-nvidia force-pushed the kjafarisadeg/yarn-hf-defaults branch from c9d6412 to 04c9ff7 Compare October 10, 2026 02:36
@kamran-nvidia

Copy link
Copy Markdown
Contributor Author

/ok to test 04c9ff7

@kamran-nvidia
kamran-nvidia marked this pull request as ready for review October 13, 2026 15:54
@cuichenx cuichenx added the needs-review PR is ready for code review and waiting on a reviewer label Oct 13, 2026
@cuichenx
cuichenx merged commit f46e89e into main Oct 14, 2026
224 of 228 checks passed
@cuichenx
cuichenx deleted the kjafarisadeg/yarn-hf-defaults branch October 14, 2026 03:39

This branch was successfully deployed

1 active deployment
test — 04c9ff7a Deployed Oct 10, 2026 by copy-pr-bot[bot] via cicd-wait-in-queue #22725
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:model Model implementations and HF bridge logic bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants