Skip to content

refactor(model): build OLMoE and GLM-4.5 from builder-backed model configs - #6383

Open
kamran-nvidia wants to merge 3 commits into
mainfrom
kjafarisadeg/olmoe-glm45-model-config
Open

kamran-nvidia wants to merge 3 commits into
mainfrom
kjafarisadeg/olmoe-glm45-model-config

Conversation

@kamran-nvidia

@kamran-nvidia kamran-nvidia commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do ?

Moves OLMoE and GLM-4.5 onto the builder-backed ModelConfig / ModelBuilder path, for recipes and checkpoint conversion, following Muse Glimmer.

Changelog

  • Opt-in after fix(model): default MODEL_CONFIG_CLASS to None and opt in per family #6305: MODEL_CONFIG_CLASS now defaults to None, so each migrated bridge sets MODEL_CONFIG_CLASS = BridgeGPTModelConfig next to USE_MODEL_CONFIG_FOR_CONVERSION = True (as Qwen3 and Llama do), and its tests assert it.
  • OlMoEBridge: provider_bridge() is replaced by hf_config_to_model_config_kwargs(), and the provider path reuses the same kwargs. The builder config carries the OLMoE layer spec (transformer_layer_spec=olmoe_layer_spec), so both paths build OLMoESelfAttention with QK layernorm. USE_MODEL_CONFIG_FOR_CONVERSION = True.
  • olmoe_layer_spec now also accepts a builder-backed GPT model config, using the builder-side TE layer helper that reads config.transformer. Previously it called the provider's default_layer_spec, which reads provider-only fields (use_transformer_engine_full_layer_spec, dense_grouped_gemm) and fails on a builder config. The provider path is unchanged.
  • GLM45Bridge: same kwargs migration. The builder config leaves transformer_layer_spec unset because MCore's builder default for MoE models is already get_gpt_decoder_block_spec, which honors the dense-first moe_layer_freq. The deprecated provider path keeps its explicit decoder block spec in hf_config_to_provider_kwargs(). USE_MODEL_CONFIG_FOR_CONVERSION = True.
  • Recipes: all six GLM-4.5 H100 recipes use get_model_config(). OLMoE recipes construct OlMoEModelProvider directly and are unchanged.
  • Tests:
    • Builder mapping, provider/builder parity, and an OLMoE layer-spec equivalence check between provider and builder config.
    • GLM-4.5 recipe fakes provide get_model_config().
    • A strict recipe test runs every GLM-4.5 recipe against a real BridgeGPTModelConfig at the real model depth (46/92 layers).

Verification

Each family was validated separately on tiny random-weight checkpoints (single GB200 node, NeMo 26.10 container).

Conversion paths:

  • scripts/conversion GPU import at TP2 and export.
  • AutoBridge.import_ckpt / export_ckpt.
  • Load + forward as in compare.py.
  • roundtrip_checkpoint.

Each was run on this branch and on main.

Check OLMoE (MoE) GLM-4.5 (dense-first MoE)
Provider fields, main vs this branch identical identical (incl. MTP=1 config)
get_model_config() vs provider 0 diffs (18 on main) 0 diffs (24 on main)
Export vs source HF weights (GPU TP2, CPU, and from a main-imported checkpoint) bit-exact bit-exact except e_score_correction_bias dtype (see below)
Logits: this branch vs main, and this branch on a main-imported checkpoint bitwise-identical bitwise-identical
Logits vs HF cosine 0.999984 cosine 0.999945
roundtrip_checkpoint TP2 pass pass
Unit tests (family run) 63 passed 45 passed

Regression: 9 OLMoE/GLM-4.5 recipe factories and 295 tests (1 skipped) across models/olmoe, models/glm, test_auto_bridge.py, test_gpt_model_config.py, the OLMoE/GLM-4.5/GLM-5 recipe tests and doc_consistency.

Lint: ruff check and ruff format --check are clean on the changed code (ruff 0.16.7; pre-commit itself not run).

Known, pre-existing: a bf16 source checkpoint exports e_score_correction_bias as fp32, because Megatron keeps the router expert bias in fp32. The values are identical after a cast, and main exports the same way.

After rebasing onto main with #6305 (Megatron Core a98aed553): 268 passed, 1 skipped, including #6305's all-bridges opt-in check in test_gpt_model_config.py; ruff clean.

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?
  • Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
    • Reviewer: Does the PR have correct import guards for all optional libraries?

Additional Information

🤖 Generated with Claude Code

@kamran-nvidia kamran-nvidia added feature New capabilities, enhancements, or enablement work area:model Model implementations and HF bridge logic labels Oct 9, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 9, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@kamran-nvidia

Copy link
Copy Markdown
Contributor Author

/ok to test 64df9ee

kamran-nvidia and others added 3 commits October 14, 2026 06:57
Move OLMoE onto the ModelConfig/ModelBuilder path, following Muse Glimmer:

- OlMoEBridge maps Hugging Face configs in hf_config_to_model_config_kwargs(),
  including its QK-layernorm layer spec; the deprecated provider path reuses
  the same kwargs. USE_MODEL_CONFIG_FOR_CONVERSION = True.
- olmoe_layer_spec accepts a builder-backed GPT model config as well as the
  provider, so both paths build the same OLMoESelfAttention layer.
- OLMoE recipes construct OlMoEModelProvider directly and are unchanged.

Signed-off-by: Kamran Jafari <kjafarisadeg@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Move GLM-4.5 onto the ModelConfig/ModelBuilder path, following Muse Glimmer:

- GLM45Bridge maps Hugging Face configs in hf_config_to_model_config_kwargs().
  The builder's default MoE layer spec is the GPT decoder block spec, which
  honors moe_layer_freq; the deprecated provider path keeps its explicit
  decoder block spec. USE_MODEL_CONFIG_FOR_CONVERSION = True.
- All six GLM-4.5 H100 recipes use get_model_config().

Signed-off-by: Kamran Jafari <kjafarisadeg@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…configs

After #6305, MODEL_CONFIG_CLASS defaults to None and each family opts in.
Set MODEL_CONFIG_CLASS = BridgeGPTModelConfig on the bridges this PR
migrates, matching Qwen3 and Llama, and assert it in their tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Kamran Jafari <kjafarisadeg@nvidia.com>
@kamran-nvidia

Copy link
Copy Markdown
Contributor Author

/ok to test 584dc25

@github-actions

Copy link
Copy Markdown
Contributor

Automatic Claude reviews have been retired. To request a pull-request review, post a comment containing:

/review

Add model=claude to use a Claude reviewer (the default is model=codex). mode=light|strict selects the review depth; for example, /review model=claude mode=strict. Comment /review help for all options.

This branch was successfully deployed

1 active deployment
test — 584dc259 Deployed Oct 14, 2026 by copy-pr-bot[bot] via cicd-wait-in-queue #22856
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:model Model implementations and HF bridge logic feature New capabilities, enhancements, or enablement work full-test-suite needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants