Repository navigation
refactor(model): build OLMoE and GLM-4.5 from builder-backed model configs - #6383
Open
kamran-nvidia wants to merge 3 commits into
Open
kamran-nvidia wants to merge 3 commits into
kamran-nvidia wants to merge 3 commits into
Conversation
Contributor
Author
|
/ok to test 64df9ee |
2 of 5 tasks
This was referenced Oct 9, 2026
kamran-nvidia
marked this pull request as ready for review
October 13, 2026 15:54
Move OLMoE onto the ModelConfig/ModelBuilder path, following Muse Glimmer: - OlMoEBridge maps Hugging Face configs in hf_config_to_model_config_kwargs(), including its QK-layernorm layer spec; the deprecated provider path reuses the same kwargs. USE_MODEL_CONFIG_FOR_CONVERSION = True. - olmoe_layer_spec accepts a builder-backed GPT model config as well as the provider, so both paths build the same OLMoESelfAttention layer. - OLMoE recipes construct OlMoEModelProvider directly and are unchanged. Signed-off-by: Kamran Jafari <kjafarisadeg@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Move GLM-4.5 onto the ModelConfig/ModelBuilder path, following Muse Glimmer: - GLM45Bridge maps Hugging Face configs in hf_config_to_model_config_kwargs(). The builder's default MoE layer spec is the GPT decoder block spec, which honors moe_layer_freq; the deprecated provider path keeps its explicit decoder block spec. USE_MODEL_CONFIG_FOR_CONVERSION = True. - All six GLM-4.5 H100 recipes use get_model_config(). Signed-off-by: Kamran Jafari <kjafarisadeg@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…configs After #6305, MODEL_CONFIG_CLASS defaults to None and each family opts in. Set MODEL_CONFIG_CLASS = BridgeGPTModelConfig on the bridges this PR migrates, matching Qwen3 and Llama, and assert it in their tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Kamran Jafari <kjafarisadeg@nvidia.com>
2 of 5 tasks
kamran-nvidia
force-pushed
the
kjafarisadeg/olmoe-glm45-model-config
branch
from
October 14, 2026 16:18
64df9ee to
584dc25
Compare
Contributor
Author
|
/ok to test 584dc25 |
Contributor
|
Automatic Claude reviews have been retired. To request a pull-request review, post a comment containing: Add |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Moves OLMoE and GLM-4.5 onto the builder-backed
ModelConfig/ModelBuilderpath, for recipes and checkpoint conversion, following Muse Glimmer.Changelog
MODEL_CONFIG_CLASSnow defaults toNone, so each migrated bridge setsMODEL_CONFIG_CLASS = BridgeGPTModelConfignext toUSE_MODEL_CONFIG_FOR_CONVERSION = True(as Qwen3 and Llama do), and its tests assert it.OlMoEBridge:provider_bridge()is replaced byhf_config_to_model_config_kwargs(), and the provider path reuses the same kwargs. The builder config carries the OLMoE layer spec (transformer_layer_spec=olmoe_layer_spec), so both paths buildOLMoESelfAttentionwith QK layernorm.USE_MODEL_CONFIG_FOR_CONVERSION = True.olmoe_layer_specnow also accepts a builder-backed GPT model config, using the builder-side TE layer helper that readsconfig.transformer. Previously it called the provider'sdefault_layer_spec, which reads provider-only fields (use_transformer_engine_full_layer_spec,dense_grouped_gemm) and fails on a builder config. The provider path is unchanged.GLM45Bridge: same kwargs migration. The builder config leavestransformer_layer_specunset because MCore's builder default for MoE models is alreadyget_gpt_decoder_block_spec, which honors the dense-firstmoe_layer_freq. The deprecated provider path keeps its explicit decoder block spec inhf_config_to_provider_kwargs().USE_MODEL_CONFIG_FOR_CONVERSION = True.get_model_config(). OLMoE recipes constructOlMoEModelProviderdirectly and are unchanged.get_model_config().BridgeGPTModelConfigat the real model depth (46/92 layers).Verification
Each family was validated separately on tiny random-weight checkpoints (single GB200 node, NeMo 26.10 container).
Conversion paths:
scripts/conversionGPU import at TP2 and export.AutoBridge.import_ckpt/export_ckpt.compare.py.roundtrip_checkpoint.Each was run on this branch and on
main.mainvs this branchget_model_config()vs providermain)main)main-imported checkpoint)e_score_correction_biasdtype (see below)main, and this branch on amain-imported checkpointroundtrip_checkpointTP2Regression: 9 OLMoE/GLM-4.5 recipe factories and 295 tests (1 skipped) across
models/olmoe,models/glm,test_auto_bridge.py,test_gpt_model_config.py, the OLMoE/GLM-4.5/GLM-5 recipe tests anddoc_consistency.Lint:
ruff checkandruff format --checkare clean on the changed code (ruff 0.16.7;pre-commititself not run).Known, pre-existing: a bf16 source checkpoint exports
e_score_correction_biasas fp32, because Megatron keeps the router expert bias in fp32. The values are identical after a cast, andmainexports the same way.After rebasing onto
mainwith #6305 (Megatron Core a98aed553): 268 passed, 1 skipped, including #6305's all-bridges opt-in check intest_gpt_model_config.py;ruffclean.Before your PR is "Ready for review"
Pre checks:
Additional Information
maindirectly.🤖 Generated with Claude Code