[ckpt, build] fix: support pip-installed conversion on Megatron-Core 0.18.x by yaoyu-33 · Pull Request #5193 · NVIDIA-NeMo/Megatron-Bridge

yaoyu-33 · 2026-07-30T16:53:14Z

What

Makes a pip-installed Megatron Bridge work against the released Megatron-Core 0.18.x line, not just the exact source pin. This is a concrete step toward #5148.

Current main pins Megatron-Core commit 24bad8e677 (0.20.0), while the public PyPI index currently ends at 0.18.2: there is no public 0.19.x or 0.20.x package to install. The supported pip range is therefore declared explicitly as >=0.18.0,<0.21 and tested against both the current source pin and the actual public 0.18.x line.

On 0.18.x, megatron.training.models.gpt does not exist. Without this change, that missing module breaks megatron.bridge.training and every checkpoint import/export path that depends on it.

How

megatron.bridge.compat.mcore_version declares the public compatibility floor (0.18.0), current source pin (0.20.0), and exclusive next-minor ceiling (0.21). Tests keep the source gitlink, code declaration, and package metadata aligned.
megatron.bridge.compat.mcore_gpt is the stable facade for GPT builder/config symbols and serialized targets. It prefers the upstream implementation when present and uses the vendored fallback only when the whole upstream module is absent; unrelated transitive import errors still propagate.
The fallback is refreshed from the current 0.20 GPT implementation. Its model-builder call projects the current arguments onto the callable exposed by older MCore releases, including omitting the new layerwise optimizer/layout arguments on 0.18.x.
All current-main consumers, including the newer GPT model-config path, import through the facade. Legacy upstream serialized targets are normalized when loading existing run configs.
Package metadata and user documentation now state megatron-core[dev,mlm]>=0.18.0,<0.21.

No file under 3rdparty/Megatron-LM is modified, no dependency is added, and no public conversion API signature changes.

Validation

Standard project container with uv:

Environment	Result
Current source pin (`0.20.0`)	822 Bridge modules imported; focused compatibility/version/model load-save/conversion matrix passed (100 tests), with one separately identified workstation-driver-only NCCL mock failure
Exact public PyPI `0.18.0`	822 Bridge modules imported with zero failures; 94 focused tests passed, 6 expected upstream-parity tests skipped, workstation-only NCCL case deselected
Exact public PyPI `0.18.2`	822 Bridge modules imported with zero failures; 94 focused tests passed, 6 expected upstream-parity tests skipped, workstation-only NCCL case deselected
Built wheel	METADATA contains `megatron-core[dev,mlm]<0.21,>=0.18.0`

The focused matrix covers support boundaries, facade/fallback selection, upstream API parity, old-builder argument projection, stable and legacy config targets, checkpoint load/save, CPU conversion launcher paths, and a tiny exact HF→MCore→HF tied/untied-logit roundtrip.

Authoritative Qwen3-30B-A3B verification card, on one 8-GPU node with exact public MCore 0.18.0 and model revision ad44e777bcd18fa416d9da3bd8f70d33ebb85d39:

TP4/PP2/EP4 HF→Megatron import completed all 3,411 mappings and wrote the distributed checkpoint.
The pinned HF/MCore forward comparison predicted token 12095 (" Paris") on both paths; cosine similarity was 0.999317 (card gate >0.99), with the card-recorded max/mean logit diagnostics 0.593750 / 0.130064.
Two independent imported-checkpoint generation runs completed exactly 32 tokens with the card's byte-identical expected text.
TP4/PP2/EP4 distributed Megatron→HF export completed all 3,411 mappings.
Strict roundtrip verification compared all 18,867 BF16 tensors: exact key, shape, dtype, and bitwise value equality. The export reloaded as Qwen3MoeForCausalLM, and norm_topk_prob remained true.

uv run pre-commit run --all-files passed before every new DCO-signed merge/conflict-resolution commit.

Main integration

Latest main was merged normally into this branch without rebasing or force-pushing. Conflicts were resolved by retaining current main's optional ModelOpt behavior and tests while routing its new GPT model-config consumer through the compatibility facade.

Refs #5148

…0.18.x Bridge only worked against the exact Megatron-Core commit pinned in 3rdparty/Megatron-LM. On the released 0.18.x line, 64 of the 806 megatron.bridge modules failed to import because megatron.training.models.gpt (GPTModelConfig / GPTModelBuilder / mtp_block_spec) was added upstream after 0.18.x was cut, which broke every checkpoint save/load path — including AutoBridge.import_ckpt and export_ckpt. Declare the supported window as a rule relative to the pin (previous minor release N-1 through the next minor, i.e. >=0.18.0,<0.20 for the 0.19.0 pin), bound the megatron-core dependency to it so installing Bridge no longer replaces a compatible preinstalled Megatron-Core, and fall back to a vendored copy of megatron/training/models/gpt.py when the upstream module is absent. Verified on 8xH100 against megatron-core 0.18.0 installed from PyPI with the Qwen3-30B-A3B model card flow: HF -> Megatron load and generation, AutoBridge.import_ckpt, generation from the Megatron checkpoint with EP=8, and export_ckpt back to HF. All 806 bridge modules import on both 0.18.0 and the 0.19.0 pin. Refs #5148 Signed-off-by: Yu Yao <yuya@nvidia.com>

copy-pr-bot · 2026-07-30T16:53:18Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

yaoyu-33 · 2026-07-30T22:22:39Z

/ok to test 84c98ed

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

yaoyu-33 · 2026-08-12T23:58:01Z

/ok to test 4dd6ae2

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

yaoyu-33 · 2026-08-13T01:15:15Z

/ok to test 11605ee

yaoyu-33 · 2026-08-28T07:05:03Z

/ok to test 5a21fa5

Signed-off-by: Yu Yao <yaoyu.094@gmail.com>

yaoyu-33 · 2026-09-16T19:25:48Z

/ok to test 386a429

Signed-off-by: Yu Yao <yaoyu.094@gmail.com>

yaoyu-33 · 2026-09-29T07:16:17Z

/ok to test 2f4b050

yaoyu-33 · 2026-09-29T19:07:06Z

/ok to test 74cb7d3

yaoyu-33 · 2026-09-30T19:07:40Z

/ok to test 2510618

github-actions · 2026-09-30T19:08:01Z

Automatic Claude reviews have been retired. To request a pull-request review, post a comment containing:

/review

Add model=claude to use a Claude reviewer (the default is model=codex). mode=light|strict selects the review depth; for example, /review model=claude mode=strict. Comment /review help for all options.

yaoyu-33 · 2026-10-01T07:08:20Z

/ok to test 7f9e9c6

github-actions · 2026-10-01T07:09:32Z

🌿 Preview your docs: https://nvidia-preview-pr-5193-5451d2ed86c1.docs.buildwithfern.com/nemo/megatron-bridge

yaoyu-33 · 2026-10-01T19:08:00Z

/ok to test 9dff99c

yaoyu-33 · 2026-10-02T07:09:27Z

/ok to test 0f6afa6

Signed-off-by: Yu Yao <yaoyu.094@gmail.com>

yaoyu-33 · 2026-10-02T07:31:58Z

/ok to test 76c54d1

yaoyu-33 · 2026-10-02T19:12:56Z

/ok to test a4ba423

yaoyu-33 · 2026-10-03T07:07:48Z

/ok to test 5cee906

yaoyu-33 · 2026-10-03T19:10:56Z

/ok to test 5451d2e

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

yaoyu-33 added area:build Dependencies, packaging, images, and environment setup area:ckpt Checkpoint conversion, loading, export, and save paths bug Something isn't working labels Jul 30, 2026

yaoyu-33 mentioned this pull request Jul 30, 2026

[Tracking] Support pip-installed Bridge conversion across the previous MCore minor release (N-1) to the current pin #5148

Closed

6 tasks

[ckpt, build] refactor: centralize MCore compatibility

84c98ed

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

copy-pr-bot Bot temporarily deployed to public July 30, 2026 22:23 Inactive

copy-pr-bot Bot temporarily deployed to test July 30, 2026 22:23 Inactive

copy-pr-bot Bot temporarily deployed to public July 30, 2026 22:46 Inactive

copy-pr-bot Bot temporarily deployed to public July 30, 2026 22:47 Inactive

yaoyu-33 added 3 commits August 12, 2026 15:58

merge: refresh MCore 0.18 compatibility against main

0e292d9

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

merge: update branch to latest main

c83f9c8

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

merge: update branch to latest main

4dd6ae2

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

copy-pr-bot Bot temporarily deployed to public August 12, 2026 23:58 Inactive

copy-pr-bot Bot temporarily deployed to test August 12, 2026 23:59 Inactive

copy-pr-bot Bot temporarily deployed to public August 13, 2026 00:21 Inactive

copy-pr-bot Bot temporarily deployed to public August 13, 2026 00:22 Inactive

docs: hide internal MCore fallback from API reference

11605ee

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

copy-pr-bot Bot temporarily deployed to public August 13, 2026 01:16 Inactive

copy-pr-bot Bot temporarily deployed to test August 13, 2026 01:16 Inactive

copy-pr-bot Bot temporarily deployed to public August 13, 2026 01:40 Inactive

copy-pr-bot Bot temporarily deployed to public August 13, 2026 01:41 Inactive

copy-pr-bot Bot temporarily deployed to public August 13, 2026 02:15 Inactive

Merge branch 'main' into yuya/mcore-018-compat

5a21fa5

copy-pr-bot Bot temporarily deployed to test August 28, 2026 07:06 Inactive

[build] chore: reconcile MCore compatibility with main

386a429

Signed-off-by: Yu Yao <yaoyu.094@gmail.com>

copy-pr-bot Bot deployed to test September 16, 2026 19:26 Active

[ckpt, build] chore: reconcile MCore compatibility with main

2f4b050

Signed-off-by: Yu Yao <yaoyu.094@gmail.com>

copy-pr-bot Bot deployed to test September 29, 2026 07:17 Active

Merge branch 'main' into yuya/mcore-018-compat

74cb7d3

Merge branch 'main' into yuya/mcore-018-compat

2510618

Merge branch 'main' into yuya/mcore-018-compat

7f9e9c6

Merge branch 'main' into yuya/mcore-018-compat

9dff99c

Merge branch 'main' into yuya/mcore-018-compat

0f6afa6

[ckpt, build] fix: align fallback MTP signature with pinned Core

76c54d1

Signed-off-by: Yu Yao <yaoyu.094@gmail.com>

Merge branch 'main' into yuya/mcore-018-compat

a4ba423

Merge branch 'main' into yuya/mcore-018-compat

5cee906

Merge branch 'main' into yuya/mcore-018-compat

5451d2e

fix(compat): preserve MTP specs and reconcile main

1dfc47d

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

Conversation

yaoyu-33 commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Validation

Main integration

Uh oh!

copy-pr-bot Bot commented Jul 30, 2026

Uh oh!

yaoyu-33 commented Jul 30, 2026

Uh oh!

yaoyu-33 commented Aug 12, 2026

Uh oh!

yaoyu-33 commented Aug 13, 2026

Uh oh!

yaoyu-33 commented Aug 28, 2026

Uh oh!

yaoyu-33 commented Sep 16, 2026

Uh oh!

yaoyu-33 commented Sep 29, 2026

Uh oh!

yaoyu-33 commented Sep 29, 2026

Uh oh!

yaoyu-33 commented Sep 30, 2026

Uh oh!

github-actions Bot commented Sep 30, 2026

Uh oh!

yaoyu-33 commented Oct 1, 2026

Uh oh!

github-actions Bot commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yaoyu-33 commented Oct 1, 2026

Uh oh!

yaoyu-33 commented Oct 2, 2026

Uh oh!

yaoyu-33 commented Oct 2, 2026

Uh oh!

yaoyu-33 commented Oct 2, 2026

Uh oh!

yaoyu-33 commented Oct 3, 2026

Uh oh!

yaoyu-33 commented Oct 3, 2026

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yaoyu-33 commented Jul 30, 2026 •

edited

Loading

github-actions Bot commented Oct 1, 2026 •

edited

Loading