Repository navigation
Conversation
Signed-off-by: Yu Yao <yaoyu.094@gmail.com>
Contributor
Author
|
/ok to test 825d491 |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
With a
GPTModelProvider,mtp_num_layers > 0, anddist.use_decentralized_pg=True, model construction reaches MCore's global pipeline-rank fallback even though Bridge has already supplied the provider's process-group collection. The global MPU pipeline group is unset in that mode, so MTP spec construction fails before training starts.Forward the rank from the provider's pipeline group to the final
get_gpt_mtp_block_speccall. This completes the Bridge caller migration started in #5752: its decoder callable receivespp_rank, but the final MTP helper still omitted it. Calls without a provider process-group collection retain their existing fallback. There are no public signature, dependency, or submodule changes.The supported path is Bridge pretrain/setup → decentralized initialization →
ModelProviderMixin.get_model→GPTModelProvider.provide→mtp_block_spec. This change is scoped to that provider path; it does not claim validation of all decentralized distributed training combinations.Validation uses Bridge base
9166429859ed9a29860ca2e1f3fc317173df8c0eand its exact MCore pin8949def81eb744b9219e2d794c80265928c40a5b. MCore discovery base wasb86e75c184483a54689d17b1a9eb6c5bcc999425; a later default-branch recheck observed6a6c8aa6e9ee258db08bed381e882c0f754d14bc, which does not change the pinned test dependency.uv run python -m pytest tests/unit_tests/models/test_gpt_provider.py -k test_mtp_spec_uses_provider_pipeline_group -q -p no:cacheprovider— exit 1, all three cases fail withAssertionError: MTP spec must not read global pipeline rank. The test calls the real pinned MCore spec builder; it covers PP=1 and both PP=2 ranks.uv run python -m pytest tests/unit_tests/models/test_gpt_provider.py tests/unit_tests/models/test_model_provider_mixin.py -q -p no:cacheprovider— exit 0, 45 passed, 1 existing GPU-dependent test skipped.uv run pre-commit run --all-files— exit 0; no files modified.uv run mypy --strict src/megatron/bridge/models/gpt_provider.py— mypy 2.2.0 exits 2 with the same internal tool error on both the unmodified base and patched source. No passing type-check result is claimed.Validation is CPU contract coverage, not a GPU training result. A separate read-only review of the final diff, source ownership, duplicate search, supported path, and red/green evidence found no blocking findings. All-state searches in both repositories found related upstream caller work in NVIDIA/Megatron-LM#8048 and fallback changes in NVIDIA/Megatron-LM#8057, but neither updates this Bridge call.