Repository navigation
fix(recipe): pin expert tensor parallelism on Qwen3.5-VL MoE pretrain recipes - #5754
Conversation
… recipes The three MoE pretrain mock recipes left expert_tensor_parallel_size unset, so Megatron-Core resolved it to the dense tensor parallel size of 4 and the expert grid demanded four times the ranks each recipe name advertises. initialize_model_parallel raised before any model was built. Adds a unit test covering the change (red-green verified). Signed-off-by: Pruthviraj Prakash <pruprakash@nvidia.com>
|
Light review — LGTM The fix pins
Without the fix an unset value resolves to TP=4, yielding expert grids of 32 / 256 / 1024 that do not divide the world size, so the change is both correct and necessary. The new parametrized unit test (test_qwen35_vl_moe_pretrain_mock_grids_divide_their_named_world_size) is the right GPU-free coverage and correctly simulates the unset case via _FakeModelCfgWithoutExpertTensorParallel. Related (out of scope, follow-up): the sibling qwen3_vl MoE mock configs — qwen3_vl_30b_a3b_pretrain_8gpu_h100_bf16_mock_config and qwen3_vl_235b_a22b_pretrain_256gpu_h100_bf16_mock_config — set expert_model_parallel_size but omit expert_tensor_parallel_size, the identical latent issue. Consider extending the same fix and test to them. Suggested test cases
|
|
/ok to test 1403572 |
|
Retrospective QA-fix tracking issue: #6394. It records the fix authored by @pruprakash and delivered by this merged PR. |
What does this PR do ?
Fixes three Qwen3.5-VL MoE pretrain recipes raising
RuntimeErroratinitialize_model_parallel, by pinning the expert tensor parallel size they left unset.Changelog
src/megatron/bridge/recipes/qwen_vl/h100/qwen35_vl.py: setexpert_tensor_parallel_size = 1on the 35B-A3B/8gpu, 122B-A10B/128gpu and 397B-A17B/512gpu pretrain mock recipes.tests/unit_tests/recipes/qwen_vl/test_qwen35_vl_recipes.py: 1 parametrized test (3 cases) asserting the dense and expert grids divide each recipe's named world size, built through a provider fake that leaves ETP unset.GitHub Actions CI
See the CI section in the Contributing doc for how to trigger the CI. A Nvidia developer will need to approve and trigger the CI for external contributors.
Before your PR is "Ready for review"
Pre checks:
Additional Information
model_parallel_config.py:506,parallel_state.py:871) and the expert grid needs 4x the ranks each recipe name advertises.tutorials/data/datacomp/train_qwen36_example.shand bothqwen3.6-35b-a3bcard leaves launch the 35B-A3B recipe on 8 GPUs with no overrides; all fail at init.4d7ce790(refactor(recipes): add h100 recipe namespace #4622). Masked in unit tests because the shared_FakeModelCfgpre-sets ETP=1.nvcr.io/nvidian/nemo:nightly.initialize_model_parallelat the recipe topology: ETP unset givesworld_size (8) is not divisible by expert_tensor_model_pipeline_parallel size (32); ETP=1 gives OK (dp=1 etp=1 ep=4). Unit red-green 3 failed to 3 passed;pytest tests/unit_tests/recipes/qwen_vl/225 passed.