Repository navigation
[recipe] fix: Resync Kimi K2.5-VL pipeline layout - #5640
Conversation
Signed-off-by: Yu Yao <yaoyu.094@gmail.com>
|
/ok to test d132e40 |
|
LGTM - light review complete. The fix registers _get_kimi_k25_vl_pipeline_layout as _pipeline_model_parallel_layout_builder, mirroring the established ownership pattern in kimi/h100/kimi_k2.py:124, deepseek/h100/deepseek_v3.py:154, and exaone/h100/exaone_moe.py:134. sync_model_pipeline_layout (scripts/training/recipe_runner.py:390) reads that attribute to rebuild recipe-owned layouts after supported -pp/-vp overrides while preserving an explicit user-provided pipeline_model_parallel_layout. Without it, a -pp 4 -vp 1 override left the stale 16-stage layout in place - the exact bug the PR describes. Verified:
No 3rdparty/Megatron-LM boundary touched; no public API or dependency change. Suggested test cases:
|
|
/ok to test d132e40 |
|
/ok to test cc3dbde |
Problem
The Kimi K2.5-VL H100 recipe materializes a static PP=16, VP=1 pipeline layout but does not register its existing layout builder. A supported public runner override such as
--recipe kimi_k25_vl_sft_config -pp 4 -vp 1therefore leaves the 16-stage layout unchanged. Megatron-Core derives VP=4 from that stale layout and rejects the explicit VP=1 setting; with only-pp 4, it can silently use VP=4 instead of the recipe's declared PP=4, VP=1 topology.The path is
run_recipeoverride application ->sync_model_pipeline_layout-> Kimi K2.5-VL recipe model config -> Megatron-Core pipeline-layout validation.Fix
Register the recipe's existing
_get_kimi_k25_vl_pipeline_layoutfunction as_pipeline_model_parallel_layout_builder. The shared runner then rebuilds recipe-owned layouts after supported PP/VP overrides while preserving explicit user-provided layouts.This is the same ownership pattern already used by the Kimi K2 recipe. No public API, dependency, CI, or Megatron-Core changes are included.
Validation
Regression test on the unmodified base revision:
The failure was the stale 16-entry layout instead of the expected four-entry PP=4, VP=1 layout.
The unchanged test after the fix:
Adjacent recipe tests:
git diff --checkanduv run pre-commit run --all-filesalso passed.Scope
This change only resynchronizes the Kimi K2.5-VL recipe-owned layout for the topologies already declared by its helper. It does not add new parallel configurations or claim GPU, convergence, performance, or full-suite validation.
Related prior Kimi K2 fix: #5008.