Skip to content

[recipe] fix: Resync Kimi K2.5-VL pipeline layout - #5640

Merged
yaoyu-33 merged 2 commits into
mainfrom
bug-hunter/kimi-k25-vl-layout-20260818
Aug 20, 2026
Merged

yaoyu-33 merged 2 commits into
mainfrom
bug-hunter/kimi-k25-vl-layout-20260818

Conversation

@yaoyu-33

Copy link
Copy Markdown
Contributor

Problem

The Kimi K2.5-VL H100 recipe materializes a static PP=16, VP=1 pipeline layout but does not register its existing layout builder. A supported public runner override such as --recipe kimi_k25_vl_sft_config -pp 4 -vp 1 therefore leaves the 16-stage layout unchanged. Megatron-Core derives VP=4 from that stale layout and rejects the explicit VP=1 setting; with only -pp 4, it can silently use VP=4 instead of the recipe's declared PP=4, VP=1 topology.

The path is run_recipe override application -> sync_model_pipeline_layout -> Kimi K2.5-VL recipe model config -> Megatron-Core pipeline-layout validation.

Fix

Register the recipe's existing _get_kimi_k25_vl_pipeline_layout function as _pipeline_model_parallel_layout_builder. The shared runner then rebuilds recipe-owned layouts after supported PP/VP overrides while preserving explicit user-provided layouts.

This is the same ownership pattern already used by the Kimi K2 recipe. No public API, dependency, CI, or Megatron-Core changes are included.

Validation

Regression test on the unmodified base revision:

uv run python -m pytest --confcutdir=tests/unit_tests/recipes/kimi_vl tests/unit_tests/recipes/kimi_vl/test_kimi_k25_vl.py -k pipeline_layout_tracks_supported_runner_override -q
1 failed, 31 deselected

The failure was the stale 16-entry layout instead of the expected four-entry PP=4, VP=1 layout.

The unchanged test after the fix:

uv run python -m pytest --confcutdir=tests/unit_tests/recipes/kimi_vl tests/unit_tests/recipes/kimi_vl/test_kimi_k25_vl.py -k pipeline_layout_tracks_supported_runner_override -q
1 passed, 31 deselected

Adjacent recipe tests:

uv run python -m pytest --confcutdir=tests/unit_tests/recipes tests/unit_tests/recipes/kimi/test_kimi_k2.py tests/unit_tests/recipes/kimi_vl/test_kimi_k25_vl.py -q
54 passed

uv run python -m pytest --confcutdir=tests/unit_tests/scripts/training tests/unit_tests/scripts/training/test_recipe_runner.py tests/unit_tests/scripts/training/test_run_recipe.py -k 'pipeline_layout or kimi_supported_pp_vp_override' -q
6 passed, 163 deselected

git diff --check and uv run pre-commit run --all-files also passed.

Scope

This change only resynchronizes the Kimi K2.5-VL recipe-owned layout for the topologies already declared by its helper. It does not add new parallel configurations or claim GPU, convergence, performance, or full-suite validation.

Related prior Kimi K2 fix: #5008.

Signed-off-by: Yu Yao <yaoyu.094@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Aug 18, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@yaoyu-33

Copy link
Copy Markdown
Contributor Author

/ok to test d132e40

@claude

claude Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

LGTM - light review complete.

The fix registers _get_kimi_k25_vl_pipeline_layout as _pipeline_model_parallel_layout_builder, mirroring the established ownership pattern in kimi/h100/kimi_k2.py:124, deepseek/h100/deepseek_v3.py:154, and exaone/h100/exaone_moe.py:134. sync_model_pipeline_layout (scripts/training/recipe_runner.py:390) reads that attribute to rebuild recipe-owned layouts after supported -pp/-vp overrides while preserving an explicit user-provided pipeline_model_parallel_layout. Without it, a -pp 4 -vp 1 override left the stale 16-stage layout in place - the exact bug the PR describes.

Verified:

  • The builder supports (4, 1) in map_pp_vp_to_layout, so the new test assertion resolves to a real layout, not the ValueError path.
  • The test import (_load_recipe_runner_module) and the compatibility alias kimi_k25_vl_sft_config both resolve.
  • No bugs, typos, or doc drift. Coverage for the changed contract is adequate.

No 3rdparty/Megatron-LM boundary touched; no public API or dependency change.

Suggested test cases:

  • tests/unit_tests/recipes/kimi_vl/test_kimi_k25_vl.py::test_sft_config_pipeline_layout_tracks_supported_runner_override
  • tests/unit_tests/recipes/kimi_vl/test_kimi_k25_vl.py::test_sft_config_pipeline_layout_matches_helper
  • No perf tests impacted.

@yaoyu-33 yaoyu-33 added area:recipe Training recipes and launch configs bug Something isn't working needs-more-tests Requires additional L0 and L1 test coverage before merge needs-review PR is ready for code review and waiting on a reviewer labels Aug 18, 2026
@yaoyu-33

Copy link
Copy Markdown
Contributor Author

/ok to test d132e40

@yaoyu-33

Copy link
Copy Markdown
Contributor Author

/ok to test cc3dbde

@yaoyu-33
yaoyu-33 merged commit b838075 into main Aug 20, 2026
127 checks passed
@yaoyu-33
yaoyu-33 deleted the bug-hunter/kimi-k25-vl-layout-20260818 branch August 20, 2026 00:41

This branch was previously deployed

2 inactive deployments
public — cc3dbde8 Deployed Aug 18, 2026 by copy-pr-bot[bot] via release / finalize / publish-docs #4731
test — cc3dbde8 Deployed Aug 18, 2026 by copy-pr-bot[bot] via cicd-wait-in-queue #20706
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:recipe Training recipes and launch configs bug Something isn't working needs-more-tests Requires additional L0 and L1 test coverage before merge needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant