Repository navigation
Conversation
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: Yu Yao <yaoyu.094@gmail.com>
yaoyu-33
force-pushed
the
codex/sync-save-sharding-validation-20261013
branch
from
October 13, 2026 07:28
6a40c9e to
8246192
Compare
Contributor
Author
|
/ok to test 8246192 |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Setting
CheckpointConfig.ckpt_load_validate_sharding_integrity=Falsecurrently still sendsvalidate_access_integrity=Trueto Core's global checkpoint loader and save path. The fully-parallel save wrapper also retains its default validation policy. This leaves the inherited checkpoint configuration ineffective in supported recipe training.Forward the policy to the global loader and to both save layers when the installed Core wrapper supports the save-side option. Preserve the development pin's load-only semantics, cached constant-structure behavior, and existing MIMO exceptions. Constructor capability is checked once at import.
Bases: Bridge
b1bb0a97ec5aad323eb2dd666e3f69aaebc1daff; upstream Core maina98aed553e25b4e86c59f50be54ac0ea4227b442. Bridge's pinned Core:e30d863a6d82a3c45f2d71355dc73fbf5610967b. Development pin:4a4de865795f123eda422ebf0a4c13d2418cf3cb.Source changes: NVIDIA/Megatron-LM#4422 and NVIDIA/Megatron-LM#5554. Related Bridge PR 6192 handles content-hash verification, a separate contract.
Validation on unchanged production base with test-only additions, then on this fix:
uv run python -m pytest tests/unit_tests/training/test_checkpointing.py -k 'save_checkpoint_global or global_checkpoint_load_honors_sharding_validation' -q -p no:cacheproviderBefore: exit 1, 11 failed and 53 passed; failure anchor
assert True is False. After: exit 0, 64 passed. Cases include both policy values, first/cached saves, constant structure, MIMO, and a strict legacy save-wrapper constructor.uv run python -m pytest tests/unit_tests/training/test_checkpointing.py -k 'TestSaveCheckpoint or TestCheckpointPathOverride or global_checkpoint_load_honors_sharding_validation' -q -p no:cacheprovider uv run pre-commit run --all-filesAdjacent: exit 0, 87 passed and one skipped. All-files hooks: exit 0, no modified files. Native CPU unit fixtures mock checkpoint I/O and distributed calls. No multi-rank resume, full development-pin runtime, or performance result is claimed.
uv run mypy --strict src/megatron/bridge/training/checkpointing.py tests/unit_tests/training/test_checkpointing.pyexits 2 with the same mypy 2.2.0 internal error on unchanged and patched sources; no type-check pass is claimed.