Skip to content

ci: use writable config locks for offline test containers - #6420

Merged
cuichenx merged 1 commit into
mainfrom
chcui/mb-chores/fix-ci-config-lock-20261012
Oct 12, 2026
Merged

cuichenx merged 1 commit into
mainfrom
chcui/mb-chores/fix-ci-config-lock-20261012

Conversation

@cuichenx

Copy link
Copy Markdown
Contributor

Problem and change

After #6162 moved configuration locks into HF_HOME, the unprivileged GB200 test container cannot create them in the shared offline model cache. This blocks main and #6414 before model loading. Main failure and bump failure both report the same lock-file permission error.

For functional jobs mounting the offline cache, pass the existing MEGATRON_CONFIG_LOCK_DIR override to the writable container cache. Every rank in these single-container jobs shares that directory. The shared model cache remains offline; the library default and unit-test configuration are unchanged. This preserves #6162's shared-directory default for multi-node users.

This is a base CI integration failure introduced by Bridge #6162, independent of the MCore bump. No upstream Megatron-LM PR causes it; the MCore cause taxonomy does not apply.

Validation

  • Regression test executes the actual container-start shell and checks Docker arguments for mounted functional and unmounted unit jobs: 1 failed / 7 passed before; 8 passed after.
  • Real safe_load_config_with_retry, AutoConfig, and FileLock with an offline local fixture and effective UID 1000: unwritable cache reproduces the permission failure; a writable lock override succeeds.
  • All-file pre-commit and git diff --check pass. Independent review found no blocking findings.
  • Requested full L0/L1/L2 CI because the action is shared across the functional matrix. Local evidence does not prove the GPU suite; current-head human review and CI are required before merge.

Signed-off-by: Chen Cui <chcui@nvidia.com>
@cuichenx
cuichenx requested a review from a team as a code owner October 12, 2026 16:19
@cuichenx cuichenx added ci CI, automation, test queue, or workflow infrastructure work area:build Dependencies, packaging, images, and environment setup needs-review PR is ready for code review and waiting on a reviewer needs-more-tests Requires additional L0 and L1 test coverage before merge full-test-suite labels Oct 12, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Automatic Claude reviews have been retired. To request a pull-request review, post a comment containing:

/review

Add model=claude to use a Claude reviewer (the default is model=codex). mode=light|strict selects the review depth; for example, /review model=claude mode=strict. Comment /review help for all options.

@cuichenx
cuichenx merged commit d8fe0b5 into main Oct 12, 2026
33 checks passed
@cuichenx
cuichenx deleted the chcui/mb-chores/fix-ci-config-lock-20261012 branch October 12, 2026 16:37
nemo-autobot-origin Bot pushed a commit to svcnemo-autobot/Megatron-Bridge that referenced this pull request Oct 13, 2026
Bring in d8fe0b5 (NVIDIA-NeMo#6420, ci: use writable config locks for offline test containers) so this PR's CI uses a writable MEGATRON_CONFIG_LOCK_DIR. The merged tree is identical to the locally validated merge(ce1b60a, a2e8061).

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>

This branch was successfully deployed

1 active deployment
test — 67c27cb8 Deployed Oct 12, 2026 by copy-pr-bot[bot] via cicd-wait-in-queue #22778
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:build Dependencies, packaging, images, and environment setup ci CI, automation, test queue, or workflow infrastructure work full-test-suite needs-more-tests Requires additional L0 and L1 test coverage before merge needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant