fix(models): default the HF config file lock to HF_HOME instead of ~/.cache/huggingface

[linmuchuiyang]

What does this PR do ?

Make safe_load_config_with_retry() create its file lock under HF_HOME (via huggingface_hub.constants.HF_HOME) instead of Path.home()/.cache/huggingface, and name the MEGATRON_CONFIG_LOCK_DIR override in the retries-exhausted error when the last failure was a lock timeout.

Why: the lock protects the Hugging Face config cache, but its default location ignored HF_HOME, so on containerised clusters it landed in /root/.cache/huggingface on the ephemeral container layer. On an 8-node × 4-GPU HF→Megatron import of DeepSeek-V4-Pro (nemo:26.08, pyxis/enroot, HF_HOME on a shared filesystem) every rank timed out 4 × 60 s:

filelock._error.Timeout: The file lock '/root/.cache/huggingface/.megatron_config_lock_fe96e83596e6dfb94d32c391d3b8772e.lock' could not be acquired.
ValueError: Failed to load configuration from deepseek-ai/DeepSeek-V4-Pro after 4 attempts. This might be due to network issues or concurrent access conflicts. Last error: ...

Setting MEGATRON_CONFIG_LOCK_DIR to a shared directory fixed it, but the override was only documented in the docstring and not mentioned by the error. With this change the default follows HF_HOME (identical behaviour when HF_HOME is unset; MEGATRON_CONFIG_LOCK_DIR still takes precedence) and the error tells the user what to do.

Changelog

• src/megatron/bridge/models/hf_pretrained/safe_config_loader.py: default lock directory os.getenv("MEGATRON_CONFIG_LOCK_DIR") or huggingface_hub.constants.HF_HOME; docstring updated; MEGATRON_CONFIG_LOCK_DIR hint appended to the final ValueError when last_exception is a filelock.Timeout.
• tests/unit_tests/utils/test_safe_config_loader.py: expected default path now Path(hf_constants.HF_HOME); new test_default_lock_directory_follows_hf_home (patched HF_HOME, lock created there, directory created) and test_lock_timeout_error_mentions_override.

GitHub Actions CI

External contributor without CI trigger permissions; happy to have a maintainer trigger CI. Local validation in the nemo:26.08 container (Bridge c932511): tests/unit_tests/utils/test_safe_config_loader.py 24/24 passed; end-to-end with the real filelock and HF_HOME set, safe_load_config_with_retry("deepseek-ai/DeepSeek-V4-Pro", trust_remote_code=True) succeeds offline and the lock file appears under HF_HOME; ruff check / ruff format --check clean.

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation? — docstring; no separate docs page mentions the lock directory.
• Does the PR affect components that are optional to install? — no (huggingface_hub is already a hard dependency via transformers).

Additional Information

• Related to Add Thread-Safe Configuration Loading to Prevent Race Conditions #606 (the PR that introduced the lock and the MEGATRON_CONFIG_LOCK_DIR override).

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[Jian-SA]

/ok to test a8d17b7

[Jian-SA]

/ok to test a8d17b7

[github-actions]

Automatic Claude reviews have been retired. To request a pull-request review, post a comment containing:

/review

Add model=claude to use a Claude reviewer (the default is model=codex). mode=light|strict selects the review depth; for example, /review model=claude mode=strict. Comment /review help for all options.

[Jian-SA]

/review

[svcnvidia-nemo-ci]

No verified findings. The loader change follows HF_HOME while preserving override precedence, lock naming, and timeout behavior. The diagnostic hint is conditional on the final exception being filelock.Timeout.

Review completed by source inspection; tests were not executed.

Suggested test cases

• test_default_lock_directory_follows_hf_home: lock path uses the patched HF_HOME, and the directory is created.
• test_custom_lock_directory_env_var: the explicit override takes precedence.
• test_lock_timeout_error_mentions_override: exhausted lock acquisition includes MEGATRON_CONFIG_LOCK_DIR in the error.
• test_error_message_includes_details: network failures retain the model path, attempt count, and original error.
• No perf tests impacted.

Review generated by codex

[linmuchuiyang]

Thanks for the branch update and the review. On the current head the only red check is link-check, and its single error is a transient 503 Service Unavailable for examples/models/stepfun/step37/README.md (the earlier 404s on the nemotron pages were from the stale base and are gone after the update). A re-run should clear it; the PR itself does not touch docs. Nothing else pending from my side.

[Jian-SA]

/ok to test 7c801b1