Repository navigation
Support for NVCRE init containers - #6375
Open
sjayaram-nv wants to merge 6 commits into
Open
sjayaram-nv wants to merge 6 commits into
sjayaram-nv wants to merge 6 commits into
Conversation
sjayaram-nv
requested review from
a team,
dingqingy-nv and
malay-nagda
as code owners
October 9, 2026 13:45
sjayaram-nv
marked this pull request as draft
October 9, 2026 13:45
cuichenx
added
feature
sjayaram-nv
marked this pull request as ready for review
October 12, 2026 07:22
sjayaram-nv
changed the title
cuichenx
added
the
needs-review
Contributor
|
I think we should invert the call logic. Right now this is a passthrough essentially of nvcre nemo-run flags. When jobs are launched via setup_experiment.py it already knows which commit is and repo location. I'd argue we should have a flag like Makes the whole flow simpler. |
sudostock
reviewed
Oct 13, 2026
sudostock
reviewed
Oct 13, 2026
sudostock
reviewed
Oct 13, 2026
sudostock
reviewed
Oct 13, 2026
sudostock
reviewed
Oct 13, 2026
sudostock
reviewed
Oct 13, 2026
cuichenx
added
waiting-on-customer
Contributor
|
/ok to test e52380c |
Contributor
|
Automatic Claude reviews have been retired. To request a pull-request review, post a comment containing: Add |
sudostock
reviewed
Oct 14, 2026
sudostock
requested changes
Oct 14, 2026
sudostock
left a comment
sudostock
left a comment
Contributor
There was a problem hiding this comment.
hf_token in arg parse from previous NVCRE PR has offline mode issue.
sudostock
pushed a commit
that referenced
this pull request
Oct 14, 2026
Signed-off-by: Smitha Jayaram <sjayaram@nvidia.com> Signed-off-by: Alex Filby <afilby@nvidia.com>
malay-nagda
requested changes
Oct 15, 2026
cuichenx
removed
the
needs-review
malay-nagda
approved these changes
Oct 15, 2026
cuichenx
added
needs-review
Signed-off-by: Smitha Jayaram <sjayaram@nvidia.com>
for NVCRE init_container in Run repo Signed-off-by: Smitha Jayaram <sjayaram@nvidia.com>
Signed-off-by: Smitha Jayaram <sjayaram@nvidia.com>
Signed-off-by: Smitha Jayaram <sjayaram@nvidia.com>
Default the NVCRE per-job timeout to 12h Signed-off-by: Smitha Jayaram <sjayaram@nvidia.com>
Signed-off-by: Alex Filby <alexfilby@gmail.com>
sjayaram-nv
force-pushed
the
sjayaram/nvcre-init-containers
branch
from
October 15, 2026 17:33
063422a to
bb0d261
Compare
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Support NVCRE init containers for llmb K8s workloads
Changelog
Added --nvcre_init_containers_json, forwarded to NvcreExecutor.init_containers, and
--nvcre_bridge_root (default /opt/Megatron-Bridge) so the generated command and
PYTHONPATH can use a Megatron-Bridge checkout that an init container placed in the
pod instead of the image's copy.