DO NOT MERGE: diagnose fork-PR nvcr.io credential loss - #6978
Closed
hujc7 wants to merge 3 commits into
Closed
Conversation
Fork-PR cuRobo builds fail with "denied: Access Denied" pulling the private isaac-sim base image, while the base image build succeeds in the same run with the same empty NGC_API_KEY. The two jobs ran on different runners, so the runner credential is doing the work and something about it differs per instance. The action repoints DOCKER_CONFIG at a rewritten copy with credsStore and credHelpers removed. If a runner's nvcr.io credential is helper-backed that rewrite destroys it; if it is a plain auths entry it survives. Printing the config shape before and after the rewrite distinguishes the two. Only registry names and booleans are printed, never credential values. Remove before merge.
The first diagnostic run passed because the deps hash covers the Dockerfile and install-relevant files but not .github/, so touching only the action left the hash unchanged, the deps-cache hit, and the image was never built. No build means no nvcr.io pull and nothing to diagnose. Remove with the diagnostic.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
1. Purpose — diagnostic only, do not merge
Fork-PR
Build cuRobo Docker Imagefails pulling the private isaac-sim baseimage:
In the same run, with the same empty
NGC_API_KEY,Build Base Docker Imagepulled the same image at the same digest successfully. The two jobsran on different runners (
ip-10-0-2-183✅ /ip-10-0-3-247❌), so therunner credential is what does the work and it differs per instance.
2. What this tests
setup-docker-configrepointsDOCKER_CONFIGat a rewritten copy withcredsStoreandcredHelpersremoved. If a runner's nvcr.io credential ishelper-backed, that rewrite destroys the only credential a fork PR has. If it is
a plain
authsentry, it survives.Printing the config shape before and after the rewrite distinguishes the two:
has_credential {'nvcr.io': True}has_credential {'nvcr.io': False}+credHelpers_for ['nvcr.io']nvcr.ioabsent3. Safety
Only dict keys (registry hostnames) and booleans are printed, never values.
Verified locally against a config containing fake credentials: no value appears
in the output.
4. Type of change