Repository navigation
ci(e2e): preload Calico images into kind through the runner mirror - #5178
Open
matthewgrossman wants to merge 2 commits into
Open
matthewgrossman wants to merge 2 commits into
matthewgrossman wants to merge 2 commits into
Conversation
The app-listener lanes install Calico into kind. The kind node's containerd pulled the Calico images from Docker Hub anonymously and hit 429 Too Many Requests, failing all four lanes in kind setup. Pull the images on the host instead, where the runner's Docker daemon uses the nv-gha-runners Docker Hub mirror, and load them into kind before applying the manifest. The image list comes from the manifest, so a Calico upgrade cannot add an image that bypasses the preload. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
matthewgrossman
requested review from
a team,
derekwaynecarr,
mrunalp and
sjenning
as code owners
November 3, 2026 00:19
Member
Author
|
/ok to test 1501e07 |
|
Label |
TaylorMutch
previously approved these changes
Nov 3, 2026
Name the three Calico images instead of extracting them from the manifest, and define the Calico version once for both the manifest URL and the image tags. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Member
Author
|
/ok to test b074fc5 |
TaylorMutch
approved these changes
Nov 3, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Kubernetes app-listener lanes fail in "Set up kind cluster" when Docker Hub rate-limits the Calico image pulls. This preloads the Calico images through the runner's Docker Hub mirror, so the kind node never pulls from Docker Hub itself.
Related Issue
No issue: this fixes flaky CI (see CONTRIBUTING.md). The failure:
All four
kubernetes-app-listener-e2ejobs fail before any test runs, with:Seen on the #5078 run 41542126954 (attempt 1). #4936's own run 41516710979 also needed a second attempt for these four lanes.
The runners' Docker daemon pulls Docker Hub through the nv-gha-runners mirror (
https://dc.local.gha-runners.nvidia.com), which is why host-side pulls such as thepython:3.12-slimfixture don't fail. Calico's images are pulled by containerd inside the kind node, which doesn't use the host's mirror or credentials, so those pulls go to Docker Hub anonymously. The repo has no Docker Hub credential to log in with.Changes
setup-e2e-kind: before applying the Calico manifest, pull Calico's three images (cni,node,kube-controllers) on the host and load them into kind, the same way the action already loads the OpenShell images. Calico usesimagePullPolicy: IfNotPresent, so the node uses the loaded copies.CALICO_VERSION) and used for both the manifest URL and the image tags, so a version bump updates both.Testing
already present on machine, so the node pulled nothing.shellcheckon the step script; YAML parsesChecklist