Re-Implementation of the Garden Linux/Gardener integration test pipeline - #422
Re-Implementation of the Garden Linux/Gardener integration test pipeline#422SujanaSubr wants to merge 12 commits into
Conversation
Gchbg
left a comment
There was a problem hiding this comment.
See suggested code edits.
Notably missing: unpublish. The unpublish operation needs to remove the cloud profiles, since unpublishing the images makes them invalid. Doing so in a reliable way is not trivial. (For example, since manifests are guaranteed to be atomic and correct, tracking the profiles in the manifests might be a way to do so, or maybe there is a more straighforward solution.)
Not reviewed: internal/nspcpfl
The reason is that I think this is the incorrect approach to doing this, since you have code to handle all cloud providers in there. The rest of GLCI is modular, where understanding and implementation for any given cloud uses the pluggable module framework to contain that knowledge into one file, with the rest of GLCI being unaware of specific cloud implementations. This is a large change and may require evolving the design of GLCI. Because of that I'm open to approving it regardless, with the intention to refactor it later.
| func (p *Publisher) Publish(ctx context.Context, version, commit string, omitComponentDescritpr bool) error { | ||
| // Publish publishes a release to all cloud providers specified in the flavors and publishing configurations. | ||
| func Publish(ctx context.Context, flavorsConfig FlavorsConfig, publishingConfig PublishingConfig, aliasesConfig AliasesConfig, version, | ||
| commit string, omitComponentDescritpr bool, omitNSCloudProfile bool, |
There was a problem hiding this comment.
| commit string, omitComponentDescritpr bool, omitNSCloudProfile bool, | |
| commit string, omitComponentDescritpr, omitNSCloudProfile bool, |
| func publish(ctx context.Context, flavorsConfig FlavorsConfig, aliasesConfig AliasesConfig, credsSource credsprovider.CredsSource, | ||
| manifestSource, manifestTarget cloudprovider.ArtifactSource, sources map[string]cloudprovider.ArtifactSource, | ||
| targets []cloudprovider.PublishingTarget, ocmTarget cloudprovider.OCMTarget, state task.StatePersistor, version, commit string, | ||
| omitComponentDescritpr bool, omitNSCloudProfile bool, |
There was a problem hiding this comment.
| omitComponentDescritpr bool, omitNSCloudProfile bool, | |
| omitComponentDescritpr, omitNSCloudProfile bool, |
|
|
||
| profiles, err := nspcpfl.BuildNSCloudProfiles(version, publications) | ||
| if err != nil { | ||
| return fmt.Errorf("error creating Namespaced Cloud Profiles %w", err) |
There was a problem hiding this comment.
| return fmt.Errorf("error creating Namespaced Cloud Profiles %w", err) | |
| return fmt.Errorf("cannot create namespaced cloud profiles: %w", err) |
| for _, profile := range profiles { | ||
| profileYAML, err := nspcpfl.ToYAML(profile) | ||
| if err != nil { | ||
| return fmt.Errorf("invalid profile configuration: %w", err) |
There was a problem hiding this comment.
| return fmt.Errorf("invalid profile configuration: %w", err) | |
| return fmt.Errorf("invalid cloud profile: %w", err) |
| NSProfileKey := fmt.Sprintf("meta/NSCloudProfile/%s/%s", version, baseName) | ||
| if err := manifestTarget.PutObject(ctx, NSProfileKey, bytes.NewReader(profileYAML)); err != nil { | ||
| return fmt.Errorf("cannot store NSCloudProfile %s: %w", profile.Name, err) | ||
| } |
There was a problem hiding this comment.
| NSProfileKey := fmt.Sprintf("meta/NSCloudProfile/%s/%s", version, baseName) | |
| if err := manifestTarget.PutObject(ctx, NSProfileKey, bytes.NewReader(profileYAML)); err != nil { | |
| return fmt.Errorf("cannot store NSCloudProfile %s: %w", profile.Name, err) | |
| } | |
| nsProfileKey := fmt.Sprintf("meta/NSCloudProfile/%s/%s", version, baseName) | |
| err = manifestTarget.PutObject(ctx, nsProfileKey, bytes.NewReader(profileYAML)) | |
| if err != nil { | |
| return fmt.Errorf("cannot store NSCloudProfile %s: %w", profile.Name, err) | |
| } |
| return fmt.Errorf("cannot store NSCloudProfile %s: %w", profile.Name, err) | ||
| } | ||
|
|
||
| shootYAML, err := nspcpfl.BuildShootSpecYAML(version, profile) |
There was a problem hiding this comment.
| shootYAML, err := nspcpfl.BuildShootSpecYAML(version, profile) | |
| var shootYAML []byte | |
| shootYAML, err = nspcpfl.BuildShootSpecYAML(version, profile) |
| return fmt.Errorf("invalid shoot spec for %s: %w", profile.Name, err) | ||
| } | ||
| shootKey := fmt.Sprintf("meta/ShootSpec/%s/%s", version, baseName) | ||
| if err := manifestTarget.PutObject(ctx, shootKey, bytes.NewReader(shootYAML)); err != nil { |
There was a problem hiding this comment.
| if err := manifestTarget.PutObject(ctx, shootKey, bytes.NewReader(shootYAML)); err != nil { | |
| err = manifestTarget.PutObject(ctx, shootKey, bytes.NewReader(shootYAML)) | |
| if err != nil { |
| if !omitNSCloudProfile { | ||
| baseName := fmt.Sprintf("gardenlinux-%s-%.8s-%s", nspcpfl.MajorVersion(version), commit, profile.Spec.Parent.Name) | ||
| NSProfileKey := fmt.Sprintf("meta/NSCloudProfile/%s/%s", version, baseName) | ||
| if err := manifestTarget.PutObject(ctx, NSProfileKey, bytes.NewReader(profileYAML)); err != nil { |
There was a problem hiding this comment.
Whether to upload to the manifestTarget is an important design decision. It would mean that cloud profiles go to the same account where the manifests go - dev for dev builds, prod for prod builds.
My understanding is that the profiles are essentially temporary objects that are only useful to run an integration test of a release, and do not need to be persisted. If that is the case, I would expect them to always go to the dev accounts, leaving the prod account for the actual manifests only.
If you believe they should go to the prod accounts instead, then manifestTarget is correct.
| } | ||
| } | ||
|
|
||
| profiles, err := nspcpfl.BuildNSCloudProfiles(version, publications) |
There was a problem hiding this comment.
The entire build+upload block for cloud profiles should be garded by the boolean flag.
688e11b to
588f828
Compare
What this PR does / why we need it:
Re-Implementation of the Garden Linux/Gardener integration test pipeline
Which issue(s) this PR fixes:
Fixes #
gardenlinux/gardenlinux#4070
Special notes for your reviewer:
Release note: