Automating resource propagation from a remote resource file with Karmada PropagationPolicy

[klerick]

Hello, I'm trying to use Karmada to propagate resources across clusters, but I am facing a challenge. Here's my setup:

• I am using Kustomize to include a remote resource file, for example:

resources:
  - https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

• This remote file contains a mix of resources:
  Namespaced resources like Deployment, Service, ServiceAccount.
  Cluster-scoped resources like ClusterRole and ClusterRoleBinding.

• My goal is to automatically propagate all resources from this remote file across multiple clusters using Karmada's PropagationPolicy, without manually listing every resource in the PropagationPolicy.

Issues I Encountered:

• The PropagationPolicy.spec.resourceSelectors requires apiVersion and kind, which means I have to manually enumerate each resource type (e.g., ClusterRole, ClusterRoleBinding, Deployment, etc.).
• It also requires the namespace for namespaced resources, which adds more manual work for large resource files with many resources in different namespaces.

Questions:

• Is there a way to dynamically select all resources from the remote resource file and propagate them without manually specifying every apiVersion, kind, and namespace?
• If not, are there any recommendations or best practices to automate this process with Kustomize and Karmada, especially for large resource files containing mixed resource types?
• Could PropagationPolicy support a wildcard option (e.g., apiVersion: *, kind: *) to simplify these use cases in the future?

[RainbowMango]

The PropagationPolicy was designed to propagate plain resources (aka. resource template), it can not resolve Kustomize resources which refers to one or more resources.

I think we can try to address this in two ways:

• 1. Make a controller on top of Karmada PropagationPolicy, it resolves Kustomize and then creates the PropagationPolicy accordingly.
• 2. Introduce another Policy in Karmada, that it dedicated to serving Kustomize, it may be built based on PropagationPlicy or not.

[klerick]

I’m confident that allowing resource selection only via labelSelector for ClusterPropagationPolicy would be a better approach. Cases where resources need to be deployed to all clusters, especially when they are managed through Kustomize or similar tools, seem to occur quite frequently. This solution would add flexibility and simplify management by making it easier to define and propagate the resulting resources without complex handling at the cluster level. What do you think about this approach?

[RainbowMango]

kustomiz allow change label out of the box for all resource in kuztomiz file

By leveraging the feature, you don't have to list all resources in ClusterPropagationPolicy, but you still need to list all resource kind.
You hope to select resource only via labelSelector, something like:

apiVersion: policy.karmada.io/v1alpha1
kind: ClusterPropagationPolicy
metadata:
  name: foo
spec:
  resourceSelectors:
    - labelSelector:
        matchLabels: 
          foo:bar

Am I right?

[klerick]

yes, you are right.
as exemple:
https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

[RainbowMango]

OK. I get it. I believe this is a useful feature, but change PropagationPolicy might be challenging, first we need to elaborate the real-world use case and then evaluate the solutions.

[klerick]

@RainbowMango sorry, could you explain about estimate?

[RainbowMango]

I mean, to achieve this goal, we may have multiple approaches and it is necessary to evaluate the advantages and disadvantages of each option.