(improve) support for kubernetes secrets in config

[cboettig]

Bug description

The helm chart linter objects to the use of a kubernetes secret instead of a literal string as the value for an ImagePullSecret.password, like so:

imagePullSecret:
      create: true
      registry: ghcr.io
      username: cboettig
      password:
        valueFrom:
          secretKeyRef:
            name: jupyter-secrets
            key: GH_REGISTRY_PULL

Error: values don't meet the specifications of the schema(s) in the following chart(s):
jupyterhub:

• at '/imagePullSecret/password': got object, want string

Other places where one would also want to use a kubernetes secret, such as hub.config.GitHubOAuthenticator fields do not trigger the linter, but for some reason still ultimately fail -- we just get the same error message about unexpected object deeper in the k8s logs when the hub doesn't start.

I think it would be reasonable to support kubernetes secrets for any of the secrets commonly used in the config.

See zulip discussion

[consideRatio]

To reference a k8s Secret in a Pod can be done by mounting something from it to an env var or to a file, but this is not something we can do here. The difference is that we want to use the information provided under imagePullSecret (without an s in the end) to render a k8s Secret resource from a Helm chart template, and a k8s Secret can't itself reference another k8s Secret like a Pod can do.

I think in this case, you may want to configure z2jh to reference an existing imagePullSecret instead, by first creating a k8s Secret yourself, and then referencing it via z2jh config imagePullSecrets (with a s in the end).

[consideRatio]

Other places where one would also want to use a kubernetes secret, such as hub.config.GitHubOAuthenticator fields do not trigger the linter, but for some reason still ultimately fail -- we just get the same error message about unexpected object deeper in the k8s logs when the hub doesn't start.

This is because hub.config.SometThing is config that is directly passed through to be read by JupyterHub the software. JupyterHub the Helm chart doesn't know about what is valid config.

If you want to configure something sensitive for GitHubOAuthenticator, like a client secret, you could configure hub.extraEnv with an environment variable that references a k8s Secret (example provided in linked docs). The environment variable you can configure for this is GITHUB_CLIENT_SECRET (based on looking in the source code of oauthenticator 🙂).

[consideRatio]

Another strategy to consider is to accept that helm will work with sensitive config values - you can then use sops to encrypt sensitive helm chart config, for example using sops (https://github.com/getsops/sops) with age, pgp, or some KMS system by a cloud provider. Then together with the helm secrets plugin (https://github.com/jkroepke/helm-secrets), you can decrypt things just-in-time so that you don't have to have decrypted files on disk.

[cboettig]

Thanks @consideRatio for all this, really appreciate your help. Still trying to work through the picture though. (For reference, here's my values.yaml, configured with k8s secrets in both the GitHub authenticator section and the image pull section. )

I think in this case, you may want to configure z2jh to reference an existing imagePullSecret instead, but first creating a k8s Secret yourself, and then referencing it via z2jh config imagePullSecrets (with a s in the end).

This sounds promising if slightly convoluted -- is there an example config you can point me too that has both the imagePullSecrets and the imagePullSecret with the one referencing the other?

I think I followed your point that z2jh is creating a secret for this on my behalf already, and it can't make a secret from a secret, but this still feels quite awkward. In any other helm chart I've seen, when we encounter a field in values.yaml that is sensitive, it's usually both recommended by the docs and straight forward to drop in a k8s secret in place of a literal string. I understand z2jh is has lot of complexity and different moving parts, but from a design aspect this feels odd.

Thanks for explaining why the situation with hub.config is quite different from the situation with imagePullSecrets. I gather here we're no longer really writing kubernetes yaml, we're just writing some verbatim configuration that is going to be pulled out and passed to jupyter software that doesn't know/assume anything about k8s, yes?

I didn't know about sops and helm-secrets, thanks much for those pointers! This looks a lot more complex to me than k8s secrets (perhaps for being unfamiliar), and I don't think I can articulate a good reason why one would use helm-secrets instead of kubernetes secrets in general, or why the latter requires sops, but I will try and learn. I guess this is what the 2i2c configs must be using? (e.g. https://github.com/2i2c-org/infrastructure/blob/a0bc7aa974c06f8643343290d0a94048bd66ae32/config/clusters/temple/enc-staging.secret.values.yaml#L5)

[minrk]

Sorry, I think I gave you the wrong pointer! Thanks @consideRatio for clearing things up.

is there an example config you can point me too that has both the imagePullSecrets and the imagePullSecret with the one referencing the other?

I think the idea is that if you already have a secret containing credentials, you use only imagePullSecrets, which is a pass-through for the imagePullSecrets field to reference existing secrets. imagePullSecret is only needed when you don't have a secret already, and want the chart to to create one for you from helm config. Since you already have a secret, there's no need to create a new one with imagePullSecret and you can just use imagePullSecrets to reference existing secrets.

imagePullSecret creates a Secret and appends it to imagePullSecrets.