|
| 1 | +# Configure credential Secrets for proxies and provisioners |
| 2 | + |
| 3 | +The operator sends Secret values to Coder and puts them into `CoderWorkspaceProxy` and `CoderProvisioner` pods. It does this only for Secrets that you mark as credentials for a destination. This page shows how to mark a Secret, which checks the operator runs, and how to upgrade from a version without these checks. |
| 4 | + |
| 5 | +!!! warning "Who may write these resources" |
| 6 | + Grant `create`, `update` and `patch` on `coderworkspaceproxies`, `coderprovisioners` and `codercontrolplanes` only to principals that may already read Secrets and create Pods in that namespace. A custom `spec.image` runs with everything the pod receives: the proxy token or the provisioner key, every `extraEnv` value, and for a `CoderProvisioner` the ServiceAccount token that manages workspace pods. The credential checks do not restrict the image. |
| 7 | + |
| 8 | +## Mark a Secret as a credential |
| 9 | + |
| 10 | +A Secret is usable only when it has both of these: |
| 11 | + |
| 12 | +- The label `coder.com/credential: "true"`. |
| 13 | +- The annotation `coder.com/credential-destinations` with the origins that may receive its values. An origin is `scheme://host`, plus `:port` for a port that is not the default. Separate more than one origin with commas. |
| 14 | + |
| 15 | +```yaml |
| 16 | +apiVersion: v1 |
| 17 | +kind: Secret |
| 18 | +metadata: |
| 19 | + name: coderworkspaceproxy-token |
| 20 | + namespace: coder |
| 21 | + labels: |
| 22 | + coder.com/credential: "true" |
| 23 | + annotations: |
| 24 | + coder.com/credential-destinations: "https://coder.example.com" |
| 25 | +type: Opaque |
| 26 | +stringData: |
| 27 | + token: "<workspace proxy token>" |
| 28 | +``` |
| 29 | +
|
| 30 | +Or mark an existing Secret: |
| 31 | +
|
| 32 | +```bash |
| 33 | +kubectl -n coder label secret coderworkspaceproxy-token coder.com/credential=true |
| 34 | +kubectl -n coder annotate secret coderworkspaceproxy-token coder.com/credential-destinations=https://coder.example.com |
| 35 | +``` |
| 36 | + |
| 37 | +The destination of each Secret input: |
| 38 | + |
| 39 | +| Kind | Secret input | Destination origin | |
| 40 | +| --- | --- | --- | |
| 41 | +| `CoderWorkspaceProxy` | `spec.proxySessionTokenSecretRef` | `spec.primaryAccessURL` | |
| 42 | +| `CoderWorkspaceProxy` | `spec.bootstrap.credentialsSecretRef` | `spec.bootstrap.coderURL`, and also the primary access URL (`spec.primaryAccessURL`, or `spec.bootstrap.coderURL` when it is empty), because the generated proxy token comes from this credential | |
| 43 | +| `CoderWorkspaceProxy` | every `spec.extraEnv[].valueFrom.secretKeyRef` | the primary access URL | |
| 44 | +| `CoderProvisioner` | every `spec.extraEnv[].valueFrom.secretKeyRef` | `status.url` of the referenced `CoderControlPlane` | |
| 45 | + |
| 46 | +Get the control plane URL for a provisioner: |
| 47 | + |
| 48 | +```bash |
| 49 | +kubectl -n coder get codercontrolplane <name> -o jsonpath='{.status.url}{"\n"}' |
| 50 | +``` |
| 51 | + |
| 52 | +## Checks the operator runs |
| 53 | + |
| 54 | +The operator also refuses these Secrets, even with the label and the annotation: |
| 55 | + |
| 56 | +- Secrets that the operator generates: Secrets owned by a `coder.com` resource, Secrets with the label `app.kubernetes.io/managed-by=coder-k8s`, and the operator token Secret of any `CoderControlPlane` in the namespace (for example `<control plane>-operator-token`). |
| 57 | +- Secrets with a type other than `Opaque`. |
| 58 | +- Values that are empty, larger than 8 KiB, or that start or end with whitespace (a trailing newline counts). |
| 59 | + |
| 60 | +It also checks these settings: |
| 61 | + |
| 62 | +- `spec.extraEnv` and `spec.extraArgs` must not override the URL or the token that the operator sets: |
| 63 | + |
| 64 | + | Kind | Reserved `extraEnv` names | Reserved `extraArgs` flags | |
| 65 | + | --- | --- | --- | |
| 66 | + | `CoderWorkspaceProxy` | `CODER_PRIMARY_ACCESS_URL`, `CODER_PROXY_SESSION_TOKEN` | `--primary-access-url`, `--proxy-session-token` | |
| 67 | + | `CoderProvisioner` | `CODER_URL`, `CODER_PROVISIONER_DAEMON_KEY` | `--url`, `--key` | |
| 68 | + |
| 69 | + Env names match without case, because the Coder CLI reads them without case. A flag matches with one or two dashes, and with or without `=value`. The operator checks each arg after the `$(VAR)` expansion that Kubernetes applies to container args. It rejects an arg where a variable whose value it cannot see (an operator-set variable, or a `valueFrom` variable) can change the flag name: a reference at the start of the arg, or before the `=` of a flag. A reference in a flag value, such as `--git-token=$(GIT_TOKEN)`, passes. Escape such a reference as `$$(VAR)` when you want the literal text. |
| 70 | + |
| 71 | +- With `spec.bootstrap`, the bootstrap credentials Secret must exist on every reconcile, also after the first registration. The operator checks that its destinations include the primary access URL, because the generated proxy token comes from it. |
| 72 | +- With `spec.bootstrap`, the generated token Secret (`spec.bootstrap.generatedTokenSecretName`, default `<proxy>-proxy-token`) must not exist yet, or must have this proxy as its controller owner. The operator does not read, mount, change or adopt any other Secret with that name. |
| 73 | +- The provisioner key Secret of a `CoderProvisioner` (`spec.key.secretName`, default `<provisioner>-provisioner-key`) must not exist yet, or must have this provisioner as its controller owner. When it does not exist, the operator creates it. The operator does not read, mount, change or adopt any other Secret with that name. Earlier versions adopted an unowned Secret with that name on the first reconcile. Such an adopted Secret already has this provisioner as its controller owner, so the operator keeps using it, and Kubernetes deletes it with the provisioner. |
| 74 | + |
| 75 | +Proxy pods run with `automountServiceAccountToken: false`. The proxy talks only to coderd and does not need a Kubernetes API token. Provisioner pods keep their token. |
| 76 | + |
| 77 | +## Read the result |
| 78 | + |
| 79 | +Each `CoderWorkspaceProxy` and `CoderProvisioner` has the condition `CredentialsReady`. When a check fails, the condition is `False`, and the operator sends nothing to Coder and does not change the Deployment. A Deployment that already runs keeps its last applied configuration. The operator applies spec changes again after you fix the cause. |
| 80 | + |
| 81 | +```bash |
| 82 | +kubectl get coderworkspaceproxies,coderprovisioners -A -o jsonpath='{range .items[*]}{.kind} {.metadata.namespace}/{.metadata.name}: {.status.conditions[?(@.type=="CredentialsReady")].status} {.status.conditions[?(@.type=="CredentialsReady")].reason}{"\n"}{end}' |
| 83 | +``` |
| 84 | + |
| 85 | +| Reason | Cause | Fix | |
| 86 | +| --- | --- | --- | |
| 87 | +| `CredentialSecretNotFound` | The referenced Secret does not exist. | Create the Secret. | |
| 88 | +| `CredentialKeyMissing` | The Secret does not have the referenced key. | Add the key, or fix the reference. | |
| 89 | +| `CredentialNotOptedIn` | The label or the annotation is missing. | Add both. | |
| 90 | +| `CredentialDestinationNotAllowed` | The annotation does not list the destination origin. | Add the origin that the message names. | |
| 91 | +| `CredentialRefused` | The operator generated the Secret, or the type is not `Opaque`. | Store the credential in its own `Opaque` Secret. For a proxy, create a dedicated Coder token instead of the operator token. | |
| 92 | +| `CredentialMalformed` | The value or the annotation is not valid, or a URL or a required reference in the spec is missing or not valid. | Fix what the message names. | |
| 93 | +| `TokenSecretNotOwned` | The generated token Secret exists and another owner, or no owner, controls it. | Remove `generatedTokenSecretName` to use the default name, or set a name that does not exist yet. | |
| 94 | +| `KeySecretNotOwned` | The provisioner key Secret exists and another owner, or no owner, controls it. | Delete that Secret, or set `spec.key.secretName` to a name that does not exist yet. The operator then creates a new provisioner key in Coder. | |
| 95 | +| `ReservedSettingOverride` | `extraEnv` or `extraArgs` sets a reserved name or flag. | Remove the entry. | |
| 96 | + |
| 97 | +Messages name Secrets, keys, fields and origins. They never contain Secret values. |
| 98 | + |
| 99 | +A Deployment that already runs keeps its last applied configuration after a failed check. This includes any reference it already has to a Secret that the operator now refuses. The references go away at the first rollout after you fix the cause. |
| 100 | + |
| 101 | +## Upgrade from a version without these checks |
| 102 | + |
| 103 | +Do these steps before you upgrade the operator. Resources that fail a check after the upgrade keep running with their last applied configuration, so an upgrade causes no outage. But the operator does not apply their spec changes until you fix them. |
| 104 | + |
| 105 | +1. List the referenced Secrets. Lines that start with `consent` need the label and the annotation. Lines that start with `owned` name a generated proxy token Secret or a provisioner key Secret, which must not exist or must be owned by that resource: |
| 106 | + |
| 107 | + ```bash |
| 108 | + kubectl get coderworkspaceproxies,coderprovisioners -A -o json | jq -r '.items[] | |
| 109 | + .kind as $k | .metadata.namespace as $ns | .metadata.name as $cr | |
| 110 | + (([.spec.bootstrap.credentialsSecretRef.name?, .spec.proxySessionTokenSecretRef.name?] |
| 111 | + + [.spec.extraEnv[]?.valueFrom.secretKeyRef.name?]) | map(select(. != null)) | unique[] | |
| 112 | + "consent \($ns)/\(.) (\($k) \($cr))"), |
| 113 | + (if $k == "CoderWorkspaceProxy" and .spec.bootstrap then |
| 114 | + "owned \($ns)/\(.spec.bootstrap.generatedTokenSecretName // ($cr + "-proxy-token")) (\($k) \($cr) uid=\(.metadata.uid))" |
| 115 | + elif $k == "CoderProvisioner" then |
| 116 | + "owned \($ns)/\(.spec.key.secretName // ($cr + "-provisioner-key")) (\($k) \($cr) uid=\(.metadata.uid))" |
| 117 | + else empty end)' |
| 118 | + ``` |
| 119 | + |
| 120 | +2. Check the owner of each `owned` Secret. Lines that start with `ACTION` name a Secret that the operator refuses after the upgrade. The script reads only owner references, not Secret data: |
| 121 | + |
| 122 | + ```bash |
| 123 | + kubectl get coderworkspaceproxies,coderprovisioners -A -o json | jq -r '.items[] | |
| 124 | + (if .kind == "CoderProvisioner" then .spec.key.secretName // (.metadata.name + "-provisioner-key") |
| 125 | + elif .spec.bootstrap then .spec.bootstrap.generatedTokenSecretName // (.metadata.name + "-proxy-token") |
| 126 | + else empty end) as $s | |
| 127 | + "\(.metadata.namespace) \($s) \(.metadata.uid) \(.kind)/\(.metadata.name)"' | |
| 128 | + while read -r ns name uid cr; do |
| 129 | + out=$(kubectl -n "$ns" get secret "$name" --ignore-not-found \ |
| 130 | + -o jsonpath='found {.metadata.ownerReferences[?(@.controller==true)].uid}') || |
| 131 | + { echo "ERROR $ns/$name ($cr): kubectl failed"; continue; } |
| 132 | + case $out in |
| 133 | + "") echo "ok $ns/$name ($cr): does not exist yet" ;; |
| 134 | + "found $uid") echo "ok $ns/$name ($cr): owned" ;; |
| 135 | + *) echo "ACTION $ns/$name ($cr): not owned by $cr" ;; |
| 136 | + esac |
| 137 | + done |
| 138 | + ``` |
| 139 | + |
| 140 | + For each `ACTION` line: |
| 141 | + |
| 142 | + - For a proxy, remove `generatedTokenSecretName`, or set it to a name that does not exist yet. |
| 143 | + - For a provisioner, delete the Secret, or set `spec.key.secretName` to a name that does not exist yet. In both cases the operator creates a new provisioner key in Coder, stores it in a new Secret that it owns, and rolls out the provisioner pods once. The old key stops working when the operator replaces it. The operator does not delete a Secret that it does not own. |
| 144 | + |
| 145 | + Also check each `owned` Secret for data keys that the operator does not manage. An owned proxy token Secret must hold only the `token` key, and an owned provisioner key Secret only its `spec.key.secretKey` (default `key`). Another key means that an earlier version adopted a Secret that you created. Kubernetes deletes that Secret, with all its keys, when you delete the resource. Move any other key to its own Secret. |
| 146 | + |
| 147 | +3. List proxies and provisioners with reserved overrides, and remove those entries: |
| 148 | + |
| 149 | + ```bash |
| 150 | + kubectl get coderworkspaceproxies,coderprovisioners -A -o json | jq -r '.items[] | |
| 151 | + (if .kind == "CoderWorkspaceProxy" then ["CODER_PRIMARY_ACCESS_URL", "CODER_PROXY_SESSION_TOKEN"] |
| 152 | + else ["CODER_URL", "CODER_PROVISIONER_DAEMON_KEY"] end) as $env | |
| 153 | + (if .kind == "CoderWorkspaceProxy" then "^--?(primary-access-url|proxy-session-token)(=|$)" |
| 154 | + else "^--?(url|key)(=|$)" end) as $arg | |
| 155 | + select(([.spec.extraEnv[]?.name | ascii_upcase] | any(IN($env[]))) or ([.spec.extraArgs[]?] | any(test($arg)))) | |
| 156 | + "\(.kind) \(.metadata.namespace)/\(.metadata.name)"' |
| 157 | + ``` |
| 158 | + |
| 159 | + The operator also rejects an `extraArgs` entry that starts with `$(VAR)`, or has `$(VAR)` before the `=` of a flag, when the variable is set from `valueFrom` or by the operator. Search for `$(` in `extraArgs` and check each match. |
| 160 | + |
| 161 | +4. For each `consent` Secret, add the label and the destinations annotation. If a bootstrap credentials Secret no longer exists, create it again (any valid token for that Coder deployment), labeled and annotated. Use the destination table in [Mark a Secret as a credential](#mark-a-secret-as-a-credential): |
| 162 | + |
| 163 | + ```bash |
| 164 | + kubectl -n <ns> label secret <name> coder.com/credential=true |
| 165 | + kubectl -n <ns> annotate secret <name> coder.com/credential-destinations=https://coder.example.com |
| 166 | + ``` |
| 167 | + |
| 168 | +5. If a proxy uses the operator token as `spec.bootstrap.credentialsSecretRef`, create a dedicated Coder API token for it. Store the token in its own labeled and annotated Secret, and point `credentialsSecretRef` at that Secret. No label makes the operator token usable. |
| 169 | + |
| 170 | +6. If a custom proxy image reads the Kubernetes API, change it. Proxy pods no longer get a service account token. Each proxy that passes the checks rolls out once after the upgrade because of this change. |
| 171 | + |
| 172 | +7. Upgrade the operator. |
| 173 | + |
| 174 | +8. Check every proxy and provisioner. Every line must end in `True`: |
| 175 | + |
| 176 | + ```bash |
| 177 | + kubectl get coderworkspaceproxies,coderprovisioners -A -o jsonpath='{range .items[*]}{.kind} {.metadata.namespace}/{.metadata.name}: {.status.conditions[?(@.type=="CredentialsReady")].status}{"\n"}{end}' |
| 178 | + ``` |
0 commit comments