Skip to content

Commit 06ce2b9

Browse files
authored
🤖 feat: add the CoderMCPServer kill switches and the user_oidc correction (#329)
## Summary After a full apply, the dormant `CoderMCPServer` reconciler now treats a dashboard disable, credential clear or auth type change as a kill switch: it reports `Stalled=True` and writes nothing until the spec changes. An observed `user_oidc` auth type is always set back, with a PATCH that holds only auth fields. An in-memory hash skips the credential resend after a metadata-only Secret change. The kind is not registered, so users see no change. **Base branch:** `main`. #328 landed as `45f27ee560`. I merged `main` into this branch with a merge commit, and the diff against `main` is the reviewed M6b change only. ## Background #328 reads the config on every reconcile and sets back every difference. That also undoes an emergency disable or credential clear in the dashboard. Plan 4.7 item 2 (D9) says an admin's disable, delete or credential clear wins until someone edits the spec. #328 already holds the delete. This PR adds the other three. ## Kill switches (plan 4.7) Only when `status.appliedGeneration == metadata.generation`: | Coder shows | Reason | Writes | |---|---|---| | `enabled: false` while the spec says enabled | `DisabledExternally` | none | | a `has_*` flag false for a credential of the spec's auth type, and the auth type is not `user_oidc` | `CredentialClearedExternally` | none | A dashboard change to another auth type clears the old credentials (plan F9), so it lands in the second row. Each stall resyncs every 10 minutes, because no Kubernetes event follows a dashboard edit. A spec edit makes the generation unapplied, so the next sync sends the full state again: enable, credentials, auth type. An apply that fails keeps the old `appliedGeneration`. ## `user_oidc` (plan 4.7 item 4, S19) `user_oidc` sends each chat user's login token to `url` (plan F14), and the CRD never allows it. An observed `user_oidc` is always drift, never a kill switch, also when the switch cleared a credential. When a disable kill switch also holds, the controller sends only the auth fields of the sparse PATCH (auth type, header, credentials, OAuth2 fields) plus `url`, so the config stays disabled, and the stall message says that the auth type was set back. `url` goes with them because the auth fields can carry credentials, and Coder must never store a credential next to a URL that the Secret does not allow (review round 3). ## Metadata-only Secret changes (plan 4.6 item 4) A label or annotation change moves the Secret's resourceVersion, so the change token differs. The reconciler keeps a SHA-256 of each value that Coder holds, in process memory only, keyed by object UID and credential field. When the token differs but the hash matches, it records the new token and sends nothing. The create and every PATCH that sends a credential update the map. After a restart the map is empty, so one resend can happen. Every sync first drops the hashes of fields that the spec's auth type does not use, because a change of the auth type cleared them in Coder, also when its answer was lost. Before a PATCH that sends a credential, the reconciler also drops that field's hash: Coder can commit the PATCH while the answer is lost, so a retry must send the value again (review round 3). A PATCH also drops the hashes of fields that the spec's auth type does not use, because Coder cleared those credentials. Otherwise an auth type round trip could skip a needed resend (review round 1). ## Review record Codex round 1 (normal review of `0ea11aa`): three findings. - Fixed in `c2de88cf`: stale hashes after a PATCH that clears a credential. - Fixed in #330 (`4e84a445`): `user_oidc` while the credential read fails. A read failure is the #330 layer. - Not fixed, by design: the removal of one custom header of several. Coder shows one `has_custom_headers` flag and no header names, so this is plan risk R2, like a dashboard replacement of a value. The security review of `0ea11aa` found no issues. Round 2 (`c2de88cf`): normal review "Didn't find any major issues", security review no issues. Round 3 (refresh on `main` as `01e015a1`): the security review found no issues. The normal review found one item, fixed in `bbd8cbb2`: hashes were updated only after a good answer, so a lost answer followed by a Secret revert skipped a needed resend. I checked the other parts of the sparse PATCH for the same pattern: every other field is compared with Coder's GET on each reconcile, the change tokens in status only move forward (a Secret's resourceVersion grows and a recreated Secret has a new UID), credentials of other auth types are cleared from Coder's `has_*` flags, and the create records hashes only after a matching 201. The review of `bbd8cbb2` (round 3, normal and security) found two more items. I fixed both as blockers in `632a7686`, and I requested no further review: - (security) the `user_oidc` correction while a kill switch holds sent the credentials without `url`. A dashboard edit of `url`, `user_oidc` and a disable then made the controller store the Secret's credential next to a URL the Secret does not allow. - a change of the auth type whose answer was lost kept the old field's hash, because the retry found nothing to send. The round-3 check missed the cleared fields. ## Validation New tests in `internal/controller/codermcpserver_sync_test.go`: - `TestCoderMCPServerSyncKillSwitches` (S14): a dashboard disable, credential clear and auth type change to `none` each give `Stalled=True` with their reason and a 10-minute requeue on two reconciles, with zero writes. A spec edit whose PATCH fails gives `CoderUnavailable` and keeps the old `appliedGeneration`. The next reconcile applies: enabled, `api_key`, the key, the new description. - `TestCoderMCPServerSyncUserOIDC` (S19): for `none`, `api_key`, `custom_headers` and `oauth2` configs, for an `api_key` config that is also disabled, and for one that is also disabled with its `url` moved to another origin, a dashboard switch to `user_oidc` gives exactly one PATCH. Every key of its body is an auth field or `url`, its `auth_type` is the spec's, and a moved `url` is set back in the same PATCH. Coder then holds the applied state again (the disabled config stays disabled with `DisabledExternally`). The next reconcile sends nothing. - `TestCoderMCPServerSyncSecretMetadataChange` (S13): an annotation change on the Secret gives no write and the new token in status. After a restart (a new reconciler), the next annotation change sends `{"api_key_value":...}` once. - `TestCoderMCPServerSyncAuthRoundTripResends`: `api_key` to `none`, then a dashboard key, then the spec back to `api_key` sends the Secret's key again. A second case loses the answer of the PATCH to `none`. - `TestCoderMCPServerSyncLostAnswerResends`: the Secret goes from A to B, the PATCH with B commits in Coder but answers 502, and the Secret goes back to A. The next reconcile sends exactly `{"api_key_value":A}`, and Coder holds A. - Mutation checks (run on the stack head): each of these 10 changes makes a test fail: - no `DisabledExternally` kill switch - no `CredentialClearedExternally` kill switch - `user_oidc` held by a kill switch instead of set back - the full sparse PATCH instead of the auth fields while a kill switch holds - no hash check (a metadata-only change resends) - no hash recorded at create - no hash dropped before the PATCH (two variants) - no hash pruning on every sync - no `url` in the auth-only PATCH One change survives: narrowing the ACL also while a kill switch holds. It only removes access, and no test combines an ACL change with a `user_oidc` switch on a disabled config. - Local gates on this head: `make verify-vendor`, `make build`, `make lint`, `GOFLAGS=-mod=vendor go tool govulncheck ./...`, `make test`, `make test-integration`, `go test -race ./internal/controller/`, and `make docs-check` pass. `make codegen`, `make manifests`, `make build-installer` and `make docs-reference` leave no diff. ## Size 402 lines added and 19 removed, all hand-written: 164 changed lines in `codermcpserver_apply.go`, 12 in the reconciler and the create step, and 245 in the new tests. Refs #155 --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high` • Cost: `$127.81`_ <!-- mux-attribution: model=anthropic:claude-opus-5-5 thinking=high costs=127.81 --> --------- Signed-off-by: Thomas Kosiewski <tk@coder.com>
1 parent 1f336cf commit 06ce2b9

4 files changed

Lines changed: 402 additions & 19 deletions

File tree

‎internal/controller/codermcpserver_apply.go‎

Lines changed: 146 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,9 @@ package controller
33
import (
44
"cmp"
55
"context"
6+
"crypto/sha256"
67
"fmt"
8+
"maps"
79
"net/http"
810
"slices"
911
"strings"
@@ -27,12 +29,14 @@ import (
2729
// it, so the ACL never holds a group that neither generation allows (plan
2830
// S20).
2931
//
30-
// A dashboard delete after a full apply of this generation is a kill switch:
31-
// the controller reports it and writes nothing until the spec changes (plan
32-
// 4.7 item 2). The other kill switches come in the next change. recreate is
33-
// true when the config is gone before a full apply of this generation: the
34-
// caller then creates it again, if the slug is free.
35-
func syncMCPServer(
32+
// After a full apply of this generation, a dashboard delete, disable or
33+
// credential clear is a kill switch: the controller reports it and writes
34+
// nothing until the spec changes (plan 4.7 item 2). An auth type user_oidc
35+
// is always set back, with only the auth fields and url, so a disable stays.
36+
//
37+
// recreate is true when the config is gone before a full apply of this
38+
// generation: the caller then creates it again, if the slug is free.
39+
func (r *CoderMCPServerReconciler) syncMCPServer(
3640
ctx context.Context, sdk *codersdk.Client, server *coderv1alpha1.CoderMCPServer, orgID uuid.UUID, values map[string]credentialsecret.Value,
3741
) (step *mcpServerStep, recreate bool, err error) {
3842
id, err := uuid.Parse(server.Status.ConfigID)
@@ -47,6 +51,7 @@ func syncMCPServer(
4751
"MCP server config %s was deleted in Coder. The controller does not create it again until the spec changes.", id), false, nil
4852
case isCoderNotFound(err):
4953
server.Status.ConfigID, server.Status.Credentials = "", nil
54+
r.sentCredentials.Delete(server.UID)
5055
return nil, true, nil
5156
case err != nil:
5257
return mcpServerWait(mcpServerRetry, "CoderUnavailable", "get MCP server config %s: %s", id, coderErrorSummary(err)), false, nil
@@ -56,14 +61,46 @@ func syncMCPServer(
5661
"assertion failed: Coder answered MCP server config %s with slug %s in organization %s when asked for %s", found.ID, found.Slug, found.OrganizationID, id), false, nil
5762
}
5863

64+
// user_oidc sends each chat user's login token to url (plan F14), so it is
65+
// set back even when a kill switch holds every other write.
66+
oidc := found.AuthType == "user_oidc"
67+
enabled := server.Spec.Enabled == nil || *server.Spec.Enabled
68+
var kill *mcpServerStep
69+
switch active := mcpServerActiveCredentials(server.Spec.Auth); {
70+
case applied && enabled && !found.Enabled:
71+
kill = mcpServerStalled(mcpServerResync, "DisabledExternally",
72+
"MCP server config %s was disabled in Coder. The controller does not enable it again until the spec changes.", id)
73+
case applied && !oidc && (active.apiKey && !found.HasAPIKey || active.headers && !found.HasCustomHeaders || active.secret && !found.HasOAuth2Secret):
74+
// A dashboard change of the auth type lands here too: Coder clears
75+
// the credentials of the old type (plan F9).
76+
kill = mcpServerStalled(mcpServerResync, "CredentialClearedExternally",
77+
"A credential of MCP server config %s was cleared in Coder (auth type %s). The controller does not send it again until the spec changes.", id, found.AuthType)
78+
}
79+
if kill != nil && !oidc {
80+
return kill, false, nil
81+
}
82+
5983
full, err := mcpServerUpdateRequest(server, values)
6084
if err != nil {
6185
return nil, false, err
6286
}
63-
if req := mcpServerSparseRequest(server, full, found, values); req != (codersdk.UpdateMCPServerConfigRequest{}) {
64-
if step, err := narrowMCPServerACL(ctx, sdk, orgID, id, server.Spec.Access); step != nil || err != nil {
65-
return step, false, err
87+
// A hash is only for a field of the spec's auth type: a change of the
88+
// auth type cleared the others in Coder, also when its answer was lost.
89+
r.pruneSentCredentials(server)
90+
req, sent := r.mcpServerSparseRequest(server, full, found, values)
91+
if kill != nil {
92+
req = mcpServerAuthOnly(req)
93+
}
94+
if req != (codersdk.UpdateMCPServerConfigRequest{}) {
95+
if kill == nil {
96+
if step, err := narrowMCPServerACL(ctx, sdk, orgID, id, server.Spec.Access); step != nil || err != nil {
97+
return step, false, err
98+
}
6699
}
100+
// Coder can commit the PATCH and the answer still get lost. Until a
101+
// good answer, the controller does not know which value Coder holds,
102+
// so a retry must send these credentials again.
103+
r.forgetSentCredentials(server, sent)
67104
updated, err := sdk.UpdateMCPServerConfig(ctx, orgID, id, req)
68105
switch status := coderStatus(err); {
69106
case status == http.StatusBadRequest:
@@ -74,12 +111,21 @@ func syncMCPServer(
74111
// so the GET of the next reconcile decides whether the config is gone.
75112
return mcpServerWait(mcpServerRetry, "CoderUnavailable", "update MCP server config %s: %s", id, coderErrorSummary(err)), false, nil
76113
}
77-
if field := mcpServerMismatch(server.Spec.Auth, full, updated); field != "" || updated.ID != id {
114+
check := full
115+
if kill != nil {
116+
check = mcpServerAuthOnly(full)
117+
}
118+
if field := mcpServerMismatch(server.Spec.Auth, check, updated); field != "" || updated.ID != id {
78119
return mcpServerStalled(mcpServerResync, "CoderAnswerMismatch", "assertion failed: Coder answered the update of MCP server config %s with another %s", id, cmp.Or(field, "ID")), false, nil
79120
}
121+
r.rememberSentCredentials(server, sent, values)
80122
} else if field := mcpServerMismatch(server.Spec.Auth, full, found); field != "" {
81123
return nil, false, fmt.Errorf("assertion failed: MCP server %s/%s differs in %s, but the sync request is empty", server.Namespace, server.Name, field)
82124
}
125+
if kill != nil {
126+
kill.message = "The controller set the auth type back from user_oidc. " + kill.message
127+
return kill, false, nil
128+
}
83129
if step, err := syncMCPServerACL(ctx, sdk, orgID, id, server.Spec.Access); step != nil || err != nil {
84130
return step, false, err
85131
}
@@ -97,11 +143,12 @@ func mcpServerActiveCredentials(auth coderv1alpha1.CoderMCPServerAuth) mcpServer
97143

98144
// mcpServerSparseRequest drops every field of full that found already holds.
99145
// It keeps a credential when a URL or the auth type changes, when its change
100-
// token changed, or when Coder lost it. It clears credentials of the other
101-
// auth types that Coder holds (#314).
102-
func mcpServerSparseRequest(
146+
// token changed and Coder does not hold the value already, or when Coder lost
147+
// it. It clears credentials of the other auth types that Coder holds (#314).
148+
// It returns the credential fields that the request sends.
149+
func (r *CoderMCPServerReconciler) mcpServerSparseRequest(
103150
server *coderv1alpha1.CoderMCPServer, full codersdk.UpdateMCPServerConfigRequest, found codersdk.MCPServerConfig, values map[string]credentialsecret.Value,
104-
) codersdk.UpdateMCPServerConfigRequest {
151+
) (codersdk.UpdateMCPServerConfigRequest, []string) {
105152
req := full
106153
authChanged := found.AuthType != *full.AuthType
107154
for _, f := range mcpServerStringFields(&req, found) {
@@ -125,7 +172,11 @@ func mcpServerSparseRequest(
125172
}
126173
token := values[field].Token
127174
c, ok := recorded[field]
128-
return !ok || c.SecretName != token.SecretName || c.SecretKey != token.SecretKey || c.SecretUID != string(token.SecretUID) || c.SecretResourceVersion != token.ResourceVersion
175+
if ok && c.SecretName == token.SecretName && c.SecretKey == token.SecretKey && c.SecretUID == string(token.SecretUID) && c.SecretResourceVersion == token.ResourceVersion {
176+
return false
177+
}
178+
// Only the Secret's metadata changed when Coder holds this value.
179+
return !r.sentCredentialMatches(server, field, values[field])
129180
}
130181
var sent []string
131182
for _, c := range mcpServerCredentialRefs(server.Spec.Auth) {
@@ -138,8 +189,13 @@ func mcpServerSparseRequest(
138189
sends := func(prefix string) bool {
139190
return slices.ContainsFunc(sent, func(f string) bool { return strings.HasPrefix(f, prefix) })
140191
}
141-
// Custom headers are one Coder field, so they all go together.
142-
if !active.headers || !sends("header:") {
192+
if active.headers && sends("header:") {
193+
// Custom headers are one Coder field, so they all go together.
194+
sent = slices.DeleteFunc(sent, func(f string) bool { return strings.HasPrefix(f, "header:") })
195+
for _, h := range server.Spec.Auth.CustomHeaders {
196+
sent = append(sent, "header:"+h.Name)
197+
}
198+
} else {
143199
req.CustomHeaders = nil
144200
}
145201
if !active.apiKey || !sends("apiKey") {
@@ -159,7 +215,19 @@ func mcpServerSparseRequest(
159215
if !active.secret && found.HasOAuth2Secret {
160216
req.OAuth2ClientSecret = &empty
161217
}
162-
return req
218+
return req, sent
219+
}
220+
221+
// mcpServerAuthOnly keeps only the auth fields of req and url. The auth
222+
// fields can carry credentials, so url goes with them: Coder must never
223+
// store a credential next to a url that the Secret does not allow (plan 4.6
224+
// item 5).
225+
func mcpServerAuthOnly(req codersdk.UpdateMCPServerConfigRequest) codersdk.UpdateMCPServerConfigRequest {
226+
return codersdk.UpdateMCPServerConfigRequest{
227+
URL: req.URL, AuthType: req.AuthType, APIKeyHeader: req.APIKeyHeader, APIKeyValue: req.APIKeyValue, CustomHeaders: req.CustomHeaders,
228+
OAuth2ClientID: req.OAuth2ClientID, OAuth2ClientSecret: req.OAuth2ClientSecret, OAuth2AuthURL: req.OAuth2AuthURL,
229+
OAuth2TokenURL: req.OAuth2TokenURL, OAuth2RevocationURL: req.OAuth2RevocationURL, OAuth2Scopes: req.OAuth2Scopes,
230+
}
163231
}
164232

165233
type mcpServerStringField struct {
@@ -244,6 +312,66 @@ func mcpServerMismatch(auth coderv1alpha1.CoderMCPServerAuth, want codersdk.Upda
244312
return ""
245313
}
246314

315+
// rememberSentCredentials keeps a SHA-256 of each credential value that Coder
316+
// now holds, in process memory only (plan 4.6 item 4), so a metadata-only
317+
// Secret change sends nothing. After a restart the map is empty, so one
318+
// resend can happen. It drops the hashes of fields that the spec's auth type
319+
// does not use: the PATCH cleared those credentials, so a later round trip
320+
// of the auth type must send them again.
321+
func (r *CoderMCPServerReconciler) rememberSentCredentials(server *coderv1alpha1.CoderMCPServer, sent []string, values map[string]credentialsecret.Value) {
322+
sums := map[string][sha256.Size]byte{}
323+
if old, ok := r.sentCredentials.Load(server.UID); ok {
324+
for _, c := range mcpServerCredentialRefs(server.Spec.Auth) {
325+
if sum, ok := old.(map[string][sha256.Size]byte)[c.field]; ok {
326+
sums[c.field] = sum
327+
}
328+
}
329+
}
330+
for _, field := range sent {
331+
sums[field] = sha256.Sum256([]byte(values[field].Reveal()))
332+
}
333+
r.sentCredentials.Store(server.UID, sums)
334+
}
335+
336+
// pruneSentCredentials drops the hashes of fields that the spec's auth type
337+
// does not use.
338+
func (r *CoderMCPServerReconciler) pruneSentCredentials(server *coderv1alpha1.CoderMCPServer) {
339+
old, ok := r.sentCredentials.Load(server.UID)
340+
if !ok {
341+
return
342+
}
343+
sums := map[string][sha256.Size]byte{}
344+
for _, c := range mcpServerCredentialRefs(server.Spec.Auth) {
345+
if sum, ok := old.(map[string][sha256.Size]byte)[c.field]; ok {
346+
sums[c.field] = sum
347+
}
348+
}
349+
r.sentCredentials.Store(server.UID, sums)
350+
}
351+
352+
// forgetSentCredentials drops the hashes of fields before a write that sends
353+
// them.
354+
func (r *CoderMCPServerReconciler) forgetSentCredentials(server *coderv1alpha1.CoderMCPServer, fields []string) {
355+
old, ok := r.sentCredentials.Load(server.UID)
356+
if !ok || len(fields) == 0 {
357+
return
358+
}
359+
sums := maps.Clone(old.(map[string][sha256.Size]byte))
360+
for _, field := range fields {
361+
delete(sums, field)
362+
}
363+
r.sentCredentials.Store(server.UID, sums)
364+
}
365+
366+
func (r *CoderMCPServerReconciler) sentCredentialMatches(server *coderv1alpha1.CoderMCPServer, field string, value credentialsecret.Value) bool {
367+
sums, ok := r.sentCredentials.Load(server.UID)
368+
if !ok {
369+
return false
370+
}
371+
sum, ok := sums.(map[string][sha256.Size]byte)[field]
372+
return ok && sum == sha256.Sum256([]byte(value.Reveal()))
373+
}
374+
247375
// mcpServerReady reports a config that matches the spec.
248376
func mcpServerReady(server *coderv1alpha1.CoderMCPServer) *mcpServerStep {
249377
return &mcpServerStep{

‎internal/controller/codermcpserver_controller.go‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ import (
66
"fmt"
77
"net/url"
88
"strings"
9+
"sync"
910
"time"
1011
"unicode/utf8"
1112

@@ -52,6 +53,10 @@ const (
5253
type CoderMCPServerReconciler struct {
5354
client.Client
5455
Scheme *runtime.Scheme
56+
57+
// sentCredentials maps types.UID -> credential field -> SHA-256 of the
58+
// value that Coder holds. Memory only (plan 4.6 item 4).
59+
sentCredentials sync.Map
5560
}
5661

5762
// mcpServerStep is the outcome of one reconcile. A ready step reports that
@@ -230,7 +235,7 @@ func (r *CoderMCPServerReconciler) check(
230235
*before = *server.Status.DeepCopy()
231236
}
232237
var recreate bool
233-
if step, recreate, err = syncMCPServer(ctx, sdk, server, org.ID, values); err != nil || !recreate {
238+
if step, recreate, err = r.syncMCPServer(ctx, sdk, server, org.ID, values); err != nil || !recreate {
234239
break
235240
}
236241
}

‎internal/controller/codermcpserver_create.go‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -141,6 +141,11 @@ func (r *CoderMCPServerReconciler) createMCPServer(
141141
"The create answer does not match the request. The controller reads Coder instead of sending it again."), nil
142142
}
143143
server.Status.ConfigID, server.Status.CreateAttemptTime = created.ID.String(), nil
144+
var sent []string
145+
for _, c := range mcpServerCredentialRefs(server.Spec.Auth) {
146+
sent = append(sent, c.field)
147+
}
148+
r.rememberSentCredentials(server, sent, values)
144149
return nil, nil
145150
case status == 0 && createHadNoEffect(err):
146151
// A failed dial or TLS handshake never sent the request.

0 commit comments

Comments
 (0)