🤖 fix: require authentication for the aggregated API and MCP servers (#136)

## Summary

Before this change, the aggregated API server and the MCP HTTP server
accepted requests without authentication, and both acted with the
operator's authority. The aggregated API server used an anonymous
authenticator and an allow-all authorizer, then called Coder with the
control plane's owner token. So any client that could reach port `6443`
directly (pod IP or the `coder-k8s-apiserver` Service) could read
template source and create, change or delete templates and workspaces.
The MCP server on `:8090` also had no authentication, and `--app=all`
(the default) started both servers. Requests through kube-apiserver and
the APIService were already RBAC-checked; direct traffic was not.

A security advisory will be published with the release that contains
this fix.

## What changed

**Aggregated API server** (`fix(apiserver): require delegated
authentication and authorization`)

- Uses the vendored `k8s.io/apiserver` delegated authentication and
authorization: kube-apiserver's front-proxy client certificates (CA from
`kube-system/extension-apiserver-authentication`), cluster client
certificates, TokenReview for bearer tokens, and SubjectAccessReview for
every request.
- Refuses the anonymous identity everywhere except exact `/healthz`,
`/livez` and `/readyz`.
- Fails closed: without read access to the auth ConfigMap it does not
start; if SubjectAccessReview is unavailable, requests are denied.
- One Kubernetes authority for both checks: `KUBECONFIG` (exactly one
valid file, no fallback), else the in-cluster ServiceAccount, else
`~/.kube/config`.
- Keeps the vendored `system:masters` bypass and 10 s caches, and
documents them.
- Docs: RBAC on `aggregation.coder.com` is owner-equivalent inside Coder
(the server still calls Coder with the operator's token for the
namespace's control plane).

**MCP server** (`fix(mcp): require a bearer token and stop serving MCP
by default`)

- `--app=all` no longer starts MCP. `--app=mcp-http` listens on
`127.0.0.1:8090` only.
- `--mcp-token-file` is mandatory for `--app=mcp-http`; every request
except exact `/healthz` and `/readyz` needs `Authorization: Bearer
<token>`, including requests on an existing session. Constant-time
compare.
- Removes `deploy/mcp-service.yaml` and the 8090 container port.

## Upgrade notes

1. Apply `config/rbac/auth-delegator-binding.yaml` and
`config/rbac/authentication-reader-binding.yaml` for the operator
ServiceAccount (both are in every documented aggregated-API install).
2. Clients that called port `6443` directly must send a Kubernetes
bearer token with RBAC on `aggregation.coder.com`, or go through
kube-apiserver (`kubectl`).
3. `--app=all` no longer runs MCP; delete the old `coder-k8s` MCP
Service. To use MCP, run `--app=mcp-http --mcp-token-file=<file>` and
reach it with `kubectl port-forward`. The token file is read once at
startup.
4. Outside a cluster, the aggregated server needs `KUBECONFIG` (one
file) or `~/.kube/config`.
5. Grant RBAC on `codertemplates`/`coderworkspaces` only to subjects you
would trust as Coder owners.

The install bundle `dist/install.yaml` installs controller mode only and
is unaffected.

## Validation

- Unit tests with a fake Kubernetes API and envtest with a real
kube-apiserver (request-header flags, RBAC): forged `X-Remote-*` headers
alone and with a valid unprivileged token, unknown CAs, a disallowed
front-proxy CN, SubjectAccessReview attributes for every verb on both
resources and namespace boundaries, delegation outages,
missing/empty/malformed/forbidden ConfigMap, startup without each
binding. Every denial asserts zero Coder backend calls. MCP tests reuse
a live session without the token (POST, GET with `Last-Event-ID`, DELETE
all get 401).
- 11 mutation checks (restore anonymous, restore allow-all, remove the
anonymous filter, prefix path matching, tolerate lookup failure,
optional kubeconfig, MCP back on `:8090`, MCP back in `--app=all`, token
check removed, filter accepting on error, session ID skipping the
token): all caught.
- Gates on this head: `make verify-vendor`, `make build`, `make lint`,
`make test`, the installer freshness step, markdownlint, cspell and
`make docs-check` pass; `make test-integration` and actionlint passed on
the same change before a docs-free rebase.
- New CI E2E step: RBAC through kube-apiserver for a non-admin identity;
direct requests to `6443` from another pod (anonymous, forged headers,
ServiceAccount token without RBAC) are rejected; `8090` is not served.
- Live KIND run with the fixed image:

![Fixed build: direct anonymous and forged-header requests get 401, an
unprivileged ServiceAccount token gets 403, the template is unchanged,
8090 is
closed](https://github.com/user-attachments/assets/10536f46-1652-4a44-89e3-b766ba37d395)

![Fail closed: without the auth bindings the server refuses to start or
denies
requests](https://github.com/user-attachments/assets/41561824-a008-4972-adfb-cdec6c65c7ab)

![MCP: not reachable on the pod network; through port-forward, requests
without the token or with a wrong token get 401, including on an
existing session; the MCP process refuses to start without a token
file](https://github.com/user-attachments/assets/fc076992-e0b1-4075-9d97-8bfabbce1403)

Recording of the fixed-build run (rendered from the terminal recording;
idle time capped, so shorter than wall time):


https://github.com/user-attachments/assets/48e11bf0-5eaf-4932-a893-5dc472f75580

Out of scope, tracked separately: `deploy/apiserver-apiservice.yaml`
still sets `insecureSkipTLSVerify: true` (documented as
development-only).

---
_Generated with [`xum`](https://github.com/coder/xum) • Model:
`anthropic:claude-opus-5-5` • Thinking: `medium` • Cost: `$1742.21`_

Author: ThomasK33

.cspell.json

@@ -52,7 +52,11 @@
     "pooler",
     "finalizer",
     "superfences",
-    "tolerations"
+    "tolerations",
+    "portforward",
+    "livez",
+    "requestheader",
+    "subjectaccessreviews"
   ],
   "ignorePaths": [
     ".git/**",

.github/workflows/ci.yaml

@@ -372,6 +372,50 @@ jobs:
           test -n "$TOKEN_SECRET"
           kubectl -n coder get secret "$TOKEN_SECRET"
 
+      # Delegated authentication/authorization of the aggregated API server:
+      # RBAC via kube-apiserver for a non-admin identity, and direct requests to
+      # port 6443 from another pod (anonymous, forged front-proxy headers, and a
+      # ServiceAccount token without RBAC) are rejected. MCP is not served in --app=all.
+      - name: Verify aggregated API authentication and authorization
+        if: env.E2E_FULL == 'true'
+        env:
+          PROBE_IMAGE: curlimages/curl:8.16.0@sha256:463eaf6072688fe96ac64fa623fe73e1dbe25d8ad6c34404a669ad3ce1f104b6
+        run: |
+          set -euo pipefail
+          kubectl -n coder create serviceaccount e2e-reader
+          kubectl -n coder create role e2e-template-reader --verb=get,list --resource=codertemplates.aggregation.coder.com
+          kubectl -n coder create rolebinding e2e-template-reader --role=e2e-template-reader --serviceaccount=coder:e2e-reader
+          reader=(--as=system:serviceaccount:coder:e2e-reader)
+          kubectl "${reader[@]}" -n coder get codertemplates.aggregation.coder.com
+          expect_forbidden() {
+            local out
+            if out=$("$@" 2>&1); then
+              echo "expected Forbidden, command succeeded: $*" >&2
+              return 1
+            fi
+            grep -q Forbidden <<<"$out" || { echo "expected Forbidden, got: $out" >&2; return 1; }
+          }
+          expect_forbidden kubectl "${reader[@]}" -n default get codertemplates.aggregation.coder.com
+          expect_forbidden kubectl "${reader[@]}" -n coder get coderworkspaces.aggregation.coder.com
+          expect_forbidden kubectl "${reader[@]}" -n coder delete codertemplates.aggregation.coder.com coder.e2e-authz-probe --dry-run=server
+
+          kubectl -n default run e2e-authn-probe --image="$PROBE_IMAGE" --restart=Never --command -- sleep 600
+          kubectl -n default wait --for=condition=Ready pod/e2e-authn-probe --timeout=180s
+          probe() { kubectl -n default exec e2e-authn-probe -- sh -c "$1"; }
+          api=https://coder-k8s-apiserver.coder-system.svc
+          list="$api/apis/aggregation.coder.com/v1alpha1/namespaces/coder/codertemplates"
+          code() { probe "curl -sk -o /dev/null -w '%{http_code}' $1"; }
+          test "$(code "$list")" = 401
+          test "$(code "-H 'X-Remote-User: kubernetes-admin' -H 'X-Remote-Group: system:masters' $list")" = 401
+          test "$(code "-H \"Authorization: Bearer \$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)\" $list")" = 403
+          test "$(code "$api/healthz")" = 200
+          pod_ip=$(kubectl -n coder-system get pod -l app=coder-k8s -o jsonpath='{.items[0].status.podIP}')
+          if probe "curl -s --max-time 5 http://$pod_ip:8090/healthz"; then
+            echo "MCP must not listen on the pod network" >&2
+            exit 1
+          fi
+          kubectl -n default delete pod e2e-authn-probe --wait=false
+
       # The driver creates the CoderTemplate itself (kubectl apply of config/e2e/codertemplate.yaml) after its setup checks.
       - name: Template and workspace lifecycle (apply, re-apply, watch, rename, delete, recreate)
         if: env.E2E_FULL == 'true'

AGENTS.md

@@ -73,7 +73,7 @@ Run from repository root.
 - **Workspace lifecycle E2E driver tests (offline, stubbed tools):** `bash ./hack/e2e-workspace-lifecycle_test.sh` (also run by `make test-scripts`)
 - **Lint (workflows):** `go run github.com/rhysd/actionlint/cmd/actionlint@v1.7.10`
 - **Development run (controller mode):** `GOFLAGS=-mod=vendor go run . --app=controller` (requires Kubernetes config via your env, e.g. `KUBECONFIG`)
-- **Development run (aggregated API mode):** `GOFLAGS=-mod=vendor go run . --app=aggregated-apiserver`
+- **Development run (aggregated API mode):** `GOFLAGS=-mod=vendor go run . --app=aggregated-apiserver` (needs `KUBECONFIG` or `~/.kube/config` with a cluster that can serve TokenReview/SubjectAccessReview and `kube-system/extension-apiserver-authentication`; the server does not start without it)
 - **Vendor consistency:** `make verify-vendor`
 - **Manifest generation:** `make manifests` (or `bash ./hack/update-manifests.sh`)
 - **Code generation:** `make codegen` (or `bash ./hack/update-codegen.sh`)

README.md

@@ -23,10 +23,10 @@ Pick what runs with `--app`:
 
 | `--app` | Runs |
 | --- | --- |
-| `all` (default) | Everything in one process |
+| `all` (default) | Operator and aggregated API server in one process |
 | `controller` | Operator only |
 | `aggregated-apiserver` | Aggregated API server only |
-| `mcp-http` | MCP server only |
+| `mcp-http` | MCP server only (never part of `all`; needs `--mcp-token-file`) |
 
 ## Quick start
 

app_dispatch.go

@@ -36,6 +36,7 @@ func run(args []string) error {
 		coderSessionToken   string
 		coderNamespace      string
 		coderRequestTimeout time.Duration
+		mcpTokenFile        string
 	)
 	fs.StringVar(&appMode, "app", "all", "Application mode (all, controller, aggregated-apiserver, mcp-http)")
 	fs.StringVar(
@@ -62,6 +63,12 @@ func run(args []string) error {
 		30*time.Second,
 		"Timeout for Coder SDK API requests",
 	)
+	fs.StringVar(
+		&mcpTokenFile,
+		"mcp-token-file",
+		"",
+		"Path to a file holding the bearer token that every MCP HTTP request must present (required for --app=mcp-http)",
+	)
 	if err := fs.Parse(args); err != nil {
 		return err
 	}
@@ -83,6 +90,10 @@ func run(args []string) error {
 		}
 	}
 
+	if mcpTokenFile != "" && appMode != "mcp-http" {
+		return fmt.Errorf("--mcp-token-file is only used with --app=mcp-http; --app=%s does not run the MCP server", appMode)
+	}
+
 	switch appMode {
 	case "all":
 		return runAllApp(setupSignalHandler(), coderRequestTimeout)
@@ -97,7 +108,7 @@ func run(args []string) error {
 		}
 		return runAggregatedAPIServerApp(setupSignalHandler(), opts)
 	case "mcp-http":
-		return runMCPHTTPApp(setupSignalHandler())
+		return runMCPHTTPApp(setupSignalHandler(), mcpTokenFile)
 	default:
 		return fmt.Errorf("assertion failed: unsupported --app value %q; must be one of: %s", appMode, supportedAppModes)
 	}

config/default/controller-mode-patch.yaml

@@ -14,5 +14,3 @@ spec:
           ports:
             - containerPort: 6443
               $patch: delete
-            - containerPort: 8090
-              $patch: delete

deploy/deployment.yaml

@@ -22,8 +22,6 @@ spec:
               name: health
             - containerPort: 6443
               name: https
-            - containerPort: 8090
-              name: mcp
           livenessProbe:
             httpGet:
               path: /healthz

deploy/mcp-service.yaml

@@ -4,33 +4,31 @@
 
 | `--app` | Runs |
 | --- | --- |
-| `all` (default) | Controller, aggregated API server, and MCP server in one process |
+| `all` (default) | Controller and aggregated API server in one process |
 | `controller` | Controller-runtime manager and reconcilers |
 | `aggregated-apiserver` | Aggregated API server (`aggregation.coder.com/v1alpha1`) |
-| `mcp-http` | MCP HTTP server |
+| `mcp-http` | MCP HTTP server (only in this mode; requires `--mcp-token-file`) |
 
 ## All-in-one mode
 
-In `all` mode, `internal/app/allapp` creates one controller-runtime manager with one shared cache. It registers the reconcilers, then starts the aggregated API server and MCP server as non-leader runnables. One process, one cache, coordinated startup.
+In `all` mode, `internal/app/allapp` creates one controller-runtime manager with one shared cache. It registers the reconcilers, then starts the aggregated API server as a non-leader runnable. The MCP server is not part of `all` mode because it acts with the operator's authority; run it on purpose with `--app=mcp-http`. One process, one cache, coordinated startup.
 
 ```mermaid
 graph TD
   entry["coder-k8s (--app=all)"] --> mgr["controller-runtime manager"]
   mgr --> ctrl["Controller reconcilers"]
   mgr --> agg["Aggregated API server runnable"]
-  mgr --> mcp["MCP HTTP runnable"]
 
   ctrl --> crds["coder.com/v1alpha1 CRDs"]
   agg --> api["aggregation.coder.com/v1alpha1"]
-  mcp --> tools["MCP tools over /mcp"]
 ```
 
 ## Components
 
 | | Controller | Aggregated API server | MCP server |
 | --- | --- | --- | --- |
 | **Code** | `internal/app/controllerapp/`, `internal/controller/` | `internal/app/apiserverapp/`, `internal/aggregated/storage/`, `internal/aggregated/coder/` | `internal/app/mcpapp/` |
-| **Listens on** | `:8081` (`/healthz`, `/readyz`) | `:6443` HTTPS (default) | `:8090` (`/mcp`, `/healthz`, `/readyz`) |
+| **Listens on** | `:8081` (`/healthz`, `/readyz`) | `:6443` HTTPS (default) | `127.0.0.1:8090` only (`/mcp` needs the bearer token; `/healthz`, `/readyz` do not) |
 | **Resources** | `CoderControlPlane`, `CoderProvisioner`, `CoderWorkspaceProxy` | `coderworkspaces`, `codertemplates` | Tools for control planes, templates, workspaces, events, pod logs, and run state |
 
 ### Controller
@@ -41,6 +39,10 @@ Uses controller-runtime with leader election. For each `CoderControlPlane`, it c
 
 Storage is backed by the Coder SDK, not memory or etcd: each request becomes a Coder API call. See [Aggregated API behavior](../reference/aggregated-api-behavior.md) for the consequences.
 
+The Coder calls use the control plane's operator credentials, so the server checks every Kubernetes caller first. It uses delegated authentication (front-proxy client certificates from kube-apiserver, TokenReview for bearer tokens) and delegated authorization (SubjectAccessReview), and fails closed when those checks are unavailable. Only exact `/healthz`, `/livez`, and `/readyz` answer anonymous callers. See [How callers are checked](../how-to/deploy-aggregated-apiserver.md#how-callers-are-checked).
+
+Kubernetes users are not mapped to Coder users. Kubernetes RBAC on `aggregation.coder.com` in a namespace therefore grants owner-equivalent access in the Coder deployment of the control plane that serves that namespace.
+
 How it finds its Coder backend:
 
 - **`all` mode:** `ControlPlaneClientProvider` discovers eligible `CoderControlPlane` resources and reads their operator token Secrets dynamically.
@@ -64,4 +66,3 @@ graph TD
 | `config/rbac/` | ServiceAccount, `manager-role`, and bindings (including auth-delegator) |
 | `deploy/deployment.yaml` | The `coder-k8s` Deployment (defaults to `--app=all`) |
 | `deploy/apiserver-service.yaml`, `deploy/apiserver-apiservice.yaml` | Expose the aggregated API |
-| `deploy/mcp-service.yaml` | MCP Service on port `8090` |

docs/explanation/architecture.md

@@ -12,6 +12,8 @@ kubectl apply -f config/rbac/
 kubectl apply -f deploy/apiserver-service.yaml -f deploy/apiserver-apiservice.yaml
 ```
 
+`config/rbac/` includes two bindings the aggregated API server needs to check callers: `auth-delegator-binding.yaml` (create TokenReviews and SubjectAccessReviews) and `authentication-reader-binding.yaml` (read `kube-system/extension-apiserver-authentication`; the default `manager-role` also grants cluster-wide ConfigMap reads). Both name the `coder-k8s` ServiceAccount in `coder-system`; edit them if you install elsewhere. The server fails closed without these permissions: without read access to that ConfigMap it does not start, and without permission to create SubjectAccessReviews it answers every request with an error (members of `system:masters` excepted).
+
 ## 2. Deploy
 
 ### Option A: all-in-one (recommended)
@@ -67,6 +69,34 @@ kubectl -n coder-system patch deployment coder-k8s --type=strategic -p '{
 }'
 ```
 
+## How callers are checked
+
+The aggregated API server authenticates and authorizes every request with the Kubernetes API:
+
+| Caller | Authentication | Authorization |
+| --- | --- | --- |
+| `kubectl` and other clients through kube-apiserver (the normal path) | kube-apiserver's front-proxy client certificate, verified against `requestheader-client-ca-file` from `kube-system/extension-apiserver-authentication`; the user comes from the `X-Remote-*` headers | SubjectAccessReview for that user, verb, resource, and namespace |
+| Direct requests to port `6443` with a bearer token | TokenReview | SubjectAccessReview |
+| Direct requests with a client certificate signed by the cluster client CA | Certificate subject | SubjectAccessReview |
+| Anything else | None | Rejected with `401`, except exact `/healthz`, `/livez`, `/readyz` |
+
+`X-Remote-*` headers are trusted only on connections that present a valid front-proxy client certificate. Access to the resources is controlled with Kubernetes RBAC on `aggregation.coder.com` (`codertemplates`, `coderworkspaces`), but read the warning below before you grant it.
+
+!!! warning "RBAC on these resources is owner access in Coder"
+    Treat Kubernetes RBAC on `codertemplates` and `coderworkspaces` as owner-equivalent inside Coder. Grant it only to subjects you would trust as Coder owners, and prefer namespaced Roles over ClusterRoles.
+
+The server does not map Kubernetes users to Coder users. After Kubernetes RBAC allows a request, the server calls Coder with the control plane's operator token, which has owner rights in Coder. Each request uses only the control plane that serves the request's namespace (in standalone mode, the server is pinned to `--coder-namespace`).
+
+So a subject allowed to create, update, patch, or delete `codertemplates` or `coderworkspaces` in a namespace acts in that Coder deployment as an owner, and `get`, `list`, or `watch` shows that deployment's templates (including their source files) and workspaces as an owner sees them. A ClusterRole binding grants this for every namespace that has a control plane.
+
+Kept Kubernetes defaults, so you know what to expect:
+
+- Members of `system:masters` are authorized without a SubjectAccessReview, as in kube-apiserver.
+- TokenReview results are cached for 10 seconds, allowed SubjectAccessReview results for 10 seconds, and denied results for 10 seconds. A permission change can take effect only after the matching cache entry expires.
+- The server reads `extension-apiserver-authentication` at startup and keeps watching it. If the ConfigMap is deleted later, the server keeps trusting the CA it last loaded (retained trust) until it restarts or sees a new one. If the ConfigMap or its request-header CA is missing at startup, front-proxy requests fail with `401`.
+
+The server uses the same Kubernetes API for all of these checks: `KUBECONFIG` if set (exactly one file, never a fallback), otherwise the in-cluster ServiceAccount, otherwise `~/.kube/config`. With none, it does not start.
+
 ## 3. Verify
 
 ```bash