cluster-manager

The Agent Platform's MCP-only cluster write surface: node pools first (bumblebee-plans#46, giantswarm/giantswarm#37637, epic giantswarm/giantswarm#37639).

cluster-manager is the third sibling of cluster-manager and agent-manager: a Go MCP server (streamable HTTP, an mcp-oauth resource server) registered with muster by its own MCPServer CR. Agents call its tools through muster, and so does the Dev Portal — as the signed-in person. Every Kubernetes call is presented to the API server with the caller's forwarded IdP token (forwardToken), so the caller's RBAC governs what a tool may read or write. There is no REST API.

Tools

Through muster the tools appear as x_cluster-manager_<tool>.

Tool	Purpose
get_info	Version, mode capabilities (apply, commit), the tool list and whether the installation serves the Cluster API (clusterApi)
list_clusters	The installation's clusters: name, organization, release version, own-cluster flag, GPU operator and serving presence with their provider (serving is present with a KServe controller — its Deployment, its release or the platform's discovery ConfigMap — never with the KServe CRDs alone, which Helm leaves behind when a serving layer goes: those are absent with the served APIs noted; a child HelmRelease of cluster-manager's slice release that is not Ready is named in the evidence with its condition's reason and message, since the slice's meta release reports Ready whether or not its children installed), the GPU pool releases (ready, and deleting while one is removed), the commit target; empty with a clusterApi note on an installation without the Cluster API. LLMInferenceServiceConfigs left terminating in the release namespace are named in serving.evidence. Beside status/provider/evidence both components carry readiness, structured, nothing to parse — gpuOperator.readiness: release{name, namespace, ready, reason, message, since, deletedAt} (the provider's operator release), clusterPolicy{name, state} (its status.state: ready, notReady, ignored), operands[]{name, namespace, desired, ready} (the device plugin and GPU feature discovery DaemonSets on the GPU nodes), operandsMessage (why operands is empty although a ClusterPolicy exists: the operator creates them once a GPU node joins, so a pool at scale-to-zero has none), operandsError (the DaemonSets could not be listed); serving.readiness: release (cluster-manager's slice release), children[] (every child release of the slice with its Ready condition — the five KServe charts, the connectivity child, kserve-runtime-configs), controllers[]{name, namespace, available, replicas} (the KServe controller Deployments), configs{count} (the LLMInferenceServiceConfigs of the release namespace), backend{registered, namespace, name, error} (the model-backend-kserve ConfigMap cluster-manager registered with model-manager for this cluster; registered is true or false when the ConfigMap was read and omitted, with error set, when it could not be read — never a bare false for a failed read), presets{count} (the published serving presets), modelsGateway{name, namespace, ready, reason} (the Gateway listening on models.<domain>, its Programmed condition), cacheClaim{namespace, name, phase, volume, zone, error} (the model cache PersistentVolumeClaim of the serving namespace — the connectivity chart's hf-cache, kept when the pool goes — with the zone its bound volume's node affinity names: where every GPU pool created while the claim exists lands; null for none); a count is null where the cluster cannot be read. Every read is concurrent: the call stays inside the aggregator's deadline.
list_node_pools	The pools of one cluster — its MachinePools, and the pool releases whose MachinePool is not there yet (right after create_node_pool) or gone already (while helm-controller uninstalls it) — with the pool's Kubernetes version and the control plane's as two fields, replicas, instance and accelerator types, the owning HelmRelease, and the pool's lifecycle: phase is creating (release not Ready, or MachinePool not created or not ready), ready (every step done; readyReplicas may be 0 at scale-to-zero), scaling (NodeClaims launching or terminating, nodes registering, or a node Karpenter could not launch), removing (release or MachinePool deleted, or a teardown re-run pending) or failed (release Ready=False, the reason in the step); steps[]{name, state, since, finishedAt, message} are release (the pool release Ready), machinePool (its Ready condition, else infrastructureReady) and nodes (Karpenter's NodeClaims of the pool on the cluster — launching, ready, terminating, with counts, a terminating node named with its deletion time: 0 NodeClaim(s) launching, 0 ready, 1 terminating — ip-… terminating since 2026-09-18T03:36:08Z (Karpenter drains the node and terminates its instance; the NodeClaim goes once the instance is terminated, minutes later) —, on a GPU pool of cluster-manager's the ready nodes that hold nothing (no GPU workload, no predictor) named idle since their last pod left (1 node(s) ready (aws:///…), 1 idle since 2026-09-17T22:55:41Z (ip-…): delete_node_pool removes an idle node with the pool) and a MachinePool still listing instances the cluster no longer has said so (0 nodes on the cluster: the MachinePool still lists 1 gone (…), its list follows within minutes), else readyReplicas of replicas where the cluster cannot be read; a node Karpenter could not launch keeps the step inProgress since the last refusal, never done with none, with Karpenter's reason and message verbatim and the way around it — 2 NodeClaims could not launch, the last (<pool>-zgbdh) at 2026-09-18T02:26:37Z — InsufficientInstanceCapacity for g6e.2xlarge in eu-central-1a (InsufficientCapacityError); Karpenter: "creating instance, insufficient capacity, …"; it retries while a pod waits — wider sizes or another accelerator (a re-run of create_node_pool) give it more to choose from — read from the NodeClaims that carry the refusal (Launched or Registered False) and, for a claim Karpenter deleted at once for want of capacity, from its Warning event on the claim in the default namespace, which outlives the claim for the API server's event TTL; a refusal counts until Karpenter creates a claim after it), each pending, inProgress, done or failed, since the condition's lastTransitionTime or the object's creation (RFC3339), finishedAt for a done step. A pool created with prewarm carries a fourth step, prewarm: the placeholder Job's pod in the release namespace — pending while the pool release is not Ready and the Job does not exist (the chart creates it with the release's install; the release step says where the install stands), inProgress while the pod is pending (the first node launching, the scheduler's message beside it — or, when Karpenter could not launch it, the refusal in a line: the placeholder waits for the pool's first node, which Karpenter could not launch — InsufficientInstanceCapacity for g6e.2xlarge in eu-central-1a (the nodes step carries Karpenter's message)), holding node <name> or being preempted, done once preempted by the first workload or finished (the hold ended without one), failed when no node came within the Job's deadline (naming Karpenter's refusal when the nodes step has one), and done with a message when the Job is absent after the release installed (its TTL removed it, or prewarm was set on an existing pool); the step never decides the phase, and a pool without the option shows no such step. A pool under delete_node_pool stays listed with phase: removing, deleting: true and pending[] — the teardown's objects still present, pending or terminating — until its HelmRelease is gone, and by its MachinePool — removing, the terminating node named in the nodes step — until Karpenter's NodeClaim has gone with the terminated instance, minutes later (the release's uninstall does not wait for it; see create_node_pool). Every read is concurrent: the call stays inside the aggregator's deadline.
create_node_pool	Create a GPU node pool for a cluster, or update the pool of that name: the pool release of the gpu-node-pool chart (HelmRelease and OCIRepository in org-<org>, chart pinned exactly) with the Kubernetes version and Flatcar machine image of the cluster's current release — refused when the release runs ahead of the control plane — and a credential-free snapshot of the cluster's settings (registry credentials go into a valuesFrom Secret). mode: apply lands the objects as the caller, owned by the Cluster; dryRun returns the manifests and, on a re-run, the difference. A GitOps-owned object is never patched. When no GPU operator runs on the cluster (detected as the caller on the cluster itself), the <cluster>-gpu-operator release of the catalog's gpu-operator chart comes with the pool, configured from the two-row table read off the nodes (Flatcar: driver and toolkit off — the image carries both and the pool chart's bootstrap makes the toolkit serve them, see the chart's image contract; nvidia.com/gpu.deploy.driver=pre-installed: toolkit on; anything else refused), with Node Feature Discovery's worker pinned to the cluster's GPU pools (gpu-operator.node-feature-discovery.worker.affinity: giantswarm.io/machine-pool In [<cluster>-<pool>, …], the pool being created included; a re-run moves the pin) — the worker is what asserts nvidia.com/gpu.present, so GPU-family nodes of the general pool without a driver get neither the label nor the operands — and polling every 10 s (gpu-operator.node-feature-discovery.worker.config.core.sleepInterval: 10s; upstream's default is 60 s): the operator rolls the toolkit, device plugin and GPU feature discovery onto a node only once the worker has labelled it, so on a node that joins for one predictor a minute's poll was most of the ~100 s between Ready and nvidia.com/gpu allocatable; the rest of worker.config (the chart's PCI device class whitelist) merges in unchanged, and a release cluster-manager does not compose keeps the chart's defaults; an operator the platform's release or a person provides is never re-created. When nothing provides serving on the cluster, the cluster's <cluster>-agent-platform slice release comes with the pool too (see enable_model_serving), updated in place on a re-run. After the pool, the cluster's kserve backend is registered with model-manager — the document naming the pool's instance shapes (spec.kserve.gpuPool.instances, the sizes below) when the pool is the cluster's only one, so model-manager's check_fit judges a preset against the pool before it has a node and load_model refuses what no size hosts (model-manager 0.23.7 or newer; with several pools none is pinned and the document names no sizes). The answer lists the pool's sizes (sizes: the node as AWS lists it and what it leaves a predictor after the kubelet's reservations and the fleet's daemonsets — a g6.xlarge leaves 3 vCPU / 11.9 GiB) and, where the cluster publishes serving presets, the smallest size that hosts each (presetFit); a preset the accelerator could serve but no size of the pool hosts is a warning (warnings) — its predictor would sit Pending while Karpenter refuses every size (giantswarm/agent-platform#502). A size the family does not have is refused. prewarm: true launches the pool's first node with the release instead of with the first predictor (the chart's pool.prewarm.enabled): a one-shot placeholder Job in the release namespace holds one GPU at negative priority until the first workload preempts it, or its hold (the chart's default, 15 minutes) ends and Karpenter consolidates the empty node — a model named up front is served minutes earlier. The pool release does not wait for the chart's Jobs (install/upgrade.disableWaitForJobs: true; the operator and slice releases keep Helm's wait): Helm counts a Job as ready only once it completes, and the placeholder is meant to run until a workload preempts it — waited for, the install ran into its timeout and was uninstalled. Nor does its uninstall wait for the chart's objects to be gone (uninstall.disableWait: true; uninstall.timeout stays the default): the MachinePool goes only when EC2 has terminated the pool's last instance, minutes after its NodeClaim's delete and longer than helm-controller's timeout — waited for, the uninstall failed (UninstallFailed) and the release went with the retry minutes later; the MachinePool and the instance terminate on their own, list_node_pools naming the terminating node until they are gone. Only for the installation's own pool, where the release namespace is on the cluster the nodes join; a workload cluster's pool is refused, naming why. The Job is created with the release's install only: a re-run flips the value in place (would-update) but never launches a placeholder again; prewarm: false removes the block. The pool follows the model cache (#59): the serving namespace's cache claim (hf-cache, the chart's serving.cacheClaim) outlives the pool and is one EBS volume, bound in one zone by the first predictor that mounted it — a pool whose nodes come up elsewhere strands the next predictor Pending (with prewarm the placeholder's node took the cheapest zone, the single-GPU limit forbade a second node, and the pool read ready with an idle node). So with the claim Bound the pool's nodes are pinned to its volume's zone — pool.zones: [eu-central-1b] on the pool release, a topology.kubernetes.io/zone In requirement on the Karpenter NodePool, placeholder and predictors alike — and the answer says so: zones, zonesNote (nodes pinned to eu-central-1b: the model cache (claim model-serving/hf-cache, volume pvc-…) lives there …; remove the cache claim to lift the pin …, or pick an accelerator offered in eu-central-1b) and cacheClaim{namespace, name, phase, volume, zone} as read on the cluster as you; dryRun shows it, a re-run on an existing pool moves the pin (would-update). No claim: no zones block and nothing said. A claim that is not Bound yet, or whose volume names no zone (a network file system), pins nothing and zonesNote says what was found; a claim or volume not readable as you pins nothing and is a warnings entry naming why — never a guessed zone. A family not offered in the cache's zone fails its launch visibly, named in list_node_pools' nodes step. LLMInferenceServiceConfigs a serving layer that went left terminating in the release namespace are removed first, finalizer and all, when no llm-d controller runs (objects lists them, warnings says why), so the slice's release creates them afresh instead of adopting and losing them.
delete_node_pool	Remove what create_node_pool created, the pool's idle nodes first. The pool's nodes are read on the cluster as you — Karpenter's NodeClaims of the pool, the Nodes registered from them and the pods on each —, never from the MachinePool's provider IDs while the cluster is readable (that list lags a terminated instance by minutes). A node that holds nothing — no pod with a GPU or a KServe predictor's on it; a DaemonSet's pods, finished pods and pods preemptible by design (the pool's prewarm placeholder) do not count — is idle: its NodeClaim is deleted first, as you (Karpenter drains the node and terminates the instance within the pool's terminationGracePeriod), listed in objects, and the teardown completes in the same call — the pool release's uninstall does not wait for the MachinePool, which goes with the terminated instance minutes later, list_node_pools naming the terminating node meanwhile; dryRun lists the NodeClaims it would delete. Refused while a node of the pool is busy unless force — the refusal names each busy node with what holds it (node ip-… runs model-serving/qwen3-8b-fp8-kserve-… (1 GPU), a NodeClaim still launching, pods not readable as you) and the models served on the cluster (LLMInferenceService model-serving/qwen3-4b-instruct (Qwen/Qwen3-4B-Instruct-2507), a Pending one included) to unload first — and refused, with no busy node, while a model model-manager serves in the serving namespace has its predictor on no node (#59): every LLMInferenceService and InferenceService carrying model-manager's app.kubernetes.io/managed-by label counts, whatever its phase — one on a node of the pool is a busy node, one on no node is waiting for one (the kept cache's zone, or a size the family lacks there) and the delete would strand it: with the cluster's last pool the serving slice, its controller and the backend go too, and the serving object is left behind with a finalizer nothing clears. The refusal names the model and its pod (LLMInferenceService model-serving/qwen3-8b (Qwen/Qwen3-8B): pod model-serving/qwen3-8b-kserve-…-xnkd2 Pending on no node, or no predictor pod yet) and the idle nodes that go with the pool once it is unloaded; a hand-made serving object does not count; the serving namespace's pods not readable as you is a refusal too. Beside the text the refusal carries a structured refused{nodes[], idle[], models[], unscheduled[], hint, readFrom} block as a second text content of the tool error ({"refused": {...}}), unscheduled naming the waiting models, readFrom being cluster, or machinePool when the cluster cannot be read as you and the MachinePool's replicas and provider IDs had to decide (the text says so). Refused for a pool cluster-manager did not create. With the cluster's last pool go the operator release, the slice release — unless another slice is on in it (sliceKept) — and the backend registration cluster-manager created. The slice goes in order (#28): its kserve-runtime-configs child release first, then every LLMInferenceServiceConfig of the release namespace is seen gone from the cluster — waited for while the llm-d controller runs to clear them, removed with serving.kserve.io/llmisvcconfig-finalizer taken off where nothing else will (the controller gone, force, or two minutes over; warnings says which) — then the pool, the operator, the slice release and the backend, so nothing is left terminating for the next slice to adopt and lose. Without force a cluster that cannot be read as the caller is refused before anything is deleted.
enable_model_serving	Switch model serving on for a cluster, with or without a GPU pool: the cluster's one release of the agent-platform chart, <cluster>-agent-platform (HelmRelease and OCIRepository in org-<org>, delivered by the installation's Flux; the chart pinned exactly to the version the installation's own platform release runs — status.history[0].chartVersion of its HelmRelease, released by construction and known to work on the installation — with a floor of 4.27.0, the first chart whose serving slice places the predictors on a tainted GPU pool; a platform below it is refused, naming why; --slice-chart-version / serving.sliceChartVersion pins another version for a lab), with the serving slice on — the five KServe components and the llm-d well-known configs, modelServing with the nvidia RuntimeClass, the models Gateway at models.<domain> with the login issuer's JWT policy — its key set fetched from the platform's Dex service in-cluster on the installation's own cluster (dex.<namespace>.svc.cluster.local:5556 in plaintext, the namespace and port from the platform's gateway.jwksEgress, the source the platform's own JWT policies use — and gateway.jwksEgress itself in the slice's values, enabled with that namespace and port, the connectivity chart's precondition for an in-cluster host, without which the slice's connectivity child fails its render and serves no models Gateway; the chart's default, the public issuer on 443, yields no key set there and every id_token was refused as signed by an unknown key) and from the public issuer on a workload cluster, which cannot reach the installation's Service; the answer's slice.jwks names the source. global.domain, global.identity and the wildcard certificate where the platform names one, else a Certificate of models.<domain> from the fleet's ClusterIssuer — --models-certificate-issuer / serving.certificateIssuer, default letsencrypt-giantswarm are read from the installation's own platform release, never invented; every cluster gets the target knob (gitops.target.kubeConfig.secretRef.name: <cluster>-kubeconfig, see Delivery below); a workload cluster gets global.domain: <cluster>.<base domain> and components.agentgateway on, the installation's own cluster the platform's domain and agentgateway off (the platform's release owns the data plane); the chart's Flux engine and model-manager are always off. With exactly one GPU pool on the cluster the predictors are pinned to it (modelServing.gpuPool.nodeSelector, and modelServing.serving.nodeSelector for the pinned chart). A release that exists is updated in place — the re-run moves the pin to the version the platform runs today; a release of the chart under another name is refused, naming it; a chart-provided or hand-installed serving layer is left alone. Registers the kserve backend, its document naming the pinned pool's instance shapes, read from the pool's release (see create_node_pool), when the cluster has exactly one pool. Stranded terminating LLMInferenceServiceConfigs are healed first, as create_node_pool does.
disable_model_serving	Remove the slice release cluster-manager created and its backend registration. Refused while models are served on the cluster (named) unless force, and plainly when the cluster cannot be read as the caller; refused for a release cluster-manager did not create. The same ordered teardown as delete_node_pool's last pool, the slice release last.

Every write tool takes dryRun (the rendered manifests, or on a re-run the difference) and mode: apply | commit; commit (a pull request as the caller) follows in the epic's later stage. An apply answers within the aggregator's deadline for a tool call: everything it reads is read once and concurrently before anything is composed, the objects are planned together (every refusal before the first write) and land one after the other — create_node_pool: pool, slice, the slice's backend registration, operator, the objects that do not exist yet before the updates. A write there is no budget left for (--apply-budget, default 8 s; the request's own deadline when earlier) is not started: the answer comes back with partial: true, the objects not reached as pending and nextStep naming the re-run, which writes them first. The deletes take the same budget: delete_node_pool's last pool and disable_model_serving remove the serving slice in order — the llm-d controller's child release; its well-known LLMInferenceServiceConfigs once no controller runs on the cluster (its webhook denies every delete of them while it does, and nothing clears their finalizer once it is gone, so cluster-manager removes them, finalizer and all); the configs' child release (a child whose uninstall failed before is asked to retry now); the operator, the backend registration, the slice release; and the pool's own objects last, its HelmRelease the very last, so the re-run finds the pool and continues where the teardown stands. Every phase is timed in the log at debug (--verbose).

Delivery

Giant Swarm installations enforce Flux multi-tenancy on every HelmRelease outside the platform's namespaces (the flux-multi-tenancy Kyverno policy, admission-enforced, on org-<org> among others): a release carries spec.serviceAccountName or spec.kubeConfig.secretRef.name, and leaves its namespace (targetNamespace, storageNamespace) only through a kubeconfig. The releases cluster-manager composes take the two shapes the fleet's own releases in org-<org> take, by what they render — for the installation's own cluster exactly as for a workload cluster:

release	objects live	delivery
<cluster>-gpu-operator	on the cluster, in kube-system	kubeConfig.secretRef.name: <cluster>-kubeconfig — the cluster's Cluster API kubeconfig Secret, which exists for the installation's own cluster too and points at its own API server
<cluster>-<pool>	in org-<org> on the installation (the Cluster API objects)	serviceAccountName: automation — the org's tenant ServiceAccount (--tenant-service-account / flux.tenantServiceAccount; empty renders none, for a lab without the policy)
<cluster>-agent-platform	in org-<org> on the installation (the meta chart's child HelmReleases and OCIRepositories)	serviceAccountName: automation; the target knob gitops.target.kubeConfig.secretRef.name: <cluster>-kubeconfig in its values puts the kubeconfig on every child, which then installs into the cluster

The dry run cannot see admission, so the goldens hold the policy's constraint on every composed HelmRelease.

Running

cluster-manager serve --kubeconfig ~/.kube/config

Every flag has an environment variable named next to it in --help; flags win. The chart in helm/cluster-manager renders the Deployment, the Service, the MCPServer CR for muster and, with oauth.enabled, the mcp-oauth resource-server flags from the platform identity contract (global.identity).

Development

make build-linux-amd64   # the binary the Dockerfile expects
go test ./...
make helm-lint helm-template

Releases are automatic: every merge to main is tagged from Conventional Commits and CircleCI publishes the image to gsoci.azurecr.io/giantswarm/cluster-manager and the chart to the Giant Swarm catalog.