Skip to content

Commit 409361b

Browse files
committed
feat(helm): make cluster-scoped RBAC optional
The gateway chart always rendered the ClusterRole and ClusterRoleBinding, so every install and upgrade required cluster-admin even when only namespaced objects were needed. Installers that are namespace-admin GitOps or platform controllers could not run the release at all, and clusters where cluster-scoped RBAC is owned by a separate team had no supported way to split the install. Add an rbac values block so a cluster-admin can apply the cluster-scoped objects once and a namespace-admin can install and upgrade the release without cluster-scoped permissions: rbac: create: true clusterScoped: create: true clusterRoleName: "" clusterRoleBindingName: "" rbac.clusterScoped.create gates the ClusterRole and ClusterRoleBinding, and is independent of the workspace mode. rbac.create additionally gates the namespaced sandbox Role and RoleBinding, which matters because Kubernetes escalation prevention stops an installer holding only the built-in admin role from creating a Role that grants agents.x-k8s.io verbs it does not itself hold. The certgen hook and credential driver RBAC keep their existing flags. Both flags default to true, so current installs are unchanged. The helpers treat a missing rbac block as enabled so upgrades with --reuse-values do not drop RBAC, matching the existing workspaceResources pattern. The ClusterRoleBinding roleRef follows clusterRoleName so a separately applied ClusterRole can carry a name the cluster-admin chooses. Document the migration for a release that already owns the cluster-scoped objects: Helm deletes objects that leave the manifest, so annotate them with helm.sh/resource-policy=keep before setting the flag, otherwise the gateway loses TokenReview until a cluster-admin re-applies them. Signed-off-by: ansjindal <ansjindal@nvidia.com>
1 parent 87929ad commit 409361b

14 files changed

Lines changed: 488 additions & 6 deletions

‎deploy/helm/openshell/README.md‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,77 @@ namespace. The gateway and workspace releases can then be upgraded and removed
1717
independently. Use Kubernetes `operator` workspace mode when one gateway serves
1818
multiple pre-provisioned workspace namespaces.
1919

20+
## Cluster-scoped vs namespaced objects
21+
22+
Most objects in this chart are namespaced and land in the release namespace.
23+
Only two are cluster-scoped:
24+
25+
| Object | Default name |
26+
| --- | --- |
27+
| `ClusterRole` | `<fullname>-node-reader-<release namespace>` |
28+
| `ClusterRoleBinding` | `<fullname>-node-reader-<release namespace>` |
29+
30+
By default the release creates both, so an install by a cluster-admin is
31+
unchanged. On clusters where cluster-scoped RBAC is owned by a different team,
32+
split the install in two.
33+
34+
A cluster-admin applies the cluster-scoped objects once per gateway
35+
ServiceAccount, rendered from the same values the release uses:
36+
37+
```shell
38+
helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
39+
--namespace openshell -f my-values.yaml \
40+
--set rbac.clusterScoped.create=true \
41+
--set agentSandbox.preflight.enabled=false \
42+
--show-only templates/clusterrole.yaml \
43+
--show-only templates/clusterrolebinding.yaml | kubectl apply -f -
44+
```
45+
46+
A namespace-admin then installs and upgrades the release with cluster-scoped
47+
objects omitted, using [`ci/values-namespace-admin.yaml`](ci/values-namespace-admin.yaml)
48+
or the equivalent `--set`:
49+
50+
```shell
51+
helm upgrade --install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
52+
--namespace openshell -f my-values.yaml \
53+
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
54+
--set rbac.clusterScoped.create=false
55+
```
56+
57+
The gateway ServiceAccount name and namespace do not change, so the
58+
pre-created `ClusterRoleBinding` keeps matching the release. This works with
59+
`serviceAccount.create=false` too: the `ClusterRoleBinding` subject follows
60+
`serviceAccount.name`, so render the admin step with the same values.
61+
62+
### Migrating an existing release
63+
64+
Helm deletes objects that leave a release manifest, so setting
65+
`rbac.clusterScoped.create=false` on a release that already owns the
66+
`ClusterRole` and `ClusterRoleBinding` deletes them. The gateway then loses
67+
TokenReview until a cluster-admin re-applies them. Hand ownership over first, as
68+
cluster-admin, so nothing is deleted:
69+
70+
```shell
71+
kubectl annotate clusterrole "openshell-node-reader-<namespace>" \
72+
helm.sh/resource-policy=keep --overwrite
73+
kubectl annotate clusterrolebinding "openshell-node-reader-<namespace>" \
74+
helm.sh/resource-policy=keep --overwrite
75+
```
76+
77+
The objects then survive the upgrade that sets the flag, and the cluster-admin
78+
owns them from that point on. Fresh installs need no such step.
79+
80+
`rbac.clusterScoped.create` is independent of
81+
`server.drivers.kubernetes.workspaceMode`. Managed and operator modes change
82+
what the `ClusterRole` contains, but they never force the namespaced release to
83+
apply it. Re-run the cluster-admin step after changing values that affect the
84+
`ClusterRole` rules.
85+
86+
Set `rbac.create=false` to also omit the namespaced sandbox `Role` and
87+
`RoleBinding`. The certgen hook and credential driver RBAC keep their own flags
88+
(`pkiInitJob.enabled` and
89+
`server.credentialDrivers.kubernetesSecrets.rbac.create`).
90+
2091
## Prerequisites
2192

2293
> **Required:** Your cluster CNI MUST enforce Kubernetes `NetworkPolicy` for
@@ -260,6 +331,10 @@ discovery endpoint or its TLS CA.
260331
| probes.startup.failureThreshold | int | `30` | Startup probe failure threshold before the container is killed. |
261332
| probes.startup.periodSeconds | int | `2` | Startup probe period, in seconds. |
262333
| probes.startup.timeoutSeconds | int | `1` | Startup probe timeout, in seconds. |
334+
| rbac.clusterScoped.clusterRoleBindingName | string | `""` | Name for the ClusterRoleBinding. Empty uses the `<fullname>-node-reader-<release namespace>` default. |
335+
| rbac.clusterScoped.clusterRoleName | string | `""` | Name for the ClusterRole. Empty uses the `<fullname>-node-reader-<release namespace>` default. |
336+
| rbac.clusterScoped.create | bool | `true` | Create the cluster-scoped ClusterRole and ClusterRoleBinding. Disable for a namespace-admin install where a cluster-admin applies them separately; the gateway ServiceAccount name and namespace are unchanged, so a pre-created ClusterRoleBinding still matches. |
337+
| rbac.create | bool | `true` | Create the RBAC objects that grant the gateway ServiceAccount access. Disable to supply the namespaced sandbox Role/RoleBinding and the cluster-scoped ClusterRole/ClusterRoleBinding out of band. The certgen hook and credential driver RBAC keep their own flags. |
263338
| replicaCount | int | `1` | Number of OpenShell gateway replicas. Values greater than 1 require server.externalDbSecret because the default SQLite backend is per pod. |
264339
| resources | object | `{}` | Gateway pod resource requests and limits. |
265340
| sandbox.image.digest | string | `""` | Sandbox image digest. When set, this takes precedence over tag. |

‎deploy/helm/openshell/README.md.gotmpl‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,77 @@ namespace. The gateway and workspace releases can then be upgraded and removed
1717
independently. Use Kubernetes `operator` workspace mode when one gateway serves
1818
multiple pre-provisioned workspace namespaces.
1919

20+
## Cluster-scoped vs namespaced objects
21+
22+
Most objects in this chart are namespaced and land in the release namespace.
23+
Only two are cluster-scoped:
24+
25+
| Object | Default name |
26+
| --- | --- |
27+
| `ClusterRole` | `<fullname>-node-reader-<release namespace>` |
28+
| `ClusterRoleBinding` | `<fullname>-node-reader-<release namespace>` |
29+
30+
By default the release creates both, so an install by a cluster-admin is
31+
unchanged. On clusters where cluster-scoped RBAC is owned by a different team,
32+
split the install in two.
33+
34+
A cluster-admin applies the cluster-scoped objects once per gateway
35+
ServiceAccount, rendered from the same values the release uses:
36+
37+
```shell
38+
helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
39+
--namespace openshell -f my-values.yaml \
40+
--set rbac.clusterScoped.create=true \
41+
--set agentSandbox.preflight.enabled=false \
42+
--show-only templates/clusterrole.yaml \
43+
--show-only templates/clusterrolebinding.yaml | kubectl apply -f -
44+
```
45+
46+
A namespace-admin then installs and upgrades the release with cluster-scoped
47+
objects omitted, using [`ci/values-namespace-admin.yaml`](ci/values-namespace-admin.yaml)
48+
or the equivalent `--set`:
49+
50+
```shell
51+
helm upgrade --install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
52+
--namespace openshell -f my-values.yaml \
53+
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
54+
--set rbac.clusterScoped.create=false
55+
```
56+
57+
The gateway ServiceAccount name and namespace do not change, so the
58+
pre-created `ClusterRoleBinding` keeps matching the release. This works with
59+
`serviceAccount.create=false` too: the `ClusterRoleBinding` subject follows
60+
`serviceAccount.name`, so render the admin step with the same values.
61+
62+
### Migrating an existing release
63+
64+
Helm deletes objects that leave a release manifest, so setting
65+
`rbac.clusterScoped.create=false` on a release that already owns the
66+
`ClusterRole` and `ClusterRoleBinding` deletes them. The gateway then loses
67+
TokenReview until a cluster-admin re-applies them. Hand ownership over first, as
68+
cluster-admin, so nothing is deleted:
69+
70+
```shell
71+
kubectl annotate clusterrole "openshell-node-reader-<namespace>" \
72+
helm.sh/resource-policy=keep --overwrite
73+
kubectl annotate clusterrolebinding "openshell-node-reader-<namespace>" \
74+
helm.sh/resource-policy=keep --overwrite
75+
```
76+
77+
The objects then survive the upgrade that sets the flag, and the cluster-admin
78+
owns them from that point on. Fresh installs need no such step.
79+
80+
`rbac.clusterScoped.create` is independent of
81+
`server.drivers.kubernetes.workspaceMode`. Managed and operator modes change
82+
what the `ClusterRole` contains, but they never force the namespaced release to
83+
apply it. Re-run the cluster-admin step after changing values that affect the
84+
`ClusterRole` rules.
85+
86+
Set `rbac.create=false` to also omit the namespaced sandbox `Role` and
87+
`RoleBinding`. The certgen hook and credential driver RBAC keep their own flags
88+
(`pkiInitJob.enabled` and
89+
`server.credentialDrivers.kubernetesSecrets.rbac.create`).
90+
2091
## Prerequisites
2192

2293
> **Required:** Your cluster CNI MUST enforce Kubernetes `NetworkPolicy` for
Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
# Namespace-admin overlay — renders only namespaced objects.
5+
#
6+
# Use this when a cluster-admin applies the gateway ClusterRole and
7+
# ClusterRoleBinding once, out of band, and the OpenShell release is installed
8+
# and upgraded by an installer that holds no cluster-scoped permissions.
9+
#
10+
# Generate the cluster-scoped objects for the cluster-admin step from the same
11+
# release values, then apply them as cluster-admin:
12+
# helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart \
13+
# --version <version> --namespace openshell -f my-values.yaml \
14+
# --set rbac.clusterScoped.create=true \
15+
# --set agentSandbox.preflight.enabled=false \
16+
# --show-only templates/clusterrole.yaml \
17+
# --show-only templates/clusterrolebinding.yaml | kubectl apply -f -
18+
#
19+
# The gateway ServiceAccount name and release namespace are unchanged, so the
20+
# pre-created ClusterRoleBinding still matches this release.
21+
22+
rbac:
23+
clusterScoped:
24+
create: false

‎deploy/helm/openshell/templates/_helpers.tpl‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,66 @@ default to enabled so upgrades with --reuse-values preserve the old topology.
8686
{{- if $enabled -}}true{{- end -}}
8787
{{- end }}
8888

89+
{{/*
90+
Whether this chart owns gateway RBAC objects. Missing legacy values default to
91+
enabled so upgrades with --reuse-values preserve the old topology.
92+
*/}}
93+
{{- define "openshell.rbacCreate" -}}
94+
{{- $rbac := .Values.rbac | default dict -}}
95+
{{- $create := true -}}
96+
{{- if hasKey $rbac "create" -}}
97+
{{- $create = get $rbac "create" -}}
98+
{{- end -}}
99+
{{- if $create -}}true{{- end -}}
100+
{{- end }}
101+
102+
{{/*
103+
The rbac.clusterScoped values map, tolerating missing legacy values.
104+
*/}}
105+
{{- define "openshell.clusterScopedRbacValues" -}}
106+
{{- $rbac := .Values.rbac | default dict -}}
107+
{{- $clusterScoped := dict -}}
108+
{{- if hasKey $rbac "clusterScoped" -}}
109+
{{- $clusterScoped = get $rbac "clusterScoped" | default dict -}}
110+
{{- end -}}
111+
{{- toYaml $clusterScoped -}}
112+
{{- end }}
113+
114+
{{/*
115+
Whether this chart owns the cluster-scoped ClusterRole and ClusterRoleBinding.
116+
Disable for a namespace-admin install where a cluster-admin applies them
117+
separately. Missing legacy values default to enabled.
118+
*/}}
119+
{{- define "openshell.clusterRbacCreate" -}}
120+
{{- if include "openshell.rbacCreate" . -}}
121+
{{- $clusterScoped := include "openshell.clusterScopedRbacValues" . | fromYaml -}}
122+
{{- $create := true -}}
123+
{{- if hasKey $clusterScoped "create" -}}
124+
{{- $create = get $clusterScoped "create" -}}
125+
{{- end -}}
126+
{{- if $create -}}true{{- end -}}
127+
{{- end -}}
128+
{{- end }}
129+
130+
{{/*
131+
Name of the gateway ClusterRole. The release namespace is part of the default
132+
name so multiple releases on one cluster do not collide.
133+
*/}}
134+
{{- define "openshell.clusterRoleName" -}}
135+
{{- $clusterScoped := include "openshell.clusterScopedRbacValues" . | fromYaml -}}
136+
{{- $default := printf "%s-node-reader-%s" (include "openshell.fullname" .) .Release.Namespace -}}
137+
{{- default $default (get $clusterScoped "clusterRoleName") -}}
138+
{{- end }}
139+
140+
{{/*
141+
Name of the gateway ClusterRoleBinding.
142+
*/}}
143+
{{- define "openshell.clusterRoleBindingName" -}}
144+
{{- $clusterScoped := include "openshell.clusterScopedRbacValues" . | fromYaml -}}
145+
{{- $default := printf "%s-node-reader-%s" (include "openshell.fullname" .) .Release.Namespace -}}
146+
{{- default $default (get $clusterScoped "clusterRoleBindingName") -}}
147+
{{- end }}
148+
89149
{{/* Gateway image reference. A digest takes precedence over a tag. */}}
90150
{{- define "openshell.image" -}}
91151
{{- $image := .Values.gateway.image -}}

‎deploy/helm/openshell/templates/clusterrole.yaml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,12 @@
1+
{{- if include "openshell.clusterRbacCreate" . }}
12
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
23
# SPDX-License-Identifier: Apache-2.0
34

45
{{- $workspaceMode := .Values.server.drivers.kubernetes.workspaceMode | default "shared" }}
56
apiVersion: rbac.authorization.k8s.io/v1
67
kind: ClusterRole
78
metadata:
8-
name: {{ include "openshell.fullname" . }}-node-reader-{{ .Release.Namespace }}
9+
name: {{ include "openshell.clusterRoleName" . }}
910
labels:
1011
{{- include "openshell.labels" . | nindent 4 }}
1112
rules:
@@ -128,3 +129,4 @@ rules:
128129
- update
129130
{{- end }}
130131
{{- end }}
132+
{{- end }}
Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,19 @@
1+
{{- if include "openshell.clusterRbacCreate" . }}
12
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
23
# SPDX-License-Identifier: Apache-2.0
34

45
apiVersion: rbac.authorization.k8s.io/v1
56
kind: ClusterRoleBinding
67
metadata:
7-
name: {{ include "openshell.fullname" . }}-node-reader-{{ .Release.Namespace }}
8+
name: {{ include "openshell.clusterRoleBindingName" . }}
89
labels:
910
{{- include "openshell.labels" . | nindent 4 }}
1011
roleRef:
1112
apiGroup: rbac.authorization.k8s.io
1213
kind: ClusterRole
13-
name: {{ include "openshell.fullname" . }}-node-reader-{{ .Release.Namespace }}
14+
name: {{ include "openshell.clusterRoleName" . }}
1415
subjects:
1516
- kind: ServiceAccount
1617
name: {{ include "openshell.serviceAccountName" . }}
1718
namespace: {{ .Release.Namespace }}
19+
{{- end }}

‎deploy/helm/openshell/templates/role.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{{- $workspaceMode := .Values.server.drivers.kubernetes.workspaceMode | default "shared" -}}
2-
{{- if and (eq $workspaceMode "shared") (include "openshell.workspaceResourcesEnabled" .) }}
2+
{{- if and (eq $workspaceMode "shared") (include "openshell.workspaceResourcesEnabled" .) (include "openshell.rbacCreate" .) }}
33
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
44
# SPDX-License-Identifier: Apache-2.0
55
apiVersion: rbac.authorization.k8s.io/v1

‎deploy/helm/openshell/templates/rolebinding.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{{- $workspaceMode := .Values.server.drivers.kubernetes.workspaceMode | default "shared" -}}
2-
{{- if and (eq $workspaceMode "shared") (include "openshell.workspaceResourcesEnabled" .) }}
2+
{{- if and (eq $workspaceMode "shared") (include "openshell.workspaceResourcesEnabled" .) (include "openshell.rbacCreate" .) }}
33
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
44
# SPDX-License-Identifier: Apache-2.0
55
apiVersion: rbac.authorization.k8s.io/v1

‎deploy/helm/openshell/tests/clusterrole_test.yaml‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -163,3 +163,42 @@ tests:
163163
apiGroups: [""]
164164
resources: ["services"]
165165
verbs: ["create", "get"]
166+
- it: omits the ClusterRole when cluster-scoped RBAC is disabled
167+
set:
168+
rbac.clusterScoped.create: false
169+
asserts:
170+
- hasDocuments:
171+
count: 0
172+
173+
- it: omits the ClusterRole when all chart-managed RBAC is disabled
174+
set:
175+
rbac.create: false
176+
asserts:
177+
- hasDocuments:
178+
count: 0
179+
180+
- it: omits the ClusterRole independently of the workspace mode
181+
set:
182+
rbac.clusterScoped.create: false
183+
server.drivers.kubernetes.workspaceMode: managed
184+
asserts:
185+
- hasDocuments:
186+
count: 0
187+
188+
- it: uses the configured ClusterRole name
189+
set:
190+
rbac.clusterScoped.clusterRoleName: platform-openshell-node-reader
191+
asserts:
192+
- equal:
193+
path: metadata.name
194+
value: platform-openshell-node-reader
195+
196+
- it: creates the ClusterRole when legacy values omit the rbac block
197+
set:
198+
rbac: null
199+
asserts:
200+
- hasDocuments:
201+
count: 1
202+
- equal:
203+
path: metadata.name
204+
value: openshell-node-reader-my-namespace

0 commit comments

Comments
 (0)