Skip to content

Commit d5fc1df

Browse files
committed
feat(kubernetes): bootstrap experimental gateway supervisor topology
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
1 parent 15f537f commit d5fc1df

12 files changed

Lines changed: 320 additions & 3 deletions

File tree

‎.agents/skills/helm-dev-environment/SKILL.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,23 @@ Override with env vars before running `helm:k3s:create`:
5757
`mcr.microsoft.com/dotnet/aspire-dashboard:latest`)
5858
- `HELM_K3S_COLLECTOR_HEALTH_TIMEOUT` (default: `120` seconds)
5959

60+
### Experimental supervisor plumbing
61+
62+
Set `gatewayConfig.openshell.drivers.kubernetes.experimental_supervisor_topology`
63+
using a values file (the table key is literally `openshell.drivers.kubernetes`):
64+
65+
```yaml
66+
gatewayConfig:
67+
openshell.drivers.kubernetes:
68+
experimental_supervisor_topology: gateway
69+
```
70+
71+
The gateway chart creates a separate, single-replica idle supervisor Deployment
72+
in the release namespace, regardless of shared/managed/operator workspace mode.
73+
Use a supervisor image built from this branch: the bootstrap subcommand is new.
74+
Check its rollout and exec health probes separately; no sandbox sessions route
75+
through it. `workspace` is reserved and rejected. Omit the field to disable it.
76+
6077
### 2. Deploy OpenShell
6178

6279
**Iterative dev** (rebuilds on file changes, recommended during active development):

‎crates/openshell-driver-kubernetes/src/config.rs‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -194,9 +194,20 @@ where
194194
Ok(value)
195195
}
196196

197+
/// Experimental scope of a shared supervisor deployment.
198+
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
199+
#[serde(rename_all = "snake_case")]
200+
pub enum ExperimentalSupervisorTopology {
201+
Gateway,
202+
Workspace,
203+
}
204+
197205
#[derive(Debug, Clone, Serialize, Deserialize)]
198206
#[serde(default, deny_unknown_fields)]
199207
pub struct KubernetesComputeConfig {
208+
/// Deploy an idle experimental supervisor through Helm. Does not route sessions.
209+
#[serde(skip_serializing_if = "Option::is_none")]
210+
pub experimental_supervisor_topology: Option<ExperimentalSupervisorTopology>,
200211
/// Permit caller-supplied driver JSON. Does not waive resource admission.
201212
pub allow_driver_config: bool,
202213
/// Operator-owned external attachment approval policy.
@@ -389,6 +400,7 @@ pub const ANNOTATION_SCC_SUPPLEMENTAL_GROUPS: &str = "openshift.io/sa.scc.supple
389400
impl Default for KubernetesComputeConfig {
390401
fn default() -> Self {
391402
Self {
403+
experimental_supervisor_topology: None,
392404
workspace_mode: WorkspaceMode::default(),
393405
allow_driver_config: false,
394406
resource_admission:
@@ -442,6 +454,10 @@ impl Default for KubernetesComputeConfig {
442454
impl KubernetesComputeConfig {
443455
/// Validate startup configuration without connecting to Kubernetes.
444456
pub fn validate_configuration(&self) -> Result<(), String> {
457+
if self.experimental_supervisor_topology == Some(ExperimentalSupervisorTopology::Workspace)
458+
{
459+
return Err("experimental_supervisor_topology=workspace is not implemented".into());
460+
}
445461
self.validate_workspace_mode()?;
446462
self.validate_provider_spiffe_workload_api_socket_path()?;
447463
self.validate_sandbox_identity_config()?;

‎crates/openshell-driver-kubernetes/src/lib.rs‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -12,9 +12,9 @@ mod sandbox_runtime;
1212

1313
pub use config::{
1414
DEFAULT_GATEWAY_ID, DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME, DEFAULT_WORKSPACE_STORAGE_SIZE,
15-
KubernetesComputeConfig, KubernetesControlConnection, KubernetesImagePullPolicy,
16-
KubernetesSandboxRuntimeConfig, ManagedSshIngressConfig, WorkspaceMode,
17-
managed_namespace_prefix,
15+
ExperimentalSupervisorTopology, KubernetesComputeConfig, KubernetesControlConnection,
16+
KubernetesImagePullPolicy, KubernetesSandboxRuntimeConfig, ManagedSshIngressConfig,
17+
WorkspaceMode, managed_namespace_prefix,
1818
};
1919
pub use driver::{KubernetesComputeDriver, KubernetesDriverError};
2020
pub use grpc::ComputeDriverService;

‎crates/openshell-driver-kubernetes/src/main.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -260,6 +260,7 @@ async fn main() -> Result<()> {
260260
let (shutdown_tx, shutdown_rx) = tokio::sync::watch::channel(false);
261261
let driver = KubernetesComputeDriver::new(
262262
KubernetesComputeConfig {
263+
experimental_supervisor_topology: None,
263264
allow_driver_config: args.admission_config_json.allow_driver_config,
264265
resource_admission: args.admission_config_json.resource_admission.clone(),
265266
workspace_mode: args.workspace_mode,

‎crates/openshell-supervisor/README.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,3 +60,11 @@ belong to the sandbox or isolation backend. Signals use `BoundarySignal`.
6060

6161
These portable control-plane foundations do not qualify a platform isolation
6262
backend or enable a Windows workload runtime.
63+
64+
## Experimental deployment bootstrap
65+
66+
`openshell-supervisor experimental-bootstrap --health-socket /run/openshell/health.sock`
67+
starts an idle process for the Kubernetes gateway topology experiment. The private
68+
Unix socket supports the existing `health --socket` command. No sandbox session
69+
listener, gateway authentication, or tenant routing is started. SIGTERM and SIGINT
70+
close the listener and remove the socket. This subcommand is Unix-only.
Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
// SPDX-License-Identifier: Apache-2.0
3+
4+
//! Deployment bootstrap only: no sandbox admission, credentials, or session listener.
5+
6+
use clap::Parser;
7+
#[cfg(unix)]
8+
use miette::IntoDiagnostic;
9+
use miette::Result;
10+
use std::path::PathBuf;
11+
12+
#[derive(Debug, Parser)]
13+
#[command(name = "openshell-supervisor experimental-bootstrap")]
14+
#[command(about = "Start an idle experimental supervisor; does not accept sandbox sessions")]
15+
struct Args {
16+
/// Private socket reporting process health, not sandbox readiness.
17+
#[arg(long)]
18+
health_socket: PathBuf,
19+
}
20+
21+
pub fn run(raw_args: &[String]) -> Result<()> {
22+
let args = Args::parse_from(raw_args);
23+
run_platform(args)
24+
}
25+
26+
#[cfg(unix)]
27+
fn run_platform(args: Args) -> Result<()> {
28+
let runtime = tokio::runtime::Builder::new_current_thread()
29+
.enable_all()
30+
.build()
31+
.into_diagnostic()?;
32+
runtime.block_on(async {
33+
// Install shutdown handlers before publishing health.
34+
let mut terminate =
35+
tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate())
36+
.into_diagnostic()?;
37+
let mut interrupt =
38+
tokio::signal::unix::signal(tokio::signal::unix::SignalKind::interrupt())
39+
.into_diagnostic()?;
40+
// The private emptyDir survives a container restart after an unclean exit.
41+
match std::fs::remove_file(&args.health_socket) {
42+
Ok(()) => {}
43+
Err(error) if error.kind() == std::io::ErrorKind::NotFound => {}
44+
Err(error) => return Err(error).into_diagnostic(),
45+
}
46+
let listener = tokio::net::UnixListener::bind(&args.health_socket).into_diagnostic()?;
47+
eprintln!("Experimental supervisor bootstrap started; sandbox sessions are disabled");
48+
let result = loop {
49+
tokio::select! {
50+
result = listener.accept() => {
51+
if let Err(error) = result {
52+
break Err(error).into_diagnostic();
53+
}
54+
}
55+
_ = terminate.recv() => break Ok(()),
56+
_ = interrupt.recv() => break Ok(()),
57+
}
58+
};
59+
drop(listener);
60+
std::fs::remove_file(&args.health_socket).into_diagnostic()?;
61+
result
62+
})
63+
}
64+
65+
#[cfg(not(unix))]
66+
fn run_platform(_args: Args) -> Result<()> {
67+
Err(miette::miette!("experimental-bootstrap requires Unix"))
68+
}

‎crates/openshell-supervisor/src/main.rs‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ use tracing_subscriber::EnvFilter;
1717
use tracing_subscriber::filter::LevelFilter;
1818
use tracing_subscriber::{Layer as _, layer::SubscriberExt as _, util::SubscriberInitExt as _};
1919

20+
mod experimental;
2021
mod logging;
2122

2223
const DEBUG_RPC_SUBCOMMAND: &str = "debug-rpc";
@@ -365,6 +366,10 @@ fn main() -> Result<()> {
365366
return openshell_supervisor::check_control_readiness(&args.socket);
366367
}
367368

369+
if raw_args.get(1).map(String::as_str) == Some("experimental-bootstrap") {
370+
return experimental::run(&raw_args[1..]);
371+
}
372+
368373
let args = Args::parse();
369374
validate_role_arguments(&args)?;
370375
arm_parent_liveness(args.parent_liveness_stdin)?;

‎deploy/helm/openshell/templates/_helpers.tpl‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -568,6 +568,20 @@ like user-supplied schema-v2 fields during compatibility resolution. */}}
568568
Validate chart values that Helm would otherwise accept silently.
569569
*/}}
570570
{{- define "openshell.validateValues" -}}
571+
{{- $kubernetes := include "openshell.effectiveKubernetesConfig" . | fromYaml -}}
572+
{{- if hasKey $kubernetes "experimental_supervisor_topology" -}}
573+
{{- $topology := get $kubernetes "experimental_supervisor_topology" -}}
574+
{{- if not (has $topology (list "gateway" "workspace")) -}}
575+
{{- fail "experimental_supervisor_topology must be gateway or workspace; omit it to disable the experiment" -}}
576+
{{- end -}}
577+
{{- if eq $topology "workspace" -}}
578+
{{- fail "experimental_supervisor_topology=workspace is not implemented" -}}
579+
{{- end -}}
580+
{{- $gateway := get (.Values.gatewayConfig | default dict) "openshell.gateway" | default dict -}}
581+
{{- if ne (get $gateway "compute_driver" | default "kubernetes") "kubernetes" -}}
582+
{{- fail "experimental_supervisor_topology requires the kubernetes compute driver" -}}
583+
{{- end -}}
584+
{{- end -}}
571585
{{- $workloadKind := include "openshell.workloadKind" . -}}
572586
{{- $workload := .Values.workload | default dict -}}
573587
{{- $maxReplicas := int (include "openshell.maxReplicas" .) -}}
Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
{{- include "openshell.validateValues" . }}
2+
{{- $kubernetes := include "openshell.effectiveKubernetesConfig" . | fromYaml -}}
3+
{{- $imagePullSecrets := get $kubernetes "image_pull_secrets" | default list -}}
4+
{{- range .Values.imagePullSecrets -}}
5+
{{- $imagePullSecrets = append $imagePullSecrets .name -}}
6+
{{- end -}}
7+
{{- if eq (get $kubernetes "experimental_supervisor_topology" | default "") "gateway" }}
8+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
9+
# SPDX-License-Identifier: Apache-2.0
10+
apiVersion: apps/v1
11+
kind: Deployment
12+
metadata:
13+
name: {{ include "openshell.fullname" . }}-experimental-supervisor
14+
namespace: {{ .Release.Namespace }}
15+
labels:
16+
{{- include "openshell.labels" . | nindent 4 }}
17+
app.kubernetes.io/component: experimental-supervisor
18+
spec:
19+
replicas: 1
20+
strategy:
21+
type: Recreate
22+
selector:
23+
matchLabels:
24+
app.kubernetes.io/name: openshell-experimental-supervisor
25+
app.kubernetes.io/instance: {{ .Release.Name }}
26+
app.kubernetes.io/component: experimental-supervisor
27+
template:
28+
metadata:
29+
labels:
30+
app.kubernetes.io/name: openshell-experimental-supervisor
31+
app.kubernetes.io/instance: {{ .Release.Name }}
32+
app.kubernetes.io/component: experimental-supervisor
33+
spec:
34+
automountServiceAccountToken: false
35+
terminationGracePeriodSeconds: 30
36+
{{- with ($imagePullSecrets | uniq) }}
37+
imagePullSecrets:
38+
{{- range . }}
39+
- name: {{ . | quote }}
40+
{{- end }}
41+
{{- end }}
42+
securityContext:
43+
runAsNonRoot: true
44+
runAsUser: 65534
45+
runAsGroup: 65534
46+
fsGroup: 65534
47+
seccompProfile:
48+
type: RuntimeDefault
49+
containers:
50+
- name: supervisor
51+
image: {{ get $kubernetes "supervisor_image" | quote }}
52+
imagePullPolicy: {{ get (dict "always" "Always" "if_not_present" "IfNotPresent" "never" "Never") (get $kubernetes "supervisor_image_pull_policy" | default "if_not_present") | quote }}
53+
args:
54+
- experimental-bootstrap
55+
- --health-socket
56+
- /run/openshell/health.sock
57+
securityContext:
58+
allowPrivilegeEscalation: false
59+
readOnlyRootFilesystem: true
60+
capabilities:
61+
drop: [ALL]
62+
resources:
63+
requests:
64+
cpu: 10m
65+
memory: 32Mi
66+
limits:
67+
memory: 128Mi
68+
readinessProbe:
69+
exec:
70+
command: [/openshell-supervisor, health, --socket, /run/openshell/health.sock]
71+
livenessProbe:
72+
exec:
73+
command: [/openshell-supervisor, health, --socket, /run/openshell/health.sock]
74+
volumeMounts:
75+
- name: health
76+
mountPath: /run/openshell
77+
volumes:
78+
- name: health
79+
emptyDir: {}
80+
{{- end }}
Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
suite: experimental supervisor deployment
4+
templates:
5+
- templates/experimental-supervisor.yaml
6+
- templates/gateway-config.yaml
7+
release:
8+
name: openshell
9+
namespace: gateway
10+
tests:
11+
- it: adds no supervisor by default
12+
template: templates/experimental-supervisor.yaml
13+
asserts:
14+
- hasDocuments:
15+
count: 0
16+
- it: deploys one isolated idle process in the gateway namespace
17+
template: templates/experimental-supervisor.yaml
18+
set:
19+
gatewayConfig:
20+
openshell.drivers.kubernetes:
21+
experimental_supervisor_topology: gateway
22+
supervisor_image: example.com/supervisor:experiment
23+
asserts:
24+
- equal:
25+
path: metadata.namespace
26+
value: gateway
27+
- equal:
28+
path: spec.replicas
29+
value: 1
30+
- equal:
31+
path: spec.template.metadata.labels["app.kubernetes.io/name"]
32+
value: openshell-experimental-supervisor
33+
- equal:
34+
path: spec.template.spec.containers[0].image
35+
value: example.com/supervisor:experiment
36+
- equal:
37+
path: spec.template.spec.containers[0].args[0]
38+
value: experimental-bootstrap
39+
- equal:
40+
path: spec.template.spec.automountServiceAccountToken
41+
value: false
42+
- it: renders the same setting into gateway TOML
43+
template: templates/gateway-config.yaml
44+
set:
45+
gatewayConfig:
46+
openshell.drivers.kubernetes:
47+
experimental_supervisor_topology: gateway
48+
asserts:
49+
- matchRegex:
50+
path: data["gateway.toml"]
51+
pattern: 'experimental_supervisor_topology = "gateway"'
52+
- it: rejects workspace until implemented
53+
template: templates/experimental-supervisor.yaml
54+
set:
55+
gatewayConfig:
56+
openshell.drivers.kubernetes:
57+
experimental_supervisor_topology: workspace
58+
asserts:
59+
- failedTemplate:
60+
errorMessage: experimental_supervisor_topology=workspace is not implemented
61+
- it: rejects invalid topology
62+
template: templates/experimental-supervisor.yaml
63+
set:
64+
gatewayConfig:
65+
openshell.drivers.kubernetes:
66+
experimental_supervisor_topology: invalid
67+
asserts:
68+
- failedTemplate:
69+
errorMessage: experimental_supervisor_topology must be gateway or workspace; omit it to disable the experiment
70+
- it: rejects enabled topology for another compute driver
71+
template: templates/experimental-supervisor.yaml
72+
set:
73+
gatewayConfig:
74+
openshell.gateway:
75+
compute_driver: docker
76+
openshell.drivers.kubernetes:
77+
experimental_supervisor_topology: gateway
78+
asserts:
79+
- failedTemplate:
80+
errorMessage: experimental_supervisor_topology requires the kubernetes compute driver

0 commit comments

Comments
 (0)