Skip to content

Commit 8b6b473

Browse files
committed
refactor(supervisor): probe shared supervisor health on session control
The shared supervisor's kubelet tcpSocket probes now connect to the mTLS session-control listener (5501) instead of a separate health listener. Drop the experimental-host --health-port flag, its accept loop, and the chart's 5502 container port. Probes open and close a TCP connection; every session-control request still requires mTLS. Signed-off-by: Drew Newberry <anewberry@nvidia.com>
1 parent efe31a8 commit 8b6b473

5 files changed

Lines changed: 23 additions & 50 deletions

File tree

‎.agents/skills/helm-dev-environment/SKILL.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -72,8 +72,9 @@ The gateway chart creates a single-replica shared supervisor Deployment,
7272
Service, and NetworkPolicy in the release namespace, regardless of
7373
shared/managed/operator workspace mode, and the certgen hook issues its
7474
session-control Secrets. New sandbox generations run as sessions in that
75-
process. Its process-health `tcpSocket` probes use container port `health`
76-
(5502); session control stays on 5501. Build the gateway, supervisor, and sandbox images from the same branch:
75+
process. Its process-health `tcpSocket` probes connect to the mTLS
76+
session-control port (5501); there is no separate health port. Build the
77+
gateway, supervisor, and sandbox images from the same branch:
7778
all three must support the routed runtime connection. `workspace` is reserved
7879
and rejected. Omit the field to disable it. `mise run
7980
e2e:kubernetes:shared-supervisor` runs the focused e2e

‎crates/openshell-supervisor/README.md‎

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -119,12 +119,11 @@ one process for the Kubernetes gateway topology experiment. It is Unix-only.
119119
- **Runtime connections.** One routed listener (`--runtime-listen`) accepts every
120120
session's runtime-initiated Sandbox Protocol connections; see
121121
`openshell-sandbox-backend` for the routing preface.
122-
- **Health.** `--health-port` (5502 in the chart) binds a wildcard TCP listener
123-
after the session-control and runtime listeners. It accepts and closes
124-
connections for kubelet `tcpSocket` probes, keeps no filesystem state, and
125-
reports process health only. Session readiness is reported per session over
126-
session control. SIGTERM and SIGINT stop every session, which ends their
127-
workloads' supervision.
122+
- **Health.** The chart's kubelet `tcpSocket` probes connect to the
123+
session-control port (5501) and close without completing TLS, so they report
124+
only that the process is listening; every request still requires mTLS.
125+
Session readiness is reported per session over session control. SIGTERM and
126+
SIGINT stop every session, which ends their workloads' supervision.
128127

129128
Sessions get no upstream proxy, extra CA bundle, or provider workload identity:
130129
the Kubernetes driver refuses the topology when an upstream proxy or provider

‎crates/openshell-supervisor/src/experimental.rs‎

Lines changed: 6 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -12,16 +12,12 @@ use std::path::PathBuf;
1212
#[command(name = "openshell-supervisor experimental-host")]
1313
#[command(about = "Supervise the sandboxes a gateway registers with this process")]
1414
struct Args {
15-
/// TCP port accepting connections while the process runs; reports process
16-
/// health, not session readiness.
17-
#[arg(long)]
18-
health_port: u16,
19-
2015
/// Gateway gRPC endpoint; each session authenticates with its own credentials.
2116
#[arg(long, env = openshell_core::sandbox_env::ENDPOINT)]
2217
openshell_endpoint: String,
2318

2419
/// Mutually authenticated session-control listener used by the gateway.
20+
/// Kubernetes TCP probes connect to it without completing TLS.
2521
#[arg(long)]
2622
session_control_listen: SocketAddr,
2723

@@ -82,7 +78,8 @@ fn run_platform(args: Args) -> Result<()> {
8278
)
8379
.init();
8480

85-
// Install shutdown handlers before publishing health.
81+
// Install shutdown handlers before the control listener starts
82+
// answering health probes.
8683
let mut terminate =
8784
tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate())
8885
.into_diagnostic()?;
@@ -113,11 +110,6 @@ fn run_platform(args: Args) -> Result<()> {
113110
.await
114111
.into_diagnostic()
115112
.wrap_err("bind session-control listener")?;
116-
117-
let health = openshell_supervisor::bind_readiness_tcp(args.health_port)
118-
.and_then(tokio::net::TcpListener::from_std)
119-
.into_diagnostic()
120-
.wrap_err("bind health listener")?;
121113
tracing::info!(
122114
runtime = %host.runtime_addr(),
123115
control = %args.session_control_listen,
@@ -133,20 +125,10 @@ fn run_platform(args: Args) -> Result<()> {
133125
let _ = stopped.wait_for(|stop| *stop).await;
134126
},
135127
));
136-
loop {
137-
tokio::select! {
138-
// A connection is the whole check; nothing is exchanged.
139-
result = health.accept() => {
140-
if let Err(error) = result {
141-
tracing::warn!(%error, "health accept failed; retrying");
142-
tokio::time::sleep(std::time::Duration::from_millis(100)).await;
143-
}
144-
}
145-
_ = terminate.recv() => break,
146-
_ = interrupt.recv() => break,
147-
}
128+
tokio::select! {
129+
_ = terminate.recv() => {}
130+
_ = interrupt.recv() => {}
148131
}
149-
drop(health);
150132
stop.send_replace(true);
151133
let _ = serve.await;
152134
// Sandboxes lose supervision with this process and fail closed.

‎deploy/helm/openshell/templates/experimental-supervisor.yaml‎

Lines changed: 5 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -60,8 +60,6 @@ spec:
6060
imagePullPolicy: {{ get (dict "always" "Always" "if_not_present" "IfNotPresent" "never" "Never") (get $kubernetes "supervisor_image_pull_policy" | default "if_not_present") | quote }}
6161
args:
6262
- experimental-host
63-
- --health-port
64-
- "5502"
6563
- --session-control-listen
6664
- 0.0.0.0:{{ get $shared "control_port" }}
6765
- --session-control-tls-dir
@@ -90,9 +88,6 @@ spec:
9088
- name: session-control
9189
containerPort: {{ get $shared "control_port" }}
9290
protocol: TCP
93-
- name: health
94-
containerPort: 5502
95-
protocol: TCP
9691
securityContext:
9792
allowPrivilegeEscalation: false
9893
readOnlyRootFilesystem: true
@@ -104,14 +99,15 @@ spec:
10499
memory: 64Mi
105100
limits:
106101
memory: 1Gi
107-
# Process health only; each session reports its own readiness to
108-
# the gateway through the session-control API.
102+
# Process health only: the probes open and close a TCP connection
103+
# to the mTLS session-control listener. Each session reports its own
104+
# readiness to the gateway through the session-control API.
109105
readinessProbe:
110106
tcpSocket:
111-
port: health
107+
port: session-control
112108
livenessProbe:
113109
tcpSocket:
114-
port: health
110+
port: session-control
115111
volumeMounts:
116112
- name: run
117113
mountPath: /run/openshell

‎deploy/helm/openshell/tests/experimental_supervisor_test.yaml‎

Lines changed: 4 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -47,8 +47,6 @@ tests:
4747
path: spec.template.spec.containers[0].args
4848
value:
4949
- experimental-host
50-
- --health-port
51-
- "5502"
5250
- --session-control-listen
5351
- 0.0.0.0:5501
5452
- --session-control-tls-dir
@@ -58,24 +56,21 @@ tests:
5856
- --session-root
5957
- /run/openshell/sessions
6058
documentIndex: 0
61-
- contains:
59+
- lengthEqual:
6260
path: spec.template.spec.containers[0].ports
63-
content:
64-
name: health
65-
containerPort: 5502
66-
protocol: TCP
61+
count: 2
6762
documentIndex: 0
6863
- equal:
6964
path: spec.template.spec.containers[0].readinessProbe
7065
value:
7166
tcpSocket:
72-
port: health
67+
port: session-control
7368
documentIndex: 0
7469
- equal:
7570
path: spec.template.spec.containers[0].livenessProbe
7671
value:
7772
tcpSocket:
78-
port: health
73+
port: session-control
7974
documentIndex: 0
8075
- contains:
8176
path: spec.template.spec.containers[0].volumeMounts

0 commit comments

Comments
 (0)