Skip to content

Commit eeccc21

Browse files
authored
feat(supervisor): advertise accepted application route installations (#4940)
* feat(supervisor): advertise accepted application route installations Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * fix(otel): identify the application endpoint RPC extension service Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * fix(gateway): preserve advertisement cleanup progress on timeout Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * perf(gateway): reuse app endpoint resolution within each lease exchange Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * test(supervisor): preserve gateway affinity in app-listener startup fixtures Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
1 parent dbc0517 commit eeccc21

34 files changed

Lines changed: 2435 additions & 29 deletions

File tree

‎.agents/skills/helm-dev-environment/SKILL.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -195,6 +195,17 @@ every member one at a time. The PID must survive each case, while an open
195195
replacement confirmed within the recovery timeout, so the runtime terminated
196196
the workload.
197197

198+
For RFC 0017 app-listener validation, enable the listener and supply the app TLS
199+
Secret on the shared Deployment, plus a separate NetworkPolicy allowing the test
200+
client/proxy to reach its app port. Runtime-port access does not grant app access.
201+
Discovery returns the acknowledged owner's Pod IP and app port; no headless app
202+
Service is needed. Exercise HTTP/WebSocket reconnection through graceful downscale
203+
and abrupt loss, preserving workload generation/PID/in-memory state and the public
204+
hostname. Verify stale-owner rejection independently of the replacement's success.
205+
Shared serving leases expire within fifteen seconds of issuance and do not renew
206+
from configuration alone. Do not claim seamless socket migration or infer external
207+
ingress readiness from an advertisement.
208+
198209
### 2. Deploy OpenShell
199210

200211
**Iterative dev** (rebuilds on file changes, recommended during active development):

‎crates/openshell-core/src/app_listener.rs‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,12 @@ impl AppListenerConfig {
5050
}
5151
}
5252

53+
/// A synchronous view of installed ingress, evaluated on the sandbox instance
54+
/// runtime. The requested lease must fit entirely within serving freshness.
55+
pub type InstallationSource = std::sync::Arc<
56+
dyn Fn(std::time::Duration) -> crate::proto::SupervisorServiceAdvertisement + Send + Sync,
57+
>;
58+
5359
#[cfg(test)]
5460
mod tests {
5561
use super::*;

‎crates/openshell-core/tests/compute_driver_compat.rs‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -186,6 +186,16 @@ async fn roundtrip(serve_image_updates: bool) {
186186
.connect()
187187
.await
188188
.unwrap();
189+
let mut apps = openshell_core::proto::compute::v1::compute_driver_app_endpoints_client::ComputeDriverAppEndpointsClient::new(channel.clone());
190+
assert_eq!(
191+
apps.resolve_supervisor_app_endpoint(
192+
openshell_core::proto::compute::v1::ResolveSupervisorAppEndpointRequest::default()
193+
)
194+
.await
195+
.unwrap_err()
196+
.code(),
197+
tonic::Code::Unimplemented,
198+
);
189199
let mut baseline = ComputeDriverClient::new(channel.clone());
190200
assert_eq!(
191201
baseline

‎crates/openshell-driver-kubernetes/src/driver.rs‎

Lines changed: 195 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2224,6 +2224,201 @@ impl KubernetesComputeDriver {
22242224
Ok(())
22252225
}
22262226

2227+
/// Resolve only a unique, live driver-owned supervisor. Multi-member pool
2228+
/// resolution must use the HA assignment/member API rather than guess.
2229+
pub async fn supervisor_app_endpoint(
2230+
&self,
2231+
sandbox_id: &str,
2232+
) -> Result<
2233+
Option<openshell_core::proto::compute::v1::SupervisorAppEndpoint>,
2234+
KubernetesDriverError,
2235+
> {
2236+
if !self.config.sandbox_runtime.app_listener.common.enabled {
2237+
return Ok(None);
2238+
}
2239+
let api = self
2240+
.supported_sandbox_api_for_lookup(self.client.clone())
2241+
.await
2242+
.map_err(KubernetesDriverError::Message)?;
2243+
let objects = api
2244+
.api
2245+
.list(&ListParams::default().labels(&self.sandbox_lookup_selector(sandbox_id)))
2246+
.await
2247+
.map_err(KubernetesDriverError::from_kube)?
2248+
.items;
2249+
let [object]: [DynamicObject; 1] = match objects.try_into() {
2250+
Ok(objects) => objects,
2251+
Err(_) => return Ok(None),
2252+
};
2253+
let namespace = object.metadata.namespace.as_deref().ok_or_else(|| {
2254+
KubernetesDriverError::Precondition("sandbox namespace missing".into())
2255+
})?;
2256+
let uid = object
2257+
.metadata
2258+
.uid
2259+
.as_deref()
2260+
.ok_or_else(|| KubernetesDriverError::Precondition("sandbox UID missing".into()))?;
2261+
let placement = supervisor_placement(&object)?;
2262+
let mut pool_route = String::new();
2263+
let mut ownership_epoch = 0;
2264+
let mut member_address = None;
2265+
let pods = if let SupervisorPlacement::Shared { route } = &placement {
2266+
let Some(shared) = &self.shared_supervisor else {
2267+
return Ok(None);
2268+
};
2269+
let controller = shared.supervision().controller();
2270+
let Some(view) = controller.pool_view() else {
2271+
return Ok(None);
2272+
};
2273+
let Some(owner) = view
2274+
.owners
2275+
.get(route)
2276+
.filter(|_| view.desired.contains(route))
2277+
else {
2278+
return Ok(None);
2279+
};
2280+
let Some(member) = view.members.iter().find(|member| {
2281+
member.member_id == owner.member_id
2282+
&& member.process_id == owner.process_id
2283+
&& !member.leaving
2284+
}) else {
2285+
return Ok(None);
2286+
};
2287+
if !matches!(controller.observed(route),
2288+
openshell_supervisor_control::ObservedInstance::Reported(status)
2289+
if status.sandbox_id == sandbox_id && status.ready
2290+
&& status.phase == openshell_supervisor_control::proto::SandboxInstancePhase::Supervising as i32)
2291+
{
2292+
return Ok(None);
2293+
}
2294+
let address = member
2295+
.runtime_address
2296+
.parse::<std::net::SocketAddr>()
2297+
.map_err(|_| {
2298+
KubernetesDriverError::Precondition(
2299+
"pool member runtime address is not an IP endpoint".into(),
2300+
)
2301+
})?;
2302+
let pod =
2303+
Api::<Pod>::namespaced(self.client.clone(), &shared.config().service_namespace)
2304+
.get_opt(&owner.member_id)
2305+
.await
2306+
.map_err(KubernetesDriverError::from_kube)?;
2307+
let Some(pod) = pod else {
2308+
return Ok(None);
2309+
};
2310+
let service =
2311+
Api::<Service>::namespaced(self.client.clone(), &shared.config().service_namespace)
2312+
.get(&shared.config().service_name)
2313+
.await
2314+
.map_err(KubernetesDriverError::from_kube)?;
2315+
let Some(selector) = service
2316+
.spec
2317+
.as_ref()
2318+
.and_then(|spec| spec.selector.as_ref())
2319+
.filter(|selector| !selector.is_empty())
2320+
else {
2321+
return Ok(None);
2322+
};
2323+
if !selector.iter().all(|(key, value)| {
2324+
pod.metadata
2325+
.labels
2326+
.as_ref()
2327+
.and_then(|labels| labels.get(key))
2328+
== Some(value)
2329+
}) {
2330+
return Ok(None);
2331+
}
2332+
pool_route = route.clone();
2333+
ownership_epoch = owner.auth_epoch;
2334+
member_address = Some(address.ip());
2335+
vec![pod]
2336+
} else {
2337+
let selector =
2338+
format!("openshell.ai/sandbox-id={sandbox_id},openshell.ai/component=supervisor");
2339+
Api::<Pod>::namespaced(self.client.clone(), namespace)
2340+
.list(&ListParams::default().labels(&selector))
2341+
.await
2342+
.map_err(KubernetesDriverError::from_kube)?
2343+
.items
2344+
};
2345+
// Includes terminating members in the ambiguity check; never guess a
2346+
// replacement's ownership during a rollout.
2347+
let [pod]: [Pod; 1] = match pods.try_into() {
2348+
Ok(pods) => pods,
2349+
Err(_) => return Ok(None),
2350+
};
2351+
if pod.metadata.deletion_timestamp.is_some() {
2352+
return Ok(None);
2353+
}
2354+
let app_port = i32::from(self.config.sandbox_runtime.app_listener.common.port);
2355+
if !pod
2356+
.spec
2357+
.as_ref()
2358+
.and_then(|spec| {
2359+
spec.containers
2360+
.iter()
2361+
.find(|container| container.name == "supervisor")
2362+
})
2363+
.is_some_and(|container| {
2364+
container.ports.as_ref().is_some_and(|ports| {
2365+
ports.iter().any(|port| {
2366+
port.name.as_deref() == Some("app") && port.container_port == app_port
2367+
})
2368+
}) && container.env.as_ref().is_some_and(|env| {
2369+
env.iter().any(|entry| {
2370+
entry.name == "OPENSHELL_APP_LISTENER_ENABLED"
2371+
&& entry.value.as_deref() == Some("true")
2372+
})
2373+
})
2374+
})
2375+
{
2376+
return Ok(None);
2377+
}
2378+
let Some(pod_uid) = pod.metadata.uid.as_deref() else {
2379+
return Ok(None);
2380+
};
2381+
if matches!(placement, SupervisorPlacement::Dedicated)
2382+
&& !pod
2383+
.metadata
2384+
.owner_references
2385+
.as_ref()
2386+
.is_some_and(|owners| owners.iter().any(|o| o.uid == uid && o.kind == "Sandbox"))
2387+
{
2388+
return Ok(None);
2389+
}
2390+
let Some(status) = pod.status else {
2391+
return Ok(None);
2392+
};
2393+
if status.phase.as_deref() != Some("Running") {
2394+
return Ok(None);
2395+
}
2396+
let Some(host) = status
2397+
.pod_ip
2398+
.filter(|ip| ip.parse::<std::net::IpAddr>().is_ok())
2399+
else {
2400+
return Ok(None);
2401+
};
2402+
if member_address
2403+
.is_some_and(|address| host.parse::<std::net::IpAddr>().ok() != Some(address))
2404+
{
2405+
return Ok(None);
2406+
}
2407+
let component = match placement {
2408+
SupervisorPlacement::Dedicated => pod_uid.to_string(),
2409+
SupervisorPlacement::Shared { route } => route,
2410+
};
2411+
Ok(Some(
2412+
openshell_core::proto::compute::v1::SupervisorAppEndpoint {
2413+
host,
2414+
port: u32::from(self.config.sandbox_runtime.app_listener.common.port),
2415+
runtime_identity: kubernetes_runtime_identity(namespace, uid, &component),
2416+
pool_route,
2417+
ownership_epoch,
2418+
},
2419+
))
2420+
}
2421+
22272422
pub async fn get_sandbox(&self, sandbox_id: &str) -> Result<Option<Sandbox>, String> {
22282423
info!(
22292424
sandbox_id = %sandbox_id,

‎crates/openshell-driver-kubernetes/src/grpc.rs‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -337,6 +337,40 @@ impl ComputeDriver for ComputeDriverService {
337337
}
338338
}
339339

340+
#[tonic::async_trait]
341+
impl openshell_core::proto::compute::v1::compute_driver_app_endpoints_server::ComputeDriverAppEndpoints for ComputeDriverService {
342+
async fn resolve_supervisor_app_endpoint(
343+
&self,
344+
request: Request<openshell_core::proto::compute::v1::ResolveSupervisorAppEndpointRequest>,
345+
) -> Result<
346+
Response<openshell_core::proto::compute::v1::ResolveSupervisorAppEndpointResponse>,
347+
Status,
348+
> {
349+
self.rpc_tracer
350+
.trace(
351+
openshell_otel::rpc::RESOLVE_SUPERVISOR_APP_ENDPOINT,
352+
async {
353+
let request = request.into_inner();
354+
if request.sandbox_id.is_empty() {
355+
return Err(Status::invalid_argument("sandbox_id is required"));
356+
}
357+
let endpoint = self
358+
.driver
359+
.supervisor_app_endpoint(&request.sandbox_id)
360+
.await
361+
.map_err(|e| Status::unavailable(e.to_string()))?;
362+
Ok(Response::new(
363+
openshell_core::proto::compute::v1::ResolveSupervisorAppEndpointResponse {
364+
sandbox_id: request.sandbox_id,
365+
supervisor_app_endpoint: endpoint,
366+
},
367+
))
368+
},
369+
)
370+
.await
371+
}
372+
}
373+
340374
#[tonic::async_trait]
341375
impl openshell_core::compute::ComputeDriver for ComputeDriverService {
342376
async fn update_sandbox_configuration(

‎crates/openshell-driver-kubernetes/src/main.rs‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -327,6 +327,7 @@ async fn main() -> Result<()> {
327327

328328
let driver_service = ComputeDriverService::new(driver);
329329
let configuration_updates_service = openshell_core::proto::compute::v1::compute_driver_configuration_updates_server::ComputeDriverConfigurationUpdatesServer::new(driver_service.clone());
330+
let app_endpoints_service = openshell_core::proto::compute::v1::compute_driver_app_endpoints_server::ComputeDriverAppEndpointsServer::new(driver_service.clone());
330331
let service = ComputeDriverServer::new(driver_service);
331332
let shutdown = async move {
332333
shutdown_signal().await;
@@ -342,6 +343,7 @@ async fn main() -> Result<()> {
342343
.layer(openshell_otel::compute_driver_rpc_layer())
343344
.add_service(service)
344345
.add_service(configuration_updates_service)
346+
.add_service(app_endpoints_service)
345347
.serve_with_incoming_shutdown(
346348
openshell_core::external_driver_socket::SameUidUnixIncoming::new(listener),
347349
shutdown,
@@ -354,6 +356,7 @@ async fn main() -> Result<()> {
354356
.layer(openshell_otel::compute_driver_rpc_layer())
355357
.add_service(service)
356358
.add_service(configuration_updates_service)
359+
.add_service(app_endpoints_service)
357360
.serve_with_shutdown(args.bind_address, shutdown)
358361
.await
359362
.into_diagnostic()

‎crates/openshell-gateway/src/lib.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -319,6 +319,7 @@ impl openshell_server::ComputeDriverFactory for KubernetesFactory {
319319
let driver = std::sync::Arc::new(
320320
openshell_driver_kubernetes::ComputeDriverService::new_in_process(driver),
321321
);
322+
context.register_app_endpoints(driver.clone())?;
322323
Ok((
323324
openshell_server::ComputeDriverInstance::InProcess(driver.clone()),
324325
Some(driver),

‎crates/openshell-otel/src/grpc.rs‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,11 @@ pub mod rpc {
111111
"GetSandbox",
112112
"openshell.compute.v1.ComputeDriver/GetSandbox",
113113
);
114+
pub const RESOLVE_SUPERVISOR_APP_ENDPOINT: ComputeDriverRpc = ComputeDriverRpc {
115+
service: "openshell.compute.v1.ComputeDriverAppEndpoints",
116+
method: "ResolveSupervisorAppEndpoint",
117+
operation: "openshell.compute.v1.ComputeDriverAppEndpoints/ResolveSupervisorAppEndpoint",
118+
};
114119
pub const LIST_SANDBOXES: ComputeDriverRpc = ComputeDriverRpc::new(
115120
"ListSandboxes",
116121
"openshell.compute.v1.ComputeDriver/ListSandboxes",
@@ -201,6 +206,7 @@ pub fn compute_driver_rpc_operation(path: &str) -> Option<ComputeDriverRpc> {
201206
Some("CreateSandbox") => Some(rpc::CREATE_SANDBOX),
202207
Some("UpdateSandboxConfiguration") => Some(rpc::UPDATE_SANDBOX_CONFIGURATION),
203208
Some("GetSandbox") => Some(rpc::GET_SANDBOX),
209+
Some("ResolveSupervisorAppEndpoint") => Some(rpc::RESOLVE_SUPERVISOR_APP_ENDPOINT),
204210
Some("ListSandboxes") => Some(rpc::LIST_SANDBOXES),
205211
Some("StopSandbox") => Some(rpc::STOP_SANDBOX),
206212
Some("StartSandbox") => Some(rpc::START_SANDBOX),
@@ -325,6 +331,7 @@ mod tests {
325331
rpc::CREATE_SANDBOX,
326332
rpc::UPDATE_SANDBOX_CONFIGURATION,
327333
rpc::GET_SANDBOX,
334+
rpc::RESOLVE_SUPERVISOR_APP_ENDPOINT,
328335
rpc::LIST_SANDBOXES,
329336
rpc::STOP_SANDBOX,
330337
rpc::START_SANDBOX,

‎crates/openshell-server/README.md‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -233,3 +233,31 @@ runs under `mise run test:rust:postgres` (see `TESTING.md`) and still accepts
233233
`OPENSHELL_REFRESH_TEST_DATABASE_URL` pointing to a disposable PostgreSQL database.
234234
It exercises independent sessions without the local mutex. Do not point this
235235
fixture at an existing deployment database.
236+
237+
## Application route installation records
238+
239+
Supervisor installation reports arrive only through the authenticated sandbox
240+
session. The gateway issues bounded, single-use lease challenges and validates
241+
reports under the sandbox mutation guard. Accepted records contain current
242+
service declarations, configuration/runtime identity, session fencing, a
243+
trusted driver-resolved app endpoint, and gateway lease expiry. The private
244+
`ComputeDriverAppEndpoints.ResolveSupervisorAppEndpoint` extension RPC resolves this endpoint separately
245+
from `GetSandbox`: endpoint lookup failures do not affect ordinary sandbox
246+
observation. Unsupported drivers return `UNIMPLEMENTED`; an absent endpoint
247+
withholds application publication. The gateway resolves the endpoint once before
248+
issuing each challenge and binds that result to the session and lease sequence.
249+
Acceptance rechecks durable ownership, shared member/readiness authority, runtime,
250+
and configuration without another driver lookup. The next challenge resolves
251+
afresh; the saved result never extends the original lease expiry.
252+
In-process factories opt in with
253+
`ComputeDriverBuildContext::register_app_endpoints`; baseline factories need no
254+
changes. Standalone drivers register the extension on their existing endpoint.
255+
The generic
256+
store key is prefixed `service-installation:`; records are ephemeral and do not
257+
change durable sandbox protobufs. Reads revalidate owner, configuration,
258+
generation, and expiry. Cleanup scans bounded pages and uses conditional deletes
259+
so concurrent renewals are preserved. Cursor progress survives partial sweep
260+
timeouts; bounded record attempts prevent a malformed or stalled record from
261+
starving later cleanup. Failed attempts are revisited on the next scan.
262+
Discovery consumers must not treat these
263+
records as application health or external ingress readiness.

0 commit comments

Comments
 (0)