Skip to content

Commit 75a317e

Browse files
st-grelezar
andauthored
feat(server): support out-of-tree compute drivers via --compute-driver-socket (ai-openshell#1703)
* feat(server): support remote compute driver endpoints Add named remote compute driver endpoint support to the gateway. Remote drivers are selected by a non-reserved compute driver name and either a CLI/env socket endpoint or [openshell.drivers.<name>].socket_path. The VM driver now enters ComputeRuntime through the same acquired remote endpoint path, while Docker, Podman, and Kubernetes retain their in-process drivers. Require --drivers/OPENSHELL_DRIVERS when pairing an ad-hoc socket endpoint so the socket does not imply a magic driver name, and keep reserved in-tree names unavailable for unmanaged socket endpoints. Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: st-gr <38470677+st-gr@users.noreply.github.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> * test(server): cover remote compute driver UDS lifecycle Signed-off-by: Evan Lezar <elezar@nvidia.com> --------- Signed-off-by: st-gr <38470677+st-gr@users.noreply.github.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> Co-authored-by: Evan Lezar <elezar@nvidia.com>
1 parent f2ecadf commit 75a317e

10 files changed

Lines changed: 1082 additions & 117 deletions

File tree

‎architecture/compute-runtimes.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,8 @@ when a sandbox create request asks for GPU resources.
3333
| Docker | Local development with Docker available. | Container plus nested sandbox namespace. | Uses host networking so loopback gateway endpoints work from the supervisor. |
3434
| Podman | Rootless or single-machine deployments. | Container plus nested sandbox namespace. | Uses the Podman REST API, OCI image volumes, and CDI GPU devices when available. |
3535
| Kubernetes | Cluster deployment through Helm. | Pod plus nested sandbox namespace. | Uses Kubernetes API objects, service accounts, secrets, PVC-backed workspace storage, and GPU resources. |
36-
| VM | Experimental microVM isolation. | Per-sandbox libkrun VM. | Gateway spawns `openshell-driver-vm` as a subprocess over a private, state-local Unix socket. The VM driver boots a cached bootstrap `rootfs.ext4`, prepares requested OCI images inside a bootstrap VM with `umoci`, attaches the prepared image disk read-only, and gives each sandbox a writable `overlay.ext4` for merged-root changes and runtime material. The driver persists each accepted launch request beside the overlay and restarts those VMs on driver startup without recreating the overlay. |
36+
| VM | Experimental microVM isolation. | Per-sandbox libkrun VM. | Managed endpoint-backed driver. The gateway spawns `openshell-driver-vm`, waits for its Unix socket, and then consumes it through the same remote `compute_driver.proto` path used by unmanaged endpoint drivers. The VM driver boots a cached bootstrap `rootfs.ext4`, prepares requested OCI images inside a bootstrap VM with `umoci`, attaches the prepared image disk read-only, and gives each sandbox a writable `overlay.ext4` for merged-root changes and runtime material. The driver persists each accepted launch request beside the overlay and restarts those VMs on driver startup without recreating the overlay. |
37+
| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a non-reserved custom `compute_drivers = ["<name>"]` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--drivers <name>` with `--compute-driver-socket=<path>`. Reserved built-in names such as `vm`, `docker`, `podman`, and `kubernetes` cannot be used as unmanaged socket endpoints. The gateway connects to a UDS the operator already provisioned, runs `GetCapabilities`, logs the advertised `driver_name`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |
3738

3839
Per-sandbox CPU and memory values currently enter the driver layer through
3940
template resource limits. Docker and Podman apply them as runtime limits.
@@ -82,6 +83,7 @@ The supervisor must be available inside each sandbox workload:
8283
| Podman | Read-only OCI image volume containing the supervisor binary. |
8384
| Kubernetes | Sandbox pod image or pod template configuration. |
8485
| VM | Embedded in the guest rootfs bundle. |
86+
| Extension | Defined by the out-of-tree driver. |
8587

8688
Driver-controlled environment variables must override sandbox image or template
8789
values for sandbox ID, sandbox name, gateway endpoint, relay socket path, TLS

‎crates/openshell-core/src/config.rs‎

Lines changed: 64 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44
//! Configuration management for `OpenShell` components.
55
66
use serde::{Deserialize, Serialize};
7+
use std::collections::BTreeMap;
78
use std::fmt;
89
#[cfg(unix)]
910
use std::io::{Read, Write};
@@ -69,6 +70,27 @@ impl ComputeDriverKind {
6970
}
7071
}
7172

73+
/// Normalize a configured compute driver name.
74+
///
75+
/// Built-in driver names and custom remote driver names share the same
76+
/// selection namespace. The normalized value is lowercase ASCII and may contain
77+
/// letters, digits, `-`, and `_`.
78+
pub fn normalize_compute_driver_name(value: &str) -> Result<String, String> {
79+
let value = value.trim();
80+
if value.is_empty() {
81+
return Err("compute driver name cannot be empty".to_string());
82+
}
83+
if !value
84+
.bytes()
85+
.all(|b| b.is_ascii_alphanumeric() || matches!(b, b'-' | b'_'))
86+
{
87+
return Err(format!(
88+
"invalid compute driver name '{value}'. use ASCII letters, digits, '-' or '_'"
89+
));
90+
}
91+
Ok(value.to_ascii_lowercase())
92+
}
93+
7294
impl fmt::Display for ComputeDriverKind {
7395
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
7496
f.write_str(self.as_str())
@@ -358,7 +380,14 @@ pub struct Config {
358380
/// The config shape allows multiple drivers so the gateway can evolve
359381
/// toward multi-backend routing. Current releases require exactly one
360382
/// configured driver.
361-
pub compute_drivers: Vec<ComputeDriverKind>,
383+
pub compute_drivers: Vec<String>,
384+
385+
/// Operator-provided endpoints for named remote compute drivers.
386+
///
387+
/// This is populated by CLI/env inputs such as `--compute-driver-socket`.
388+
/// TOML-authored endpoints live under `[openshell.drivers.<name>]` and are
389+
/// resolved by the gateway config loader.
390+
pub compute_driver_endpoints: BTreeMap<String, PathBuf>,
362391

363392
/// TTL for SSH session tokens, in seconds. 0 disables expiry.
364393
pub ssh_session_ttl_secs: u64,
@@ -559,6 +588,7 @@ impl Config {
559588
gateway_jwt: None,
560589
database_url: String::new(),
561590
compute_drivers: vec![],
591+
compute_driver_endpoints: BTreeMap::new(),
562592
ssh_session_ttl_secs: default_ssh_session_ttl_secs(),
563593
grpc_rate_limit_requests: None,
564594
grpc_rate_limit_window_secs: None,
@@ -614,11 +644,27 @@ impl Config {
614644

615645
/// Create a new configuration with the configured compute drivers.
616646
#[must_use]
617-
pub fn with_compute_drivers<I>(mut self, drivers: I) -> Self
647+
pub fn with_compute_drivers<I, D>(mut self, drivers: I) -> Self
618648
where
619-
I: IntoIterator<Item = ComputeDriverKind>,
649+
I: IntoIterator<Item = D>,
650+
D: ToString,
620651
{
621-
self.compute_drivers = drivers.into_iter().collect();
652+
self.compute_drivers = drivers
653+
.into_iter()
654+
.map(|driver| driver.to_string())
655+
.collect();
656+
self
657+
}
658+
659+
/// Register a Unix domain socket endpoint for a named remote driver.
660+
#[must_use]
661+
pub fn with_compute_driver_endpoint(
662+
mut self,
663+
name: impl Into<String>,
664+
socket: impl Into<PathBuf>,
665+
) -> Self {
666+
self.compute_driver_endpoints
667+
.insert(name.into(), socket.into());
622668
self
623669
}
624670

@@ -766,8 +812,8 @@ mod tests {
766812
use super::is_reachable_unix_socket;
767813
use super::{
768814
ComputeDriverKind, Config, DEFAULT_SERVICE_ROUTING_DOMAIN, GatewayJwtConfig, detect_driver,
769-
docker_host_unix_socket_path, is_unix_socket, podman_socket_candidates_from_env,
770-
podman_socket_responds,
815+
docker_host_unix_socket_path, is_unix_socket, normalize_compute_driver_name,
816+
podman_socket_candidates_from_env, podman_socket_responds,
771817
};
772818
#[cfg(unix)]
773819
use std::io::{Read as _, Write as _};
@@ -803,6 +849,18 @@ mod tests {
803849
assert!(err.contains("unsupported compute driver 'firecracker'"));
804850
}
805851

852+
#[test]
853+
fn compute_driver_name_normalization_accepts_builtin_and_custom_names() {
854+
assert_eq!(normalize_compute_driver_name(" VM ").unwrap(), "vm");
855+
assert_eq!(
856+
normalize_compute_driver_name("Kyma_GPU-1").unwrap(),
857+
"kyma_gpu-1"
858+
);
859+
860+
let err = normalize_compute_driver_name("kyma/gpu").unwrap_err();
861+
assert!(err.contains("invalid compute driver name"));
862+
}
863+
806864
#[test]
807865
fn config_defaults_to_loopback_bind_address() {
808866
let expected: SocketAddr = "127.0.0.1:17670".parse().expect("valid address");

‎crates/openshell-server/src/cli.rs‎

Lines changed: 184 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -109,7 +109,17 @@ struct RunArgs {
109109
value_delimiter = ',',
110110
value_parser = parse_compute_driver
111111
)]
112-
drivers: Vec<ComputeDriverKind>,
112+
drivers: Vec<String>,
113+
114+
/// Path to a Unix domain socket served by a remote compute driver
115+
/// implementing `compute_driver.proto`.
116+
///
117+
/// When set, the socket is associated with the single driver name supplied
118+
/// by `--drivers` or `OPENSHELL_DRIVERS`. Reserved built-in driver names
119+
/// such as Docker, Podman, Kubernetes, and VM do not accept socket
120+
/// endpoints.
121+
#[arg(long, env = "OPENSHELL_COMPUTE_DRIVER_SOCKET")]
122+
compute_driver_socket: Option<PathBuf>,
113123

114124
/// Disable TLS entirely — listen on plaintext HTTP.
115125
/// Use this when the gateway sits behind a reverse proxy or tunnel
@@ -235,6 +245,7 @@ async fn run_from_args(mut args: RunArgs, matches: ArgMatches) -> Result<()> {
235245
if let Some(file) = file.as_ref() {
236246
merge_file_into_args(&mut args, &file.openshell.gateway, &matches);
237247
}
248+
normalize_compute_driver_socket_args(&mut args, &matches)?;
238249

239250
let local_tls = apply_runtime_defaults(&mut args)?;
240251
let local_jwt = defaults::complete_local_jwt_config()?;
@@ -371,6 +382,13 @@ async fn run_from_args(mut args: RunArgs, matches: ArgMatches) -> Result<()> {
371382
args.grpc_rate_limit_requests,
372383
args.grpc_rate_limit_window_seconds,
373384
)?;
385+
if let Some(socket) = args.compute_driver_socket.clone() {
386+
let driver = args
387+
.drivers
388+
.first()
389+
.expect("normalize_compute_driver_socket_args sets a driver for socket endpoints");
390+
config = config.with_compute_driver_endpoint(driver.clone(), socket);
391+
}
374392

375393
if let Some(ttl) = file
376394
.as_ref()
@@ -457,8 +475,8 @@ async fn run_from_args(mut args: RunArgs, matches: ArgMatches) -> Result<()> {
457475
.into_diagnostic()
458476
}
459477

460-
fn parse_compute_driver(value: &str) -> std::result::Result<ComputeDriverKind, String> {
461-
value.parse()
478+
fn parse_compute_driver(value: &str) -> std::result::Result<String, String> {
479+
openshell_core::config::normalize_compute_driver_name(value)
462480
}
463481

464482
fn resolve_config_path(args: &RunArgs) -> Result<Option<PathBuf>> {
@@ -657,10 +675,52 @@ fn validate_grpc_rate_limit_args(requests: Option<u64>, window_seconds: Option<u
657675
Ok(())
658676
}
659677

678+
fn normalize_compute_driver_socket_args(args: &mut RunArgs, matches: &ArgMatches) -> Result<()> {
679+
let Some(socket) = args.compute_driver_socket.as_ref() else {
680+
return Ok(());
681+
};
682+
if socket.as_os_str().is_empty() {
683+
return Err(miette::miette!(
684+
"--compute-driver-socket must not be an empty path"
685+
));
686+
}
687+
if arg_defaulted(matches, "drivers") {
688+
return Err(miette::miette!(
689+
"--compute-driver-socket requires --drivers <name> or OPENSHELL_DRIVERS=<name> to select a non-reserved compute driver name"
690+
));
691+
}
692+
693+
match args.drivers.as_slice() {
694+
[driver] => {
695+
let driver = openshell_core::config::normalize_compute_driver_name(driver)
696+
.map_err(|err| miette::miette!("{err}"))?;
697+
if matches!(
698+
driver.parse::<ComputeDriverKind>().ok(),
699+
Some(
700+
ComputeDriverKind::Docker
701+
| ComputeDriverKind::Podman
702+
| ComputeDriverKind::Kubernetes
703+
| ComputeDriverKind::Vm
704+
)
705+
) {
706+
return Err(miette::miette!(
707+
"--compute-driver-socket cannot be combined with reserved built-in compute driver '{driver}'"
708+
));
709+
}
710+
args.drivers[0] = driver;
711+
Ok(())
712+
}
713+
drivers => Err(miette::miette!(
714+
"--compute-driver-socket requires exactly one compute driver name, got: {}",
715+
drivers.join(",")
716+
)),
717+
}
718+
}
719+
660720
fn effective_single_driver(args: &RunArgs) -> Option<ComputeDriverKind> {
661721
match args.drivers.as_slice() {
662722
[] => openshell_core::config::detect_driver(),
663-
[driver] => Some(*driver),
723+
[driver] => driver.parse().ok(),
664724
_ => None,
665725
}
666726
}
@@ -1561,6 +1621,126 @@ ssh_session_ttl_secs = 1234
15611621
assert!(!super::is_singleplayer_driver(&multi));
15621622
}
15631623

1624+
#[test]
1625+
fn compute_driver_socket_flag_uses_explicit_driver_name() {
1626+
let _lock = ENV_LOCK
1627+
.lock()
1628+
.unwrap_or_else(std::sync::PoisonError::into_inner);
1629+
let _g1 = EnvVarGuard::remove("OPENSHELL_COMPUTE_DRIVER_SOCKET");
1630+
let _g2 = EnvVarGuard::remove("OPENSHELL_DRIVERS");
1631+
1632+
let (mut args, matches) = parse_with_args(&[
1633+
"openshell-gateway",
1634+
"--db-url",
1635+
"sqlite::memory:",
1636+
"--drivers",
1637+
"Kyma",
1638+
"--compute-driver-socket",
1639+
"/run/openshell/kyma.sock",
1640+
]);
1641+
super::normalize_compute_driver_socket_args(&mut args, &matches).unwrap();
1642+
assert_eq!(
1643+
args.compute_driver_socket.as_deref(),
1644+
Some(std::path::Path::new("/run/openshell/kyma.sock"))
1645+
);
1646+
assert_eq!(args.drivers, ["kyma"]);
1647+
assert!(super::effective_single_driver(&args).is_none());
1648+
}
1649+
1650+
#[test]
1651+
fn compute_driver_socket_requires_explicit_driver_name() {
1652+
let _lock = ENV_LOCK
1653+
.lock()
1654+
.unwrap_or_else(std::sync::PoisonError::into_inner);
1655+
let _g1 = EnvVarGuard::remove("OPENSHELL_COMPUTE_DRIVER_SOCKET");
1656+
let _g2 = EnvVarGuard::remove("OPENSHELL_DRIVERS");
1657+
1658+
let (mut args, matches) = parse_with_args(&[
1659+
"openshell-gateway",
1660+
"--db-url",
1661+
"sqlite::memory:",
1662+
"--compute-driver-socket",
1663+
"/run/openshell/kyma.sock",
1664+
]);
1665+
let err = super::normalize_compute_driver_socket_args(&mut args, &matches).unwrap_err();
1666+
1667+
assert!(
1668+
err.to_string().contains("requires --drivers <name>"),
1669+
"unexpected error: {err}"
1670+
);
1671+
}
1672+
1673+
#[test]
1674+
fn compute_driver_socket_rejects_reserved_builtin_drivers() {
1675+
let _lock = ENV_LOCK
1676+
.lock()
1677+
.unwrap_or_else(std::sync::PoisonError::into_inner);
1678+
let _g1 = EnvVarGuard::remove("OPENSHELL_COMPUTE_DRIVER_SOCKET");
1679+
let _g2 = EnvVarGuard::remove("OPENSHELL_DRIVERS");
1680+
1681+
let (mut args, matches) = parse_with_args(&[
1682+
"openshell-gateway",
1683+
"--db-url",
1684+
"sqlite::memory:",
1685+
"--drivers",
1686+
"docker",
1687+
"--compute-driver-socket",
1688+
"/run/openshell/extension.sock",
1689+
]);
1690+
let err = super::normalize_compute_driver_socket_args(&mut args, &matches).unwrap_err();
1691+
assert!(
1692+
err.to_string()
1693+
.contains("cannot be combined with reserved built-in compute driver 'docker'"),
1694+
"unexpected error: {err}"
1695+
);
1696+
}
1697+
1698+
#[test]
1699+
fn compute_driver_socket_rejects_vm_endpoint() {
1700+
let _lock = ENV_LOCK
1701+
.lock()
1702+
.unwrap_or_else(std::sync::PoisonError::into_inner);
1703+
let _g1 = EnvVarGuard::remove("OPENSHELL_COMPUTE_DRIVER_SOCKET");
1704+
let _g2 = EnvVarGuard::remove("OPENSHELL_DRIVERS");
1705+
1706+
let (mut args, matches) = parse_with_args(&[
1707+
"openshell-gateway",
1708+
"--db-url",
1709+
"sqlite::memory:",
1710+
"--drivers",
1711+
"vm",
1712+
"--compute-driver-socket",
1713+
"/run/openshell/vm.sock",
1714+
]);
1715+
let err = super::normalize_compute_driver_socket_args(&mut args, &matches).unwrap_err();
1716+
assert!(
1717+
err.to_string()
1718+
.contains("cannot be combined with reserved built-in compute driver 'vm'"),
1719+
"unexpected error: {err}"
1720+
);
1721+
}
1722+
1723+
#[test]
1724+
fn compute_driver_socket_reads_from_env_var() {
1725+
let _lock = ENV_LOCK
1726+
.lock()
1727+
.unwrap_or_else(std::sync::PoisonError::into_inner);
1728+
let _g1 = EnvVarGuard::set(
1729+
"OPENSHELL_COMPUTE_DRIVER_SOCKET",
1730+
"/var/run/openshell/kyma.sock",
1731+
);
1732+
let _g2 = EnvVarGuard::set("OPENSHELL_DRIVERS", "kyma");
1733+
1734+
let (mut args, matches) =
1735+
parse_with_args(&["openshell-gateway", "--db-url", "sqlite::memory:"]);
1736+
super::normalize_compute_driver_socket_args(&mut args, &matches).unwrap();
1737+
assert_eq!(
1738+
args.compute_driver_socket.as_deref(),
1739+
Some(std::path::Path::new("/var/run/openshell/kyma.sock"))
1740+
);
1741+
assert_eq!(args.drivers, ["kyma"]);
1742+
}
1743+
15641744
#[test]
15651745
fn file_populates_service_routing_fields() {
15661746
let _lock = ENV_LOCK

0 commit comments

Comments
 (0)