Skip to content

Commit 2adddaa

Browse files
feat: Adding qemu vm driver support with GPU pass-through (ai-openshell#992)
* Adding qemu vm driver support with GPU pass-through Signed-off-by: Vincent Caux-Brisebois <vcauxbrisebo@nvidia.com> * Add GPU rootfs variant, harden VFIO binding, and fix networking and supervisor reliability issues discovered during GPU VM bring-up. Signed-off-by: Vincent Caux-Brisebois <vcauxbrisebo@nvidia.com> --------- Signed-off-by: Vincent Caux-Brisebois <vcauxbrisebo@nvidia.com>
1 parent c0ffa93 commit 2adddaa

30 files changed

Lines changed: 3801 additions & 142 deletions

File tree

‎Cargo.lock‎

Lines changed: 14 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎architecture/podman-rootless-networking.md‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ For rootful bridge networking:
3535
6. Netavark configures iptables/nftables rules -- masquerade for outbound, DNAT for port mappings
3636
7. Netavark starts aardvark-dns if DNS is enabled, listening on the bridge gateway address
3737

38-
```
38+
```text
3939
Host Kernel
4040
|
4141
+-- Bridge interface (e.g., "podman0") <-- created by Netavark
@@ -60,7 +60,7 @@ Unprivileged users cannot create network interfaces on the host. They cannot cre
6060

6161
Pasta (part of the `passt` project -- same binary, different command name) operates entirely in userspace, translating between the container's L2 TAP interface and the host's L4 sockets. It requires no capabilities or privileges.
6262

63-
```
63+
```text
6464
Container Network Namespace
6565
|
6666
+-- TAP device (e.g., "eth0")
@@ -131,7 +131,7 @@ Unlike bridge networking, pasta containers are isolated from each other by defau
131131

132132
The Podman compute driver creates three layers of network isolation:
133133

134-
```
134+
```text
135135
Namespace 1: Host
136136
|
137137
pasta manages port forwarding (127.0.0.1:<ephemeral>)
@@ -164,7 +164,7 @@ client.ensure_network(&config.network_name).await?;
164164

165165
This creates a bridge network named `"openshell"` (default from `DEFAULT_NETWORK_NAME` in `openshell-core/src/config.rs`) with `dns_enabled: true`. In rootless mode, this bridge exists inside a user namespace managed by pasta. The bridge IP range (e.g., `10.89.x.x`) is not routable from the host.
166166

167-
```
167+
```text
168168
Host (your machine)
169169
|
170170
127.0.0.1:<ephemeral> <--- pasta binds this on the host
@@ -212,7 +212,7 @@ The bridge gateway IP does NOT work for this purpose in rootless mode because it
212212

213213
Inside the container, the supervisor creates another network namespace (`netns.rs:53-178`, setup at lines 53-63, `ip netns add` at line 77) for the user workload:
214214

215-
```
215+
```text
216216
Container (10.89.1.2 on the Podman bridge)
217217
|
218218
[Supervisor process - runs in container's default netns]
@@ -247,7 +247,7 @@ A tmpfs is mounted at `/run/netns` in the container spec (`container.rs:458-463`
247247

248248
### SSH Session: Client to Sandbox Shell
249249

250-
```
250+
```text
251251
Client (CLI on user's machine)
252252
|
253253
1. gRPC: CreateSshSession -> gateway (returns token, connect_path)
@@ -281,7 +281,7 @@ The SSH daemon listens on a Unix socket (not a TCP port) with 0600 permissions.
281281

282282
### Outbound HTTP Request from Sandbox Process
283283

284-
```
284+
```text
285285
User's code (inner netns, 10.200.0.2)
286286
|
287287
1. curl https://api.example.com
@@ -306,7 +306,7 @@ Supervisor proxy (10.200.0.1:3128 in container netns)
306306

307307
### Supervisor gRPC Callback to Gateway
308308

309-
```
309+
```text
310310
Supervisor (container netns, 10.89.x.2)
311311
|
312312
1. gRPC connect to http://host.containers.internal:8080

‎crates/openshell-cli/src/main.rs‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1138,6 +1138,11 @@ enum SandboxCommands {
11381138
#[arg(long)]
11391139
gpu: bool,
11401140

1141+
/// Target a specific GPU by PCI address (e.g. "0000:2d:00.0") or index (e.g. "0", "1").
1142+
/// Only valid with --gpu. When omitted with --gpu, the first available GPU is assigned.
1143+
#[arg(long, requires = "gpu")]
1144+
gpu_device: Option<String>,
1145+
11411146
/// SSH destination for remote bootstrap (e.g., user@hostname).
11421147
/// Only used when no cluster exists yet; ignored if a cluster is
11431148
/// already active.
@@ -2307,6 +2312,7 @@ async fn main() -> Result<()> {
23072312
no_keep,
23082313
editor,
23092314
gpu,
2315+
gpu_device,
23102316
remote,
23112317
ssh_key,
23122318
providers,
@@ -2402,6 +2408,7 @@ async fn main() -> Result<()> {
24022408
upload_spec.as_ref(),
24032409
keep,
24042410
gpu,
2411+
gpu_device.as_deref(),
24052412
editor,
24062413
remote.as_deref(),
24072414
ssh_key.as_deref(),
@@ -2425,6 +2432,7 @@ async fn main() -> Result<()> {
24252432
upload_spec.as_ref(),
24262433
keep,
24272434
gpu,
2435+
gpu_device.as_deref(),
24282436
editor,
24292437
remote.as_deref(),
24302438
ssh_key.as_deref(),

‎crates/openshell-cli/src/run.rs‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1923,6 +1923,7 @@ pub async fn sandbox_create_with_bootstrap(
19231923
upload: Option<&(String, Option<String>, bool)>,
19241924
keep: bool,
19251925
gpu: bool,
1926+
gpu_device: Option<&str>,
19261927
editor: Option<Editor>,
19271928
remote: Option<&str>,
19281929
ssh_key: Option<&str>,
@@ -1954,6 +1955,7 @@ pub async fn sandbox_create_with_bootstrap(
19541955
upload,
19551956
keep,
19561957
gpu,
1958+
gpu_device,
19571959
editor,
19581960
remote,
19591961
ssh_key,
@@ -2010,6 +2012,7 @@ pub async fn sandbox_create(
20102012
upload: Option<&(String, Option<String>, bool)>,
20112013
keep: bool,
20122014
gpu: bool,
2015+
gpu_device: Option<&str>,
20132016
editor: Option<Editor>,
20142017
remote: Option<&str>,
20152018
ssh_key: Option<&str>,
@@ -2117,6 +2120,7 @@ pub async fn sandbox_create(
21172120
let request = CreateSandboxRequest {
21182121
spec: Some(SandboxSpec {
21192122
gpu: requested_gpu,
2123+
gpu_device: gpu_device.unwrap_or_default().to_string(),
21202124
policy,
21212125
providers: configured_providers,
21222126
template,

‎crates/openshell-cli/tests/sandbox_create_lifecycle_integration.rs‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -574,6 +574,7 @@ async fn sandbox_create_keeps_command_sessions_by_default() {
574574
None,
575575
None,
576576
None,
577+
None,
577578
&[],
578579
None,
579580
None,
@@ -615,6 +616,7 @@ async fn sandbox_create_deletes_command_sessions_with_no_keep() {
615616
None,
616617
None,
617618
None,
619+
None,
618620
&[],
619621
None,
620622
None,
@@ -659,6 +661,7 @@ async fn sandbox_create_deletes_shell_sessions_with_no_keep() {
659661
None,
660662
None,
661663
None,
664+
None,
662665
&[],
663666
None,
664667
None,
@@ -703,6 +706,7 @@ async fn sandbox_create_keeps_sandbox_with_hidden_keep_flag() {
703706
None,
704707
None,
705708
None,
709+
None,
706710
&[],
707711
None,
708712
None,
@@ -744,6 +748,7 @@ async fn sandbox_create_keeps_sandbox_with_forwarding() {
744748
None,
745749
None,
746750
None,
751+
None,
747752
&[],
748753
None,
749754
Some(openshell_core::forward::ForwardSpec::new(8080)),

‎crates/openshell-driver-docker/src/lib.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -231,6 +231,7 @@ impl DockerComputeDriver {
231231
driver_version: self.config.daemon_version.clone(),
232232
default_image: self.config.default_image.clone(),
233233
supports_gpu: false,
234+
gpu_count: 0,
234235
}
235236
}
236237

‎crates/openshell-driver-docker/src/tests.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ fn test_sandbox() -> DriverSandbox {
3030
platform_config: None,
3131
}),
3232
gpu: false,
33+
gpu_device: String::new(),
3334
}),
3435
status: None,
3536
}

‎crates/openshell-driver-kubernetes/src/driver.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -150,6 +150,7 @@ impl KubernetesComputeDriver {
150150
driver_version: openshell_core::VERSION.to_string(),
151151
default_image: self.config.default_image.clone(),
152152
supports_gpu: self.has_gpu_capacity().await.unwrap_or(false),
153+
gpu_count: 0,
153154
})
154155
}
155156

‎crates/openshell-driver-podman/src/driver.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -161,6 +161,7 @@ impl PodmanComputeDriver {
161161
driver_version: openshell_core::VERSION.to_string(),
162162
default_image: self.config.default_image.clone(),
163163
supports_gpu,
164+
gpu_count: 0,
164165
})
165166
}
166167

‎crates/openshell-driver-vm/Cargo.toml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ path = "src/main.rs"
2020

2121
[dependencies]
2222
openshell-core = { path = "../openshell-core" }
23+
openshell-vfio = { path = "../openshell-vfio" }
2324

2425
tokio = { workspace = true }
2526
tonic = { workspace = true, features = ["transport"] }
@@ -32,6 +33,8 @@ tracing = { workspace = true }
3233
tracing-subscriber = { workspace = true }
3334
miette = { workspace = true }
3435
url = { workspace = true }
36+
serde = { workspace = true }
37+
serde_json = { workspace = true }
3538
libc = "0.2"
3639
libloading = "0.8"
3740
tar = "0.4"

0 commit comments

Comments
 (0)