fix(macos): restore Homebrew sandbox callbacks (#2739)

* fix(macos): restore Docker gateway callbacks

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(gateway): reuse reachable primary callback listener

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

---------

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

Author: drew

.agents/skills/debug-openshell-cluster/SKILL.md

@@ -178,6 +178,13 @@ Common findings:
 - A workdir rejected as a special filesystem or OpenShell control-path collision cannot be made valid with permissions. Move the image workdir away from kernel-backed mounts and the concrete supervisor, TLS, token, runtime, and socket paths named in the error.
 - Docker driver cannot initialize because it cannot find `openshell-sandbox`: verify `OPENSHELL_DOCKER_SUPERVISOR_BIN`, the sibling binary next to `openshell-gateway`, or the configured supervisor image contains `/openshell-sandbox`.
 - Sandbox never registers: check gateway logs and supervisor callback endpoint.
+- On macOS, repeated `Policy fetch failed after 5 attempts` messages with a
+  Homebrew gateway bound to `[::1]:17670` indicate that the Docker
+  `host-gateway` IPv4 route has no matching callback listener. Current releases
+  leave `bind_address` unset in the Homebrew config, use the built-in
+  `127.0.0.1:17670` primary listener, and reuse it for authenticated sandbox
+  callbacks. On an older release, set `bind_address = "127.0.0.1:17670"` or
+  upgrade.
 - Supervisor image exits before printing `openshell-sandbox --version`: the image should be the scratch supervisor image from `deploy/docker/Dockerfile.supervisor` and must contain a static executable at `/openshell-sandbox`.
 - `mise run e2e:docker:gpu` fails with `docker info --format json did not report any discovered NVIDIA CDI GPU devices`: Docker may report `CDISpecDirs` while still having no generated NVIDIA CDI specs. Verify `.DiscoveredDevices` contains entries such as `nvidia.com/gpu=all`, verify `/etc/cdi` or `/var/run/cdi` contains a generated NVIDIA spec, and check that `nvidia-cdi-refresh.service` and `nvidia-cdi-refresh.path` from NVIDIA Container Toolkit are enabled and healthy. The service is a one-shot unit, so `inactive (dead)` can be normal after a successful run; use `systemctl status` and `journalctl` to distinguish success from a skipped or failed refresh. NVIDIA recommends enabling the path and service units, and restarting `nvidia-cdi-refresh.service` to regenerate missing or stale CDI specs. If specs are generated but Docker still reports no discovered devices, restart Docker or reload the daemon and re-check `docker info`.
 
@@ -207,14 +214,9 @@ Common findings:
   error: inspect `podman info --debug`, the configured Podman network, and the
   host's IPv4 default route. Rootless pasta uses the private source address
   selected by that route; rootful Podman uses the bridge gateway address.
-- Callback discovery reports that the requested address equals the primary
-  listener: configure a distinct primary address. For Podman Machine, bind the
-  primary listener to IPv6 loopback, for example
-  `bind_address = "[::1]:17670"`, and register the CLI endpoint as
-  `https://localhost:17670`. The generated certificate includes `localhost`,
-  while a raw `https://[::1]:17670` endpoint can fail TLS setup with
-  `invalid dns name`. This leaves `127.0.0.1:17670` available for the
-  callback-only listener.
+- Current gateways reuse the primary listener when it covers Podman's callback
+  address. If the primary does not cover that address, inspect the gateway
+  startup logs for the additional callback-only listener and its provenance.
 - Rootless slirp4netns, another named helper, or missing helper metadata
   requires an explicitly remote `grpc_endpoint`. An explicit `host_gateway_ip`
   cannot bypass slirp4netns host-loopback isolation. Do not work around
@@ -544,7 +546,7 @@ openshell logs <sandbox-name>
 | `openshell status` fails | Gateway endpoint unreachable or auth mismatch | `openshell gateway info`, gateway logs |
 | Gateway starts but sandbox create fails | Compute driver cannot reach runtime | Docker/Podman/Kubernetes/VM driver logs |
 | Gateway exits while resolving compute-driver listener requirements | Callback alias topology is unsupported, the Podman network cannot be inspected, or the selected address is not private/authorized | Gateway startup error, `podman info --debug`, Podman network inspection, host IPv4 default route |
-| Admin, health, reflection, or HTTP request is denied on a Docker/Podman callback address | Negotiated callback listeners intentionally expose only sandbox-callable gRPC methods | Retry through the gateway's primary endpoint; inspect the listener-purpose startup log if the address was unexpected |
+| Admin, health, reflection, or HTTP request is denied on an additional Docker/Podman callback-only listener | Additional callback listeners intentionally expose only sandbox-callable gRPC methods | Retry through the gateway's primary endpoint; inspect the listener-purpose startup log if the address was unexpected |
 | Docker or Podman sandbox never registers | Wrong callback endpoint or supervisor startup failure | Gateway logs and sandbox container logs |
 | Docker GPU e2e fails before GPU sandbox comparison | NVIDIA CDI specs are missing or Docker has not discovered them | `docker info --format '{{json .DiscoveredDevices}}'`, `/etc/cdi`, `/var/run/cdi`, `nvidia-cdi-refresh.service` |
 | Kubernetes gateway pod pending | PVC unbound, taint, selector, or insufficient resources | `kubectl -n openshell describe pod <pod>` |

architecture/gateway.md

@@ -37,19 +37,17 @@ health, metrics, or tunnel routes. The plaintext service router also rejects
 browser requests whose Fetch Metadata, Origin, or Referer headers indicate a
 cross-origin or sibling-subdomain request.
 
-Docker and Podman may negotiate additional listeners that make the gateway
-reachable from their local sandbox network topology. Those listeners accept
+Docker and Podman report the local address through which their sandboxes can
+reach the gateway. When the primary listener covers that address, the gateway
+reuses it; sandbox JWT authentication and its RPC allowlist remain the callback
+authorization boundary. When the primary listener does not cover the address,
+the gateway adds a callback-only listener. Additional callback listeners accept
 only gRPC methods classified as sandbox-callable by the gateway's generated
 authorization metadata. They reject user and administrator APIs, health,
 reflection, non-callback inference APIs, and HTTP routes before normal request
 authentication. The operator-configured primary listener retains the full
 multiplexed API surface.
 
-The gateway rejects a callback requirement that resolves to the exact primary
-listener address because one socket cannot preserve two authorization scopes.
-A wildcard primary listener may cover a callback address because the accepted
-connection's concrete local address still selects the callback-only scope.
-
 The `rpc_auth` classification is also the source of truth for negotiated
 listener exposure: marking an RPC as `sandbox` or `dual` makes it callable on
 these listeners. Review such changes as both authorization and network-surface
@@ -699,8 +697,10 @@ system entry instead of pretending to delete package-manager owned state.
 - Compute runtimes own the mechanics of starting workloads and injecting
   callback configuration.
 - Docker-backed local gateways use Docker's `host-gateway` callback alias on
-  macOS and Docker Desktop-style runtimes. Native Linux Docker may expose an
-  additional bridge-gateway listener because the host can bind that bridge IP.
+  macOS and Docker Desktop-style runtimes. They request IPv4 loopback callback
+  reachability and add a listener only when the primary does not cover it.
+  Native Linux Docker may expose an additional bridge-gateway listener because
+  the host can bind that bridge IP.
 - Podman-backed macOS gateways use gvproxy's host-loopback IP for sandbox host
   aliases by default so stale Podman machine images do not need Podman's
   `host-gateway` resolver. Linux Podman keeps the resolver unless

crates/openshell-cli/src/tls.rs

@@ -22,6 +22,7 @@ use tokio::sync::Mutex;
 use tonic::service::interceptor::InterceptedService;
 use tonic::transport::{Certificate, Channel, ClientTlsConfig, Endpoint, Identity};
 use tracing::debug;
+use url::{Host, Url};
 
 /// Concrete gRPC client type used by all commands.
 pub type GrpcClient = OpenShellClient<InterceptedService<Channel, EdgeAuthInterceptor>>;
@@ -223,12 +224,33 @@ pub fn build_rustls_config(materials: &TlsMaterials) -> Result<rustls::ClientCon
         .into_diagnostic()
 }
 
-pub fn build_tonic_tls_config(materials: &TlsMaterials) -> ClientTlsConfig {
+pub fn build_tonic_tls_config(server: &str, materials: &TlsMaterials) -> Result<ClientTlsConfig> {
     let ca_cert = Certificate::from_pem(materials.ca.clone());
     let identity = Identity::from_pem(materials.cert.clone(), materials.key.clone());
-    ClientTlsConfig::new()
-        .ca_certificate(ca_cert)
-        .identity(identity)
+    tls_config_with_server_name(
+        server,
+        ClientTlsConfig::new()
+            .ca_certificate(ca_cert)
+            .identity(identity),
+    )
+}
+
+fn tls_config_with_server_name(server: &str, config: ClientTlsConfig) -> Result<ClientTlsConfig> {
+    Ok(config.domain_name(tls_server_name(server)?))
+}
+
+fn tls_server_name(server: &str) -> Result<String> {
+    let endpoint = Url::parse(server)
+        .into_diagnostic()
+        .wrap_err_with(|| format!("invalid gateway endpoint '{server}'"))?;
+    let host = endpoint
+        .host()
+        .ok_or_else(|| miette::miette!("gateway endpoint '{server}' has no host"))?;
+    Ok(match host {
+        Host::Domain(domain) => domain.to_string(),
+        Host::Ipv4(address) => address.to_string(),
+        Host::Ipv6(address) => address.to_string(),
+    })
 }
 
 #[derive(Debug)]
@@ -402,30 +424,28 @@ pub async fn build_channel(server: &str, tls: &TlsOptions) -> Result<Channel> {
         // Bearer auth over HTTPS: use mTLS certs for the transport layer when
         // available (server may still require client certs), and layer the
         // Bearer token on top via the interceptor.
-        require_tls_materials(server, tls).map_or_else(
-            |_| {
-                let resolved = tls.with_default_paths(server);
-                resolved
-                    .ca
-                    .as_ref()
-                    .and_then(|ca_path| std::fs::read(ca_path).ok())
-                    .map_or_else(
-                        || ClientTlsConfig::new().with_enabled_roots(),
-                        |ca_pem| {
-                            ClientTlsConfig::new().ca_certificate(Certificate::from_pem(ca_pem))
-                        },
-                    )
-            },
-            |materials| build_tonic_tls_config(&materials),
-        )
+        if let Ok(materials) = require_tls_materials(server, tls) {
+            build_tonic_tls_config(server, &materials)?
+        } else {
+            let resolved = tls.with_default_paths(server);
+            let config = resolved
+                .ca
+                .as_ref()
+                .and_then(|ca_path| std::fs::read(ca_path).ok())
+                .map_or_else(
+                    || ClientTlsConfig::new().with_enabled_roots(),
+                    |ca_pem| ClientTlsConfig::new().ca_certificate(Certificate::from_pem(ca_pem)),
+                );
+            tls_config_with_server_name(server, config)?
+        }
     } else if tls.edge_token.is_some() {
         // Edge bearer mode — routed through tunnel above; if we reach here
         // the server is not HTTPS so connect plaintext.
         return endpoint.connect().await.into_diagnostic();
     } else {
         // Standard mTLS: private CA + client cert.
         let materials = require_tls_materials(server, tls)?;
-        build_tonic_tls_config(&materials)
+        build_tonic_tls_config(server, &materials)?
     };
     endpoint = endpoint.tls_config(tls_config).into_diagnostic()?;
     endpoint.connect().await.into_diagnostic()
@@ -451,3 +471,25 @@ pub async fn grpc_inference_client(server: &str, tls: &TlsOptions) -> Result<Grp
     let interceptor = interceptor_from_tls(tls)?;
     Ok(InferenceClient::with_interceptor(channel, interceptor))
 }
+
+#[cfg(test)]
+mod tests {
+    use super::tls_server_name;
+
+    #[test]
+    fn tls_server_name_normalizes_bracketed_ipv6_endpoint() {
+        assert_eq!(
+            tls_server_name("https://[::1]:17670").unwrap(),
+            "::1",
+            "rustls accepts an unbracketed IPv6 address as an IP server name"
+        );
+    }
+
+    #[test]
+    fn tls_server_name_preserves_dns_endpoint() {
+        assert_eq!(
+            tls_server_name("https://localhost:17670").unwrap(),
+            "localhost"
+        );
+    }
+}

crates/openshell-driver-docker/README.md

@@ -67,8 +67,10 @@ Docker containers join an OpenShell-managed bridge network. The driver injects
 `host.openshell.internal` and `host.docker.internal` so supervisors have stable
 names for reaching the gateway host. On Docker Desktop, Colima, Rancher
 Desktop, OrbStack, and macOS-hosted gateways, those names use Docker's
-`host-gateway` alias. On native Linux Docker, the gateway also binds the bridge
-gateway IP so containers can call back to the host process.
+`host-gateway` alias. The driver requests a separate IPv4 loopback callback
+listener when the primary listener does not already cover it. On native Linux
+Docker, the gateway also binds the bridge gateway IP so containers can call
+back to the host process.
 
 ## Container Contract
 

crates/openshell-driver-docker/src/lib.rs

@@ -55,7 +55,7 @@ use openshell_core::proto_struct::{
 use openshell_core::{Config, Error, Result as CoreResult};
 use std::collections::{HashMap, HashSet};
 use std::io::Read;
-use std::net::{IpAddr, SocketAddr};
+use std::net::{IpAddr, Ipv4Addr, SocketAddr};
 use std::path::{Path, PathBuf};
 use std::pin::Pin;
 use std::sync::Arc;
@@ -177,6 +177,7 @@ struct DockerDriverRuntimeConfig {
     grpc_endpoint: String,
     network_name: String,
     gateway_route: DockerGatewayRoute,
+    gateway_callback_bind_address: Option<SocketAddr>,
     ssh_socket_path: String,
     stop_timeout_secs: u32,
     log_level: String,
@@ -428,6 +429,8 @@ impl DockerComputeDriver {
         let host_gateway_ip = parse_optional_host_gateway_ip(&docker_config.host_gateway_ip)?;
         let gateway_route =
             docker_gateway_route(&info, bridge_gateway_ip, gateway_port, host_gateway_ip);
+        let gateway_callback_bind_address =
+            docker_gateway_callback_bind_address(&gateway_route, config.bind_address);
         let mut docker_config = docker_config.clone();
         if docker_config.grpc_endpoint.trim().is_empty() {
             let scheme = if docker_guest_tls_configured(&docker_config) {
@@ -456,6 +459,7 @@ impl DockerComputeDriver {
                 grpc_endpoint,
                 network_name,
                 gateway_route,
+                gateway_callback_bind_address,
                 ssh_socket_path: docker_config.ssh_socket_path.clone(),
                 stop_timeout_secs: DEFAULT_STOP_TIMEOUT_SECS,
                 log_level: config.log_level.clone(),
@@ -1577,15 +1581,19 @@ impl ComputeDriver for DockerComputeDriver {
         &self,
         _request: Request<GetGatewayListenerRequirementsRequest>,
     ) -> Result<Response<GetGatewayListenerRequirementsResponse>, Status> {
-        let requirements = match self.config.gateway_route {
-            DockerGatewayRoute::Bridge { bind_address, .. } => {
-                vec![GatewayListenerRequirement {
-                    reason: "docker managed bridge gateway".to_string(),
-                    selector: Some(Selector::ExactBindAddress(bind_address.to_string())),
-                }]
-            }
-            DockerGatewayRoute::HostGateway => Vec::new(),
-        };
+        let requirements =
+            self.config
+                .gateway_callback_bind_address
+                .map_or_else(Vec::new, |bind_address| {
+                    vec![GatewayListenerRequirement {
+                        reason: match self.config.gateway_route {
+                            DockerGatewayRoute::Bridge { .. } => "docker managed bridge gateway",
+                            DockerGatewayRoute::HostGateway => "docker host-gateway IPv4 loopback",
+                        }
+                        .to_string(),
+                        selector: Some(Selector::ExactBindAddress(bind_address.to_string())),
+                    }]
+                });
         Ok(Response::new(GetGatewayListenerRequirementsResponse {
             requirements,
         }))
@@ -2793,6 +2801,22 @@ fn docker_gateway_route_for_host(
     }
 }
 
+fn docker_gateway_callback_bind_address(
+    route: &DockerGatewayRoute,
+    primary_bind_address: SocketAddr,
+) -> Option<SocketAddr> {
+    match route {
+        DockerGatewayRoute::Bridge { bind_address, .. } => Some(*bind_address),
+        DockerGatewayRoute::HostGateway => match primary_bind_address.ip() {
+            IpAddr::V4(ip) if ip.is_unspecified() || ip == Ipv4Addr::LOCALHOST => None,
+            _ => Some(SocketAddr::new(
+                IpAddr::V4(Ipv4Addr::LOCALHOST),
+                primary_bind_address.port(),
+            )),
+        },
+    }
+}
+
 fn host_runtime_requires_host_gateway_alias() -> bool {
     cfg!(target_os = "macos")
 }

crates/openshell-driver-docker/src/tests.rs

@@ -102,6 +102,10 @@ fn runtime_config() -> DockerDriverRuntimeConfig {
             ),
             host_alias_ip: IpAddr::V4(Ipv4Addr::new(172, 18, 0, 1)),
         },
+        gateway_callback_bind_address: Some(SocketAddr::new(
+            IpAddr::V4(Ipv4Addr::new(172, 18, 0, 1)),
+            DEFAULT_SERVER_PORT,
+        )),
         ssh_socket_path: "/run/openshell/ssh.sock".to_string(),
         stop_timeout_secs: DEFAULT_STOP_TIMEOUT_SECS,
         log_level: "info".to_string(),
@@ -186,6 +190,7 @@ async fn gateway_listener_requirements_report_managed_bridge_address() {
 async fn gateway_listener_requirements_are_empty_for_host_gateway_route() {
     let mut config = runtime_config();
     config.gateway_route = DockerGatewayRoute::HostGateway;
+    config.gateway_callback_bind_address = None;
     let driver = test_driver_with_config(config);
 
     let response = driver
@@ -197,6 +202,26 @@ async fn gateway_listener_requirements_are_empty_for_host_gateway_route() {
     assert!(response.requirements.is_empty());
 }
 
+#[tokio::test]
+async fn host_gateway_route_reports_ipv4_loopback_callback_listener() {
+    let mut config = runtime_config();
+    config.gateway_route = DockerGatewayRoute::HostGateway;
+    config.gateway_callback_bind_address = Some("127.0.0.1:17670".parse().unwrap());
+    let driver = test_driver_with_config(config);
+
+    let response = driver
+        .get_gateway_listener_requirements(Request::new(GetGatewayListenerRequirementsRequest {}))
+        .await
+        .unwrap()
+        .into_inner();
+
+    assert_eq!(response.requirements.len(), 1);
+    assert_eq!(
+        response.requirements[0].selector,
+        Some(Selector::ExactBindAddress("127.0.0.1:17670".to_string()))
+    );
+}
+
 #[test]
 fn container_visible_endpoint_rewrites_loopback_hosts() {
     assert_eq!(
@@ -298,6 +323,31 @@ fn docker_gateway_route_uses_host_gateway_for_docker_desktop() {
     );
 }
 
+#[test]
+fn host_gateway_route_requests_ipv4_loopback_for_ipv6_primary() {
+    assert_eq!(
+        docker_gateway_callback_bind_address(
+            &DockerGatewayRoute::HostGateway,
+            "[::1]:17670".parse().unwrap(),
+        ),
+        Some("127.0.0.1:17670".parse().unwrap())
+    );
+}
+
+#[test]
+fn host_gateway_route_reuses_ipv4_primary_when_it_covers_loopback() {
+    for primary in ["127.0.0.1:17670", "0.0.0.0:17670"] {
+        assert_eq!(
+            docker_gateway_callback_bind_address(
+                &DockerGatewayRoute::HostGateway,
+                primary.parse().unwrap(),
+            ),
+            None,
+            "{primary} already covers the IPv4 loopback callback"
+        );
+    }
+}
+
 #[test]
 fn docker_gateway_route_uses_host_gateway_for_colima() {
     let info = SystemInfo {