feat(kubernetes): warm pooling for sandboxes created from templates

Signed-off-by: Gordon Sim <gsim@redhat.com>

Author: grs

Cargo.lock

@@ -141,7 +141,7 @@ delete, reconciliation removes the row; otherwise it can remain `Deleting`.
 |---|---|---|---|
 | Docker | Local development with Docker available. | Container plus nested sandbox namespace. | Uses host networking so loopback gateway endpoints work from the supervisor. |
 | Podman | Rootless or single-machine deployments. | Container plus nested sandbox namespace. | Uses the Podman REST API and CDI GPU devices when available. Delivers the supervisor via OCI image volume by default; falls back to extracting the binary to a host-side cache and bind-mounting it when `userns` is configured (overlay does not support idmapped mounts). |
-| Kubernetes | Cluster deployment through Helm. | Pod plus nested sandbox namespace. | Uses Kubernetes API objects, service accounts, secrets, PVC-backed workspace storage, and GPU resources. |
+| Kubernetes | Cluster deployment through Helm. | Pod plus nested sandbox namespace. | Uses Kubernetes API objects, service accounts, secrets, PVC-backed workspace storage, GPU resources, and optionally Agent Sandbox v1beta1 warm-pool claims. |
 | VM | Experimental microVM isolation. | Per-sandbox libkrun VM. | Managed endpoint-backed driver. The gateway spawns `openshell-driver-vm`, waits for its Unix socket, and then consumes it through the same remote `compute_driver.proto` path used by unmanaged endpoint drivers. The VM driver boots a cached bootstrap `rootfs.ext4`, prepares requested OCI images inside a bootstrap VM with `umoci`, attaches the prepared image disk read-only, and gives each sandbox a writable `overlay.ext4` for merged-root changes and runtime material. The driver persists each accepted launch request beside the overlay and restarts those VMs on driver startup without recreating the overlay. |
 | Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a non-reserved custom `compute_drivers = ["<name>"]` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--drivers <name>` with `--compute-driver-socket=<path>`. Reserved built-in names such as `vm`, `docker`, `podman`, and `kubernetes` cannot be used as unmanaged socket endpoints. The gateway connects to a UDS the operator already provisioned, runs `GetCapabilities`, logs the advertised `driver_name`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |
 
@@ -248,14 +248,75 @@ The supervisor itself remains root so it can establish isolation before
 starting unprivileged children.
 
 Kubernetes supervisors authenticate to the gateway in two stages. They first
-call `RegisterSupervisorPod` with the projected ServiceAccount token; the
+call `RegisterSupervisor` with the projected ServiceAccount token; the
 gateway validates the pod-bound token and live Agent Sandbox owner state, then
 activates already-bound cold pods by streaming back a gateway-minted sandbox
 JWT. The supervisor installs that JWT in memory and starts the normal
 `ConnectSupervisor` session as the activated sandbox. The gateway does not dial
 pod IPs or require an inbound activation port; activation is supervisor
 initiated over the existing outbound gRPC connection.
 
+When compatible OpenShell-enabled Agent Sandbox warm pools exist, the
+Kubernetes driver can create a `SandboxClaim` instead of a direct `Sandbox`.
+The driver watches warm pools and templates into a local cache so create
+requests do not list or fetch extension objects on the hot path. This remains
+behind the compute-driver boundary: other drivers and the driver-agnostic
+gateway lifecycle continue to operate on OpenShell sandbox identity and status.
+Warm-pool activation uses the same trust boundary as direct Kubernetes
+activation. The direct path validates `pod token -> live pod -> owning Sandbox
+CR -> Sandbox sandbox-id metadata`; the warm path validates `pod token -> live
+pod -> owning Sandbox CR -> associated SandboxClaim -> SandboxClaim sandbox-id
+metadata`. In both paths, sandbox-id metadata is not cryptographic proof by
+itself. It is trusted because the gateway and Agent Sandbox controllers own the
+relevant Kubernetes objects, and RBAC must prevent sandbox workloads and
+untrusted users from creating or mutating trusted `Sandbox`, `SandboxClaim`, pod
+metadata, or sandbox service-account identity in the gateway-managed namespace.
+Under that model, warm pooling does not require a separate anti-spoofing token
+or gateway-side claim mapping beyond the same live Kubernetes ownership and
+metadata consistency checks used by the direct path. If an operator grants
+untrusted principals write access to those objects, both warm and direct
+activation require a broader common proof model.
+Claim activation is level-triggered rather than dependent on a single watch
+event. The controller periodically relists OpenShell-managed claims and retries
+failed watch streams, because a claim can become visible before its selected
+`Sandbox`, pod, or outbound supervisor registration. A newly authenticated
+warm-supervisor registration clears completed-claim hints and triggers an immediate
+relist, so both a restarted process with the same pod UID and a replacement pod
+with a new UID can reactivate against an existing claim. It reconciles different
+claims concurrently, deduplicates work by claim UID, and caps concurrency so
+one late registration or slow Kubernetes lookup cannot block activation for
+the namespace or create unbounded work.
+The gateway retains activated pod-UID tombstones for one hour to reject
+duplicate registration and activation races, then prunes them opportunistically
+on registry access so long-running gateways do not accumulate one entry per
+historical pod. A new authenticated registration supersedes its pod-UID
+tombstone and receives a new local session ID. Activation and failure delivery
+are conditional on that session ID, preventing work started for an old process
+from consuming or terminating its replacement stream.
+Warm claim creation is idempotent by the deterministic claim name. After a
+create timeout, transport failure, server error, or conflict, the driver reads
+that name back and retries the same claim create when it is not yet visible. It
+never switches to direct Sandbox creation after an ambiguous claim write. An
+existing claim is accepted only when its OpenShell identity, workspace,
+allocation marker, and warm-pool reference match the request. If Kubernetes
+still cannot determine whether the claim exists, the driver returns
+`Unavailable` and the gateway retains the durable `Provisioning` record. This
+prevents another create from assigning a new sandbox ID to the same name while
+the original Kubernetes write may still be live; the normal driver watcher can
+subsequently reconcile an accepted claim.
+Drivers opt into OpenShell `SandboxTemplate` lifecycle notifications through
+the compute-driver capability snapshot. The Kubernetes driver uses those
+notifications, plus a startup sync of persisted templates, to generate Agent
+Sandbox `SandboxTemplate` and `SandboxWarmPool` resources for templates whose
+`desired_service_level.startup.ready_within` is strictly below the configured
+threshold. The generated pool size follows
+`desired_service_level.startup.max_burst` capped by the Kubernetes driver
+configuration. Sandbox creation and template reconciliation share one
+workspace-to-namespace mapping boundary; the current shared-namespace
+implementation maps every workspace to the configured Kubernetes namespace.
+The read-side warm-pool cache still runs on every replica so local create
+requests do not depend on a separate writer lease.
+
 Kubernetes can run the supervisor in the default combined topology or in a
 sidecar topology. Combined mode keeps network and process supervision in the
 agent container. Sidecar mode runs network enforcement, the proxy, and gateway

architecture/compute-runtimes.md

@@ -210,11 +210,24 @@ identity inspection without client-side token decoding.
 Sandbox secrets are gateway-signed JWTs bound to a single sandbox ID. Docker,
 Podman, and VM drivers deliver the initial token through supervisor-only
 runtime material; Kubernetes supervisors exchange a projected ServiceAccount
-token through `RegisterSupervisorPod`. The gateway validates that projected
+token through `RegisterSupervisor`. The gateway validates that projected
 token with Kubernetes `TokenReview`, requires the configured sandbox service
 account, checks the returned pod binding against the live pod UID, and verifies
 the pod's controlling `Sandbox` ownerReference against the live Sandbox CR UID
 and sandbox-id label before sending an activation message with the gateway JWT.
+For warm-pooled Kubernetes sandboxes, the same check is re-anchored through the
+claim that adopted the warm pod: the live pod must still be controlled by a
+live `Sandbox`, that `Sandbox` must be associated with the live
+`SandboxClaim`, and the sandbox-id metadata on that claim must identify the
+same OpenShell sandbox record before activation can complete. This is not a
+separate warm-only spoofing defense; it is the warm-path equivalent of the
+direct path's owner-reference and sandbox-id consistency check. Both paths
+depend on the same Kubernetes RBAC boundary: sandbox workloads and untrusted
+users must not be able to create or mutate trusted Agent Sandbox objects, pod
+metadata, or the configured sandbox ServiceAccount in the gateway-managed
+namespace. If that boundary is weakened, a fix must cover both direct
+`Sandbox` and warm `SandboxClaim` activation rather than adding a warm-only
+proof.
 The bootstrap path accepts both `agents.x-k8s.io/v1beta1` ownerReferences from
 newer Agent Sandbox controllers and `agents.x-k8s.io/v1alpha1` ownerReferences
 from existing deployments. `IssueSandboxToken` remains as a compatibility shim

architecture/gateway.md

@@ -619,14 +619,14 @@ impl OpenShell for TestOpenShell {
         Err(Status::unimplemented("not implemented in test"))
     }
 
-    type RegisterSupervisorPodStream = tokio_stream::wrappers::ReceiverStream<
-        Result<openshell_core::proto::PodActivationMessage, Status>,
+    type RegisterSupervisorStream = tokio_stream::wrappers::ReceiverStream<
+        Result<openshell_core::proto::SupervisorActivationMessage, Status>,
     >;
 
-    async fn register_supervisor_pod(
+    async fn register_supervisor(
         &self,
-        _request: tonic::Request<openshell_core::proto::RegisterSupervisorPodRequest>,
-    ) -> Result<Response<Self::RegisterSupervisorPodStream>, Status> {
+        _request: tonic::Request<openshell_core::proto::RegisterSupervisorRequest>,
+    ) -> Result<Response<Self::RegisterSupervisorStream>, Status> {
         Err(Status::unimplemented("not implemented in test"))
     }
 

crates/openshell-cli/tests/ensure_providers_integration.rs

@@ -498,14 +498,14 @@ impl OpenShell for TestOpenShell {
         Err(Status::unimplemented("not implemented in test"))
     }
 
-    type RegisterSupervisorPodStream = tokio_stream::wrappers::ReceiverStream<
-        Result<openshell_core::proto::PodActivationMessage, Status>,
+    type RegisterSupervisorStream = tokio_stream::wrappers::ReceiverStream<
+        Result<openshell_core::proto::SupervisorActivationMessage, Status>,
     >;
 
-    async fn register_supervisor_pod(
+    async fn register_supervisor(
         &self,
-        _request: tonic::Request<openshell_core::proto::RegisterSupervisorPodRequest>,
-    ) -> Result<Response<Self::RegisterSupervisorPodStream>, Status> {
+        _request: tonic::Request<openshell_core::proto::RegisterSupervisorRequest>,
+    ) -> Result<Response<Self::RegisterSupervisorStream>, Status> {
         Err(Status::unimplemented("not implemented in test"))
     }
 

crates/openshell-cli/tests/mtls_integration.rs

@@ -1022,14 +1022,14 @@ impl OpenShell for TestOpenShell {
         Err(Status::unimplemented("not implemented in test"))
     }
 
-    type RegisterSupervisorPodStream = tokio_stream::wrappers::ReceiverStream<
-        Result<openshell_core::proto::PodActivationMessage, Status>,
+    type RegisterSupervisorStream = tokio_stream::wrappers::ReceiverStream<
+        Result<openshell_core::proto::SupervisorActivationMessage, Status>,
     >;
 
-    async fn register_supervisor_pod(
+    async fn register_supervisor(
         &self,
-        _request: tonic::Request<openshell_core::proto::RegisterSupervisorPodRequest>,
-    ) -> Result<Response<Self::RegisterSupervisorPodStream>, Status> {
+        _request: tonic::Request<openshell_core::proto::RegisterSupervisorRequest>,
+    ) -> Result<Response<Self::RegisterSupervisorStream>, Status> {
         Err(Status::unimplemented("not implemented in test"))
     }
 

crates/openshell-cli/tests/provider_commands_integration.rs

@@ -792,14 +792,14 @@ impl OpenShell for TestOpenShell {
         Err(Status::unimplemented("not implemented in test"))
     }
 
-    type RegisterSupervisorPodStream = tokio_stream::wrappers::ReceiverStream<
-        Result<openshell_core::proto::PodActivationMessage, Status>,
+    type RegisterSupervisorStream = tokio_stream::wrappers::ReceiverStream<
+        Result<openshell_core::proto::SupervisorActivationMessage, Status>,
     >;
 
-    async fn register_supervisor_pod(
+    async fn register_supervisor(
         &self,
-        _request: tonic::Request<openshell_core::proto::RegisterSupervisorPodRequest>,
-    ) -> Result<Response<Self::RegisterSupervisorPodStream>, Status> {
+        _request: tonic::Request<openshell_core::proto::RegisterSupervisorRequest>,
+    ) -> Result<Response<Self::RegisterSupervisorStream>, Status> {
         Err(Status::unimplemented("not implemented in test"))
     }
 

crates/openshell-cli/tests/sandbox_create_lifecycle_integration.rs

@@ -586,14 +586,14 @@ impl OpenShell for TestOpenShell {
         Err(Status::unimplemented("not implemented in test"))
     }
 
-    type RegisterSupervisorPodStream = tokio_stream::wrappers::ReceiverStream<
-        Result<openshell_core::proto::PodActivationMessage, Status>,
+    type RegisterSupervisorStream = tokio_stream::wrappers::ReceiverStream<
+        Result<openshell_core::proto::SupervisorActivationMessage, Status>,
     >;
 
-    async fn register_supervisor_pod(
+    async fn register_supervisor(
         &self,
-        _request: tonic::Request<openshell_core::proto::RegisterSupervisorPodRequest>,
-    ) -> Result<Response<Self::RegisterSupervisorPodStream>, Status> {
+        _request: tonic::Request<openshell_core::proto::RegisterSupervisorRequest>,
+    ) -> Result<Response<Self::RegisterSupervisorStream>, Status> {
         Err(Status::unimplemented("not implemented in test"))
     }
 

crates/openshell-cli/tests/sandbox_name_fallback_integration.rs

@@ -380,11 +380,28 @@ pub fn build_capabilities_response(
     driver_name: &str,
     driver_version: impl Into<String>,
     default_image: impl Into<String>,
+) -> GetCapabilitiesResponse {
+    build_capabilities_response_with_template_lifecycle(
+        driver_name,
+        driver_version,
+        default_image,
+        false,
+    )
+}
+
+/// Build a [`GetCapabilitiesResponse`] and opt into sandbox-template lifecycle
+/// notifications.
+pub fn build_capabilities_response_with_template_lifecycle(
+    driver_name: &str,
+    driver_version: impl Into<String>,
+    default_image: impl Into<String>,
+    supports_sandbox_template_lifecycle: bool,
 ) -> GetCapabilitiesResponse {
     GetCapabilitiesResponse {
         driver_name: driver_name.to_string(),
         driver_version: driver_version.into(),
         default_image: default_image.into(),
+        supports_sandbox_template_lifecycle,
     }
 }
 

crates/openshell-core/src/driver_utils.rs

@@ -122,6 +122,10 @@ pub enum ComputeDriverError {
     /// A precondition for the operation was not met.
     #[error("{0}")]
     Precondition(String),
+    /// The backend may have accepted the operation, but its result could not be
+    /// determined. Callers must preserve durable intent for reconciliation.
+    #[error("{0}")]
+    Unavailable(String),
     /// Generic error message.
     #[error("{0}")]
     Message(String),
@@ -134,6 +138,7 @@ impl From<ComputeDriverError> for tonic::Status {
             ComputeDriverError::NotFound => Self::not_found("sandbox not found"),
             ComputeDriverError::InvalidArgument(m) => Self::invalid_argument(m),
             ComputeDriverError::Precondition(m) => Self::failed_precondition(m),
+            ComputeDriverError::Unavailable(m) => Self::unavailable(m),
             ComputeDriverError::Message(m) => Self::internal(m),
         }
     }