Skip to content

bug: Kubernetes v1beta1 stop-start can hang waiting for supervisor readiness #3566

Open

Description

@drew

User Story

As an OpenShell contributor, I want the Kubernetes sandbox stop/start lifecycle test to complete reliably, so that required CI reflects product regressions instead of intermittent supervisor startup hangs.

Problem Statement

The Kubernetes Agent Sandbox v1beta1 E2E lane intermittently times out during the sandbox-lifecycle stop/start scenario. The gateway completes StartSandbox successfully and Kubernetes starts the replacement supervisor pod, but the supervisor never creates its readiness socket or reconnects to the gateway. Its only log line is Starting sandbox supervision.

The same v1beta1 lane passed on the preceding commit with identical Kubernetes code, while the v1alpha1, workspace-managed, workspace-operator, and external-driver Kubernetes lanes passed in the failing run.

Failed job: https://github.com/NVIDIA/OpenShell/actions/runs/35776641970/job/106915241802

Manual retry: https://github.com/NVIDIA/OpenShell/actions/runs/35776641970/job/106938509231

Impact / Why This Matters

This intermittently fails the required Core E2E gate, delays otherwise valid pull requests, and consumes additional CI capacity through manual reruns. A generic 120-second CLI timeout also makes the underlying supervisor startup stage difficult to diagnose. Retrying the job is the current workaround, but it does not prevent recurrence or protect the stop/start lifecycle from real regressions.

Acceptance Criteria

  • The v1beta1 stop/start scenario no longer intermittently stalls after the replacement supervisor pod starts.
  • A regression test or repeatable stress test covers stop followed by start and verifies that the replacement supervisor reconnects and becomes ready.
  • If supervisor startup cannot complete, CI reports the blocked startup stage or cause instead of only timing out at the CLI after 120 seconds.
  • The existing v1alpha1 and other Kubernetes deployment-mode E2E lanes continue to pass.

Reproduction Steps

  1. Run the Kubernetes E2E workflow with Agent Sandbox v0.5.0 and the sandbox-lifecycle conformance scenario.
  2. Create a sandbox and wait for it to become ready.
  3. Stop the sandbox.
  4. Start the sandbox again.
  5. Intermittently, observe openshell sandbox start time out after 120 seconds while the replacement supervisor pod remains unready.

Environment

  • OpenShell: 0c6c60c1eb70b1c4bc8e8a4f6e06d74ea09069bc
  • OS: Linux amd64 GitHub Actions runner
  • Runtime, deployment, or integration: kind Kubernetes cluster, Agent Sandbox API v1beta1, Agent Sandbox controller v0.5.0

Logs

[sandbox-lifecycle/stop-start/start] timed out after 120.0s
command: openshell sandbox start ct-869gjwhrud-ss

[pod/os-supervisor-.../supervisor] Starting sandbox supervision
Readiness probe failed: connect supervisor readiness socket /run/openshell/health.sock

Activity

  1. github-actions commented on Oct 7, 2026

    @github-actions

    This issue has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity. Comment or remove the state:stale label to keep it open.

  2. github-actions commented on Oct 22, 2026

    @github-actions

    Closing this issue after 14 days with no activity since it was marked stale. Reopen it if the work is still relevant.

  3. cv commented on Oct 27, 2026

    @cv
    Contributor

    We hit what looks like the same hang in NVIDIA/NemoClaw's CI, but when a sandbox was created rather than on stop and start, and with OpenShell v0.1.5.

    • Setup: kind on linux_arm64 (Kubernetes 1.37), Agent Sandbox 0.5.0, OpenShell v0.1.5 (bdb956f). The test creates three sandboxes behind one gateway in a namespace with the OpenShift profile. Run: https://github.com/NVIDIA/NemoClaw/actions/runs/40464315173/job/122058329950
    • Symptom: one of the three supervisor pods logged only Starting sandbox supervision and never logged Fetching sandbox policy via gRPC; its readiness probe on :5501 kept refusing connections. The other two supervisors fetched their policy about 7 s after starting. All three workload pods were running and ready, and the gateway kept polling GetSandbox until our two-minute limit.
    • Frequency: the same test passed on linux_amd64 in that run, on a re-run of the failed job, and locally on linux_arm64.

    So the stall seems to affect first start too, not only the replacement supervisor after a stop. Happy to share more of the log if it helps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions