Skip to content

fix(kubernetes): prevent stop-start relay race - #3064

Merged
drew merged 1 commit into
mainfrom
fix-kubernetes-stop-start/jm
Aug 31, 2026
Merged

fix(kubernetes): prevent stop-start relay race#3064
drew merged 1 commit into
mainfrom
fix-kubernetes-stop-start/jm

Conversation

@johntmyers

Copy link
Copy Markdown
Collaborator

Summary

Fix a Kubernetes v1beta1 stop/start race that allowed a sandbox to report Stopped and restart while its previous pod was still terminating. The old supervisor could then close the newly claimed exec relay before reporting an exit status.

The regression was exposed by #2884: intentional lifecycle shutdown discards the canonical process exit result, but the new terminal-finalization loop treated that discarded result as unfinalized and retried until Kubernetes exhausted its 30-second termination grace period.

Affected CI evidence

Each linked job failed in sandbox_stop_start_preserves_workspace with exec relay closed before the command reported an exit status:

Related Issue

No issue required: this is a localized regression fix with an existing deterministic lifecycle E2E test and CI evidence across unrelated PRs.

Changes

  • Acknowledge main-process finalization during intentional Deleting, Stopping, and Stopped lifecycle transitions so the supervisor can terminate promptly.
  • Require both Suspended=True and deletion of the old pod before a Kubernetes v1beta1 stop completes.
  • Preserve the v1alpha1 fallback, where pod deletion alone is the usable stop-completion signal.
  • Add unit coverage for the finalization and stop-completion invariants.
  • Document the stronger Kubernetes stop contract in the driver and published compute-driver reference.

Testing

  • mise run pre-commit passes
  • Unit tests added/updated
  • Existing Kubernetes lifecycle E2E regression coverage exercised
  • cargo test -p openshell-driver-kubernetes --lib (228 passed)
  • OPENSHELL_E2E_KUBE_TEST=sandbox_lifecycle mise run e2e:kubernetes (7 passed, including sandbox_stop_start_preserves_workspace)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (not applicable; driver and published behavior references updated)

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown

@johntmyers johntmyers added the test:e2e-kubernetes Requires Kubernetes end-to-end coverage label Aug 31, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e-kubernetes applied for 37962b6. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute Kubernetes HA and credential-driver E2E after building the required gateway and supervisor images once. This is an optional proof-of-life suite; failures are visible in the workflow run but do not publish a required CI gate status.

@johntmyers johntmyers added the test:e2e Requires end-to-end coverage label Aug 31, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for 37962b6. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@drew
drew added this pull request to the merge queue Aug 31, 2026
Merged via the queue into main with commit 5c541e1 Aug 31, 2026
108 checks passed
@drew
drew deleted the fix-kubernetes-stop-start/jm branch August 31, 2026 23:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage test:e2e-kubernetes Requires Kubernetes end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants