fix(run): stop healthy retained runs from losing workspace ownership

[coygeek]

Summary

An active retained local-container run using Crabbox 0.48.0 lost its workspace-owner renewal after a sustained dependency-install workload and before required artifact collection. Crabbox canceled the run lifecycle, refused collection and cleanup, and returned a workspace-owner failure even though the lease remained ready and later accepted reused runs successfully.

The reported exit status 75 is also too opaque to distinguish the remote protocol's MISMATCH and EXPIRED outcomes. A healthy active run should keep its owner renewed; a genuine ownership loss should remain fail-closed but identify the state that was actually observed.

Steps to reproduce

1. Install Crabbox 0.48.0, initialize a small Git repository, and create a retained local-container lease:

   crabbox run --provider local-container --keep --slug owner-renew-repro --sync-only

2. Reuse that lease for a setup command that spans many 10-second workspace-owner renewal intervals and requests a required artifact. The observed failing command used this shape:

   crabbox run \
     --provider local-container \
     --id owner-renew-repro \
     --preflight \
     --preflight-tools default,node,npm,python3 \
     --timing-json \
     --artifact-glob 'reports/**' \
     --require-artifact reports/summary.json \
     --download reports/summary.json=proof/summary.json \
     --keep-on-failure \
     --stop-after success \
     --script ./verify.sh

3. In the observed run, verify.sh installed the clean Ubuntu runner's Node/npm and other test prerequisites, verified its source inputs, and entered npm ci. Instead of reaching artifact generation and collection, Crabbox ended with:

   warning: workspace owner release failed: remote workspace owner renewal failed closed: exit status 75
   remote command child ownership remains active; refusing collection and cleanup: remote workspace owner renewal failed closed: exit status 75
   

4. Inspect the lease after the outer run exits. The observed lease remained ready, had no lingering npm, Node, make, or verification process, and accepted a subsequent synced run. That rerun completed the same source checks and later stopped the lease normally.

5. The smallest missing reproduction fact is a deterministic way to produce the same active-owner exit status 75 without changing or tampering with workspace-owner state. Exact current-main controls using the same retained local-container route completed a 37-second dependency install and a separate 120-second command across twelve renewal intervals without failure.

Expected behavior

A retained or reused SSH-backed run should maintain workspace ownership while its controlling client and witnessed child remain active. A transient transport or response loss should not become terminal while the existing owner is still provably valid. A confirmed token mismatch, expiry, or ambiguous child state must continue to fail closed and prevent unsafe collection, cleanup, or reuse.

The regression is closed when a deterministic run-level test can hold or lose a renewal response during an active child, prove that still-valid ownership survives within its confirmed deadline, and prove that real mismatch or expiry remains exit 7 with the exact protocol state preserved. A live retained-run control should cross repeated renewal intervals, complete its workload, collect the required artifact, release ownership, and leave no owner or child residue.

Actual behavior

The renewal path returned only exit status 75, canceled the owner context, and caused the normal post-command ownership check to return before result and artifact collection. The output did not identify whether the remote owner reported MISMATCH or EXPIRED, and the still-usable lease required a separate manual inspection and rerun.

Affected area

• Retained and reused SSH-backed crabbox run workflows that acquire a remote workspace owner.
• Renewal response handling in internal/cli/workspace_owner.go.
• Post-command child quiescence and the collection boundary in internal/cli/run.go.
• Required artifacts, downloads, parsed results, failure capture, and cleanup that occur only after workspace ownership is confirmed.

Runtime or environment

• Observed release: v0.48.0 at 26b735abbaf6571d5f46620e07514b7ce9ff0d7b.
• Observed installed binary: Crabbox 0.48.0, SHA-256 44539a257aa51f885881aaf11f77ab800a48f72103e1072752f84dae3ab60a4e.
• Observed failing direct run: run_9cdf95149d63, lease cbx_6f6398d094a5, provider local-container, Ubuntu 26.04 ARM64, Docker 29.7.2.
• Exact current main: 697ac171ecca19ecdcad748ae9a66bd0d108ea28, three commits after the release baseline.
• Current-main binary: dev, SHA-256 fc235b0216475c2748b44d2a426b8c703f4106d499c4faa07702e21ff04f19a8.
• Build and control host: Go 1.26.5 on macOS 27.0 build 26A5421a, ARM64.
• Current-main retained-run controls: dependency-install run run_4db22791540e and 120-second run run_4ae2c3d58f25, both exit 0 with workspace owner released.

Evidence

Current source maps a renewal token mismatch or expired owner state to remote exit 75. renewLoopWithTicks stores any non-RENEWED result as remote workspace owner renewal failed closed, cancels the lifecycle context, and does not retain the nonzero response token in the user-facing error. waitWorkspaceOwnerNoChild then returns that stored renewal error before results, required artifacts, downloads, or ordinary cleanup run.

The workspace-owner, run, and associated test paths are byte-identical between v0.48.0 and exact current main. The complete three-commit range from the release baseline through current main, all 20 live open issues, all 16 open issues authored by coygeek, all 28 live open pull requests, and relevant closed issues and merged pull requests were checked for duplicates, fixes, or supersession. No open item owns active-run renewal loss before evidence collection.

Closed issue #1390 and merged pull request #1392 fixed a different teardown race in which destructive --stop-after always cleanup killed an in-flight renewal after a successful command and evidence download. That fix deliberately preserves exit 7 for genuine renewal ambiguity before stop, which is the phase reached here.

Open issue #1691 concerns coordinator cleanup that preserves a claim while final output incorrectly reports the lease as stopped. This report instead concerns workspace-owner renewal loss during an active direct run, before collection and cleanup are allowed to begin.

Closed pull request #1442 implemented bounded same-token reconciliation for lost acquire and renewal responses, while keeping mismatch and expiry terminal. It was closed without merge because the broader patch's native-Windows authority proof remained incomplete. Current main does not contain that reconciliation; this report does not request the PR's Windows supervisor or POSIX descendant-containment scope.

The exact current-main focused workspace-owner tests pass, as do the retained 37-second dependency-install and 120-second live controls. Those controls establish that the failure is intermittent and that a deterministic trigger remains the evidence gap; they do not supersede the exact v0.48.0 observed failure or prove the unchanged renewal path cannot recur.

Impact

Long-running retained jobs can lose their final result, required artifacts, downloads, parsed test evidence, and automatic cleanup after paying the full setup and compute cost. Automation receives an infrastructure failure instead of the workload result, while the retained lease can remain usable but requires manual inspection. The opaque exit 75 also prevents operators from distinguishing a real token mismatch from expiry after delayed or lost renewal.

Additional context

The current-main control lease was explicitly stopped after testing, and local-container inventory confirmed its removal. This is an ordinary lifecycle and reliability bug under Crabbox's trusted-user model. It does not request a new preflight probe, weaken fail-closed ownership, or claim that artifact collection is safe after ownership is genuinely lost.

[clawsweeper]

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed August 31, 2026, 6:02 PM ET / 22:02 UTC.

Summary
Keep open for maintainer decision: current code intentionally fails closed on every renewal transport/protocol failure, while this issue proposes bounded same-token reconciliation before the confirmed ownership deadline.

Reproducibility: no. deterministic current-main reproduction of the observed lost response is supplied, but source inspection conclusively shows that any non-RENEWED renewal response cancels the lifecycle before collection.

Ways to help us reproduce this

• Add a screenshot or short recording showing the behavior.
• Add expected vs actual behavior.
• Share version, platform, channel/provider, and relevant config details.

Maintainer decision needed

Question	Recommendation
Should Crabbox reconcile a lost renewal response with the same token until the last confirmed ownership deadline, rather than immediately canceling an active retained run?	Approve bounded reconciliation: Retry only the same token within a recorded confirmed deadline, while explicit mismatch, expiry, and ambiguous child state remain terminal before collection or cleanup.

Why: Current source and docs intentionally fail closed on all renewal failures; allowing a bounded exception changes the ownership authority contract even though the proposed guardrails are narrow.

Next step
A maintainer must choose the bounded reconciliation policy before a focused implementation can safely alter the documented fail-closed ownership behavior.

Review details

Best possible solution:

Define and document a bounded same-token renewal-reconciliation rule, preserve immediate terminal handling for explicit mismatch and expiry, retain the remote state in diagnostics, and prove both outcomes through the run-level collection boundary.

Do we have a high-confidence way to reproduce the issue?

No deterministic current-main reproduction of the observed lost response is supplied, but source inspection conclusively shows that any non-RENEWED renewal response cancels the lifecycle before collection.

Is this the best way to solve the issue?

Unclear pending the authority-policy choice; bounded same-token reconciliation is the narrowest proposed path, but it deliberately changes the documented immediate fail-closed renewal behavior.

AGENTS.md: found and applied where relevant.

Remaining risk / open question:

• Retrying an ambiguous renewal changes an ownership authority boundary; reconciliation must reject mismatch and expiry before any artifact collection, cleanup, reuse, or other side effect.
• The reported active-owner response loss remains intermittent, so a deterministic injected-response test is needed before selecting the exact retry/deadline policy.

Codex review notes: model internal, reasoning high; reviewed against 697ac171ecca.

Label changes

Label justifications:

• P1: An active retained run can terminate before required artifacts and workload results are collected.
• impact:other: The affected surface is remote-run lifecycle reliability and evidence collection rather than a more specific owned impact class.

Evidence reviewed

What I checked:

• Current renewal behavior: The renewal loop treats every result other than a successful RENEWED response as a terminal exit-7 error and cancels the owner lifecycle; when the remote command returns both output and a nonzero status, the stored error is the transport/status error rather than the response token. (internal/cli/workspace_owner.go:376, 697ac171ecca)
• Collection is gated by ownership: After the workload returns, runCommand calls waitWorkspaceOwnerNoChild before collecting results, required artifacts, downloads, and run artifacts, so a stored renewal error blocks those outputs. (internal/cli/run.go:2377, 697ac171ecca)
• Remote protocol preserves distinct terminal states: The POSIX owner protocol emits MISMATCH and EXPIRED with remote exit 75, confirming the diagnostic distinction that is currently lost by the renewal-loop error path. (internal/cli/workspace_owner.go:654, 697ac171ecca)
• Existing contract requires immediate fail-closed renewal: Current tests require a failed renewal to cancel the lifecycle and later return a renewal-failed-closed error; the documentation likewise says token-bound renewal and release fail closed. (internal/cli/workspace_owner_test.go:256, 697ac171ecca)
• No main-branch fix since v0.48.0: The workspace-owner and run paths named by the report are unchanged between the v0.48.0 release commit and current main, so the reported behavior remains present rather than being release-stale. (internal/cli/workspace_owner.go:367, 697ac171ecca)
• Related teardown fix is distinct: The merged cleanup fix changed the workspace-owner and run cleanup paths for post-stop sequencing; it does not establish same-token reconciliation during an active retained run. (internal/cli/run_workspace_owner_cleanup_test.go:1, e6dd2151171b)

Likely related people:

• steipete: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
• Vincent Koc: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.