[vineethsai7]

Summary

Add a local VM driver command to inspect and prune prepared image disks without guessing cache filenames. Pruning protects every disk referenced by persisted sandboxes, including stopped sandboxes.

Related Issue

Part of #5150.

Changes

• Add openshell-driver-vm image-cache list and image-cache prune --unused [--dry-run] for the configured VM state directory.
• Record image reference, last use, and bootstrap cache identity in driver-owned state. Older cache entries remain readable and prunable without this metadata.
• Add a shared cache lease for the driver and preparation workers. A real prune requires an exclusive lease and checks the default socket used by older drivers.
• Document the operator workflow in the VM driver README, published runtime guide, and gateway debugging skill.

Design

The VM driver owns the cache directory and already persists the prepared cache identity for each sandbox. The command scans those identities and removes only committed root disks with no persisted references. A sandbox from an older release without a saved bootstrap identity conservatively protects all bootstrap disks. List and dry-run remain available while the driver runs; deletion requires the gateway to be stopped. This keeps cleanup explicit and leaves normal sandbox behavior unchanged.

A gateway and compute-driver RPC would support the eventual openshell image cache command, but it changes the shared driver contract and public API. Automatic startup cleanup and a size cap also need an eviction policy. Middleware, gateway interceptors, providers, and OCSF events do not own VM disk storage, so they cannot safely perform this operation.

Usage

Use the existing VM driver state_dir setting:

[openshell.drivers.vm]
state_dir = "/var/lib/openshell/vm-driver"

On the gateway host:

openshell-driver-vm --state-dir /var/lib/openshell/vm-driver image-cache list
openshell-driver-vm --state-dir /var/lib/openshell/vm-driver image-cache prune --unused --dry-run

After stopping the VM gateway:

openshell-driver-vm --state-dir /var/lib/openshell/vm-driver image-cache prune --unused

The list is tab-separated and includes cache ID, image reference, allocated and apparent bytes, last-use Unix milliseconds, layout version, and referencing sandbox IDs. Prune reports the number of entries removed and their pre-removal allocated bytes. No new gateway API or event shape is introduced.

Compatibility

No cache is deleted automatically. Existing cache keys and root disks remain unchanged. Entries created before this change show - for image reference and last use until reused. Incomplete sandbox state blocks pruning; older kept sandboxes without bootstrap metadata protect all bootstrap disks. The allocated-byte total is an estimate of space reclaimed because filesystem snapshots and shared blocks can affect free space.
Overlay templates and in-progress preparation directories are outside this command's scope.

Follow-ups

• Add a compute-driver capability, gateway API, and openshell image cache CLI with a portable "no cache" response for other drivers.
• Expose resolved source image IDs, entry-specific and age-based pruning, and structured output.
• Design optional stale-version cleanup and a size cap that evicts only unreferenced entries.

The issue is still state:triage-needed; design feedback is welcome.

Release Notes

VM gateway operators can list prepared image disks and, after stopping the gateway, prune disks unused by persisted sandboxes.

Testing

• cargo fmt --check -p openshell-driver-vm
• cargo clippy -p openshell-driver-vm --all-targets -- -D warnings
• cargo test -p openshell-driver-vm --quiet (library, binary, and integration targets)
• uv run tasks/scripts/check_docs_nav.py
• Fern 5.112.0 check (0 errors; 3 warnings outside the changed page)
• Unit tests added for cache references, pruning, legacy state, symlink safety, locking, and CLI parsing
• VM host E2E not run

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Relevant documentation updated

Review follow-up

Review against upstream main is recorded in REVIEW-5159.md. Upstream main was merged as signed commit cfc4ae9f8 before the review. The follow-up fix commit is d7e3a24ab.

Finding	Severity	Result
F1: identity write order and partial writes	P1	Fixed in d7e3a24ab; bootstrap identity is published atomically before the main identity, and failure stops provisioning.
F2: implicit or missing state directory	P2	Fixed in d7e3a24ab; cache commands require an explicit existing state directory.
F3: rootfs tar listed as a container image	P2	Fixed in d7e3a24ab; tar entries have no image reference and retain last-use time.
F4: legacy socket guard lacked a test	P2	Fixed in d7e3a24ab; regression test keeps a disk while the default legacy socket is live.
F5: custom-socket legacy driver caveat	P3	Fixed in documentation in d7e3a24ab; older standalone drivers must be stopped explicitly.
F6: docs described only prepared disks	P3	Fixed in documentation in d7e3a24ab; the command covers bootstrap and prepared image disks.

GitHub had zero inline review comments and no reviews. The copy-pr-bot workflow vetting notice is not applicable to the code because it contains no code finding.

Before the fixes, the new identity failure and cache CLI regression tests failed. After the fixes, these checks passed on the build box through dcb up-os-5150 with nice -n 15:

• cargo fmt --check
• cargo clippy -p openshell-driver-vm --all-targets -- -D warnings
• cargo test -p openshell-driver-vm --quiet: 279 library, 24 binary, and 6 integration tests passed.
• uv run tasks/scripts/check_docs_nav.py
• Fern 5.112.0 check: 0 errors and 3 warnings outside the changed page (existing gateway configuration MDX parse, unauthenticated redirect check, and light theme contrast).

Manual verification

Result: PASS for the macOS VM driver's local cache commands. Upstream main rejected image-cache; PR head listed the cache, kept both disks referenced by a sandbox state directory, refused deletion while its driver was running, and removed only the unused disk after that driver stopped.

Environment

• macOS 15.8, Apple silicon arm64. One disposable local account, private HOME and state directory, and a private non-default loopback port. Existing OpenShell and DefenseClaw services were untouched.
• Before: upstream main 8d0797a52, the PR's merge base. After: PR head d7e3a24ab4. Both VM driver binaries were built through dcb up-os-5150 using nice -n 15 cargo build --release --locked -p openshell-driver-vm --bin openshell-driver-vm --target aarch64-apple-darwin, with Zig 0.14 for C compilation and LLVM LLD 18 plus macOS 15.4 SDK link stubs. Both were signed ad hoc and verified on the host.
• The private cache fixture had three 16 MiB, valid ext4 rootfs.ext4 files in the driver's committed-cache layout. One prepared disk and one bootstrap disk had identity files in a sandbox state directory representing a stopped sandbox; the third prepared disk had no reference. The two prepared entries had seeded cache-metadata.json; the bootstrap entry had no metadata. The sandbox request marker was a fixture, not a real VM-created protobuf.

Steps

1. Run the main binary's --state-dir <private-state> image-cache list against the fixture.
2. Run the PR binary's image-cache list and image-cache prune --unused --dry-run against the same state.
3. Start the PR driver on a private loopback port with that state, without creating a VM. While it holds the shared cache lease, repeat list and dry-run, then attempt real prune.
4. Stop the driver by its exact PID. Run image-cache prune --unused, inspect the remaining files, and list again. Also try list with an explicitly missing state directory.

Before: upstream main 8d0797a52

The command exited 2 and left all three cache entries present:

error: unexpected argument 'image-cache' found
cache entries after command: 3

After: PR head d7e3a24ab4

image-cache list exited 0. Excerpt of its tab-separated output (the three data rows):

sandbox-bootstrap-rootfs-ext4-v6-openshell-proof-boot	-	16777216	16777216	-	sandbox-bootstrap-rootfs-ext4-v6	stopped-proof
sandbox-prepared-rootfs-ext4-umoci-v4-openshell-proof-kept	example.invalid/proof:kept	16777216	16777216	1793681058740	sandbox-prepared-rootfs-ext4-umoci-v4	stopped-proof
sandbox-prepared-rootfs-ext4-umoci-v4-openshell-proof-unused	example.invalid/proof:unused	16777216	16777216	1793681058751	sandbox-prepared-rootfs-ext4-umoci-v4	-

The columns are cache ID, image reference, allocated bytes, apparent bytes, last-use Unix milliseconds, layout version, and referencing sandbox IDs. Dry-run exited 0 and left all three entries present:

would remove 1 entries, 16777216 allocated bytes

With the PR driver running, list still returned three rows and dry-run still reported one unused entry. Real prune exited 1 with image cache is busy; all three entries remained. After stopping that exact driver process, real prune exited 0:

removed 1 entries, 16777216 allocated bytes
kept disk: present
bootstrap disk: present
unused disk: absent
post-prune list: 2 entries

List with an explicitly missing state directory exited 1 and did not create that directory. Cleanup removed the disposable account, HOME, state, binaries, and test data. Readback found no disposable processes or private listener; the pre-existing VM driver PID was unchanged.

Not covered

This was a host CLI and lock proof against a constructed on-disk cache fixture. The binaries did not embed the guest/runtime payload, so this run did not boot a VM, pull an OCI image, create or stop a real sandbox, or verify metadata publication during preparation. The legacy-driver default-socket guard and Linux KVM were not exercised.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-03T05:25:29.907635Z	d7e3a24	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d7e3a24ab4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Keep newly created locks usable by the gateway owner

When an administrator runs image-cache list or prune as root against a pre-upgrade state directory owned by a non-root gateway account, this path creates .image-cache.lock as root with mode 0600. The later gateway starts as the directory owner but lock_cache reopens that file read/write and exits with PermissionDenied, so merely inspecting the legacy cache can prevent the VM gateway from restarting until the file is manually repaired; reject cross-owner invocation or create the lock for the state-directory owner.

Useful? React with 👍 / 👎.