Skip to content

[CI] Cache renderer runtime artifacts - #7242

Closed
StafaH wants to merge 1 commit into
isaac-sim:developfrom
StafaH:ci/cache-renderer-runtime
Closed

[CI] Cache renderer runtime artifacts#7242
StafaH wants to merge 1 commit into
isaac-sim:developfrom
StafaH:ci/cache-renderer-runtime

Conversation

@StafaH

@StafaH StafaH commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Description

Persist the Kit, RTX, and NVIDIA compute caches used by the isaaclab_tasks CI shards. Fresh temporary bind mounts currently force camera-enabled tests to rebuild the renderer cache in every job.

The cache follows the existing Warp-cache trust model:

  • pull-request jobs restore but never publish
  • the post-merge warm job is the sole writer
  • compatibility is keyed by the Isaac Sim image/version, runner OS/architecture, GPU model, and NVIDIA driver
  • only cache directories are persisted; logs, settings, and user data remain temporary
  • snapshots larger than 8 GB are reported but not published

Test selection, assertions, process isolation, and coverage are unchanged. The inventory output will provide cache size and growth data during the rollout.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Validation

  • uv run python -m pytest .github/actions/run-package-tests/test_renderer_cache_inventory.py
  • bash -n .github/actions/run-tests/run_tests.sh
  • parsed all changed action/workflow YAML
  • uv run isaaclab -f

Checklist

  • I have read and understood the contribution guidelines
  • I have run the pre-commit checks
  • My changes generate no new warnings
  • I have added focused tests for cache inventory and change detection
  • No source package changed, so no package changelog fragment is required
  • My name is already present in CONTRIBUTORS.md

@StafaH
StafaH requested a review from a team August 21, 2026 07:51
@greptile-apps

greptile-apps Bot commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds hardware- and Isaac-Sim-specific persistence for renderer runtime artifacts in task-test CI shards.

  • Computes cache compatibility from the image, Isaac Sim version, runner platform, GPU model, and driver.
  • Restores cache snapshots into isolated Kit, RTX, and NVIDIA compute-cache bind mounts.
  • Restricts publication to the post-merge warm job and skips empty, unchanged, or oversized snapshots.
  • Adds inventory, fingerprinting, and focused unit tests for cache reporting and change detection.

Confidence Score: 5/5

The PR appears safe to merge with no concrete blocking or independently actionable non-blocking issues identified.

Cache compatibility, per-job restore isolation, renderer-only mounts, publication trust boundaries, and snapshot validation are consistently wired through the changed workflow and actions.

Important Files Changed

Filename Overview
.github/actions/renderer-cache-key/action.yml Adds a renderer-cache key derived from the Isaac Sim image, runner platform, GPU inventory, and driver version.
.github/actions/run-package-tests/action.yml Integrates renderer-cache restore, inventory, growth checks, and trusted warm-job publication into package tests.
.github/actions/run-package-tests/renderer_cache_inventory.py Adds metadata-based cache fingerprinting and size inventory grouped by top-level directory.
.github/actions/run-package-tests/test_renderer_cache_inventory.py Tests inventory grouping and fingerprint changes when an artifact is modified.
.github/actions/run-tests/action.yml Forwards the renderer cache host directory to the container test script.
.github/actions/run-tests/run_tests.sh Redirects only renderer-related Kit, XDG, and NVIDIA compute-cache paths to persistent bind mounts.
.github/workflows/build.yaml Enables restore-only renderer caching for PR task shards and save mode for the trusted post-merge warm job.

Sequence Diagram

sequenceDiagram
    participant Workflow as CI workflow
    participant Key as Renderer cache key action
    participant Cache as GitHub Actions cache
    participant Tests as Package-test container
    participant Inventory as Cache inventory

    Workflow->>Key: Image/version and runner hardware
    Key-->>Workflow: Compatibility collection and run key
    Workflow->>Cache: Restore newest compatible snapshot
    Cache-->>Workflow: Renderer cache directory
    Workflow->>Tests: Bind-mount cache directories
    Tests-->>Workflow: Updated runtime artifacts
    Workflow->>Inventory: Measure and fingerprint cache
    alt "Trusted warm job and changed snapshot <= 8 GB"
        Workflow->>Cache: Save immutable snapshot
    else PR job, unchanged, empty, or oversized
        Workflow-->>Workflow: Skip publication
    end
Loading

Reviews (1): Last reviewed commit: "Cache renderer runtime artifacts in CI" | Re-trigger Greptile

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac Lab Review Bot

The renderer-cache flow follows the existing restore-only/save-only trust model, but two scope and contract issues should be addressed before merge: the new cache input silently depends on volume-mount-source, and the persisted home cache includes all of XDG_CACHE_HOME rather than only renderer artifacts.

  • Design and architecture: The compatibility key, restore fallback, sole-writer warm job, size limit, and fingerprint-based publish gates are consistent with the Warp-cache design. However, bind-mounting /tmp/isaaclab-ci-home/.cache persists every cache under XDG_CACHE_HOME, broadening the artifact set beyond the documented Kit, RTX, and NVIDIA compute caches and allowing unrelated entries to consume the shared 8 GB budget. Narrow the persisted paths to renderer-specific directories.
  • API: The new inputs default to empty, preserving existing callers, and the added positional argument matches the updated action invocation. However, renderer-cache-host-dir is documented as an independent optional input while its implementation is nested under the unrelated volume-mount-source branch. Supplying the renderer cache alone therefore silently has no effect; either support it independently or validate and document the dependency.
  • Implementation: Key generation, restore/save handoff, inventory reporting, fingerprint comparison, and empty/oversized publish guards are internally consistent. The remaining implementation fixes are to move or validate renderer mount setup outside the source-mount condition and to avoid publishing the entire home XDG cache tree.

Minor fixes needed. Posted 2 actionable findings inline.

Automated review; human maintainers own approval decisions.

isaacsim_cache_dir="${docker_runtime_dir}/isaac-sim/cache"
isaacsim_compute_cache_dir="${docker_runtime_dir}/isaac-sim/computecache"
home_cache_args=""
if [ -n "$renderer_cache_host_dir" ]; then

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Warning · Api — Renderer cache silently requires unrelated source mount

renderer-cache-host-dir is declared as an independent optional input, but its mount setup is nested inside the if [ -n "$volume_mount_source" ] block, while the analogous warp-cache-host-dir is handled outside it. A caller that supplies only the renderer cache directory gets no mounts and no diagnostic, so the cache silently does nothing. Handle the renderer mounts outside that branch, or validate and document the dependency.

isaacsim_cache_dir="${renderer_cache_host_dir}/isaac-sim/cache"
isaacsim_compute_cache_dir="${renderer_cache_host_dir}/isaac-sim/computecache"
home_cache_args="\
-v ${renderer_cache_host_dir}/home/cache:/tmp/isaaclab-ci-home/.cache:rw \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Warning · Design Architecture — Cache mount covers entire XDG cache home

/tmp/isaaclab-ci-home/.cache is both HOME/.cache and XDG_CACHE_HOME, so this mount persists and publishes every tool cache written there (including pip/uv caches from extra-pip-packages/extra-uv-packages), not just Kit/RTX artifacts as the input documents. Those entries are keyed only on GPU, driver and Isaac Sim image and consume the 8 GB publish budget. Mount only the renderer subdirectories instead.

@StafaH StafaH closed this Aug 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants