Add OVRTX shader cache for CI rendering tests - #6905
Conversation
Cache the NVIDIA Vulkan driver PSO blobs (nv_shadercache/GLCache, ~140 MB
each) that are compiled on every cold CI run and account for the ~600 s
COLD_CACHE_BUFFER in conftest.py.
Two cache sub-trees are persisted via GitHub Actions cache keyed on
ovrtx version + GPU driver version + OS/arch:
kit/ — nv_shadercache for Kit/AppLauncher rendering, bind-mounted
over /isaac-sim/kit/cache/nv_shadercache so the existing
kit/cache tmpdir mount no longer wipes it each run.
kitless/ — nv_shadercache for the standalone OVRTXRenderer path,
redirected via new OVRTX_SHADER_CACHE_PATH env var that
applies /rtx/shaderDb/driverShaderCachePath before Renderer
construction.
Added:
- .github/actions/ovrtx-shader-cache-key/ — cache key action and
collection_id.py (reads ovrtx version from uv.lock)
- .github/actions/run-package-tests/ovrtx_shader_cache_inventory.py
- tools/verify_ovrtx_shader_cache.py — container-side mount check
- warm-ovrtx-cache post-merge job in build.yaml
- ovrtx-shader-cache: restore on both rendering test jobs
Changed:
- OVRTXRenderer.__init__ now calls _redirect_ovrtx_shader_cache() before
Renderer() to apply the carb setting when the env var is set
- COLD_CACHE_BUFFER docstring explains the cache mechanism
Greptile SummaryThe PR adds persistent, architecture-aware OVRTX shader caching for CI rendering jobs, with separate Kit and kitless cache trees.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains; the cache key now includes the compute capability of the GPU selected by the affected rendering jobs. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart LR
Key[Compute platform, GPU, driver, and runtime keys] --> Restore[Restore Kit and kitless snapshots]
Restore --> Mount[Mount cache trees into test container]
Mount --> Kit[Kit RTX rendering]
Mount --> Kitless[OVRTX kitless rendering]
Kit --> Report[Measure cache growth]
Kitless --> Report
Report --> Verify[Verify both trees are populated]
Verify --> Save[Conditionally publish changed snapshots]
Reviews (3): Last reviewed commit: "Skip republishing unchanged OVRTX shader..." | Re-trigger Greptile |
There was a problem hiding this comment.
Isaac Lab Review Bot
Adds persistent OVRTX shader caching for Kit and kitless rendering through shared CI cache wiring, Docker mounts, a post-merge warmer, and an OVRTXRenderer cache-path redirect. The overall producer/consumer flow is coherent, but the cache compatibility key omits the GPU architecture despite the PSO blobs being architecture-specific.
- Design and architecture: The two-subtree
kit/andkitless/design, shared key action, restore-only test jobs, and post-merge writer follow a consistent cache architecture. However, the collection key usesRUNNER_ARCH, which identifies the host CPU architecture, rather than the GPU architecture required for PSO compatibility. Add a stable, sanitized GPU architecture identifier to prevent incompatible runners from sharing a collection. - API: The new
OVRTX_SHADER_CACHE_PATHenvironment variable is documented in the package changelog, while the composite-action inputs default to disabled and preserve existing callers. No breaking public API change is evident. - Implementation: The restore path was traced through
run-package-tests,run-tests, the nested Docker mounts, the verification script, and the renderer redirect. The actionable defect is in the key computation: OS, CPU architecture, OVRTX version, and driver version are included, but GPU architecture is not, contrary to the compatibility requirement documented beside the code.
Minor fixes needed. Posted 1 actionable finding inline.
Automated review; human maintainers own approval decisions.
Add SM compute capability to the collection key so heterogeneous GPU runner fleets never share incompatible PSO blobs. Previously RUNNER_ARCH captured only the CPU architecture; runners with different GPU generations (e.g. Ampere vs Hopper) but the same driver version would compute the same key and risk a cross-architecture cache hit.
|
@greptile review |
|
How cache key is generated
|
AntoineRichard
left a comment
There was a problem hiding this comment.
AI-generated review
Requesting changes. The main blockers are that the shared collection is not keyed by the exact runtime that produces each subtree, and the warmer selects Newton Warp cases while omitting the Kit RTX renderer it is intended to warm. The current failure handling can also publish a partial cache or silently continue after the renderer redirect breaks.
I also left focused inline requests to remove duplicated cache orchestration and repeated implementation documentation, shorten the timeout/changelog text to functional contracts, and add a non-GPU regression test around the redirect boundary.
Verification: reviewed all 11 changed files plus the renderer matrices, cache helpers, workflow callers, prior review threads, and current CI logs. GitHub pre-commit passed. The two rendering jobs populated their individual cold-cache subtrees, but failed golden-image assertions; the combined warm job is skipped on pull requests, so this PR has not demonstrated that the proposed warmer populates both subtrees. I did not run local tests or pre-commit because benchmarks are active on the machine, and I ran no GPU workload.
The kitless rendering matrix was split into explicit legacy and ovstage jobs on develop. Reapply ovrtx-shader-cache: restore to both variants, matching the coverage the matrix job had.
The two cache trees are compiled by different runtimes, but shared one entry keyed on the uv.lock ovrtx version. kit/ is produced by the RTX renderer inside the Isaac Sim image, which never reads that version, so a Kit upgrade silently reused blobs from the previous renderer. kitless/ is produced by the ovrtx wheel the container installs from the pyproject range, which uv.lock does not have to agree with, so a newly published wheel in that range would inherit and then overwrite its predecessor's entry. Cache each tree separately and key it on what actually compiles it: the Isaac Sim image tag for kit/, the exact wheel the pyproject range resolves to for kitless/. Both keep the driver version and GPU compute capability, which gate PSO validity regardless of producer. Resolving the wheel reads metadata from the index the container installs from and falls back to its own collection, so an unreachable index costs a cold compile rather than the job. Gate the writeback per tree on its file count instead of the combined directory size, and fail the warmer when either tree is empty: an empty tree means the selected tests never exercised that render path, which otherwise publishes a half-warm snapshot whose missing half still compiles cold on every consumer.
The warmer selected "newton_warp or ovrtx", but the newton_warp ids run the Warp rasterizer, which compiles no driver shaders. In the Kit file that left zero RTX cases selected, so kit/ was never populated at all, and in the kitless file it added four Warp cases that cannot contribute to either tree. Select the two RTX paths instead: isaacsim_rtx ids come from the Kit file and fill kit/, ovrtx ids from the kitless file and fill kitless/.
The redirect caught every exception and downgraded it to a warning, then logged success unconditionally, so a rejected setting or a change to the private ovrtx bindings left rendering working but compiling cold with nothing in the logs to say the redirect had stopped working. Tolerate only the one expected outcome: a runtime that reports through ovrtx_query_extension that it has no settings extension, which warns and returns. A setting the extension rejects now raises, since the environment variable is an explicit request to redirect, and success is logged only once every setting has applied. Move the policy into its own module so it imports without the ovrtx runtime, and cover it with a fake applier: both setting strings, the unavailable-extension path, and the rejection paths, none of which need a GPU.
The mount layout, cache invalidation cases and a measured compile duration were restated across the timeout constant, the verification script and the action inputs, none of which change how those are used. Keep the layout once beside the mounts that create it in run_tests.sh and leave each boundary describing only its own contract.
Nothing consumed it; the resolved version already reaches the key through the kitless collection prefix.
…dia/IsaacLab into mataylor/ovrtx-shader-cache
StafaH
left a comment
There was a problem hiding this comment.
Thanks @mataylor-nvidia! Looks like a good prototype. I would maybe change the architecture to be more like:
build.yaml
── rendering tests
└── restore-ovrtx-cache
── outputs host-dir
└── warm-ovrtx-cache
── restore-ovrtx-cache
── run warming workload
── verify both shader paths produced actual files
── save-ovrtx-cache
Current prototype is also good, but alot of logic is in different places, we could collect all ovrtx shader cache into one action, keep it nice and containted.
We can do similar to the warp cache if theres a mismatch between their implementations.
WDYT?
|
Yes I think I can try to unify / condense some of the logic to keep it self contained warp cache pr for context: #6809 |
The cache state machine was split across two places: a key-only ovrtx-shader-cache-key action, and ~160 lines of restore, report, save and verify steps in run-package-tests interleaved with the unrelated Warp cache blocks. Callers had to know about kit/kitless trees, the per-tree file counts the writeback gates on, and the environment variables carrying the baseline across the test run. Replace both with a single ovrtx-shader-cache action that owns all of it. A composite action cannot span the caller's test step, so the phases are separate invocations selected by mode: restore before the tests, then report or save after them. run-package-tests now only decides when each phase runs; job-status gating stays at the call site where it already worked. Every mode recomputes the keys from the same key.sh, so the collection a job reads and the one it writes cannot drift. That requires the computation to be constant for the lifetime of a job, which the previous nvidia-smi fallback broke: `|| echo unknown` bound to the pipeline rather than the query, and head/tr both succeed on empty input, so a driver that could not be read yielded a key with blank components instead of failing. Check the captured values instead. Keys are otherwise byte-identical, so collections already published by the warmer stay reachable.
The flag was not needed. ovrtx and ovphysx publish wheel-stub placeholder sdists on public PyPI whose build backend reads index_url from their own pyproject and fetches the real wheel from pypi.nvidia.com, so the unflagged install already resolves them. develop uses this same line for both packages in isaaclab_ov, arm-ci and daily-compatibility. TEST_EXTRA_PIP_PACKAGES is shared by around a dozen callers that install pytetwild and leapp, and pip gives no priority between indexes, so adding a second one changed resolution for packages unrelated to shader caching. Any hardening of the stub download path belongs in its own change.
Trim the rationale comments across the cache action, its scripts, the test runner wiring and the renderer redirect down to the reasoning a reader cannot recover from the code, keeping the non-obvious constraints: the nested kit/ mount winning on destination depth, the renderer config having to reach library initialization, and the writeback gating on per-tree file counts rather than size. Merge the two settings-rejection tests into one parametrized case and the two redirect tests into one, so each behavior is asserted once. Replace the loop in verify.sh with a check_tree function, dropping the per-iteration variable reassignment. No behavior change.
|
🤖 AI-assisted review (batch triage of open CI/infra PRs, with automated code-review/silent-failure/test-coverage passes on a checkout of this branch) Strong PR — cache staleness is designed out (keys cover OS/arch/ Two blocking asks, both small:
Non-blocking findings:
Feel free to disregard anything that doesn't match your intent — happy to be corrected. |
Every later step in the composite reads this step's outputs, but it lacked the always() gate the growth, save and verify steps carry. On a failed job the caller's always()-gated report invocation ran with the key step skipped, leaving host-dir empty and aborting report.sh; in save mode the gates then saw empty counts and published nothing, discarding a whole warm run over one failed assertion.
Each save writes a new immutable entry, so a warm run on develop, main or a release branch published ~270 MB of fresh snapshots per push even when it only read back what it restored. Fingerprint each tree's contents during the restore pass and gate its save on the digest changing, the way the Warp kernel cache already does. Contents rather than size or file count, since the driver rewrites blobs in place as well as appending them. Only jobs that later publish take the baseline, so consumers pay nothing, and an unmeasured baseline still publishes: the gate can cost a duplicate snapshot, never a lost one.
- Key the ovrtx collection on [tool.isaaclab.versions].ovrtx from pyproject.toml instead of the uv.lock pin, since CI installs from resolve-ov-pins, not uv.lock - Let a job restore/save only the tree(s) it needs via a new 'trees' input, instead of always restoring and saving both kit and kitless - Use absolute imports in ovrtx_renderer.py, and move the stdlib ctypes import in ovrtx_shader_cache.py to the top of the file - Raise instead of warn when the ovrtx runtime has no settings extension, since a silent fallback to the default cache path would leave CI reporting a restore hit even though the run recompiled from scratch elsewhere
…hader-cache # Conflicts: # source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_renderer.py
|
run-ci |
|
added pr comment to backport to release |
Backports #6905 to `release/3.0.0`. The original automatic cherry-pick conflicted in `ovrtx_renderer.py`. Its inferred resolution incorrectly replaced the release branch's newer renderer with the older develop-side file, producing a `+444/-1130` renderer diff and removing ovstage behavior. Follow-up commit `64ae8edb1` corrects that resolution: - The other 13 changed files remain exact patch replays of #6905. - `ovrtx_renderer.py` retains the complete release implementation. - The renderer receives only the original shader-cache integration: the `redirect_shader_cache` import and the call immediately before `Renderer(config)`. - The net renderer diff is now seven additions and no deletions. - The net PR diff is 866 additions and 7 deletions. | Field | Commit | |---|---| | Original merged change | `9847b71e3324cb46d4f5882d226ac666bbc22f4d` | | Release base used | `2af02510cc7cac6171fa25bf912f1c4dc83c3279` | | Initial automatic resolution | `88ef9ed54831eb33ea9a93fdc0c860089488f71e` | | Corrected backport | `64ae8edb1` | ## Validation - `uv run --no-project --with pytest --with lazy-loader python -m pytest source/isaaclab_ov/test/test_ovrtx_shader_cache_redirect.py -q` - 6 passed. - Shell syntax checks passed for the shader-cache and test-runner scripts. - All changed YAML files parsed successfully. - Applicable file-scoped pre-commit hooks passed. - The branch-wide changelog hook was skipped locally because it compares historical release differences against `develop`; the existing #6905 `isaaclab_ov` changelog fragment remains included. - The canonical `uv run isaaclab -f` command cannot resolve the release branch's Linux/Windows-only lockfile on macOS; equivalent file-scoped hooks were run directly. --------- Co-authored-by: Matthew Taylor <mataylor@nvidia.com> Co-authored-by: Kelly Guo <kellyg@nvidia.com>
Summary
Each CI run starts with an empty
nv_shadercachedirectory, so the NVIDIA driver recompiles every pipeline state object (PSO) before the first render — roughly 600 s, which is whatCOLD_CACHE_BUFFER = 700intools/conftest.pyexists to absorb. The cost is paid on every rendering job, twice per run (kit-based and kitless).This persists the driver's PSO blobs across runs with GitHub Actions cache, so warm runs skip the compile entirely.
Two independent cache trees live under
${RUNNER_TEMP}/isaaclab-ovrtx-shader-cache/, because the two render paths are compiled by different runtimes and a bump to one must not invalidate the other:kit//isaac-sim/kit/cache/nv_shadercache, which the enclosingkit/cachetmpdir mount would otherwise present as emptykitless/ovrtxwheelOVRTX_SHADER_CACHE_PATH, applied byOVRTXRendereras a carb settingChanges
.github/actions/ovrtx-shader-cache/(new)One action owns the whole lifecycle — key computation, restore, reporting and writeback — so callers only decide when each phase runs. A composite action cannot span the caller's test step, so the phases are separate invocations selected by
mode:restore— reads the newest compatible snapshot of each tree, never writesreport— summarises how far the run compiled beyond what was restoredsave— reports, then writes each populated tree back as a new snapshot; only the warmer uses itEvery mode recomputes keys from the same
key.sh, so the collection a job reads and the one it writes cannot drift.Cache key composition:
RUNNER_OS,RUNNER_ARCHsm<cc>nvidia-smi --query-gpu=compute_capdrv<version>nvidia-smi --query-gpu=driver_versionisaacsim<tag>(kit only)isaacsim-versioninputkit/ovrtx<version>(kitless only)uv.lockpinverify.shfails the warmer when either tree came out empty, which would otherwise publish a silent half-warm snapshot that every consumer prefers over the last good one.CI wiring
run-package-tests: newovrtx-shader-cacheinput (''/restore/save); restores before the tests, reports or publishes after. Publish is gated on!cancelled()rather thansuccess()— compiled PSO blobs stay valid when a golden-image assertion fails.run-tests/run_tests.sh: newovrtx-shader-cache-host-dirinput; owns the canonical mount layout and runs the pre-test mount check.build.yaml:ovrtx-shader-cache: restoreonrendering-correctness,rendering-correctness-kitless-legacyandrendering-correctness-kitless-ovstage; new post-mergewarm-ovrtx-cachejob (continue-on-error, publishes only onmain/develop/release/*) that renders one Kit and one kitless cartpole case to fill both trees.Source
source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_shader_cache.py(new): readsOVRTX_SHADER_CACHE_PATHand points both/rtx/shaderDb/driverShaderCachePathand/rtx/shaderDb/driverAppShaderCachePathat it via the ovrtx settings extension. No-ops when the variable is unset; warns when the runtime has no settings extension; raises when a setting is rejected, since a partial redirect still compiles cold. Kept out ofovrtx_rendererso the policy imports without the ovrtx runtime.ovrtx_renderer.py: callsredirect_shader_cache(OVRTX_CONFIG)before constructingRenderer()— the settings are read at renderer-creation time and cannot be changed on a live instance.tools/verify_ovrtx_shader_cache.py(new): in-container check that the mounted trees are present and writable, run before tests start. A shadowedkit/mount is otherwise invisible — the restore step still reports a hit and the only symptom is a slower job.tools/conftest.py:COLD_CACHE_BUFFERdocstring condensed to its functional contract; the value is unchanged.Tests
source/isaaclab_ov/test/test_ovrtx_shader_cache_redirect.pycovers the redirect boundary with a fake settings applier — no GPU, no ovrtx runtime, no renderer: both settings redirected, rejection raises and is not logged as success, unset env var queries nothing, the renderer config reaches the applier (it is what initializes the library), and a runtime without the extension degrades to a warning.Type of change
Release backport
developTest plan
warm-ovrtx-cacheruns post-merge and publishes both entries — step summary shows a non-zero file count forkit/andkitless/rendering-correctnessandrendering-correctness-kitless-*runs restore the cache and reportfully coveredCOLD_CACHE_BUFFERallowance is no longer consumed on warm runs — the first camera test completes within its normal budgetChecklist
pre-commitchecks with./isaaclab.sh --formatsource/isaaclab_ov/changelog.d/CONTRIBUTORS.md