Fix Warp camera runtime errors

[kellyguo11]

Description

Fixes NVBug 6675392 and NVBug 6684416

This PR fixes two camera-rendering failures at their respective boundaries:

• Newton Warp rendering now marks deformable triangle-mesh render work as non-graph-capturable because Warp mesh refits record allocation nodes that conditional CUDA graph bodies do not support. Other graph-safe sensor tasks remain capturable.
• Isaac RTX rendering now treats an empty annotator warm-up frame as not ready, clears the destination buffer, and skips Warp slicing and reshape work for that frame.

No new dependencies are required.

Validation

NVBug 6675392 exact command

uv run --extra isaacsim,all,rlinf,mimic,teleop,tetrahedralization,video,leapp isaaclab train --rl_library rsl_rl --task Isaac-Lift-Cloth-Franka-Camera --info --max_iterations 5

• PR parent 3639364a: reproduced Conditional body graph contains an unsupported operation (memory allocation) and the sensor CUDA graph capture failed traceback, then completed learning iterations 0/5 through 4/5 via the existing eager fallback.
• This PR: completed learning iterations 0/5 through 4/5 without the conditional-body or sensor-capture traceback.

NVBug 6684416 exact commands

Primary command (Windows path separators translated to Linux path separators only):

uv run --extra all,isaacsim,rlinf,mimic,teleop,tetrahedralization,video,leapp python scripts/environments/zero_agent.py --task IsaacContrib-Stack-Cube-Franka-IK-Rel-Visuomotor-Cosmos --visualizer newton_gl

Ubuntu multi-visualizer command:

uv run --extra isaacsim,all,rlinf,mimic,teleop,tetrahedralization,video,leapp python scripts/environments/zero_agent.py --task IsaacContrib-Stack-Cube-Franka-IK-Rel-Visuomotor-Cosmos --visualizer kit,rerun,newton_gl,viser physics=isaacsim_physx

• PR parent and this PR: both commands completed environment setup, reached Zero agent is running, and stepped until an external watchdog stopped the intentionally unbounded process (90-120 seconds). Neither revision raised Invalid indexing in slice on the locally available repository-pinned Isaac Sim 6.0.1.0 stack.
• The ticket reports Isaac Sim 6.1.0.0-rc.12, which is not installed locally. A run using the closest local 6.1 source build (6.1.0-alpha.56) at the ticket's reported Isaac Lab revision stopped on an unrelated 4096-environment RTX allocation/OOM failure before reaching the reported invalid-slice path, so it is not counted as a reproduction.
• The focused empty-annotator-frame regression test deterministically fails on the PR parent with the reported invalid-slice behavior and passes on this PR.

The unbounded zero-agent commands were wrapped only in a timeout, and OMNI_KIT_ACCEPT_EULA=YES was scoped to the processes after confirming an existing accepted-EULA marker. Their command arguments were otherwise unchanged.

Focused and repository checks

• uv run --frozen --extra test python -m pytest source/isaaclab_newton/test/physics/test_newton_manager_abstraction.py source/isaaclab_physx/test/renderers/test_isaac_rtx_renderer_contract.py -q — 200 passed
• The new focused regression tests were verified to fail before the fixes and pass afterward.
• uv run --frozen isaaclab -f
• uv run --frozen --extra test python tools/changelog/cli.py check develop

Type of change

• Bug fix (non-breaking change which fixes an issue)

Release backport

• Backport this pull request to the active release branch after it merges into develop

Screenshots

Not applicable.

Checklist

Docker and GPU tests run on demand. Push the commits you want tested, then
comment run-ci on the pull request.

• I have read and understood the contribution guidelines
• I have run the pre-commit checks with uv run isaaclab -f
• I have made corresponding changes to the documentation (changelog fragments; no public API change)
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package (do not edit CHANGELOG.rst or bump extension.toml — CI handles that)
• I have added my name to the CONTRIBUTORS.md or my name already exists there

[greptile-apps]

Greptile Summary

This PR fixes two renderer boundary failures:

• Excludes Newton Warp deformable triangle-mesh rendering from conditional CUDA graph capture while retaining capture for compatible sensor tasks.
• Clears output for empty Isaac RTX annotator warm-up frames and skips invalid slicing and reshape work.
• Adds focused regression coverage and changelog fragments for both fixes.

Confidence Score: 5/5

The PR appears safe to merge; both renderer failure paths are handled without breaking the surrounding scheduling or output contracts.

No actionable failures remain: eager Newton tasks maintain consistent graph lifecycle and sensor state, while empty RTX frames clear the appropriate buffer before safely skipping frame processing.

Important Files Changed

Filename	Overview
source/isaaclab_newton/isaaclab_newton/physics/newton_manager.py	Separates non-capturable sensor tasks from conditional CUDA graph capture while preserving task lifecycle, BVH refitting, and eager fallback behavior.
source/isaaclab_newton/isaaclab_newton/renderers/newton_warp_renderer.py	Marks rendering with deformable triangle indices as eager to avoid unsupported allocation nodes inside conditional graph bodies.
source/isaaclab_newton/test/physics/test_newton_manager_abstraction.py	Verifies eager task execution and the renderer’s triangle-mesh capture classification.
source/isaaclab_physx/isaaclab_physx/renderers/isaac_rtx_renderer.py	Detects empty annotator frames, clears the correct destination, and bypasses invalid frame processing.
source/isaaclab_physx/test/renderers/test_isaac_rtx_renderer_contract.py	Verifies that an empty warm-up frame clears output without launching the reshape kernel.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Newton sensor update requested] --> B{Requested task marked eager?}
    B -->|Yes| C[Refit dirty sensor BVHs]
    C --> D[Run requested task eagerly]
    B -->|No| E[Capture or launch conditional sensor graph]

    F[RTX annotator returns frame] --> G{Frame empty?}
    G -->|Yes| H[Clear destination or HDR scratch buffer]
    H --> I[Skip conversion, slicing, and reshape]
    G -->|No| J[Convert and reshape tiled frame]
    I --> K[Run PPISP when configured]
    J --> K

Loading

_{Reviews (1): Last reviewed commit: "Fix Warp camera runtime errors" | Re-trigger Greptile}

[kellyguo11]

run-ci

[isaaclab-review-bot]

Isaac Lab Review Bot

Reviewed the Newton sensor-task graph-capture opt-out and the Isaac RTX empty-frame handling, tracing task registration, graph invalidation and indexing, eager execution, and output-buffer selection. The changes are focused and supported by regression coverage.

• Design and architecture: The keyword-only capturability flag and eager-task registry cleanly separate unsupported work from conditional CUDA graph capture. Eager tasks are consistently excluded from graph construction and flag indexing, while registration, unregistration, and reset paths maintain registry consistency. A mixed request containing an eager task conservatively executes all requested tasks eagerly for that frame, which is a reasonable non-blocking performance tradeoff.
• API: The changed sensor registration method is private and adds a backward-compatible keyword-only argument with a default preserving existing behavior. Existing registrations remain valid, lifecycle reset behavior includes the new state, and both affected packages include appropriate changelog fragments.
• Implementation: Newton Warp rendering classifies nonempty triangle-index workloads at initial task registration and routes them through eager execution, avoiding unsupported allocation work inside conditional graph bodies. Isaac RTX resolves the correct user or HDR scratch destination before checking for an empty annotator frame, clears that destination, and skips conversion, slicing, and reshape work. Focused tests cover both boundaries.

No blocking issues. No inline issue met the actionable-evidence threshold; the assessment above records the review feedback.

Automated review; human maintainers own approval decisions.

[hujc7]

Superseded — re-posted as inline comments anchored to the relevant lines, so each finding sits on the code it concerns. See the inline thread(s) on this PR.

[hujc7]

approved to unblock

[hujc7]

[P1] Agent review: this test uses data_type="rgba", which never reaches the slice that raises the reported error, so it does not cover the reported failure.

Measured on the PR parent with an empty (0, 0, 4) frame:

data_type	Result
rgba	no exception at all; wp.launch still called
normals	RuntimeError: Invalid indexing in slice: 0:0:1

The test does fail on the parent, but on the zero_ assertion rather than the reported behaviour. Only normals, motion_vectors, SIMPLE_SHADING_MODES and rgb_hdr reach the slice.

Suggested: parametrize over one of those, threading data_type through the annotators / output_data dicts.

@pytest.mark.parametrize("data_type", ["rgba", "normals"])
def test_render_treats_empty_annotator_frame_as_not_ready(monkeypatch, data_type):

Verified both halves: normals fails without the fix for the reported reason, and passes with it.

[kellyguo11]

Addressed in 65b282a. The test now covers both rgba and normals. I verified the normals case against the PR parent: it fails with the reported RuntimeError: Invalid indexing in slice: 0:0:1; both parameter cases pass with the fix. The combined focused suites pass with 201 tests, and uv run --frozen isaaclab -f passes against the PR base.

[kellyguo11]

run-ci