Proposal: single-environment OVRTX LiDAR adapter through the shared renderer lifecycle

[StriverAlex]

Proposal

I have a working proof of concept for exposing OVRTX 0.4 PointCloud LiDAR products as an IsaacLab
sensor in the kitless isaaclab_ov stack.

The proposed design adds a narrow renderer transaction/registration seam and an
OVRTXLiDARCfg / OVRTXLiDARData / OVRTXLiDAR adapter. Camera and LiDAR products share the same
simulation-scoped OVStage and OVRTX renderer. The sensor owns only its product specification and
handle; it does not create a second stage or renderer. PointCloud outputs remain on CUDA and retain
valid counts, flags, coordinates, intensity, time offset, emitter, channel, tick, and echo fields.
Because OVRTX 0.4 EchoId readback was not reliable in the proof, the initial adapter explicitly
requires maxReturns=1 instead of claiming multi-return support.

LiDAR profile authoring remains explicit and asset-owned. IsaacLab core would not contain Livox or
other vendor scan tables, and the adapter would not fall back to Newton or another backend when
OVRTX is unavailable.

Motivation

IsaacLab's current OVRTX integration exposes camera rendering, while OVRTX itself also provides a
generic LiDAR schema and CUDA PointCloud output. Users who need intensity, timing, emitter/channel
metadata, material response, or OVRTX motion compensation currently have no IsaacLab sensor
lifecycle for that product.

Earlier IsaacLab RTX LiDAR work in PR #802 and
PR #1372 targeted the Isaac Sim/Replicator stack;
issue #865 and
Discussion #291 record the related request
history. This proposal instead targets the kitless isaaclab_ov renderer and shares its existing
scene sync and renderer ownership.

Proven boundary

The proof branch has passed its single-environment correctness and integration gates:

• deterministic plane/box range, occlusion, coordinate-frame, count, and channel-layout checks;
• CUDA-backed PointCloud decoding without steady-state host copies;
• LiDAR-only and Camera+LiDAR operation with one OVStage and one OVRTX renderer;
• sensor mounting on a moving rigid body, reset, and renderer lifecycle checks;
• explicit maxReturns=1 validation for the current OVRTX 0.4 EchoId limitation;
• 72 renderer/sensor contract tests and 9 isolated CUDA integration tests;
• Camera-only p95 regression below the 10% gate on the test GPU.

There is one intentional hard boundary. In paired low-level probes on OVRTX 0.4.0.346409,
non-tiled PointCloud products did not trace scene-partitioned geometry. With only partitioned
geometry both products returned zero points; with an unpartitioned control object they hit only that
control geometry. The adapter therefore rejects num_envs != 1 before opening the renderer scene.
It does not disable partitioning, post-filter points, or allocate one renderer per environment.

The initial contribution would be documented as a generic single-environment OVRTX LiDAR. It would
not claim batched support, strict Livox scan equivalence, or calibrated real-device noise.

Proposed pull-request split

1. Renderer transaction and sensor-registration lifecycle
   • add a default no-op sensor renderer-preparation hook;
   • register renderer-backed sensors before stage preparation;
   • advance time-integrating shared renderers once per simulation step, independent of consumer
     read order;
   • preserve existing on-demand camera behavior;
   • cover the contract with fake-renderer and Camera regression tests;
   • add no OVRTX LiDAR public API.
2. OVRTX PointCloud product and LiDAR sensor
   • add the OVRTX product registry/readback and LiDAR cfg/data/sensor API;
   • require one shared OVRTX renderer and authored OmniLidar profiles;
   • preserve raw CUDA channel contracts and fail fast on unsupported configuration;
   • enforce the current single-environment boundary;
   • add unit, CUDA integration, API documentation, and isaaclab_ov changelog coverage.

The second pull request would depend on the first. Vendor scan profiles and application-specific
robot mounts would remain outside IsaacLab.

Questions for maintainers

1. Would you accept an initial single-environment adapter with the explicit OVRTX 0.4 fail-fast
   boundary, or should the contribution wait for a public PointCloud geometry-isolation contract?
2. Is the two-PR split above the preferred ownership boundary?
3. The contribution guide says to target main, but source/isaaclab_ov exists on develop and not
   on the audited main. Should these PRs target develop?
4. Should a new scoped issue be opened after this Discussion, or should it cross-link/reopen Add RTX Lidar sensor #865
   even though Add RTX Lidar sensor #865 and its successor PRs concerned the Isaac Sim RTX path?
5. Is there an OVRTX-side issue or planned API for per-PointCloud scene-partition isolation that this
   work should track?

Build info

• IsaacLab version: 3.0.0
• proof base: 135cf98904cbd73f3b8bdd89188e1a463ebcbb97
• current audited upstream/develop: 45ca595b11c271f1f202d00f66099480e8f77e32
• OVRTX: 0.4.0.346409 (ovrtx>=0.4.0,<0.5.0)
• OVStage: 0.1.0.346039
• execution path: kitless Newton + OVRTX; Isaac Sim is not used
• GPU used for the full application performance matrix: NVIDIA RTX 4090 D

Acceptance criteria proposed for the initial contribution

• One OVRTX renderer and one OVStage are shared by Camera and LiDAR products.
• A LiDAR-only scene initializes, resets, steps, reads CUDA PointCloud data, and closes.
• Deterministic range, occlusion, coordinate-frame, valid-count, and channel-layout tests pass.
• Moving-mount and full-reset behavior pass.
• Existing OVRTX Camera tests and the Camera-only performance regression gate pass.
• Unsupported multi-environment use fails before scene loading with an actionable error.
• Missing OVRTX/CUDA/schema requirements fail explicitly; no fallback backend is selected.
• maxReturns != 1 fails explicitly until OVRTX provides reliable echo identity readback.
• Public docs state the single-environment and generic-fidelity boundaries.

[RandomOakForest]

Thank you for posting this. I'll move it to our issues so we can address this as a proper feature request proposal.