Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/check-links.yml
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,7 @@ jobs:
--exclude 'andrew\.cmu\.edu/course/10-703/textbook/BartoSutton\.pdf'
--exclude 'www\.nvidia\.com/en-us/security'
--exclude 'bostondynamics\.com/reinforcement-learning-researcher-kit'
--exclude 'opensource\.org'
--max-retries 5
--retry-wait-time 10
--timeout 20
Expand Down
1 change: 1 addition & 0 deletions docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -328,6 +328,7 @@ def _read_pinned_versions() -> dict:
html_static_path = [
"source/_static/css",
"source/_static/how-to",
"source/_static/visualizers",
"source/_static/tasks/previews",
"source/_static/benchmarks",
]
Expand Down
5 changes: 5 additions & 0 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,8 @@ Table of Contents
source/concepts/backends_and_presets
source/concepts/actuators
source/concepts/coupled_solvers
source/concepts/scene_data_providers
source/concepts/visualization
source/concepts/reinforcement_learning


Expand Down Expand Up @@ -140,6 +142,9 @@ Table of Contents
source/features/population_based_training
source/features/ray
source/features/reproducibility
source/features/draw_markers
source/features/record_video
source/features/visualizer_tiled_camera


.. toctree::
Expand Down
8 changes: 8 additions & 0 deletions docs/source/_static/css/custom.css
Original file line number Diff line number Diff line change
Expand Up @@ -187,3 +187,11 @@ figure.nurec-gif-contain img {
button.copybtn {
opacity: 1;
}

/* Force readable code-block text in dark mode; some rendering paths were
showing unstyled text in a dark color against the dark code background. */
html[data-theme="dark"] pre,
html[data-theme="dark"] .highlight,
html[data-theme="dark"] .highlight pre {
color: #ffffff !important;
}
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed docs/source/_static/visualizers/newton_viz.jpg
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file removed docs/source/_static/visualizers/ov_viz.jpg
Binary file not shown.
Binary file removed docs/source/_static/visualizers/rerun_viz.jpg
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@
Scene Data Provider
===================

The :class:`~isaaclab.scene_data.SceneDataProvider` bridges physics simulation
backends and the visualizers/renderers that consume scene data. It exposes a single Warp-native
read path for body transforms regardless of which physics backend (PhysX or Newton) is active,
so renderers and visualizers can stay backend-agnostic.
:class:`~isaaclab.scene_data.SceneDataProvider` bridges physics simulation backends and the
visualizers/renderers that consume scene data. It exposes a single Warp-native read path for
body transforms regardless of which physics backend (PhysX or Newton) is active, so renderers
and visualizers can stay backend-agnostic.

Overview
--------

Isaac Lab supports multiple physics backends (PhysX and Newton) and multiple visualizers
(Omniverse Kit, Newton, Rerun, Viser). Each combination needs scene data to flow from the
physics engine into the renderer or visualizer. The :class:`SceneDataProvider` owns this flow:
the physics manager provides a :class:`~isaaclab.scene_data.SceneDataBackend` that wraps its
native tensor views, and the provider handles format conversion and re-mapping on top of it.
physics engine into the renderer or visualizer. :class:`SceneDataProvider` owns this flow: the
physics manager provides a :class:`~isaaclab.scene_data.SceneDataBackend` that wraps its native
tensor views, and the provider handles format conversion and re-mapping on top of it.

.. code-block:: python

Expand All @@ -28,55 +28,55 @@ Architecture

The system has three layers:

1. :class:`~isaaclab.scene_data.SceneDataBackend` small interface implemented by each physics
1. :class:`~isaaclab.scene_data.SceneDataBackend`: a small interface implemented by each physics
manager. It exposes the backend's transform array directly as one of the
:class:`~isaaclab.scene_data.SceneDataFormat` Warp structs, plus the per-transform prim paths
and total count. There is no per-frame "update" call the property accessors return live
and total count. There is no per-frame "update" call; the property accessors return live
views into the underlying tensor each time they're read.

- :attr:`SceneDataBackend.transforms` current transforms as a Warp struct (one of
- :attr:`SceneDataBackend.transforms`: current transforms as a Warp struct (one of
:class:`SceneDataFormat.Vec3_Quat`, :class:`SceneDataFormat.Transform`,
:class:`SceneDataFormat.Matrix44`, :class:`SceneDataFormat.Vec3_Matrix33`).
- :attr:`SceneDataBackend.transform_count` number of transforms.
- :attr:`SceneDataBackend.transform_paths` list of USD prim paths, one per transform.
- :attr:`SceneDataBackend.points` flattened deformable nodal positions as
- :attr:`SceneDataBackend.transform_count`: number of transforms.
- :attr:`SceneDataBackend.transform_paths`: list of USD prim paths, one per transform.
- :attr:`SceneDataBackend.points`: flattened deformable nodal positions as
:class:`SceneDataFormat.Points` (optional; rigid-only backends return an empty buffer).
- :attr:`SceneDataBackend.point_count` total number of geometry points.
- :attr:`SceneDataBackend.geometry_paths` one USD prim path per deformable body instance.
- :attr:`SceneDataBackend.geometry_counts` unpadded nodal count per geometry entity.
- :attr:`SceneDataBackend.point_count`: total number of geometry points.
- :attr:`SceneDataBackend.geometry_paths`: one USD prim path per deformable body instance.
- :attr:`SceneDataBackend.geometry_counts`: unpadded nodal count per geometry entity.

2. :class:`~isaaclab.scene_data.SceneDataProvider`wraps a backend and offers
format conversion plus index re-mapping:
2. :class:`~isaaclab.scene_data.SceneDataProvider`: wraps a backend and offers format conversion
plus index re-mapping.

- :meth:`SceneDataProvider.get_transforms` — write the backend's transforms into a
- :meth:`SceneDataProvider.get_transforms`: writes the backend's transforms into a
consumer-provided :class:`SceneDataFormat` struct, optionally converting format
(e.g. ``Vec3_Quat`` ``Transform``) and applying an index mapping. When the backend
(e.g. ``Vec3_Quat`` to ``Transform``) and applying an index mapping. When the backend
format matches the output format and no mapping is provided, the result is a zero-copy
passthrough.
- :meth:`SceneDataProvider.create_mapping` — build a remap array from the backend's prim
- :meth:`SceneDataProvider.create_mapping`: builds a remap array from the backend's prim
paths to a consumer's desired ordering. Used when a renderer or visualizer wants
transforms indexed by its own body list rather than by the physics view order.
- :meth:`SceneDataProvider.get_points` — copy backend deformable nodal positions into a
- :meth:`SceneDataProvider.get_points`: copies backend deformable nodal positions into a
consumer buffer, optionally remapping entity slices via
:meth:`SceneDataProvider.create_geometry_mapping`.
- :meth:`SceneDataProvider.create_geometry_mapping` — map backend deformable entities to
- :meth:`SceneDataProvider.create_geometry_mapping`: maps backend deformable entities to
consumer particle offsets in a shadow Newton ``particle_q`` buffer.
- :meth:`SceneDataProvider.get_camera_transforms` — discover per-camera, per-env
world transforms from the USD stage.
- :attr:`SceneDataProvider.usd_stage` USD stage handle for stage-walking consumers.
- :attr:`SceneDataProvider.num_envs` environment count inferred from
- :meth:`SceneDataProvider.get_camera_transforms`: discovers per-camera, per-env world
transforms from the USD stage.
- :attr:`SceneDataProvider.usd_stage`: USD stage handle for stage-walking consumers.
- :attr:`SceneDataProvider.num_envs`: environment count inferred from
``/World/envs/env_<id>`` prims.

3. Backend implementations:

- ``PhysxSceneDataBackend`` (internal to :mod:`isaaclab_physx.physics`) wraps PhysX's
``RigidBodyView`` and exposes its transforms as :class:`SceneDataFormat.Transform`.
When deformable bodies are present it also exposes flattened simulation nodal positions
through :class:`SceneDataFormat.Points`.
- ``OvPhysxSceneDataBackend`` (internal to :mod:`isaaclab_ov.physics`) mirrors the
PhysX contract for rigid transforms and OVPhysX deformable nodal tensors.
- ``NewtonSceneDataBackend`` (internal to :mod:`isaaclab_newton.physics`) wraps the
Newton model's ``body_q`` and exposes it as :class:`SceneDataFormat.Transform`.
``RigidBodyView`` and exposes its transforms as :class:`SceneDataFormat.Transform`. When
deformable bodies are present it also exposes flattened simulation nodal positions through
:class:`SceneDataFormat.Points`.
- ``OvPhysxSceneDataBackend`` (internal to :mod:`isaaclab_ov.physics`) mirrors the PhysX
contract for rigid transforms and OVPhysX deformable nodal tensors.
- ``NewtonSceneDataBackend`` (internal to :mod:`isaaclab_newton.physics`) wraps the Newton
model's ``body_q`` and exposes it as :class:`SceneDataFormat.Transform`.

PhysX backend
-------------
Expand All @@ -87,30 +87,30 @@ The transforms are returned as :class:`SceneDataFormat.Transform` (Warp ``transf
so consumers that want this format get them zero-copy.

Newton-native consumers (Newton visualizer, Rerun, Viser, Newton Warp renderer, OVRTX renderer)
additionally need a Newton ``Model``/``State`` to render against. To satisfy that requirement,
also need a Newton ``Model``/``State`` to render against. To provide that,
:class:`~isaaclab_newton.physics.NewtonManager` builds a **shadow Newton model** from the USD
stage on first access and updates its ``body_q`` from the PhysX backend each render frame.
When the scene contains PhysX or OVPhysX deformables, the shadow model also allocates
When the scene has PhysX or OVPhysX deformables, the shadow model also allocates
``particle_q`` render slots for soft/cloth meshes, syncs simulation nodal positions through
:meth:`SceneDataProvider.get_points` with ``allow_passthrough=False`` into a separate
sim-sized buffer, and remaps or copies those positions into the render-sized
``particle_q`` buffer each frame. Volume deformables with mismatched sim and visual vertex
counts use a barycentric sim-to-visual remap so Newton Warp and OVRTX render the paired
visual mesh rather than tet simulation topology. The shadow deformable registry exposes
render-slot offsets and ``particles_per_body`` counts for OVRTX point bindings.
sim-sized buffer, and remaps or copies those positions into the render-sized ``particle_q``
buffer each frame. Volume deformables with mismatched sim and visual vertex counts use a
barycentric sim-to-visual remap so Newton Warp and OVRTX render the paired visual mesh rather
than tet simulation topology. The shadow deformable registry exposes render-slot offsets and
``particles_per_body`` counts for OVRTX point bindings.

This is hidden behind :meth:`NewtonManager.get_model` / :meth:`NewtonManager.get_state`, so
renderers don't need to know which physics backend is active.

Newton backend
--------------

When Newton is the active physics backend, the backend wraps the Newton model's ``body_q``
directly. No shadow model or per-frame sync is needed — Newton already owns the authoritative
model and state, and the provider exposes that state as
:class:`SceneDataFormat.Transform`.
directly. No shadow model or per-frame sync is needed: Newton already owns the authoritative
model and state, and the provider exposes that state as :class:`SceneDataFormat.Transform`.

Data requirements
-----------------
------------------

Visualizers and renderers declare what they need from the scene data path. This is resolved at
simulation-context construction time and is what triggers the shadow-model build for PhysX:
Expand Down Expand Up @@ -146,5 +146,5 @@ simulation-context construction time and is what triggers the shadow-model build
See Also
--------

- :doc:`renderers` renderer backends that consume scene data
- :doc:`/source/overview/core-concepts/visualization` visualizer backends that consume scene data
- :doc:`/source/overview/core-concepts/renderers`: renderer backends that consume scene data
- :doc:`/source/concepts/visualization`: visualizer backends that consume scene data
Loading
Loading