[Backport] Update Visualization Docs (#7054) - #7555
Conversation
Update these pages - Visualization, Record Video, Visualization Markers, Streaming Camera View, Stage Optimization, Scene Data Providers Updates visualization, video recording, streaming camera view docs - Adds good amount of visuals. Kept the visualization page < 20MB. - Uses mp4 files instead of GIFs, because the h.264 encoding is very efficient and keeps the clips much more lightweight - The videos are set to auto-play and loop like GIFs - Uploaded al .mp4s to our documentation assets server Bug fixes - Fixed `NewtonGLVisualizer.render_rgb_array` omitting visualization markers, so recorded videos showed the scene without its goal poses/command arrows visible in the interactive viewer - Fixed `NewtonRTXVisualizer` unconditionally reporting the streaming/tiled camera view as unsupported; `streaming_view=True` now creates the owned streaming camera sensor for headless capture (the live on-screen preview panel is still unavailable on this backend) - Implemented `RerunVisualizer.set_camera_view` / `ViserVisualizer.set_camera_view` (previously no-op stubs), enabling programmatic follow-cameras Misc - Remove unused images ## Release backport - [x] <!-- backport-active-release --> Backport this pull request to the active release branch after it merges into `develop` # Description > [!IMPORTANT] > Confirm the pull request base before submitting. Target `develop` for all > contributions. The `release/3.0.0-beta2` branch is a frozen stable landing > snapshot and is not used for ongoing maintenance. <!-- Thank you for your interest in sending a pull request. Please make sure to check the contribution guidelines. Link: https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html 💡 Please try to keep PRs small and focused. Large PRs are harder to review and merge. --> Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. Fixes # (issue) <!-- As a practice, it is recommended to open an issue to have discussions on the proposed pull request. This makes it easier for the community to keep track of what is being developed or added, and if a given feature is demanded by more than one party. --> ## Type of change <!-- As you go through the list, delete the ones that are not applicable. --> - Documentation update - Bug fix (non-breaking change which fixes an issue) ## Screenshots Please attach before and after screenshots of the change if applicable. <!-- Example: | Before | After | | ------ | ----- | | _gif/png before_ | _gif/png after_ | To upload images to a PR -- simply drag and drop an image while in edit mode and it should upload the image directly. You can then paste that source into the above before/after sections. --> ## Checklist - [ ] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html) - [ ] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have added a changelog fragment under `source/<pkg>/changelog.d/` for every touched package (do **not** edit `CHANGELOG.rst` or bump `extension.toml` -- CI handles that) - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task --> --------- Signed-off-by: matthewtrepte <mtrepte@nvidia.com> (cherry picked from commit 7cc8292)
github-actions
Bot
added
documentation
Greptile SummaryThis backport reorganizes and expands the visualization documentation while adding camera controls, streaming capture support, marker-aware Newton capture, and corrected video rollout timing.
Confidence Score: 4/5The multi-recorder rollout horizon should be corrected before merging because later video streams can otherwise be truncated. All four RL play backends stop according to only the first recorder even though configurations may retain multiple recorders with independent lengths and offsets. Files Needing Attention: source/isaaclab_rl/isaaclab_rl/entrypoints/backends/play_rl_games.py, play_rsl_rl.py, play_sb3.py, and play_skrl.py Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart LR
CLI[Play command with --video] --> Apply[Apply video recorder configuration]
Apply --> Recorders[Configured video recorders]
Recorders --> Loop[RL backend rollout loop]
Loop --> Step[Environment step]
Step --> Capture[Each recorder applies its own offset and length]
Capture --> Stop[Rollout stop horizon]
Stop --> Close[Close environment and flush clips]
Reviews (1): Last reviewed commit: "Update Visualization Docs (#7054)" | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| if video_stop is None: | ||
| recorders = getattr(env_cfg, "video_recorders", []) | ||
| video_stop = recorders[0].video_length if recorders else None | ||
| video_stop = recorders[0].video_length + recorders[0].step_offset if recorders else None |
There was a problem hiding this comment.
When multiple configured recorders have different video_length + step_offset horizons, this fallback stops the rollout at the first recorder's horizon, causing later recorders to flush clips shorter than their configured length. The same calculation is present in the RSL-RL, SB3, and SKRL play backends.
![isaaclab-review-bot[bot]](https://avatars.githubusercontent.com/u/157846462?s=60&v=4){kind=small}
There was a problem hiding this comment.
Isaac Lab Review Bot
The documentation reorganization and shared Newton streaming path are coherent, but the replacement pages omit required installation and partial-visualization guidance, contradict the newly enabled Newton RTX streaming capture behavior, and the new Rerun camera API can discard registered live-plot panels.
- Design and architecture: The concepts/features documentation split, updated toctrees, and shared Newton streaming-composite implementation are consistent. However, Rerun camera updates rebuild a reduced blueprint rather than preserving the shared layout, causing live-plot views to disappear.
- API: The moved pages preserve important reference labels, and the new Rerun/Viser set_camera_view signatures align with the existing visualizer API. The Rerun implementation must preserve registered blueprint views, and the public partial-visualization options removed from the old page should remain documented.
- Implementation: The video stop calculation consistently includes step_offset across all four play backends. Before merge, restore the video-extra prerequisite, document the retained partial-visualization controls, and correct the recording guide so Newton RTX streaming capture is distinguished from its unavailable live on-screen preview.
Minor fixes needed. Posted 4 actionable findings inline.
Automated review; human maintainers own approval decisions.
| VideoRecorderCfg(source="visualizer:kit", output_dir="videos/") | ||
| ] | ||
|
|
||
| Or pass ``--video`` on the command line to record from the default visualizer without editing |
There was a problem hiding this comment.
🟡 Warning · Implementation — Video extra install requirement dropped in move
The replaced how-to/record_video.rst had a Requirements section stating that the video extra must be installed to provide moviepy and its ffmpeg runtime (--extra video in a uv checkout). The new page never mentions moviepy, ffmpeg, or that extra, so a reader following this Quick Start --video command hits a missing-dependency failure with no guidance. Re-add the prerequisite.
| ) | ||
| self._last_camera_pose = (cam_pos, cam_target) | ||
|
|
||
| def set_camera_view( |
There was a problem hiding this comment.
🟡 Warning · Api — Camera updates discard live-plot blueprint
The new public method delegates to _apply_camera_pose, which sends a hard-coded blueprint containing only the 3D and physics-backend views. After add_live_plots registers _live_plot_manager_names, the documented per-step camera-follow call replaces the active blueprint and drops those time-series panels. Rebuild through the shared _get_blueprint layout (or skip re-sending when the pose is unchanged) so registered views survive.
| - Newton Warp renderer; use ``max_visible_envs`` to limit the number of rendered | ||
| environments. | ||
| * - Newton RTX | ||
| - Path-traced; highest per-frame cost, use ``--max_visible_envs`` to reduce load. |
There was a problem hiding this comment.
🔵 Suggestion · Implementation — Partial visualization fields no longer documented
The deleted overview/core-concepts/visualization.rst documented --max_visible_envs, max_visible_envs, visible_env_indices, randomly_sample_visible_envs, and Newton world_spacing. The Performance table here still cites max_visible_envs/--max_visible_envs, but no page in this change defines them, leaving those public config options undocumented. Add a short partial-visualization subsection under Usage.
|
|
||
| .. note:: | ||
|
|
||
| The Newton RTX viewer framebuffer can be recorded with ``"visualizer:newton_rtx"``, but |
There was a problem hiding this comment.
🔵 Suggestion · Api — RTX streaming note contradicts shipped behavior
This note says recording the Newton RTX streaming view is unsupported, but the same PR hoists render_tiled_rgb_array() onto the shared Newton base, its changelog fragment states RTX composites are now usable for headless capture through VideoRecorderCfg, and concepts/visualization.rst says headless streaming capture works. Update the note so only the live on-screen preview is described as unavailable.
|
run-ci |
isaaclab-bot
Bot
added
ci:run-docker
Signed-off-by: Kelly Guo <kellyg@nvidia.com>
|
run-ci |
isaaclab-bot
Bot
added
ci:run-docker
3 participants
Description
Backports #7054 to
release/3.0.0.The automatic backport run selected the PR and completed the cherry-pick, but its exact-patch validator then failed because this source path was omitted as a no-op:
source/isaaclab_rl/changelog.d/maximiliank-skip-unsupported-checkpoint-presets.skipThat fragment is already present on
release/3.0.0as the same empty blob (e69de29bb2d1d6434b8b29ae775ad8c2e48c5391). The repository's non-exact backport validation confirms that every other source path is preserved and reports this as the onlymissing_paths_already_presententry. No conflict resolution or manual source edits were required.7cc829274ce6050dc8288f2ea93f2b5c2175dd382af02510c1c599e23aa51f0ba81922d2c1af3cddbb07b4b09Type of change
Validation
.skipfragment described above.sphinx-lintreported no problems for added and modified RST files.Checklist
CONTRIBUTORS.md