Consolidate simulation performance guidance (#7365)

## Description

Consolidates the standalone simulation performance how-to into the
troubleshooting reference, where users are more likely to look for slow
simulation and training guidance.

The previous page remains as a compatibility pointer so existing links
continue to work. This also updates the Nsight profiling
cross-reference, points broad tuning guidance to the maintained Isaac
Sim performance guides, and synchronizes the setup-troubleshooting
skill.

## Type of change

- Documentation update

## Release backport

- [x] <!-- backport-active-release --> Backport this pull request to the
active release branch after it merges into `develop`

## Validation

- uv run --isolated --extra test -- make -C docs current-docs
- uv run --no-sync isaaclab -f
- uv run --no-project python tools/skills/cli.py check
- Baseline and skill-assisted slow-throughput routing evaluation

## Checklist

- [x] I have read the contribution guidelines.
- [x] I have run the pre-commit checks.
- [x] I have updated the relevant documentation and troubleshooting
skill.
- [x] The documentation builds with warnings treated as errors.
- [x] A changelog fragment is not required because no source package
changed.

Co-authored-by: Kelly Guo <kellyg@nvidia.com>

Author: AntoineRichard

docs/source/how-to/index.rst

@@ -188,19 +188,6 @@ with directional force feedback in Isaac Lab.
     haply_teleoperation
 
 
-Understanding Simulation Performance
-------------------------------------
-
-This guide provides tips on optimizing simulation performance for different simulation use cases.
-Additional resources are also linked to provide relevant performance guides for Isaac Sim and
-Omniverse Physics.
-
-.. toctree::
-    :maxdepth: 1
-
-    simulation_performance
-
-
 Profiling Isaac Lab with Nsight Systems
 ---------------------------------------
 

docs/source/how-to/profile_with_nsys.rst

@@ -124,6 +124,6 @@ Troubleshooting
 See Also
 --------
 
-- :doc:`simulation_performance` - broader simulation performance tuning tips.
+- :ref:`simulation-performance-troubleshooting` - checks for unexpectedly slow simulation or training.
 - `Nsight Systems User Guide <https://docs.nvidia.com/nsight-systems/UserGuide/index.html>`_ - official ``nsys`` documentation.
 - `NVTX Python package <https://nvtx.readthedocs.io/>`_ - the package nsys uses to emit NVTX ranges from Python.

docs/source/how-to/simulation_performance.rst

@@ -1,80 +1,7 @@
-PhysX Simulation Performance and Tuning
-=========================================
+:orphan:
 
-.. note::
+Simulation Performance
+======================
 
-   This guide covers performance tuning for the **PhysX** backend, which is the default when
-   running Isaac Lab with Isaac Sim. For the **Newton** backend solver parameters (e.g.
-   ``njmax``, ``nconmax``, ``ls_iterations``), see the
-   :ref:`migrating-to-isaaclab-3-0` migration guide and the Newton physics documentation.
-
-The performance of the PhysX simulation can be affected by various factors, including the number
-of objects in the scene, the complexity of the physics simulation, and the hardware being used.
-Here are some tips to improve performance:
-
-1. **Use Headless Mode**: Running the simulation in headless mode can significantly improve performance, especially
-   when rendering is not required. For commands that do not select a visualizer, no viewer is launched unless the configuration requests one. If a config or
-   command would otherwise launch visualizers, pass ``--viz none`` to force-disable them.
-2. **Avoid Unnecessary Collisions**: If possible, reduce the number of object overlaps to reduce overhead in the simulation.
-   Excessive contacts and collisions in the simulation can be expensive in the collision phase in the simulation.
-3. **Use Simplified Physics**: Consider using simplified physics collision geometries or lowering simulation fidelity
-   for better performance. This can be done by modifying the assets and adjusting the physics parameters in the simulation configuration.
-4. **Use CPU/GPU Simulation**: If your scene consists of just a few articulations or rigid bodies, consider using CPU simulation
-   for better performance. For larger scenes, using GPU simulation can significantly improve performance.
-
-Collision Geometries
---------------------
-
-Collision geometries are used to define the shape of objects in the simulation for collision detection. Using
-simplified collision geometries can improve performance and reduce the complexity of the simulation.
-
-For example, if you have a complex mesh, you can create a simplified collision geometry that approximates the shape
-of the mesh. This can be done in Isaac Sim through the UI by modifying the collision mesh and approximation methods.
-
-Additionally, we can often remove collision geometries on areas of the robot that are not important for training.
-In the AnymalC robot, we keep the collision geometries for the kneeds and feet, but remove the collision geometries
-on other parts of the legs to optimize for performance.
-
-Simpler collision geometries such as primitive shapes like spheres will also yield better performance than complex meshes.
-For example, an SDF mesh collider will be more expensive than a simple sphere.
-
-Note that cylinder and cone collision geometries have special support for smooth collisions with triangle meshes for
-better wheeled simulation behavior. This comes at a cost of performance and may not always be desired. To disable this feature,
-we can set the stage settings ``--/physics/collisionApproximateCylinders=true`` and ``--/physics/collisionApproximateCones=true``.
-
-Another item to watch out for in GPU RL workloads is warnings about GPU compatibility of ``Convex Hull`` approximated mesh collision geometry.
-If the input mesh has a high aspect ratio (e.g. a long thin shape), the convex hull approximation may be incompatible with GPU simulation,
-triggering a CPU fallback that can significantly impact performance.
-
-A CPU-fallback warning looks as follows: ``[Warning] [omni.physx.cooking.plugin] ConvexMeshCookingTask: failed to cook GPU-compatible mesh,
-collision detection will fall back to CPU. Collisions with particles and deformables will not work with this mesh.``.
-Suitable workarounds include switching to a bounding cube approximation, or using a static triangle mesh collider
-if the geometry is not part of a dynamic rigid body.
-
-CPU Governor Settings on Linux
-------------------------------
-
-CPU governors dictate the operating clock frequency range and scaling of the CPU. This can be a limiting factor for Isaac Sim performance. For maximum performance, the CPU governor should be set to ``performance``. To modify the CPU governor, run the following commands:
-
-.. code-block:: bash
-
-    sudo apt-get install linux-tools-common
-    cpupower frequency-info # Check available governors
-    sudo cpupower frequency-set -g performance # Set governor with root permissions
-
-.. note::
-
-    Not all governors are available on all systems. Governors enabling higher clock speed are typically more performance-centric and will yield better performance for Isaac Sim.
-
-Additional Performance Guides
------------------------------
-
-There are many ways to "tune" the performance of the simulation, but the way you choose largely depends on what you are trying to simulate. In general, the first place
-you will want to look for performance gains is with the `PhysX engine <https://docs.omniverse.nvidia.com/kit/docs/omni_physics/107.3/dev_guide/guides.html>`_. Next to rendering
-and running deep learning models, the PhysX engine is the most computationally costly. Tuning the PhysX sim to limit the scope to only the task of interest is a great place to
-start hunting for performance gains.
-
-We have recently released a new `gripper tuning guide <https://docs.omniverse.nvidia.com/kit/docs/omni_physics/107.3/dev_guide/guides/gripper_tuning_example.html>`_ , specific to contact and grasp tuning. Please check it first if you intend to use robot grippers. For additional details, you should also checkout these guides!
-
-* `Isaac Sim Performance Optimization Handbook <https://docs.isaacsim.omniverse.nvidia.com/latest/reference_material/sim_performance_optimization_handbook.html>`_
-* `Omni Physics Simulation Performance Guide <https://docs.omniverse.nvidia.com/kit/docs/omni_physics/latest/dev_guide/guides/physics-performance.html>`_
+This page has moved. See :ref:`simulation-performance-troubleshooting` for guidance on diagnosing
+slow simulation and training workloads.

docs/source/refs/troubleshooting.rst

@@ -219,6 +219,38 @@ Newton backends
 
 These entries apply to the ``physics=newton_mjwarp`` and ``physics=newton_kamino`` backends.
 
+.. _simulation-performance-troubleshooting:
+
+Simulation or training runs slower than expected
+------------------------------------------------
+
+Start with a representative profile before changing physics or rendering settings. The
+:doc:`Nsight Systems profiling guide <../how-to/profile_with_nsys>` can help distinguish time spent
+in physics, rendering, environment code, and policy inference.
+
+For PhysX workloads, check the following common causes:
+
+* **Unneeded visualization:** Commands that do not select a visualizer launch without a viewer by
+  default. If a configuration would otherwise launch one, pass ``--viz none`` to disable it.
+* **Excessive collision work:** Avoid duplicated or overlapping collision geometry and use the
+  simplest collider that provides the required fidelity.
+* **GPU collider fallbacks:** A warning that a convex mesh failed to cook as GPU-compatible means
+  its collision handling falls back to the CPU. Replace the collider with a primitive or bounding
+  box approximation when possible. A static triangle mesh is another option only when the geometry
+  is not part of a dynamic rigid body.
+
+For broader tuning guidance, including CPU/GPU selection, solver settings, rendering, sensors, and
+CPU configuration, consult the maintained upstream guides:
+
+* `Isaac Sim Performance Optimization Handbook
+  <https://docs.isaacsim.omniverse.nvidia.com/latest/reference_material/sim_performance_optimization_handbook.html>`_
+* `Omni Physics Simulation Performance Guide
+  <https://docs.omniverse.nvidia.com/kit/docs/omni_physics/latest/dev_guide/guides/physics-performance.html>`_
+
+For Newton-specific performance and solver parameters, see the
+:doc:`Newton physics documentation <../overview/core-concepts/physical-backends/newton/index>`.
+
+
 Joints actuate in PhysX but not in a Newton-based backend
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

skills/user/setup-troubleshooting/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: isaaclab-setup-troubleshooting
-description: Routes Isaac Lab installation, verification, and common troubleshooting issues to official docs and canonical commands. Use when installing Isaac Lab, verifying setup, debugging launch failures, or diagnosing environment problems.
+description: Use when installing Isaac Lab, verifying setup, debugging launch failures, or diagnosing environment and simulation-performance problems.
 audience: user
 status: experimental
 owners:
@@ -11,7 +11,7 @@ owners:
 
 ## When To Use
 
-Use this skill when a user asks for help installing Isaac Lab, verifying a local setup, or diagnosing common setup and launch failures.
+Use this skill when a user asks for help installing Isaac Lab, verifying a local setup, or diagnosing common setup, launch, and simulation-performance problems.
 
 Do not duplicate installation or troubleshooting docs in this skill. The official docs are the source of truth.
 
@@ -25,8 +25,9 @@ Do not duplicate installation or troubleshooting docs in this skill. The officia
 6. Ask for the smallest relevant error output when the failure mode is unclear.
 7. Prefer a minimal verification command before running examples, training, or rendering workflows.
 8. Route backend-specific setup to the relevant PhysX or Newton docs.
-9. For XR teleoperation setup, which is a separate workflow from the base installation, route to the CloudXR how-to rather than the installation guide.
-10. If the docs are incomplete or stale, update the docs rather than expanding this skill.
+9. For unexpectedly slow simulation or training, route to the performance section in the troubleshooting reference and profile a representative workload before prescribing tuning changes.
+10. For XR teleoperation setup, which is a separate workflow from the base installation, route to the CloudXR how-to rather than the installation guide.
+11. If the docs are incomplete or stale, update the docs rather than expanding this skill.
 
 ## Validation
 

skills/user/setup-troubleshooting/evaluations.md

@@ -61,3 +61,20 @@ Known failure modes:
 - Recommends reinstalling before checking the active Python environment.
 - Starts debugging the training runner before validating imports.
 - Ignores the official troubleshooting page.
+
+## Scenario 5: Slow Simulation Throughput
+
+Query: "My GPU RL training runs, but throughput is poor and PhysX reports that a convex mesh will fall back to CPU. What should I check?"
+
+Expected behavior:
+
+- Routes first to the performance section in the troubleshooting reference.
+- Recommends profiling a representative workload with the Nsight Systems guide before changing settings.
+- Explains that the collider warning indicates a CPU fallback and uses the documented collider workarounds.
+- Defers general PhysX and Isaac Sim tuning to the maintained upstream performance guides.
+
+Known failure modes:
+
+- Gives generic performance suggestions without consulting the troubleshooting reference.
+- Changes solver or RL settings before identifying the bottleneck.
+- Omits the CPU-fallback warning or recommends a dynamic triangle-mesh collider.

skills/user/setup-troubleshooting/reference.md

@@ -54,6 +54,7 @@ uv run isaaclab train --rl_library rsl_rl --task Isaac-Cartpole --max_iterations
 | Backend preset fails | `uv run python scripts/environments/list_envs.py --show_presets` |
 | Camera or renderer fails | Renderer selection and sensor docs |
 | Training starts but shapes fail | Environment reset/step smoke test before runner |
+| Simulation or training throughput is poor | Performance troubleshooting, then Nsight Systems profiling |
 
 ## Escalation Checklist