From 883b56f9cdcf125dba20c84e160b08663604fa73 Mon Sep 17 00:00:00 2001 From: Antoine Richard Date: Wed, 26 Aug 2026 18:20:19 +0200 Subject: [PATCH] Consolidate simulation performance guidance --- docs/source/how-to/index.rst | 13 --- docs/source/how-to/profile_with_nsys.rst | 2 +- docs/source/how-to/simulation_performance.rst | 83 ++----------------- docs/source/refs/troubleshooting.rst | 32 +++++++ skills/user/setup-troubleshooting/SKILL.md | 9 +- .../user/setup-troubleshooting/evaluations.md | 17 ++++ .../user/setup-troubleshooting/reference.md | 1 + 7 files changed, 61 insertions(+), 96 deletions(-) diff --git a/docs/source/how-to/index.rst b/docs/source/how-to/index.rst index a9eb67ad40c4..57e5bd62a404 100644 --- a/docs/source/how-to/index.rst +++ b/docs/source/how-to/index.rst @@ -210,19 +210,6 @@ with directional force feedback in Isaac Lab. haply_teleoperation -Understanding Simulation Performance ------------------------------------- - -This guide provides tips on optimizing simulation performance for different simulation use cases. -Additional resources are also linked to provide relevant performance guides for Isaac Sim and -Omniverse Physics. - -.. toctree:: - :maxdepth: 1 - - simulation_performance - - Optimize Stage Creation ----------------------- diff --git a/docs/source/how-to/profile_with_nsys.rst b/docs/source/how-to/profile_with_nsys.rst index 7bf2d93a5d68..8fabbf1ad1c9 100644 --- a/docs/source/how-to/profile_with_nsys.rst +++ b/docs/source/how-to/profile_with_nsys.rst @@ -124,6 +124,6 @@ Troubleshooting See Also -------- -- :doc:`simulation_performance` - broader simulation performance tuning tips. +- :ref:`simulation-performance-troubleshooting` - checks for unexpectedly slow simulation or training. - `Nsight Systems User Guide `_ - official ``nsys`` documentation. - `NVTX Python package `_ - the package nsys uses to emit NVTX ranges from Python. diff --git a/docs/source/how-to/simulation_performance.rst b/docs/source/how-to/simulation_performance.rst index c5fba271517c..528112761102 100644 --- a/docs/source/how-to/simulation_performance.rst +++ b/docs/source/how-to/simulation_performance.rst @@ -1,80 +1,7 @@ -PhysX Simulation Performance and Tuning -========================================= +:orphan: -.. note:: +Simulation Performance +====================== - This guide covers performance tuning for the **PhysX** backend, which is the default when - running Isaac Lab with Isaac Sim. For the **Newton** backend solver parameters (e.g. - ``njmax``, ``nconmax``, ``ls_iterations``), see the - :ref:`migrating-to-isaaclab-3-0` migration guide and the Newton physics documentation. - -The performance of the PhysX simulation can be affected by various factors, including the number -of objects in the scene, the complexity of the physics simulation, and the hardware being used. -Here are some tips to improve performance: - -1. **Use Headless Mode**: Running the simulation in headless mode can significantly improve performance, especially - when rendering is not required. For commands that do not select a visualizer, no viewer is launched unless the configuration requests one. If a config or - command would otherwise launch visualizers, pass ``--viz none`` to force-disable them. -2. **Avoid Unnecessary Collisions**: If possible, reduce the number of object overlaps to reduce overhead in the simulation. - Excessive contacts and collisions in the simulation can be expensive in the collision phase in the simulation. -3. **Use Simplified Physics**: Consider using simplified physics collision geometries or lowering simulation fidelity - for better performance. This can be done by modifying the assets and adjusting the physics parameters in the simulation configuration. -4. **Use CPU/GPU Simulation**: If your scene consists of just a few articulations or rigid bodies, consider using CPU simulation - for better performance. For larger scenes, using GPU simulation can significantly improve performance. - -Collision Geometries --------------------- - -Collision geometries are used to define the shape of objects in the simulation for collision detection. Using -simplified collision geometries can improve performance and reduce the complexity of the simulation. - -For example, if you have a complex mesh, you can create a simplified collision geometry that approximates the shape -of the mesh. This can be done in Isaac Sim through the UI by modifying the collision mesh and approximation methods. - -Additionally, we can often remove collision geometries on areas of the robot that are not important for training. -In the AnymalC robot, we keep the collision geometries for the kneeds and feet, but remove the collision geometries -on other parts of the legs to optimize for performance. - -Simpler collision geometries such as primitive shapes like spheres will also yield better performance than complex meshes. -For example, an SDF mesh collider will be more expensive than a simple sphere. - -Note that cylinder and cone collision geometries have special support for smooth collisions with triangle meshes for -better wheeled simulation behavior. This comes at a cost of performance and may not always be desired. To disable this feature, -we can set the stage settings ``--/physics/collisionApproximateCylinders=true`` and ``--/physics/collisionApproximateCones=true``. - -Another item to watch out for in GPU RL workloads is warnings about GPU compatibility of ``Convex Hull`` approximated mesh collision geometry. -If the input mesh has a high aspect ratio (e.g. a long thin shape), the convex hull approximation may be incompatible with GPU simulation, -triggering a CPU fallback that can significantly impact performance. - -A CPU-fallback warning looks as follows: ``[Warning] [omni.physx.cooking.plugin] ConvexMeshCookingTask: failed to cook GPU-compatible mesh, -collision detection will fall back to CPU. Collisions with particles and deformables will not work with this mesh.``. -Suitable workarounds include switching to a bounding cube approximation, or using a static triangle mesh collider -if the geometry is not part of a dynamic rigid body. - -CPU Governor Settings on Linux ------------------------------- - -CPU governors dictate the operating clock frequency range and scaling of the CPU. This can be a limiting factor for Isaac Sim performance. For maximum performance, the CPU governor should be set to ``performance``. To modify the CPU governor, run the following commands: - -.. code-block:: bash - - sudo apt-get install linux-tools-common - cpupower frequency-info # Check available governors - sudo cpupower frequency-set -g performance # Set governor with root permissions - -.. note:: - - Not all governors are available on all systems. Governors enabling higher clock speed are typically more performance-centric and will yield better performance for Isaac Sim. - -Additional Performance Guides ------------------------------ - -There are many ways to "tune" the performance of the simulation, but the way you choose largely depends on what you are trying to simulate. In general, the first place -you will want to look for performance gains is with the `PhysX engine `_. Next to rendering -and running deep learning models, the PhysX engine is the most computationally costly. Tuning the PhysX sim to limit the scope to only the task of interest is a great place to -start hunting for performance gains. - -We have recently released a new `gripper tuning guide `_ , specific to contact and grasp tuning. Please check it first if you intend to use robot grippers. For additional details, you should also checkout these guides! - -* `Isaac Sim Performance Optimization Handbook `_ -* `Omni Physics Simulation Performance Guide `_ +This page has moved. See :ref:`simulation-performance-troubleshooting` for guidance on diagnosing +slow simulation and training workloads. diff --git a/docs/source/refs/troubleshooting.rst b/docs/source/refs/troubleshooting.rst index d50244fee7f4..1b106e104471 100644 --- a/docs/source/refs/troubleshooting.rst +++ b/docs/source/refs/troubleshooting.rst @@ -169,6 +169,38 @@ To enable OmniPVD capture in Isaac Lab, add the relevant kit arguments to the co ./isaaclab.sh -p scripts/demos/bipeds.py --kit_args "--/persistent/physics/omniPvdOvdRecordingDirectory=/tmp/ --/physics/omniPvdOutputEnabled=true" +.. _simulation-performance-troubleshooting: + +Simulation or training runs slower than expected +------------------------------------------------ + +Start with a representative profile before changing physics or rendering settings. The +:doc:`Nsight Systems profiling guide <../how-to/profile_with_nsys>` can help distinguish time spent +in physics, rendering, environment code, and policy inference. + +For PhysX workloads, check the following common causes: + +* **Unneeded visualization:** Commands that do not select a visualizer launch without a viewer by + default. If a configuration would otherwise launch one, pass ``--viz none`` to disable it. +* **Excessive collision work:** Avoid duplicated or overlapping collision geometry and use the + simplest collider that provides the required fidelity. +* **GPU collider fallbacks:** A warning that a convex mesh failed to cook as GPU-compatible means + its collision handling falls back to the CPU. Replace the collider with a primitive or bounding + box approximation when possible. A static triangle mesh is another option only when the geometry + is not part of a dynamic rigid body. + +For broader tuning guidance, including CPU/GPU selection, solver settings, rendering, sensors, and +CPU configuration, consult the maintained upstream guides: + +* `Isaac Sim Performance Optimization Handbook + `_ +* `Omni Physics Simulation Performance Guide + `_ + +For Newton-specific performance and solver parameters, see the +:doc:`Newton physics documentation <../overview/core-concepts/physical-backends/newton/index>`. + + Joints actuate in PhysX but not in a Newton-based backend --------------------------------------------------------- diff --git a/skills/user/setup-troubleshooting/SKILL.md b/skills/user/setup-troubleshooting/SKILL.md index 5c78c73590e5..cd6c5f77ca8b 100644 --- a/skills/user/setup-troubleshooting/SKILL.md +++ b/skills/user/setup-troubleshooting/SKILL.md @@ -1,6 +1,6 @@ --- name: isaaclab-setup-troubleshooting -description: Routes Isaac Lab installation, verification, and common troubleshooting issues to official docs and canonical commands. Use when installing Isaac Lab, verifying setup, debugging launch failures, or diagnosing environment problems. +description: Use when installing Isaac Lab, verifying setup, debugging launch failures, or diagnosing environment and simulation-performance problems. audience: user status: experimental owners: @@ -11,7 +11,7 @@ owners: ## When To Use -Use this skill when a user asks for help installing Isaac Lab, verifying a local setup, or diagnosing common setup and launch failures. +Use this skill when a user asks for help installing Isaac Lab, verifying a local setup, or diagnosing common setup, launch, and simulation-performance problems. Do not duplicate installation or troubleshooting docs in this skill. The official docs are the source of truth. @@ -25,8 +25,9 @@ Do not duplicate installation or troubleshooting docs in this skill. The officia 6. Ask for the smallest relevant error output when the failure mode is unclear. 7. Prefer a minimal verification command before running examples, training, or rendering workflows. 8. Route backend-specific setup to the relevant PhysX or Newton docs. -9. For XR teleoperation setup, which is a separate workflow from the base installation, route to the CloudXR how-to rather than the installation guide. -10. If the docs are incomplete or stale, update the docs rather than expanding this skill. +9. For unexpectedly slow simulation or training, route to the performance section in the troubleshooting reference and profile a representative workload before prescribing tuning changes. +10. For XR teleoperation setup, which is a separate workflow from the base installation, route to the CloudXR how-to rather than the installation guide. +11. If the docs are incomplete or stale, update the docs rather than expanding this skill. ## Validation diff --git a/skills/user/setup-troubleshooting/evaluations.md b/skills/user/setup-troubleshooting/evaluations.md index 87180fab4c27..3b49fe6415c0 100644 --- a/skills/user/setup-troubleshooting/evaluations.md +++ b/skills/user/setup-troubleshooting/evaluations.md @@ -61,3 +61,20 @@ Known failure modes: - Recommends reinstalling before checking the active Python environment. - Starts debugging the training runner before validating imports. - Ignores the official troubleshooting page. + +## Scenario 5: Slow Simulation Throughput + +Query: "My GPU RL training runs, but throughput is poor and PhysX reports that a convex mesh will fall back to CPU. What should I check?" + +Expected behavior: + +- Routes first to the performance section in the troubleshooting reference. +- Recommends profiling a representative workload with the Nsight Systems guide before changing settings. +- Explains that the collider warning indicates a CPU fallback and uses the documented collider workarounds. +- Defers general PhysX and Isaac Sim tuning to the maintained upstream performance guides. + +Known failure modes: + +- Gives generic performance suggestions without consulting the troubleshooting reference. +- Changes solver or RL settings before identifying the bottleneck. +- Omits the CPU-fallback warning or recommends a dynamic triangle-mesh collider. diff --git a/skills/user/setup-troubleshooting/reference.md b/skills/user/setup-troubleshooting/reference.md index 3680a63ce6d1..009f98ae3d11 100644 --- a/skills/user/setup-troubleshooting/reference.md +++ b/skills/user/setup-troubleshooting/reference.md @@ -54,6 +54,7 @@ uv run isaaclab train --rl_library rsl_rl --task Isaac-Cartpole --max_iterations | Backend preset fails | `uv run python scripts/environments/list_envs.py --show_presets` | | Camera or renderer fails | Renderer selection and sensor docs | | Training starts but shapes fail | Environment reset/step smoke test before runner | +| Simulation or training throughput is poor | Performance troubleshooting, then Nsight Systems profiling | ## Escalation Checklist