Skip to content

Document OpenUSD parser thread-limit workaround - #7598

Merged
kellyguo11 merged 4 commits into
isaac-sim:developfrom
kellyguo11:kellyg/document-openusd-thread-limit
Sep 5, 2026
Merged

Document OpenUSD parser thread-limit workaround#7598
kellyguo11 merged 4 commits into
isaac-sim:developfrom
kellyguo11:kellyg/document-openusd-thread-limit

Conversation

@kellyguo11

@kellyguo11 kellyguo11 commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Description

Document the known OpenUSD physics-parser race that can terminate Newton workflows while loading collider-rich rigid bodies.

The Known Issues page now explains:

  • the affected kitless OpenUSD versions and Isaac Sim releases prior to 6.1;
  • that Isaac Sim 6.1 and later contain the upstream fix;
  • that kitless Isaac Lab obtains pxr from usd-exchange, while usd-core is an alternative provider with a different distribution version scheme;
  • the PXR_WORK_THREAD_LIMIT=1 workaround for affected Linux and Windows workflows; and
  • that limiting OpenUSD's process-wide worker pool can reduce USD import performance.

Related to Isaac Sim #692, OpenUSD #4002, and NVIDIA NVBug 6592534.

No dependencies are added.

Type of change

  • Documentation update

Screenshots

Not applicable.

Checklist

  • I have read and understood the contribution guidelines.
  • I have run the pre-commit checks with uv run isaaclab -f.
  • I have made corresponding changes to the documentation.
  • My changes generate no new warnings.
  • Documentation-only change; no runtime test is required.
  • Documentation-only change; no package changelog fragment is required.
  • My name already exists in CONTRIBUTORS.md.

Validation

  • make -C docs current-docs (with the project environment and non-interactive EULA acceptance)
  • UV_FROZEN=1 ISAACLAB_CHANGELOG_BASE_REF=codex-doc-base uv run isaaclab -f (the temporary local base-ref alias pointed to upstream/develop)

Release backport

  • Backport this pull request to the active release branch after it merges into develop

@kellyguo11
kellyguo11 requested a review from a team September 5, 2026 06:57
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Sep 5, 2026
@greptile-apps

greptile-apps Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR documents an OpenUSD physics-parser race affecting collider-rich Newton workflows, including affected versions, failure symptoms, thread-limit workarounds, and their performance tradeoff.

  • Adds Linux and Windows examples for setting PXR_WORK_THREAD_LIMIT.
  • Adds Kit-specific guidance using AppLauncher.limit_cpu_threads.
  • Links the upstream OpenUSD fix.
  • The Kit example currently omits the startup environment setting required by the repository's existing workaround.

Confidence Score: 4/5

This documentation should not merge until the Kit workaround also preserves the required pre-startup PXR_WORK_THREAD_LIMIT setting.

The new Kit example can leave OpenUSD's worker pool unrestricted because limit_cpu_threads is applied through SimulationApp after OpenUSD reads its startup environment, so users may still experience the crash the documentation is intended to prevent.

Files Needing Attention: docs/source/refs/issues.rst

Important Files Changed

Filename Overview
docs/source/refs/issues.rst Adds the OpenUSD parser-race documentation, but presents the later-applied Kit thread limit as a sufficient replacement for the startup environment variable.

Reviews (1): Last reviewed commit: "Document OpenUSD parser thread limit" | Re-trigger Greptile

Comment thread docs/source/refs/issues.rst Outdated

from isaaclab.app import AppLauncher

app_launcher = AppLauncher(headless=True, limit_cpu_threads=1)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Kit workaround is insufficient

For Kit-based scripts, limit_cpu_threads=1 is not equivalent to setting PXR_WORK_THREAD_LIMIT=1. AppLauncher forwards the value to SimulationApp, but only distributed launches set the environment variable. Existing tests note that OpenUSD reads this variable before SimulationApp applies its limit. Users who follow this example can therefore remain exposed to the parser race and still encounter crashes or hangs. Please document that PXR_WORK_THREAD_LIMIT must also be set before process startup.

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac Lab Review Bot

Reviewed the new Newton Known Issues entry documenting the OpenUSD collider-parser race and its single-thread workarounds. The documentation is integrated cleanly, and no actionable finding was supplied or established by the available patch context.

  • Design and architecture: The subsection is appropriately placed under "Newton backends," preserves the existing section hierarchy and anchor, and groups the new OpenUSD reference with the page’s existing link targets.
  • API: No API or configuration behavior changes. The documented compatibility surfaces are PXR_WORK_THREAD_LIMIT=1, AppLauncher(headless=True, limit_cpu_threads=1), and the physics=newton_mjwarp preset; the added RST reference resolves to the OpenUSD pull request. The supplied repository context does not independently confirm the launcher argument or affected-version boundary, so those technical claims remain a non-blocking verification consideration.
  • Implementation: The heading, tab-set, code-block indentation, and link target are structurally consistent with the surrounding RST. This is confined to documentation, adds no dependency, and does not require a source-package changelog fragment. The documented CPU-performance tradeoff is clearly disclosed.

No blocking issues. No inline issue met the actionable-evidence threshold; the assessment above records the review feedback.

Automated review; human maintainers own approval decisions.

@kellyguo11

Copy link
Copy Markdown
Contributor Author

run-ci

@isaaclab-bot isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Sep 5, 2026
@hujc7

hujc7 commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator

run-ci

@isaaclab-bot isaaclab-bot Bot added the ci:run-docker Trigger the on-demand Docker and GPU CI workflow label Sep 5, 2026

@hujc7 hujc7 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this need backport?

@isaaclab-bot isaaclab-bot Bot removed the ci:run-docker Trigger the on-demand Docker and GPU CI workflow label Sep 5, 2026

@hujc7 hujc7 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded — re-posted as inline comments anchored to the relevant lines, so each finding sits on the code it concerns. See the inline thread(s) on this PR.

Comment thread docs/source/refs/issues.rst Outdated
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Affects:** Kitless Newton workflows using OpenUSD releases earlier than 26.05
(``usd-core<26.5``). Isaac Sim workflows are not affected because the bundled OpenUSD

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Agent found bug: this is false for Isaac Sim 6.0.x, which is the release Isaac Lab pins. Capped at [P2] only because I have already approved this PR — on merit I would fix it before merge.

Kit ships a private OpenUSD, so immunity is per-build:

Line kit-kernel USD package Backport
6.0.1-ga / release/6.0 110.1.2+production 0.25.11.kit.2 absent
release/6.1 / develop 110.3.0+feature.* 0.25.11.kit.5 present

pyproject.toml:169 pins isaacsim[all,extscache]==6.0.1.0, and Newton-under-Kit is an exercised configuration — rendering_test_utils.py:109-114 parametrizes ("franka_cloth", "newton", "isaacsim_rtx_renderer"), with paired Kit and _kitless modules. As written, the entry tells users on the supported install that it does not apply to them.

Suggested:

Isaac Sim 6.1 and newer are not affected: their bundled OpenUSD contains the upstream fix. Isaac Sim 6.0.x does not — apply the workaround there as well.

Comment thread docs/source/refs/issues.rst Outdated
OpenUSD can crash while parsing collider-rich rigid bodies
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Affects:** Kitless Newton workflows using OpenUSD releases earlier than 26.05

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Agent review: this condition, and the removal condition at the end of the section, both name usd-core — a distribution a supported Isaac Lab install never has.

pyproject.toml:73 pins usd-exchange==2.3.0 as the sole USD provider, and test_source_package_metadata.py:50 asserts "usd-core" not in locked. So pip show usd-core returns nothing and a reader cannot tell whether the entry applies to them, or when to stop applying the workaround.

Suggested: key both conditions on the installed version instead.

Affects: kitless Newton workflows on OpenUSD earlier than 26.05. Check with uv run python -c "from pxr import Usd; print(Usd.GetVersion())" — below (0, 26, 5) is affected. Isaac Lab installs OpenUSD through usd-exchange, not usd-core.

@kellyguo11

Copy link
Copy Markdown
Contributor Author

run-ci

@isaaclab-bot isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Sep 5, 2026
@kellyguo11
kellyguo11 merged commit 6533a4e into isaac-sim:develop Sep 5, 2026
22 checks passed
@isaaclab-bot

isaaclab-bot Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Backported to release/3.0.0 as 282f982.

isaaclab-bot Bot pushed a commit that referenced this pull request Sep 5, 2026
# Description

Document the known OpenUSD physics-parser race that can terminate Newton
workflows while loading collider-rich rigid bodies.

The Known Issues page now explains:

- the affected kitless OpenUSD versions and Isaac Sim releases prior to
6.1;
- that Isaac Sim 6.1 and later contain the upstream fix;
- that kitless Isaac Lab obtains `pxr` from `usd-exchange`, while
`usd-core` is an alternative provider with a different distribution
version scheme;
- the `PXR_WORK_THREAD_LIMIT=1` workaround for affected Linux and
Windows workflows; and
- that limiting OpenUSD's process-wide worker pool can reduce USD import
performance.

Related to [Isaac Sim
#692](isaac-sim/IsaacSim#692), [OpenUSD
#4002](PixarAnimationStudios/OpenUSD#4002), and
NVIDIA NVBug 6592534.

No dependencies are added.

## Type of change

- Documentation update

## Screenshots

Not applicable.

## Checklist

- [x] I have read and understood the contribution guidelines.
- [x] I have run the pre-commit checks with `uv run isaaclab -f`.
- [x] I have made corresponding changes to the documentation.
- [x] My changes generate no new warnings.
- [x] Documentation-only change; no runtime test is required.
- [x] Documentation-only change; no package changelog fragment is
required.
- [x] My name already exists in `CONTRIBUTORS.md`.

## Validation

- `make -C docs current-docs` (with the project environment and
non-interactive EULA acceptance)
- `UV_FROZEN=1 ISAACLAB_CHANGELOG_BASE_REF=codex-doc-base uv run
isaaclab -f` (the temporary local base-ref alias pointed to
`upstream/develop`)

## Release backport

- [x] <!-- backport-active-release --> Backport this pull request to the
active release branch after it merges into `develop`

(cherry picked from commit 6533a4e)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants