Skip to content

[Backport] Set Warp's global determinism mode under --deterministic (#7561) - #7586

Merged
kellyguo11 merged 1 commit into
isaac-sim:release/3.0.0from
kellyguo11:codex/backport-7561-release-3.0.0
Sep 5, 2026
Merged

[Backport] Set Warp's global determinism mode under --deterministic (#7561)#7586
kellyguo11 merged 1 commit into
isaac-sim:release/3.0.0from
kellyguo11:codex/backport-7561-release-3.0.0

Conversation

@kellyguo11

Copy link
Copy Markdown
Contributor

Description

Backports #7561 to release/3.0.0.

This branch cherry-picks the canonical merged commit 838fb6d66288cc5f452c3aa1b1ad57f420e0f0e4 with -x provenance. The change applied without conflicts or release-specific edits. The repository backport validator confirms an exact patch replay across all eight source paths, and the stable source/backport patch IDs match.

Field Commit
Original merged change 838fb6d66288cc5f452c3aa1b1ad57f420e0f0e4
Release base used 1c754876008f0806fdcfda8e3a6b2f593b34d6fc
Backport commit 6044466990af2cb4b54f07f3705e37f16e6a4d58
Stable patch ID 643b6fbbd03866389488b17070db3504315b4a41

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • Documentation update

Release backport

  • This PR targets the active release/3.0.0 branch directly.

Validation

  • Repository exact-patch backport validation passed; all eight source paths are preserved.
  • Stable source/backport patch IDs match: 643b6fbbd03866389488b17070db3504315b4a41.
  • test_entrypoints_common.py — 36 passed in an isolated uv environment.
  • All changed-file pre-commit hooks passed, including changelog and Git LFS checks.
  • sphinx-lint reported no problems for the modified RST files.
  • git diff --check upstream/release/3.0.0...HEAD passed.
  • The canonical project formatter and warning-as-error docs build cannot resolve the release lock on this macOS/arm64 host; Linux CI remains authoritative for those checks.

Checklist

  • I have read and understood the contribution guidelines
  • I have run the available pre-commit checks
  • The original documentation update is preserved exactly
  • My changes generate no new warnings
  • The original regression coverage is preserved and passes
  • The isaaclab_rl changelog fragment is preserved
  • The original contributor is already listed in CONTRIBUTORS.md

…-sim#7561)

# Description

`--deterministic` did not make camera observations reproducible.
Training `Isaac-Cartpole-Camera --deterministic` on an RTX PRO 6000
Blackwell splits into two bitwise-identical outcome clusters; on an L40
it does not. Reported as NVBug 6658578 against Isaac Lab 3.0 GA.

Newton's solvers accept a `deterministic` argument and apply it as a
per-module option, so PR isaac-sim#7334 already covered the physics kernels. Its
**sensor and geometry kernels take no such argument** and fall back to
`warp.config.deterministic`, which stayed at `NOT_GUARANTEED` — Isaac
Lab has never set it (`git log --all -S "config.deterministic"` is
empty).

The concrete path: `newton._src.geometry.bvh.compute_enabled_shapes`
claims output slots with `wp.atomic_add`, so the enabled-shape order —
and with it the primitive order the scene BVH is built over — varies
between processes. Ray queries then break ties differently, and a tiled
camera renders a few pixels differently from bit-identical simulation
state. That is enough to make an image-observation policy diverge.

`apply_env_overrides` now also requests `wp.config.deterministic =
RUN_TO_RUN`. Warp reads the setting at module build time and the BVH is
built during `ModelBuilder.finalize()`, so the request must land before
the environment is created — `_apply_deterministic_request` at solver
init is too late. An explicitly chosen mode such as `GPU_TO_GPU` is left
alone.

## Type of change

- Bug fix (non-breaking change which fixes an issue)

## Validation

Where the divergence starts, from the real training path with every
renderer input fingerprinted per frame (6 runs, Blackwell). All
geometric inputs are bit-identical at the frame where the output already
differs:

```
cam_xform / cam_rays / shape_transform / shape_body : never diverge
body_q, BVH bounds                                  : render 390  (after)
rendered colour (output)                            : render 389  (first)
```

`Isaac-Cartpole-Camera --deterministic`, epoch-25 reward, unique run
dirs, 8 runs per arm:

| Branch | Configuration | Runs | Distinct outcomes |
|---|---|---|---|
| develop | before this change | 8 | 2 |
| develop | `wp.config.deterministic` set externally | 8 | 1 |
| develop | **this change**, plain `--deterministic` | 8 | **1** |
| release/3.0.0 | before | 8 | 2 |
| release/3.0.0 | `wp.config` set | 8 | 1 |

Architecture control on L40 (4gpu), 12 runs, unique run dirs: all
identical — the defect does not reproduce on Ada, which is why PR
isaac-sim#7334's L40-only validation missed it.

Unit tests: 31 pass in `test_entrypoints_common.py`, including three
added here covering the request, an explicit mode being preserved, and
no change without the flag.

Fixes NVBug 6658578.

## Follow-up (not in this PR)

Newton-side: sensor and geometry modules should accept a `deterministic`
option like the solvers do, rather than depending on a process global;
and `compute_enabled_shapes` produces order-dependent output regardless
of that setting.

## Checklist

- [x] I have read and understood the [contribution
guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html)
- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] I have added a changelog fragment under
`source/<pkg>/changelog.d/` for every touched package
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

(cherry picked from commit 838fb6d)
@kellyguo11
kellyguo11 requested a review from a team September 4, 2026 21:25
@github-actions github-actions Bot added documentation Improvements or additions to documentation isaac-lab Related to Isaac Lab team labels Sep 4, 2026
@greptile-apps

greptile-apps Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This backport extends --deterministic to Warp’s process-wide determinism setting before environment construction, ensuring Newton sensor and geometry kernels receive the requested guarantee.

  • Centralizes physics and Warp determinism handling in the shared entrypoint utilities.
  • Applies the request to all four supported play backends before environment creation; training and benchmark paths inherit it through shared overrides.
  • Preserves stronger preconfigured Warp guarantees and maps Newton’s configured mode to the corresponding Warp mode.
  • Adds regression coverage and documents the camera-observation reproducibility behavior.

Confidence Score: 5/5

The PR appears safe to merge; the determinism request is applied consistently before environment creation and is covered by focused regression tests.

No actionable failures remain: supported entrypoints are covered, the pinned Warp API matches the implementation, configured modes map correctly, and stronger process-wide guarantees are intentionally preserved.

Important Files Changed

Filename Overview
source/isaaclab_rl/isaaclab_rl/entrypoints/common.py Centralizes deterministic configuration and raises Warp’s global mode before environment construction without weakening an existing guarantee.
source/isaaclab_rl/isaaclab_rl/entrypoints/backends/play_rl_games.py Requests determinism before creating the RL-Games play environment.
source/isaaclab_rl/isaaclab_rl/entrypoints/backends/play_rsl_rl.py Requests determinism before creating the RSL-RL play environment.
source/isaaclab_rl/isaaclab_rl/entrypoints/backends/play_sb3.py Requests determinism before creating the Stable-Baselines3 play environment.
source/isaaclab_rl/isaaclab_rl/entrypoints/backends/play_skrl.py Requests determinism before creating the SKRL play environment.
source/isaaclab_rl/test/test_entrypoints_common.py Covers default, configured, stronger-existing, and disabled Warp determinism behavior.
docs/source/features/reproducibility.rst Documents Warp’s global fallback, required initialization timing, and its effect on Newton camera reproducibility.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  CLI["--deterministic"] --> Entry["Training or play entrypoint"]
  Entry --> Request["request_determinism"]
  Request --> Physics["PhysicsCfg.deterministic = true"]
  Request --> Warp["Raise warp.config.deterministic"]
  Physics --> Env["Create environment"]
  Warp --> Env
  Env --> Kernels["Solver, sensor, geometry, and BVH kernels"]
  Kernels --> Repro["Reproducible simulation and camera observations"]
Loading

Reviews (1): Last reviewed commit: "[Fix] Set Warp's global determinism mode..." | Re-trigger Greptile

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac Lab Review Bot

The backport correctly sets Warp’s global determinism mode before environment construction, but the new calls in all four play entrypoints also enable backend physics determinism, widening the behavior of play --deterministic beyond the stated Warp fix.

  • Design and architecture: Centralizing determinism handling in the shared entrypoint helper satisfies Warp’s module-build-time constraint. However, reusing the combined helper in play paths couples the intended Warp-global update to a new solver-determinism request that can be rejected by a backend.
  • API: The --deterministic CLI contract changes for RL playback: it now sets PhysicsCfg.deterministic in addition to configuring rendering and Warp. The documentation describes physics determinism for training entrypoints but does not clearly document this new play behavior or its possible startup rejection.
  • Implementation: The Warp setting is applied before environment creation and preserves stronger existing modes. The play entrypoints should either invoke only the Warp-specific helper or explicitly document and validate the intentional expansion to physics determinism.

Minor fixes needed. Posted 1 actionable finding inline.

Automated review; human maintainers own approval decisions.


env_cfg.scene.num_envs = args_cli.num_envs if args_cli.num_envs is not None else env_cfg.scene.num_envs
# Warp reads its determinism mode at module build time, so request it before the env exists.
request_determinism(args_cli, env_cfg)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Suggestion · Implementation — Play now also enables physics determinism

The added comment says the goal is Warp's global, but request_determinism additionally sets env_cfg.sim.physics.deterministic = True. These play scripts set num_envs/device/seed manually and never called apply_env_overrides, so play --deterministic now requests backend solver determinism for the first time (same in play_skrl/play_sb3/play_rsl_rl), which reproducibility.rst says a manager may reject at startup. Either call only the Warp helper here or document the widened play behavior.

@kellyguo11

Copy link
Copy Markdown
Contributor Author

run-ci

@isaaclab-bot isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Sep 4, 2026
@kellyguo11
kellyguo11 merged commit e0dc96c into isaac-sim:release/3.0.0 Sep 5, 2026
52 of 53 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants