[Backport] Set Warp's global determinism mode under --deterministic (#7561) - #7586
Conversation
…-sim#7561) # Description `--deterministic` did not make camera observations reproducible. Training `Isaac-Cartpole-Camera --deterministic` on an RTX PRO 6000 Blackwell splits into two bitwise-identical outcome clusters; on an L40 it does not. Reported as NVBug 6658578 against Isaac Lab 3.0 GA. Newton's solvers accept a `deterministic` argument and apply it as a per-module option, so PR isaac-sim#7334 already covered the physics kernels. Its **sensor and geometry kernels take no such argument** and fall back to `warp.config.deterministic`, which stayed at `NOT_GUARANTEED` — Isaac Lab has never set it (`git log --all -S "config.deterministic"` is empty). The concrete path: `newton._src.geometry.bvh.compute_enabled_shapes` claims output slots with `wp.atomic_add`, so the enabled-shape order — and with it the primitive order the scene BVH is built over — varies between processes. Ray queries then break ties differently, and a tiled camera renders a few pixels differently from bit-identical simulation state. That is enough to make an image-observation policy diverge. `apply_env_overrides` now also requests `wp.config.deterministic = RUN_TO_RUN`. Warp reads the setting at module build time and the BVH is built during `ModelBuilder.finalize()`, so the request must land before the environment is created — `_apply_deterministic_request` at solver init is too late. An explicitly chosen mode such as `GPU_TO_GPU` is left alone. ## Type of change - Bug fix (non-breaking change which fixes an issue) ## Validation Where the divergence starts, from the real training path with every renderer input fingerprinted per frame (6 runs, Blackwell). All geometric inputs are bit-identical at the frame where the output already differs: ``` cam_xform / cam_rays / shape_transform / shape_body : never diverge body_q, BVH bounds : render 390 (after) rendered colour (output) : render 389 (first) ``` `Isaac-Cartpole-Camera --deterministic`, epoch-25 reward, unique run dirs, 8 runs per arm: | Branch | Configuration | Runs | Distinct outcomes | |---|---|---|---| | develop | before this change | 8 | 2 | | develop | `wp.config.deterministic` set externally | 8 | 1 | | develop | **this change**, plain `--deterministic` | 8 | **1** | | release/3.0.0 | before | 8 | 2 | | release/3.0.0 | `wp.config` set | 8 | 1 | Architecture control on L40 (4gpu), 12 runs, unique run dirs: all identical — the defect does not reproduce on Ada, which is why PR isaac-sim#7334's L40-only validation missed it. Unit tests: 31 pass in `test_entrypoints_common.py`, including three added here covering the request, an explicit mode being preserved, and no change without the flag. Fixes NVBug 6658578. ## Follow-up (not in this PR) Newton-side: sensor and geometry modules should accept a `deterministic` option like the solvers do, rather than depending on a process global; and `compute_enabled_shapes` produces order-dependent output regardless of that setting. ## Checklist - [x] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html) - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] I have added a changelog fragment under `source/<pkg>/changelog.d/` for every touched package - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there (cherry picked from commit 838fb6d)
Greptile SummaryThis backport extends
Confidence Score: 5/5The PR appears safe to merge; the determinism request is applied consistently before environment creation and is covered by focused regression tests. No actionable failures remain: supported entrypoints are covered, the pinned Warp API matches the implementation, configured modes map correctly, and stronger process-wide guarantees are intentionally preserved. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart LR
CLI["--deterministic"] --> Entry["Training or play entrypoint"]
Entry --> Request["request_determinism"]
Request --> Physics["PhysicsCfg.deterministic = true"]
Request --> Warp["Raise warp.config.deterministic"]
Physics --> Env["Create environment"]
Warp --> Env
Env --> Kernels["Solver, sensor, geometry, and BVH kernels"]
Kernels --> Repro["Reproducible simulation and camera observations"]
Reviews (1): Last reviewed commit: "[Fix] Set Warp's global determinism mode..." | Re-trigger Greptile |
There was a problem hiding this comment.
Isaac Lab Review Bot
The backport correctly sets Warp’s global determinism mode before environment construction, but the new calls in all four play entrypoints also enable backend physics determinism, widening the behavior of play --deterministic beyond the stated Warp fix.
- Design and architecture: Centralizing determinism handling in the shared entrypoint helper satisfies Warp’s module-build-time constraint. However, reusing the combined helper in play paths couples the intended Warp-global update to a new solver-determinism request that can be rejected by a backend.
- API: The
--deterministicCLI contract changes for RL playback: it now setsPhysicsCfg.deterministicin addition to configuring rendering and Warp. The documentation describes physics determinism for training entrypoints but does not clearly document this new play behavior or its possible startup rejection. - Implementation: The Warp setting is applied before environment creation and preserves stronger existing modes. The play entrypoints should either invoke only the Warp-specific helper or explicitly document and validate the intentional expansion to physics determinism.
Minor fixes needed. Posted 1 actionable finding inline.
Automated review; human maintainers own approval decisions.
|
|
||
| env_cfg.scene.num_envs = args_cli.num_envs if args_cli.num_envs is not None else env_cfg.scene.num_envs | ||
| # Warp reads its determinism mode at module build time, so request it before the env exists. | ||
| request_determinism(args_cli, env_cfg) |
There was a problem hiding this comment.
🔵 Suggestion · Implementation — Play now also enables physics determinism
The added comment says the goal is Warp's global, but request_determinism additionally sets env_cfg.sim.physics.deterministic = True. These play scripts set num_envs/device/seed manually and never called apply_env_overrides, so play --deterministic now requests backend solver determinism for the first time (same in play_skrl/play_sb3/play_rsl_rl), which reproducibility.rst says a manager may reject at startup. Either call only the Warp helper here or document the widened play behavior.
|
run-ci |
Description
Backports #7561 to
release/3.0.0.This branch cherry-picks the canonical merged commit
838fb6d66288cc5f452c3aa1b1ad57f420e0f0e4with-xprovenance. The change applied without conflicts or release-specific edits. The repository backport validator confirms an exact patch replay across all eight source paths, and the stable source/backport patch IDs match.838fb6d66288cc5f452c3aa1b1ad57f420e0f0e41c754876008f0806fdcfda8e3a6b2f593b34d6fc6044466990af2cb4b54f07f3705e37f16e6a4d58643b6fbbd03866389488b17070db3504315b4a41Type of change
Release backport
release/3.0.0branch directly.Validation
643b6fbbd03866389488b17070db3504315b4a41.test_entrypoints_common.py— 36 passed in an isolated uv environment.sphinx-lintreported no problems for the modified RST files.git diff --check upstream/release/3.0.0...HEADpassed.Checklist
isaaclab_rlchangelog fragment is preservedCONTRIBUTORS.md