Set Warp's global determinism mode under --deterministic

Newton's solvers take a deterministic argument and apply it as a per-module
option, so the solver kernels already honoured PhysicsCfg.deterministic. Its
sensor and geometry kernels take no such argument and fall back to
warp.config.deterministic, which stayed at NOT_GUARANTEED.

The scene BVH is built over a shape list compacted with wp.atomic_add, so its
primitive order varied between processes. Ray queries then broke ties
differently and a tiled camera rendered a few pixels differently from
bit-identical simulation state, which was enough to make an image-observation
policy diverge.

Warp reads the setting at module build time, so it is requested in
apply_env_overrides, before the environment is created. An explicitly chosen
mode is left alone.

Author: hujc7

docs/source/features/reproducibility.rst

@@ -35,7 +35,18 @@ for **RL-Games**, **skrl**, **RSL-RL**, and **Stable-Baselines3**: each calls
 so library initialization is not disturbed, then training proceeds with the requested global RNG and
 optional PyTorch deterministic algorithms. Whether the **rendering** half of the flag matters depends
 on the workload: **physics-only** simulation does not render at all; **RTX** rendering (non-minimal
-mode) needs it for reproducible imagery; **Newton** rendering is already deterministic.
+mode) needs it for reproducible imagery; **Newton** rendering needs Warp's global determinism mode,
+which the flag sets (see :ref:`below <reproducibility-warp-determinism>`).
+
+.. _reproducibility-warp-determinism:
+
+**Warp determinism.** Newton's solvers accept a ``deterministic`` argument that Isaac Lab supplies
+from :attr:`~isaaclab.physics.PhysicsCfg.deterministic`, and they apply it as a per-module option.
+Its sensor and geometry kernels take no such argument and fall back to ``warp.config.deterministic``,
+so ``--deterministic`` sets that global to ``RUN_TO_RUN`` as well. Without it the scene BVH is built
+over an atomically compacted shape list whose order varies between processes, and a tiled camera can
+render a few pixels differently from identical simulation state. An explicitly chosen mode, such as
+``GPU_TO_GPU``, is left untouched.
 
 .. note::
 

source/isaaclab_rl/changelog.d/warp-determinism-global.rst

@@ -0,0 +1,9 @@
+Fixed
+^^^^^
+
+* Fixed ``--deterministic`` not making camera observations reproducible. The flag configured the
+  physics solver but left :attr:`warp.config.deterministic` at ``NOT_GUARANTEED``, and Newton's
+  sensor and geometry kernels -- unlike its solvers -- take no per-module determinism option and
+  fall back to that global. The scene BVH is built over an atomically compacted shape list, so its
+  primitive order varied between processes and a tiled camera rendered a few pixels differently
+  from identical simulation state, which was enough to make image-observation training diverge.

source/isaaclab_rl/isaaclab_rl/entrypoints/common.py

@@ -497,6 +497,29 @@ def apply_env_overrides(args_cli: argparse.Namespace, env_cfg: Any, *, apply_dev
         physics_cfg = getattr(getattr(env_cfg, "sim", None), "physics", None)
         if physics_cfg is not None:
             physics_cfg.deterministic = True
+        request_warp_determinism()
+
+
+def request_warp_determinism() -> None:
+    """Ask Warp for run-to-run deterministic atomics process-wide.
+
+    Newton's solvers take a ``deterministic`` argument and apply it as a per-module option, so a
+    solver-level request already covers the physics kernels. Its sensor and geometry modules take
+    no such argument and fall back to ``warp.config.deterministic``, which defaults to
+    ``NOT_GUARANTEED``. One of them, the BVH shape compaction in ``newton._src.geometry.bvh``,
+    claims output slots with ``wp.atomic_add``, so the enabled-shape order -- and with it the
+    order of the primitives the scene BVH is built over -- varies between processes. Ray queries
+    then break ties differently and a tiled camera renders a handful of pixels differently from
+    identical simulation state, which is enough to make an image-observation policy diverge.
+
+    Warp reads this at module build time, so it must be set before the first kernel launch;
+    :func:`apply_env_overrides` runs before the environment is created. An explicit setting is
+    left alone so a caller can still opt into ``GPU_TO_GPU``.
+    """
+    import warp as wp
+
+    if wp.config.deterministic == wp.DeterministicMode.NOT_GUARANTEED:
+        wp.config.deterministic = wp.DeterministicMode.RUN_TO_RUN
 
 
 def validate_distributed_device(args_cli: argparse.Namespace) -> None:

source/isaaclab_rl/test/test_entrypoints_common.py

@@ -420,6 +420,48 @@ def test_apply_env_overrides_leaves_physics_alone_without_the_flag(monkeypatch:
     assert env_cfg.sim.physics.deterministic is False
 
 
+def test_apply_env_overrides_requests_warp_determinism(monkeypatch: pytest.MonkeyPatch) -> None:
+    """The request reaches Warp's global too, which is all that covers Newton's sensor kernels."""
+    import warp as wp
+
+    monkeypatch.setattr(wp.config, "deterministic", wp.DeterministicMode.NOT_GUARANTEED)
+    physics = _fake_physics_cfg("NewtonCfg")
+    env_cfg = SimpleNamespace(sim=SimpleNamespace(physics=physics))
+
+    args_cli = argparse.Namespace(num_envs=None, device=None, deterministic=True)
+    _rl_common.apply_env_overrides(args_cli, env_cfg, apply_device=False)
+
+    assert wp.config.deterministic == wp.DeterministicMode.RUN_TO_RUN
+
+
+def test_apply_env_overrides_keeps_an_explicit_warp_mode(monkeypatch: pytest.MonkeyPatch) -> None:
+    """A caller that already chose a stronger guarantee keeps it."""
+    import warp as wp
+
+    monkeypatch.setattr(wp.config, "deterministic", wp.DeterministicMode.GPU_TO_GPU)
+    physics = _fake_physics_cfg("NewtonCfg")
+    env_cfg = SimpleNamespace(sim=SimpleNamespace(physics=physics))
+
+    args_cli = argparse.Namespace(num_envs=None, device=None, deterministic=True)
+    _rl_common.apply_env_overrides(args_cli, env_cfg, apply_device=False)
+
+    assert wp.config.deterministic == wp.DeterministicMode.GPU_TO_GPU
+
+
+def test_apply_env_overrides_leaves_warp_alone_without_the_flag(monkeypatch: pytest.MonkeyPatch) -> None:
+    """Without ``--deterministic`` Warp keeps its default, so no run pays for determinism."""
+    import warp as wp
+
+    monkeypatch.setattr(wp.config, "deterministic", wp.DeterministicMode.NOT_GUARANTEED)
+    physics = _fake_physics_cfg("NewtonCfg")
+    env_cfg = SimpleNamespace(sim=SimpleNamespace(physics=physics))
+
+    args_cli = argparse.Namespace(num_envs=None, device=None, deterministic=False)
+    _rl_common.apply_env_overrides(args_cli, env_cfg, apply_device=False)
+
+    assert wp.config.deterministic == wp.DeterministicMode.NOT_GUARANTEED
+
+
 @pytest.mark.parametrize("class_name", ["PhysxCfg", "OvPhysxCfg", "NewtonCfg", "SomeFutureBackendCfg"])
 def test_apply_env_overrides_records_the_request_for_every_backend(class_name: str) -> None:
     """The request is backend-agnostic, so the entrypoint needs no per-backend knowledge."""