Commit 3748a72
authored
[Fix] Make --deterministic reproduce training runs by configuring the physics backend (#7334)
# Description
`--deterministic` configured PyTorch and the Isaac RTX renderer but
never reached the physics solver, so training on Newton backends was not
reproducible even with the flag passed. Two defaults moved out from
under the flag between 3.0beta2 and GA:
| Default | v3.0.0-beta2 | GA |
|---|---|---|
| `CartpolePhysicsCfg.default` | `PhysxCfg()` | `NewtonCfg` MJWarp,
`deterministic_mode="not_guaranteed"` |
| `MultiBackendRendererCfg.default` | `IsaacRtxRendererCfg()` |
`NewtonWarpRendererCfg()` |
The physics switch landed in `0caae64dc7c` (#7066), absent from all
three `v3.0.0-beta*` tags. At beta2 the flag worked because PhysX is
run-to-run deterministic for rigid bodies **and** Isaac RTX was the
default renderer; both premises were removed without re-wiring the flag.
`--deterministic` now sets `PhysicsCfg.deterministic` on the resolved
physics config, in `apply_env_overrides()` — the existing CLI-to-cfg
seam, after `scan()` resolves the backend and before the solver is
built. That field is the backend-agnostic request; each physics manager
translates it when the simulation starts:
- **Newton** derives `deterministic_mode="run_to_run"`, applies the
MJWarp `disable_sensors` prerequisite on the GPU path, and leaves
MuJoCo-CPU alone. An explicitly set `deterministic_mode` wins.
- **PhysX / OvPhysX** enable `enable_enhanced_determinism`. OvPhysX is
best-effort and not verified end to end.
Validation stays with the backend:
`NewtonManager._validate_deterministic_solver_cfg()` rejects an
unsupported solver at solver initialization, so there is one policy and
one set of error messages rather than a copy in the RL layer. Adding a
backend no longer means editing `isaaclab_rl`.
**A determinism request that would starve a sensor is now refused.**
Disabling MuJoCo Warp's sensors also skips its `rne_postconstraint`
stage, which fills Newton's `body_qdd` / `body_parent_f`. The IMU, PVA
and joint-wrench sensors read that state, so `Isaac-Ant`,
`Isaac-Humanoid` and `Isaac-Repose-Cube-Shadow` — all defaulting to
`newton_mjwarp` and feeding `joint_wrench` into their policy
observations — would have trained on values that are never refreshed,
with no error. `NewtonManager` raises at solver initialization, the only
point where both the solver config and the registered sensors are
visible.
The guard restates `{"body_qdd", "body_parent_f"}`, which Newton also
states internally (`solver_mujoco.py:4360` as a set, `:5179` as an
equivalent `or`-chain). That duplication is tracked upstream in
newton-physics/newton#4109, which asks for two
things: Newton refusing the combination at the source, and exporting the
field set as a public constant. Either lets this guard shrink — the
constant and its sensor-tracking delete entirely once Isaac Lab pins a
Newton that raises. Until then the guard is what prevents the silent
case, so it stays.
Fixes NVBug 6658578 (P0, Isaac Lab 3.0 GA).
## Type of change
- Bug fix (non-breaking change which fixes an issue)
## Validation
Three `--deterministic` runs of `Isaac-Cartpole-Camera` (50 epochs, task
defaults) on one L40, each in its own container:
```
det1 vs det2 55/55 checkpoint tensors bitwise identical
det1 vs det3 55/55 checkpoint tensors bitwise identical
ctrl1 vs ctrl2 41/55 differ <- same task, no flag
```
The unflagged controls diverge, so the agreement above is the flag's
doing rather than a task that is trivially reproducible. `Isaac-Ant
--deterministic` fails at startup with the sensor message. MuJoCo-CPU
could not be exercised: `SolverMuJoCo.get_max_contact_count()` raises
`NotImplementedError` on that path, so it is unreachable in Isaac Lab
today.
8 unit tests added; 200 pass across `test_entrypoints_common.py` and
`test_newton_manager_abstraction.py`.
## Release backport
- [x] <!-- backport-active-release --> Backport this pull request to the
active release branch after it merges into `develop`
## Checklist
- [x] I have read and understood the [contribution
guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html)
- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] I have added a changelog fragment under
`source/<pkg>/changelog.d/` for every touched package
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there1 parent d592423 commit 3748a72
15 files changed
Lines changed: 345 additions & 13 deletions
File tree
- docs/source/features
- source
- isaaclab_newton
- changelog.d
- isaaclab_newton/physics
- test/physics
- isaaclab_ov
- changelog.d
- isaaclab_ov/physics
- isaaclab_physx
- changelog.d
- isaaclab_physx/physics
- isaaclab_rl
- changelog.d
- isaaclab_rl/entrypoints
- test
- isaaclab_tasks
- changelog.d
- test/core
- isaaclab
- changelog.d
- isaaclab/physics
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
23 | | - | |
24 | | - | |
| 23 | + | |
| 24 | + | |
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
36 | | - | |
37 | | - | |
38 | | - | |
39 | | - | |
40 | | - | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
41 | 49 | | |
42 | 50 | | |
43 | 51 | | |
| |||
66 | 74 | | |
67 | 75 | | |
68 | 76 | | |
69 | | - | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
70 | 97 | | |
71 | 98 | | |
72 | 99 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
37 | 50 | | |
38 | 51 | | |
39 | 52 | | |
| |||
Lines changed: 16 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
Lines changed: 64 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
121 | 121 | | |
122 | 122 | | |
123 | 123 | | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
124 | 134 | | |
125 | 135 | | |
126 | 136 | | |
| |||
433 | 443 | | |
434 | 444 | | |
435 | 445 | | |
| 446 | + | |
436 | 447 | | |
437 | 448 | | |
438 | 449 | | |
| |||
1188 | 1199 | | |
1189 | 1200 | | |
1190 | 1201 | | |
| 1202 | + | |
1191 | 1203 | | |
1192 | 1204 | | |
1193 | 1205 | | |
| |||
1589 | 1601 | | |
1590 | 1602 | | |
1591 | 1603 | | |
1592 | | - | |
| 1604 | + | |
| 1605 | + | |
| 1606 | + | |
1593 | 1607 | | |
1594 | 1608 | | |
| 1609 | + | |
1595 | 1610 | | |
1596 | 1611 | | |
1597 | 1612 | | |
| |||
2111 | 2126 | | |
2112 | 2127 | | |
2113 | 2128 | | |
2114 | | - | |
| 2129 | + | |
2115 | 2130 | | |
2116 | | - | |
| 2131 | + | |
2117 | 2132 | | |
2118 | 2133 | | |
2119 | 2134 | | |
| |||
2135 | 2150 | | |
2136 | 2151 | | |
2137 | 2152 | | |
| 2153 | + | |
| 2154 | + | |
| 2155 | + | |
| 2156 | + | |
| 2157 | + | |
| 2158 | + | |
| 2159 | + | |
| 2160 | + | |
| 2161 | + | |
| 2162 | + | |
| 2163 | + | |
| 2164 | + | |
| 2165 | + | |
| 2166 | + | |
| 2167 | + | |
| 2168 | + | |
| 2169 | + | |
| 2170 | + | |
| 2171 | + | |
| 2172 | + | |
| 2173 | + | |
| 2174 | + | |
| 2175 | + | |
| 2176 | + | |
| 2177 | + | |
| 2178 | + | |
| 2179 | + | |
| 2180 | + | |
| 2181 | + | |
| 2182 | + | |
| 2183 | + | |
| 2184 | + | |
| 2185 | + | |
| 2186 | + | |
| 2187 | + | |
| 2188 | + | |
| 2189 | + | |
| 2190 | + | |
| 2191 | + | |
| 2192 | + | |
| 2193 | + | |
| 2194 | + | |
| 2195 | + | |
| 2196 | + | |
| 2197 | + | |
2138 | 2198 | | |
2139 | 2199 | | |
2140 | 2200 | | |
| |||
2224 | 2284 | | |
2225 | 2285 | | |
2226 | 2286 | | |
2227 | | - | |
| 2287 | + | |
2228 | 2288 | | |
2229 | 2289 | | |
2230 | 2290 | | |
| |||
Lines changed: 84 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
| 28 | + | |
28 | 29 | | |
29 | 30 | | |
30 | 31 | | |
| |||
1592 | 1593 | | |
1593 | 1594 | | |
1594 | 1595 | | |
| 1596 | + | |
| 1597 | + | |
| 1598 | + | |
| 1599 | + | |
| 1600 | + | |
| 1601 | + | |
| 1602 | + | |
| 1603 | + | |
| 1604 | + | |
| 1605 | + | |
| 1606 | + | |
| 1607 | + | |
| 1608 | + | |
| 1609 | + | |
| 1610 | + | |
| 1611 | + | |
| 1612 | + | |
| 1613 | + | |
| 1614 | + | |
| 1615 | + | |
| 1616 | + | |
| 1617 | + | |
| 1618 | + | |
| 1619 | + | |
| 1620 | + | |
| 1621 | + | |
| 1622 | + | |
| 1623 | + | |
| 1624 | + | |
| 1625 | + | |
| 1626 | + | |
| 1627 | + | |
| 1628 | + | |
| 1629 | + | |
| 1630 | + | |
| 1631 | + | |
| 1632 | + | |
| 1633 | + | |
| 1634 | + | |
| 1635 | + | |
| 1636 | + | |
| 1637 | + | |
| 1638 | + | |
| 1639 | + | |
| 1640 | + | |
| 1641 | + | |
| 1642 | + | |
| 1643 | + | |
| 1644 | + | |
| 1645 | + | |
| 1646 | + | |
| 1647 | + | |
| 1648 | + | |
| 1649 | + | |
| 1650 | + | |
| 1651 | + | |
| 1652 | + | |
| 1653 | + | |
| 1654 | + | |
| 1655 | + | |
| 1656 | + | |
| 1657 | + | |
| 1658 | + | |
| 1659 | + | |
| 1660 | + | |
| 1661 | + | |
| 1662 | + | |
| 1663 | + | |
| 1664 | + | |
| 1665 | + | |
| 1666 | + | |
| 1667 | + | |
| 1668 | + | |
| 1669 | + | |
| 1670 | + | |
| 1671 | + | |
| 1672 | + | |
| 1673 | + | |
| 1674 | + | |
| 1675 | + | |
| 1676 | + | |
| 1677 | + | |
| 1678 | + | |
Lines changed: 6 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1120 | 1120 | | |
1121 | 1121 | | |
1122 | 1122 | | |
| 1123 | + | |
| 1124 | + | |
1123 | 1125 | | |
1124 | | - | |
| 1126 | + | |
1125 | 1127 | | |
1126 | 1128 | | |
1127 | 1129 | | |
| |||
Lines changed: 5 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
Lines changed: 7 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
797 | 797 | | |
798 | 798 | | |
799 | 799 | | |
| 800 | + | |
| 801 | + | |
| 802 | + | |
| 803 | + | |
| 804 | + | |
800 | 805 | | |
801 | 806 | | |
| 807 | + | |
| 808 | + | |
802 | 809 | | |
803 | 810 | | |
804 | 811 | | |
| |||
0 commit comments