Accelerate Newton transform sync for Isaac RTX - #7553
Conversation
Greptile SummaryThe PR accelerates Newton-to-Isaac RTX transform synchronization by deriving Fabric local matrices from Newton world poses and using GPU hierarchy propagation.
Confidence Score: 5/5The PR appears safe to merge with no concrete blocking or independently actionable non-blocking issues identified. The changed transform synchronization and cache lifecycle have plausible edge cases, but the available repository evidence does not establish a realistic supported path that produces incorrect transforms or failed rendering. Important Files Changed
Sequence DiagramsequenceDiagram
participant N as Newton state
participant M as NewtonManager
participant F as Fabric arrays
participant H as Fabric hierarchy
participant R as Isaac RTX
N->>M: Current body world poses
M->>F: Read cached world/local matrices and body indices
M->>F: Write derived local matrices
M->>H: update_world_xforms_gpu(...)
H->>F: Propagate world transforms
F->>R: Updated render transforms
Reviews (1): Last reviewed commit: "Accelerate Newton Fabric transform sync" | Re-trigger Greptile |
![isaaclab-review-bot[bot]](https://avatars.githubusercontent.com/u/157846462?s=60&v=4){kind=small}
There was a problem hiding this comment.
Isaac Lab Review Bot
The Fabric binding cache and local-matrix propagation reduce per-frame work, but the new synchronization and failure-retry paths need correction before merge: CPU devices fail at the unconditional stream synchronization, and failed GPU hierarchy propagation causes subsequent retries to compound the local transform delta.
- Design and architecture: Caching the Fabric selection and bound arrays behind
PrepareForReuse()is coherent and the new state is reset consistently. However, propagation failure occurs after local matrices have been mutated while_transforms_dirtyremains set, so the next render retries from altered local matrices and stale world matrices. The write must be idempotent across retries or failed propagation must restore/fallback before retrying. - API: No public Python symbols or signatures change, and the package changelog fragment is present. The internal Fabric path does regress the previously supported CPU-device behavior because it now unconditionally invokes CUDA-oriented stream synchronization instead of retaining a compatible synchronization/propagation path.
- Implementation:
wp.synchronize_stream(PhysicsManager._device)fails for a non-CUDA device, causing every dirty render sync to enter the broad exception handler without updating transforms. Additionally,_set_fabric_transformswritesW_target * inverse(W_old) * L_oldbefore propagation; if propagation fails, retrying applies that delta again to the already-modified local matrix. Use device-appropriate ordering and ensure the failure path does not retry against partially updated transform state.
Minor fixes needed. Posted 2 actionable findings inline.
Automated review; human maintainers own approval decisions.
|
|
||
| NewtonManager._newton_fabric_ready = True | ||
| NewtonManager._transforms_dirty = False | ||
| wp.synchronize_stream(PhysicsManager._device) |
There was a problem hiding this comment.
🟡 Warning · Implementation — CUDA-only stream sync breaks CPU device
wp.synchronize_stream resolves its argument via get_device(...).stream, which raises for non-CUDA devices, while this path still binds and launches on PhysicsManager._device. On a CPU device this now throws into the broad except, _transforms_dirty stays set, and every render frame logs a traceback with no transform update, whereas the removed update_world_xforms() fallback previously handled it. wp.synchronize_device on line 762 already provides the ordering.
| NewtonManager._newton_fabric_ready = True | ||
| NewtonManager._transforms_dirty = False | ||
| wp.synchronize_stream(PhysicsManager._device) | ||
| if not cls._fabric_hierarchy.update_world_xforms_gpu(not topology_changed): |
There was a problem hiding this comment.
🟡 Warning · Design Architecture — Retry after failed propagation compounds local writes
Local matrices are written and synchronized before this call. When update_world_xforms_gpu returns false or raises, the handler leaves _transforms_dirty true so pre_render() retries, but the kernel derives the next local from the already-mutated local combined with the still-stale world matrix, so each retry reapplies the delta and corrupts poses. Restore a fallback propagation or make the local write idempotent before allowing a retry.
|
run-ci |
isaaclab-bot
Bot
added
ci:run-docker
|
Thanks for the speedup, the numbers are convincing. I ran the new sync path against a couple of scenes the existing tests don't cover and hit two regressions, both reproduced on this branch ( 1. Nested rigid-body prims render at the wrong poseThe kernel writes Two-link floating articulation, child link nested under the base, root shifted by 1 m between renders:
Suggested fix: keep world authoritative for Newton bodies instead of feeding the previous frame's Fabric matrices back into the write. Capture each prim's parent at bind time and derive the local matrix against the parent's target world when the parent is also a Newton body, or against a cached parent world otherwise. 2. Bodies without a USD prim silently disable sync for the whole sceneThe new selection requires So Suggested fix: create the local matrix attribute alongside the world one ( Smaller points
On the bot comments: the CPU-device Repro 1: nested articulationfrom isaaclab.app import AppLauncher
simulation_app = AppLauncher(headless=True, enable_cameras=True).app
import torch
import warp as wp
from pxr import Gf, UsdGeom, UsdPhysics
from usdrt import Rt
import isaaclab.sim as sim_utils
from isaaclab.actuators import ImplicitActuatorCfg
from isaaclab.assets import Articulation, ArticulationCfg
from isaaclab.sim import SimulationCfg, build_simulation_context
from isaaclab_newton.physics import MJWarpSolverCfg, NewtonCfg, NewtonManager
def _fabric_position(path):
prim = sim_utils.get_current_stage(fabric=True).GetPrimAtPath(path)
t = Rt.Xformable(prim).GetFabricHierarchyWorldMatrixAttr().Get().ExtractTranslation()
return torch.tensor([float(t[i]) for i in range(3)])
def _author_nested_articulation(stage, root_path):
root = UsdGeom.Xform.Define(stage, root_path)
UsdPhysics.ArticulationRootAPI.Apply(root.GetPrim())
base = UsdGeom.Xform.Define(stage, f"{root_path}/base")
base.AddTranslateOp().Set(Gf.Vec3d(0.0, 0.0, 1.0))
UsdPhysics.RigidBodyAPI.Apply(base.GetPrim())
UsdPhysics.MassAPI.Apply(base.GetPrim()).CreateMassAttr(1.0)
g = UsdGeom.Cube.Define(stage, f"{root_path}/base/geom")
g.GetSizeAttr().Set(0.2)
UsdPhysics.CollisionAPI.Apply(g.GetPrim())
# child link nested UNDER the parent link prim (MJCF/URDF importer style)
arm = UsdGeom.Xform.Define(stage, f"{root_path}/base/arm")
arm.AddTranslateOp().Set(Gf.Vec3d(0.0, 0.5, 0.0))
UsdPhysics.RigidBodyAPI.Apply(arm.GetPrim())
UsdPhysics.MassAPI.Apply(arm.GetPrim()).CreateMassAttr(1.0)
g2 = UsdGeom.Cube.Define(stage, f"{root_path}/base/arm/geom")
g2.GetSizeAttr().Set(0.2)
UsdPhysics.CollisionAPI.Apply(g2.GetPrim())
j = UsdPhysics.RevoluteJoint.Define(stage, f"{root_path}/base/arm_joint")
j.CreateBody0Rel().SetTargets([f"{root_path}/base"])
j.CreateBody1Rel().SetTargets([f"{root_path}/base/arm"])
j.CreateAxisAttr("X")
j.CreateLocalPos0Attr(Gf.Vec3f(0.0, 0.5, 0.0))
j.CreateLocalPos1Attr(Gf.Vec3f(0.0, 0.0, 0.0))
def test_nested_child_body_renders_at_newton_pose():
device = "cuda:0"
sim_cfg = SimulationCfg(
device=device, gravity=(0.0, 0.0, 0.0),
physics=NewtonCfg(solver_cfg=MJWarpSolverCfg(), use_cuda_graph=False),
)
with build_simulation_context(sim_cfg=sim_cfg) as sim:
sim._app_control_on_stop_handle = None
sim_utils.create_prim("/World/Env_0", "Xform", translation=(0.0, 0.0, 0.0))
_author_nested_articulation(sim_utils.get_current_stage(), "/World/Env_0/Robot")
robot = Articulation(ArticulationCfg(
prim_path="/World/Env_0/Robot", spawn=None,
actuators={"all": ImplicitActuatorCfg(joint_names_expr=[".*"], stiffness=0.0, damping=0.0)},
))
sim.reset()
labels = list(NewtonManager.get_model().body_label)
child_idx = [i for i, l in enumerate(labels) if l.endswith("/arm")][0]
parent_idx = [i for i, l in enumerate(labels) if l.endswith("/base")][0]
def check(tag):
sim.render()
wp.synchronize_device(device)
body_q = wp.to_torch(NewtonManager.get_state_0().body_q).cpu()
fp = _fabric_position("/World/Env_0/Robot/base")
fc = _fabric_position("/World/Env_0/Robot/base/arm")
print(f"[{tag}] newton child={body_q[child_idx, :3].tolist()} fabric child={fc.tolist()}")
torch.testing.assert_close(fp, body_q[parent_idx, :3], rtol=0.0, atol=1e-4, msg=f"{tag}: parent")
torch.testing.assert_close(fc, body_q[child_idx, :3], rtol=0.0, atol=1e-4, msg=f"{tag}: nested child")
check("after reset")
for k in range(1, 4):
target = torch.tensor([[1.0 * k, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0]], dtype=torch.float32, device=device)
robot.write_root_link_pose_to_sim_index(root_pose=target)
check(f"after root shift {k}")Output on this branch: Repro 2: cable + cube scenefrom isaaclab.app import AppLauncher
simulation_app = AppLauncher(headless=True, enable_cameras=True).app
import torch
import warp as wp
from usdrt import Rt
import isaaclab.sim as sim_utils
from isaaclab.assets import CableObjectCfg, RigidObjectCfg
from isaaclab.scene import InteractiveScene, InteractiveSceneCfg
from isaaclab.sim import SimulationCfg, build_simulation_context
from isaaclab.sim.spawners.materials import CableMaterialCfg
from isaaclab.sim.spawners.shapes import CableCfg
from isaaclab.utils.configclass import configclass
from isaaclab_newton.physics import NewtonCfg, NewtonManager, VBDSolverCfg
@configclass
class _SceneCfg(InteractiveSceneCfg):
cable: CableObjectCfg = CableObjectCfg(
prim_path="{ENV_REGEX_NS}/Cable",
spawn=CableCfg(
positions=((0.0, 0.0, 1.0), (0.0, 0.2, 1.0), (0.0, 0.4, 1.0), (0.0, 0.6, 1.0)),
physics_material=CableMaterialCfg(
thickness=0.02, density=500.0, stretch_stiffness=1.0e5, bend_stiffness=1.0e3
),
),
)
cube: RigidObjectCfg = RigidObjectCfg(
prim_path="{ENV_REGEX_NS}/Cube",
spawn=sim_utils.CuboidCfg(
size=(0.2, 0.2, 0.2),
rigid_props=sim_utils.RigidBodyPropertiesCfg(disable_gravity=True),
mass_props=sim_utils.MassPropertiesCfg(mass=1.0),
collision_props=sim_utils.CollisionPropertiesCfg(),
),
init_state=RigidObjectCfg.InitialStateCfg(pos=(1.0, 0.0, 1.0)),
)
def _fabric_position(path):
prim = sim_utils.get_current_stage(fabric=True).GetPrimAtPath(path)
t = Rt.Xformable(prim).GetFabricHierarchyWorldMatrixAttr().Get().ExtractTranslation()
return torch.tensor([float(t[i]) for i in range(3)])
def test_cube_next_to_cable_still_syncs():
device = "cuda:0"
sim_cfg = SimulationCfg(
dt=1.0 / 120.0, device=device, gravity=(0.0, 0.0, 0.0),
physics=NewtonCfg(solver_cfg=VBDSolverCfg(iterations=2), num_substeps=1, use_cuda_graph=False),
)
with build_simulation_context(sim_cfg=sim_cfg) as sim:
sim._app_control_on_stop_handle = None
scene = InteractiveScene(_SceneCfg(num_envs=1, env_spacing=2.0))
sim.register_interactive_scene(scene)
try:
sim.reset()
scene.reset()
sim.render()
wp.synchronize_device(device)
target = torch.tensor([[1.5, -0.75, 2.0, 0.0, 0.0, 0.0, 1.0]], dtype=torch.float32, device=device)
scene["cube"].write_root_link_pose_to_sim_index(root_pose=target)
sim.render()
wp.synchronize_device(device)
fp = _fabric_position("/World/envs/env_0/Cube")
print("cube newton", target[0, :3].tolist(), "fabric", fp.tolist(),
"fabric_ready", NewtonManager._newton_fabric_ready)
torch.testing.assert_close(fp, target[0, :3].cpu(), rtol=0.0, atol=1e-4)
finally:
sim.register_interactive_scene(None)Output on this branch: |
2 participants
Description
Newton's Isaac RTX bridge probes
FabricHierarchyGpuUpdateOptionsandupdate_world_xforms_gpu_with_options, which are absent from the supported Kit API. Every renderedframe therefore falls back to the CPU
update_world_xforms()path.This change keeps the fix local to
NewtonManager:update_world_xforms_gpu()API;PrepareForReuse()reports a topology change.It does not depend on the ClonePlan, SDP, renderer, or backend-registry refactors.
Performance
Matched GPU 0 runs used
Isaac-Cartpole-Camera-Direct, 4096 environments, Newton MJWarp physics,Isaac RTX rendering, 10 warm-up steps, and 100 measured host-return steps on an RTX 5090.
develop(c9dca35ef)Type of change
Validation
python -m pytest -q source/isaaclab_newton/test/physics/test_newton_fabric_body_sync.pydevelop.Checklist
isaaclab_newtonCONTRIBUTORS.md