[Bug][Newton] MultiMeshRayCaster tracked targets: registered body_pattern (clone-dest glob) never matches prototype body labels

[wert93]

Isaac Lab version: v3.0.0-beta2.patch1 · Isaac Sim: 6.0.1 · backend: newton_mjwarp (PhysX same scene works)

Repro

A scene with per-env cloned kinematic RigidObjects ({ENV_REGEX_NS}/Obstacle_i) used as MultiMeshRayCasterCfg targets with RaycastTargetCfg(is_shared=False, track_mesh_transforms=True). On env creation with the Newton backend:

ValueError: Site 'ft_2' with body_pattern '/World/envs/env_*/Obstacle_0' matched no prototype bodies across 1 prototype(s).

Root cause (verified by instrumenting the error to print available labels)

Prototype builders label bodies with their full clone-SOURCE prim paths (e.g. /World/envs/env_0/Obstacle_0, /World/envs/env_0/Robot/FL_hip), and NewtonManager.cl_register_site matches body_pattern as a regex against those labels. But _NewtonRayCasterMixin._resolve_target_owner_exprs (clone-plan branch) registers the clone-DEST glob dest_glob + suffix = /World/envs/env_*/Obstacle_0 — and env_* as a regex means env + repeated underscores, which can never match env_0. (The cl_register_site docstring examples like Robot/link0 don't match the actual label format either.)

There is a second latent bug behind it: _tracked_site_labels_by_target is keyed by the registration-time owner expressions, while _create_tracked_target_view receives a third, differently-formatted expression list from _build_mesh_records — so after fixing the pattern, the dict lookup KeyErrors.

Fix (tested)

1. Register the escaped full source prim path (re.escape(owner_prim_path)) — exact literal match against the prototype label.
2. Consume tracked-site labels positionally (the base base_multi_mesh_ray_caster loop creates views exactly once per tracked target, in _raycast_targets_cfg order) instead of keying by unstable string forms.
3. (Optional QoL) extend the no-match ValueError in _cl_inject_sites to print available body labels — this is what made the root cause obvious.

With the patch below, our local-planner task (28 tracked per-env kinematic targets + robot articulation, 8×4 m grid ray caster) trains on newton_mjwarp with no errors; PhysX behavior unchanged.

diff --git a/source/isaaclab_newton/isaaclab_newton/physics/newton_manager.py b/source/isaaclab_newton/isaaclab_newton/physics/newton_manager.py
index 238253b..b5edbf1 100644
--- a/source/isaaclab_newton/isaaclab_newton/physics/newton_manager.py
+++ b/source/isaaclab_newton/isaaclab_newton/physics/newton_manager.py
@@ -853,10 +853,12 @@ class NewtonManager(PhysicsManager):
                 proto_sites.setdefault(proto_id, {})[label] = site_indices
 
             if not any_matched:
+                available = sorted({lbl for proto in proto_builders.values() for lbl in proto.body_label})
                 raise ValueError(
                     f"Site '{label}' with body_pattern '{body_pattern}' matched no prototype bodies "
                     f"across {len(proto_builders)} prototype(s). "
-                    f"Check that the pattern matches a body label in the prototype builder."
+                    f"Check that the pattern matches a body label in the prototype builder. "
+                    f"Available body labels (first 40 of {len(available)}): {available[:40]}"
                 )
 
         cls._cl_pending_sites.clear()
diff --git a/source/isaaclab_newton/isaaclab_newton/sensors/ray_caster/ray_caster.py b/source/isaaclab_newton/isaaclab_newton/sensors/ray_caster/ray_caster.py
index b514d52..033f8ee 100644
--- a/source/isaaclab_newton/isaaclab_newton/sensors/ray_caster/ray_caster.py
+++ b/source/isaaclab_newton/isaaclab_newton/sensors/ray_caster/ray_caster.py
@@ -100,12 +100,18 @@ class _NewtonRayCasterMixin:
         """Register sensor and dynamic target sites before cloning occurs."""
         super().__init__(cfg)  # pyright: ignore[reportCallIssue]
         self._sensor_site_labels = self._register_sites_for_expr(self.cfg.prim_path)
-        self._tracked_site_labels_by_target: dict[tuple[str, ...], list[str]] = {}
+        # Labels are stored PER TRACKED TARGET IN CFG ORDER and consumed positionally by
+        # _create_tracked_target_view: the expression strings computed at registration time
+        # (prototype-local body patterns) never match the differently-formatted expressions
+        # the base class passes at view-creation time (clone-dest globs / regex-expanded
+        # paths), so string keys cannot work here.
+        self._tracked_site_labels_seq: list[list[str]] = []
+        self._tracked_view_cursor: int = 0
         for target_cfg in getattr(self, "_raycast_targets_cfg", []):
             if target_cfg.track_mesh_transforms:
-                owner_exprs = self._resolve_target_owner_exprs(target_cfg.prim_expr)
-                labels = self._register_target_sites_for_exprs(owner_exprs)
-                self._tracked_site_labels_by_target[tuple(owner_exprs)] = labels
+                owner_patterns = self._resolve_target_owner_exprs(target_cfg.prim_expr)
+                labels = self._register_target_sites_for_exprs(owner_patterns)
+                self._tracked_site_labels_seq.append(labels)
 
     def _register_sites_for_expr(self, prim_expr: str) -> list[str]:
         """Register Newton sites for a prim expression and return site labels."""
@@ -145,7 +151,13 @@ class _NewtonRayCasterMixin:
                         " to dynamic targets."
                     )
                 owner_prim_path = str(body.GetPath())
-                owner_exprs.append(dest_glob + owner_prim_path[len(source_path) :])
+                # Prototype builders label bodies with their full clone-SOURCE prim paths
+                # (e.g. "/World/envs/env_0/Obstacle_0", "/World/envs/env_0/Robot/FL_hip"),
+                # and cl_register_site matches body_pattern as a REGEX against those labels.
+                # The previous clone-DEST glob ("/World/envs/env_*/...") never matches:
+                # "env_*" as a regex means "env" + repeated underscores, not "env_0".
+                # Register the escaped source path — an exact literal match.
+                owner_exprs.append(re.escape(owner_prim_path))
             return list(dict.fromkeys(owner_exprs))
 
         # Legacy fallback: no clone plan, or the target is not owned by any plan row.
@@ -240,8 +252,11 @@ class _NewtonRayCasterMixin:
 
     def _create_tracked_target_view(self: Any, target_prim_path: str | list[str]):
         """Resolve dynamic multi-mesh target sites to raw Newton site indices."""
-        target_exprs = target_prim_path if isinstance(target_prim_path, list) else [target_prim_path]
-        labels = self._tracked_site_labels_by_target[tuple(target_exprs)]
+        # Views are created once per tracked target, in cfg order, by the
+        # base_multi_mesh_ray_caster loop — consume the registered labels positionally
+        # (the modulo guards a hypothetical re-initialization pass).
+        labels = self._tracked_site_labels_seq[self._tracked_view_cursor % len(self._tracked_site_labels_seq)]
+        self._tracked_view_cursor += 1
         site_indices = self._resolve_site_indices(labels, str(target_prim_path), self._num_envs)
         return wp.array(site_indices, dtype=wp.int32, device=self._device)
 

[RandomOakForest]

Thank you for posting this. It would be great if you could submit a PR with the changes. Thank you!

[prina404]

I believe this is not exclusive to newton's backend, as I've encountered the same bug when using MultiMeshRaycaster with physx backend too.

The diff for the fix on the physx side should look something like this:

diff --git a/source/isaaclab_physx/isaaclab_physx/sensors/ray_caster/ray_caster.py b/source/isaaclab_physx/isaaclab_physx/sensors/ray_caster/ray_caster.py
--- a/source/isaaclab_physx/isaaclab_physx/sensors/ray_caster/ray_caster.py
+++ b/source/isaaclab_physx/isaaclab_physx/sensors/ray_caster/ray_caster.py
@@ -123,13 +123,14 @@ class _PhysXRayCasterMixin:
     def _create_tracked_target_view(self: Any, target_prim_paths: str | list[str]):
         """Create a PhysX rigid-body view for dynamic multi-mesh targets."""
         if isinstance(target_prim_paths, str):
             target_prim_paths = [target_prim_paths]
-        body_paths = []
+        bodies_per_expr = []
         for target_prim_path in target_prim_paths:
             prims = sim_utils.find_matching_prims(target_prim_path)
             if len(prims) == 0:
                 # ClonePlan-backed targets may not have destination mesh prims.
                 # In that case BaseMultiMeshRayCaster passes the destination owner-body expression.
-                body_paths.append(target_prim_path)
+                bodies_per_expr.append([target_prim_path])
                 continue
+            expr_bodies = []
             for prim in prims:
                 body = sim_utils.get_first_matching_ancestor_prim(prim.GetPath(), predicate=_has_rigid_body_api)
                 if body is None:
                     raise RuntimeError(
                         f"Cannot track non-physics ray-cast target '{target_prim_path}' with PhysX. "
                         "Set track_mesh_transforms=False for static targets, or apply RigidBodyAPI to dynamic targets."
                     )
-                body_paths.append(body.GetPath().pathString)
+                expr_bodies.append(body.GetPath().pathString)
+            bodies_per_expr.append(expr_bodies)

+        if len(bodies_per_expr) > 1 and len({len(bodies) for bodies in bodies_per_expr}) == 1:
+            body_paths = [path for env_bodies in zip(*bodies_per_expr) for path in env_bodies]
+        else:
+            body_paths = [path for bodies in bodies_per_expr for path in bodies]

         if len(body_paths) == 0:
             raise RuntimeError(f"No tracked target bodies resolved from: {target_prim_paths}")

[StafaH]

Thanks for bringing up this issue! This is now fixed by: #7516