Skip to content

[omniperf-agent] fix: PhysX KukaAllegro grasp via compliant contacts - #7511

Open
yts-nv wants to merge 6 commits into
isaac-sim:developfrom
yts-nv:omniperf-agent/physx-kukaallegro-compliant-contact-lift
Open

[omniperf-agent] fix: PhysX KukaAllegro grasp via compliant contacts#7511
yts-nv wants to merge 6 commits into
isaac-sim:developfrom
yts-nv:omniperf-agent/physx-kukaallegro-compliant-contact-lift

Conversation

@yts-nv

@yts-nv yts-nv commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Description

What & why

On Isaac-Lift-KukaAllegro, the PhysX backend never learns to grasp (success_rate ~ 0) while Newton converges. Root cause is a domain-randomization interaction, not a reward/sensor bug:

  • The dexterous KukaAllegro hand needs PhysX compliant (soft) contacts to hold the object.
  • But randomize_rigid_body_material writes the object's material through the rigid 3-tuple API — RigidBodyView.set_material_properties (static/dynamic friction + restitution) — which overwrites the compliant contact authored at spawn. -> rigid contacts -> the object is squirted out of the hand before a grasp forms -> good_finger_contact stays 0 -> the policy never converges.
  • Newton is immune: its DR touches only friction bindings (shape_material_mu/restitution), leaving contact stiffness/damping alone, and its contacts are soft by default.

Fix (core/lift)

  1. Give the object a PhysX compliant material — PhysxRigidBodyMaterialCfg(static_friction=0.5, compliant_contact_stiffness=300.0, compliant_contact_damping=30.0).
  2. Add core/lift/mdp/compliant_events.py -> RandomizeFrictionKeepCompliant: a startup event that randomizes friction and writes it back through RigidBodyView.set_compliant_material_properties (the 4-tuple: friction + stiffness + damping), so compliant contact survives domain randomization.
  3. Wire object_physics_material to it; robot_physics_material keeps the stock DR.

Keeps full domain randomization (object friction still randomized) and preserves compliant contact — no engine change needed; the compliant setter already exists in omni.physics.tensors.

Validation (PhysX, seed 42, 4096 envs)

Config success_rate @2000 Passed
baseline (no fix) 0.055 0
compliant only, DR on 0.000 0
disable both DRs 0.40 1
this PR (keep DR + compliant) 0.637 1
Newton (reference) 0.827 1

Converges higher than disabling DR (0.637 vs 0.40) because retaining randomization yields a more robust grasp. Validated on develop 8c3bc5e; targets current develop.

Note for reviewers

Implemented task-scoped (a core/lift/mdp event). The same idea could be upstreamed into the core randomize_rigid_body_material (write compliant via set_compliant_material_properties when the material has compliant fields) so all tasks benefit.

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context.
List any dependencies that are required for this change.

Fixes # (issue)

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (existing functionality will not work without user modification)
  • Documentation update

Release backport

  • Backport this pull request to the active release branch after it merges into develop

Screenshots

Please attach before and after screenshots of the change if applicable.

Checklist

Docker and GPU tests run on demand. Push the commits you want tested, then
comment run-ci on the pull request.

  • I have read and understood the contribution guidelines
  • I have run the pre-commit checks with ./isaaclab.sh --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package (do not edit CHANGELOG.rst or bump extension.toml — CI handles that)
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

Under rigid PhysX contacts the manipulated object is squirted out of the
dexterous KukaAllegro hand before a sustained thumb+finger pinch forms, so
the good_finger_contact grasp reward stays 0 and PPO never learns to lift;
the object leaves the workspace and episodes terminate early. Newton
(soft-constraint contacts) converges on the same task.

Fix (core/lift/lift_env_cfg.py):
- give the object a PhysX compliant (soft) contact material so the fingers
  can hold it (PhysxRigidBodyMaterialCfg, stiffness=300, damping=30);
- disable the startup material domain randomization that rebuilds+rebinds
  the object material at startup and would otherwise discard the compliant
  spawn material (that DR path does not support compliant contact).

Validated on this file (Isaac-Lift-KukaAllegro, env seed 42):
success_rate 0 -> 0.40 @2000 iters, 0.78 @4000 (Newton-level), reproduced 2x.
Note: OBJECT_PHYSICS / EventCfg here are shared by other lift configs
(e.g. franka); reviewer should confirm desired scope.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions github-actions Bot added the isaac-lab Related to Isaac Lab team label Sep 3, 2026
…g DR (true fix)

Replaces the earlier DR-disable workaround with a fix that keeps domain
randomization AND compliant contact on PhysX.

Root cause: the object needs PhysX compliant (soft) contacts to be graspable by
the dexterous KukaAllegro hand, but randomize_rigid_body_material writes the
object's material through the rigid 3-tuple API
(RigidBodyView.set_material_properties: static/dynamic friction + restitution),
overwriting the compliant contact authored at spawn -> rigid contacts -> the
object is ejected before a grasp forms -> the policy never converges. Newton is
unaffected: its DR touches only friction bindings and its contacts are soft by
default.

Fix (core/lift):
- give the object a PhysX compliant material (PhysxRigidBodyMaterialCfg,
  stiffness=300, damping=30);
- add core/lift/mdp/compliant_events.py: RandomizeFrictionKeepCompliant, a
  startup event that randomizes friction and writes it back through
  RigidBodyView.set_compliant_material_properties (the 4-tuple:
  friction + stiffness + damping), so compliant contact survives DR;
- wire object_physics_material to it; robot_physics_material keeps the stock DR.

Keeps full domain randomization (object friction still randomized) while
preserving compliant contact.

Validated (Isaac-Lift-KukaAllegro, PhysX, seed 42, 4096 envs):
  success_rate 0 -> 0.637 @2000 iters, Success Passed=1 (converged) -- higher
  than the DR-disable variant (0.40), because full randomization yields a more
  robust grasp. Validated on develop 8c3bc5e; targets current develop.

Note: implemented task-scoped (a core/lift/mdp event); could be upstreamed into
core randomize_rigid_body_material so all tasks benefit.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@yts-nv
yts-nv marked this pull request as ready for review September 3, 2026 11:05
@yts-nv
yts-nv requested a review from a team September 3, 2026 11:05
@greptile-apps

greptile-apps Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes failed PhysX grasp learning in the KukaAllegro lift task by authoring compliant object contacts and preserving their stiffness and damping while friction is randomized.

  • Adds a task-scoped compliant-contact friction-randomization event.
  • Applies compliant PhysX materials to the lift object variants.
  • Replaces object material randomization while retaining the existing robot randomization.
  • The new event is currently installed for non-PhysX lift backends whose views do not share the required API.

Confidence Score: 4/5

The backend-specific compliant-material event must be limited to PhysX or given backend-compatible behavior before this PR is safe to merge.

The shared lift configuration invokes compliant-material methods during startup even for selectable backends such as OvPhysX whose root view does not implement those methods, preventing affected environments from initializing.

Files Needing Attention: source/isaaclab_tasks/isaaclab_tasks/core/lift/lift_env_cfg.py, source/isaaclab_tasks/isaaclab_tasks/core/lift/mdp/compliant_events.py

Important Files Changed

Filename Overview
source/isaaclab_tasks/isaaclab_tasks/core/lift/lift_env_cfg.py Authors compliant object materials and installs the new randomizer globally, unintentionally exposing non-PhysX lift presets to a PhysX-only API.
source/isaaclab_tasks/isaaclab_tasks/core/lift/mdp/compliant_events.py Implements friction randomization through PhysX compliant-material tensors, but assumes every asset root view provides that backend-specific interface.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Lift EventCfg startup] --> B[RandomizeFrictionKeepCompliant]
  B --> C{Selected backend}
  C -->|PhysX| D[Compliant material tensor API]
  C -->|OvPhysX or Newton| E[Backend root view]
  E --> F[Unsupported compliant-material call]
  F --> G[Environment startup failure]
Loading

Reviews (1): Last reviewed commit: "[omniperf-agent] fix: PhysX KukaAllegro ..." | Re-trigger Greptile


object_physics_material = EventTerm(
func=mdp.randomize_rigid_body_material,
func=RandomizeFrictionKeepCompliant,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Backend-specific startup event

When a lift environment uses the OvPhysX or default Newton backend, the shared startup event calls PhysX-specific compliant-material methods on that backend's root view, causing environment initialization to fail. Install this event only for the PhysX configuration or provide backend-compatible behavior.

Knowledge Base Used: Physics backend extensions

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac Lab Review Bot

The compliant-contact fix is currently applied through backend-shared lift configuration, introducing PhysX-specific material and tensor APIs into Newton and OV startup paths. The new event also retains configuration parameters whose semantics it does not implement, and the new source is missing required repository metadata and includes experimental logging. A package changelog fragment is also required.

  • Design and architecture: EventCfg and the shared object presets serve multiple physics backends, but now directly use RandomizeFrictionKeepCompliant and PhysxRigidBodyMaterialCfg. The compliant material and event should be scoped to the PhysX-specific configuration while shared paths retain backend-neutral materials and backend-dispatched randomization.
  • API: The replacement event accepts the stock randomizer's parameters but ignores restitution_range, num_buckets, env_ids, and the call-time asset_cfg, causing configured behavior to be silently dropped. Its supported parameter contract should match its implementation, with callers updated accordingly, or the missing semantics should be implemented.
  • Implementation: The compliant material write path is coherent, but the new module lacks the required BSD-3-Clause SPDX header and contains an experimental [TRUEFIX] stdout message that should not remain in production. Because this visibly changes isaaclab_tasks physics behavior, the required changelog fragment or .skip fragment must also be added.

Needs rework. Posted 5 actionable findings inline.

Automated review; human maintainers own approval decisions.


object_physics_material = EventTerm(
func=mdp.randomize_rigid_body_material,
func=RandomizeFrictionKeepCompliant,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Critical · Design Architecture — PhysX-only event term in backend-shared EventCfg

EventCfg is documented as shared by all physics backends, yet RandomizeFrictionKeepCompliant calls root_view.max_shapes, get_compliant_material_properties() and set_compliant_material_properties(), which are PhysX RigidBodyView APIs. Newton and OV runs of this task now hit that startup term instead of the backend-dispatching mdp.randomize_rigid_body_material. Keep the shared config on the stock term and apply the compliant variant only in the PhysX-specific configuration.


OBJECT_PHYSICS = {
"physics_material": RigidBodyMaterialCfg(static_friction=0.5),
"physics_material": PhysxRigidBodyMaterialCfg(static_friction=0.5, compliant_contact_stiffness=300.0, compliant_contact_damping=30.0),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Warning · Design Architecture — PhysX material type in shared spawn presets

OBJECT_PHYSICS (and ObjectCfg.cube at line 85) now spawn with PhysxRigidBodyMaterialCfg instead of the backend-neutral RigidBodyMaterialCfg, but these presets feed default = shapes and ovphysx = cube for all backends. This changes the authored material type for non-PhysX spawners. Override the material in the PhysX configuration and leave the shared preset backend-neutral.

@@ -0,0 +1,50 @@
# PhysX "true fix" experiment: randomize friction while PRESERVING compliant contact,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Warning · Implementation — Missing SPDX header and leftover experiment artifacts

New source files must carry the repository SPDX header used by sibling modules such as lift_env_cfg.py; this file instead opens with an "experiment" note and ends with an unconditional [TRUEFIX] stdout print guarded by a _logged flag. Add the standard BSD-3-Clause header, remove the debug print, and rewrite the header comment to state the intent of preserving compliant contact.

self._dr = tuple(cfg.params.get("dynamic_friction_range", (0.5, 1.0)))
self._logged = False

def __call__(self, env, env_ids, static_friction_range=None, dynamic_friction_range=None,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Warning · Api — Term silently ignores configured randomization parameters

__call__ accepts env_ids, restitution_range, num_buckets, and asset_cfg but uses none of them: every environment and shape is written, restitution is never applied, and the configured num_buckets=250 friction discretization is dropped. Callers editing these params in EventCfg will see no effect and different behavior from the term being replaced. Honor the parameters or remove them from both the signature and the config.

cube = sim_utils.CuboidCfg(
size=(0.05, 0.05, 0.05),
physics_material=RigidBodyMaterialCfg(static_friction=0.5),
physics_material=PhysxRigidBodyMaterialCfg(static_friction=0.5, compliant_contact_stiffness=300.0, compliant_contact_damping=30.0),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Suggestion · Implementation — Missing changelog fragment for isaaclab_tasks

This changes user-visible lift task physics (compliant object material and a different startup randomization term) in the isaaclab_tasks package, but no fragment was added under source/isaaclab_tasks/changelog.d/. Repository rules require one past-tense fragment per touched source package, or a .skip fragment when no release note is needed.

yts-nv and others added 3 commits September 3, 2026 04:23
…ing DR fix

Follow IsaacLab contribution format for the PhysX KukaAllegro compliant-contact fix:
- add isaaclab_tasks CHANGELOG.rst entry (20.0.1) describing the fix;
- bump isaaclab_tasks version 20.0.0 -> 20.0.1;
- add the standard copyright/SPDX header + class docstring to
  core/lift/mdp/compliant_events.py and drop the debug print.

No functional change to the fix itself.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…x general

- Make ``RandomizeFrictionKeepCompliant`` backend-aware: PhysX uses
  ``RigidBodyView.set_compliant_material_properties``; Newton / OVPhysX delegate to the stock
  ``randomize_rigid_body_material``. The previous version called a PhysX-only tensor API
  unconditionally and would crash under the ``newton_mjwarp`` preset.
- Anchor the compliant stiffness/damping to ``2500`` / ``100`` = Newton's own default rigid
  contact ``ke`` / ``kd``, so both backends run the identical, tuned contact model. This is more
  physical than the previous ``300`` / ``30`` (0.78 mm vs 6.5 mm static penetration for the 0.2 kg
  object) and converges higher on PhysX: success_rate 0.843 vs 0.637 @2000 iters, Passed 2x @4000.

Compliant contact is a Kelvin-Voigt penalty spring-damper (stiffness [N/m], damping [N*s/m]) --
the same quantity as Newton's contact ke/kd, not a material Young's modulus.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ength)

Run the repo's pinned ruff (v0.14.10) format + lint --fix on the two changed files so they pass
pre-commit at line-length 120: wrap the long ``PhysxRigidBodyMaterialCfg`` material declarations and
the friction-sampling calls. No functional change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant