Obstacle Avoidance with Vision-Based RL for Manipulators in Isaac Lab

This project implements vision-based reinforcement learning for the UR5 robotic manipulator in Isaac Lab, enabling precise object manipulation through camera-guided control. Our framework combines state-of-the-art physics simulation with deep reinforcement learning to achieve robust reach-avoid operations in complex environments.

Key Features:

🎯 Vision-Based Control: Direct camera input for object detection and manipulation
🚀 GPU-Accelerated Training: Leverage Isaac Sim's parallel simulation capabilities
📊 Real-time Monitoring: Integrated WandB support for experiment tracking
🔧 Modular Architecture: Easy to extend and customize for different tasks

Keywords: UR5, vision-based RL, Isaac Lab, robotic manipulation, pick-and-place

🚀 Installation

Prerequisites

Ubuntu 22.04 or Windows
NVIDIA GPU with CUDA 11.7+
Python 3.11

Source installation: Isaac Sim 5.1.0 and Isaac Lab 2.3.0

This repository was developed and tested with Isaac Sim 5.1.0 and Isaac Lab 2.3.0. The instructions below follow the Isaac Lab source-installation workflow.

Important

Directory Structure is Critical! You must install Isaac Sim, Isaac Lab, and this repository in three separate directories. Do not clone one inside another.

Recommended Structure:

~/workspaces/
├── IsaacSim/           # Installation 1
├── IsaacLab/           # Installation 2
└── RL_UR5_IsaacLab/    # This Repository (Installation 3)

1) Clone and build Isaac Sim (5.1.0)

# Example workspace directory where you keep the sources
cd $HOME

# Clone Isaac Sim
git clone https://github.com/isaac-sim/IsaacSim.git
cd IsaacSim


# Build Isaac Sim from source (Linux)
./build.sh

# After a successful build, set the ISAACSIM_PATH environment variable to the built release
export ISAACSIM_PATH="${PWD}/IsaacSim"
export ISAACSIM_PYTHON_EXE="${ISAACSIM_PATH}/python.sh"

# Quick verification
${ISAACSIM_PYTHON_EXE} -c "print('Isaac Sim configuration is now complete.')"
${ISAACSIM_PATH}/isaac-sim.sh --help

2) Clone Isaac Lab (2.3.0) and link to Isaac Sim

# Move to your workspace (outside of IsaacSim) and clone Isaac Lab
cd $HOME
git clone https://github.com/isaac-sim/IsaacLab.git
cd IsaacLab


# Create a symbolic link in Isaac Lab pointing to the Isaac Sim built release
# This makes the Isaac Sim modules and extensions discoverable by Isaac Lab
ln -s ${ISAACSIM_PATH} _isaac_sim

3) Create / activate a Python environment for Isaac Lab

Recommendation: create a dedicated environment (conda or uv). For Isaac Sim 5.x the Python runtime is 3.11 — ensure your virtual env uses the same Python minor version.

# Using the helper to create a conda environment (default name: env_isaaclab)
./isaaclab.sh -c


# Activate the environment (conda example)
conda activate env_isaaclab

4) Install Isaac Lab extensions and learning frameworks

# Install all extensions (default). This installs the learning frameworks (rl_games, rsl_rl, sb3, skrl, robomimic, ...)
./isaaclab.sh -i

5) Clone and Install this Repository

Now that the base environment is set up, clone this repository in a separate folder (do not nest it inside IsaacLab or IsaacSim).

cd $HOME
# Clone this repository (replace URL with actual repository URL)
git clone https://github.com/yourusername/RL_UR5_IsaacLab.git
cd RL_UR5_IsaacLab

# Install the project in editable mode
pip install -e source/RL_UR5

6) Download the 3D assets

Download the 3D assets from: https://clemson.box.com/s/raeoeb7gcislpjend57gj5im4q5p24h2

Place the downloaded assets in the following folder (replace the existing assets in that folder):

/RL_UR5_IsaacLab/source/RL_UR5/RL_UR5/tasks/direct/rl_ur5/assets

Note: the assets must be downloaded and placed into the assets folder above before running the tasks or training — tasks expect these assets to be present.

7) Weights & Biases (WandB) integration (optional but recommended)

WandB is used for experiment tracking and visualizing training metrics. To enable WandB integration for this project:

Install the WandB client in the active Conda environment:

pip install wandb

# interactive login (recommended for local use)
wandb login

# or set the API key in CI/headless setups
export WANDB_API_KEY="<your_api_key_here>"

Enable WandB in the project configuration at:

source/RL_UR5/RL_UR5/tasks/direct/rl_ur5/agents/PPO_skrl_hierarchical_gray_depth.yaml

Set agent.experiment.wandb: true and set agent.experiment.wandb_kwargs.project / agent.experiment.wandb_kwargs.entity to your project and account.

Note: ensure you have network access and a WandB account (or set WANDB_API_KEY) before running training with WandB enabled. If you prefer not to use WandB, set agent.experiment.wandb to false.

Notes and troubleshooting:

Ensure OS is Ubuntu 22.04 LTS (required for building Isaac Sim from source).
Isaac Sim 5.x requires Python 3.11 — the Python interpreter in your virtual environment must match the simulator's Python version.
If you see ModuleNotFoundError: No module named 'isaacsim', ensure the virtual environment is activated and _isaac_sim/setup_conda_env.sh (or the corresponding setup script) has been executed.
If switching from older Isaac Sim versions, you may want to reset user data after the first run:

${ISAACSIM_PATH}/isaac-sim.sh --reset-user

If you prefer not to build from source, you can use pre-built packages for Isaac Sim (not covered here) or follow the Isaac Lab pip/binaries installation guides linked in the official docs.

📦 Model Checkpoints & Auto-Downloads

To preserve Git repository storage and avoid Git LFS budget limitations, heavy model checkpoints (.pt files) are hosted directly on GitHub Releases (release tag: v1.0.0-checkpoints).

Automatic Fetching

When you run evaluation scripts (scripts/skrl/play.py) or training resume commands (scripts/skrl/train.py --checkpoint ...), any missing checkpoint is automatically downloaded from the GitHub Release directly into the local logs/ directory on-demand.

Manual Download & Pre-fetching

To download all or specific model checkpoints prior to running evaluation:

# Download all registered model checkpoints into local logs/
python scripts/utils/download_checkpoints.py --all

# Download a specific model checkpoint by key
python scripts/utils/download_checkpoints.py --key hierarchical_depth_v1

You can also download binary weights directly from the GitHub Releases release page:
🔗 v1.0.0-checkpoints Release Page

🎯 Training

Quick Start

Train the UR5 manipulator with vision-based reinforcement learning (High-Level Policy):

python scripts/skrl/train.py \
    --task=UR5-Hierarchical-Depth-PPO \
    --num_envs 32 \
    --enable_cameras \
    --headless

Command Arguments

Argument	Description	Default
`--task`	Training environment/task name	Required
`--num_envs`	Number of parallel simulation environments	2
`--enable_cameras`	Enable camera sensors for vision-based control	False
`--headless`	Run without GUI rendering (faster training)	False
`--seed`	Random seed for reproducibility	42
`--max_iterations`	PPO update iterations; overrides the YAML trainer timesteps	Config value

Advanced Training Example

For longer training with more environments:

python scripts/skrl/train.py \
    --task=UR5-Hierarchical-Depth-PPO \
    --num_envs 64 \
    --enable_cameras \
    --headless \
    --seed 123 \
    --max_iterations 50000

Monitoring Training Progress

Training logs and checkpoints are automatically saved to:

logs/skrl/logs/<experiment_name>/<timestamp>/
├── checkpoints/
│   ├── best_agent.pt      # Best performing model
│   └── agent_XXXX.pt      # Periodic checkpoints
├── events.out.tfevents.*  # TensorBoard event file
└── params/
    ├── agent.yaml
    └── env.yaml

How It Works

The hierarchical setup splits the problem into:

UR5-Waypoint-LowLevel-PPO: a state-only low-level waypoint tracker with 6-D joint-delta actions.
UR5-Hierarchical-Depth-PPO: a high-level gray+depth visual policy with 3-D Cartesian waypoint actions. The waypoint is executed by a damped differential IK controller inside the environment.

The high-level environment does not require a trained low-level checkpoint to run. It uses DLS IK for the low-level executor, while the separate low-level PPO task gives you a learned tracker path to experiment with later.

The human arm pose is not provided in the policy state; obstacle awareness must come from the gray+depth image.

For a deeper explanation of how the high-level policy chooses and learns waypoints, see HIERARCHICAL_WAYPOINT_RL.md.

For a quick smoke test of the high-level policy:

python scripts/skrl/train.py \
    --task=UR5-Hierarchical-Depth-PPO \
    --num_envs 2 \
    --enable_cameras \
    --headless \
    --max_iterations 1

Train the Low-Level Waypoint Tracker

python scripts/skrl/train.py \
    --task=UR5-Waypoint-LowLevel-PPO \
    --num_envs 64 \
    --headless

For a quick smoke test:

python scripts/skrl/train.py \
    --task=UR5-Waypoint-LowLevel-PPO \
    --num_envs 2 \
    --headless \
    --max_iterations 1

Relevant Files

Environment: source/RL_UR5/RL_UR5/tasks/direct/rl_ur5/huber_obj_hierarchical_gray_depth.py
Low-level PPO config: source/RL_UR5/RL_UR5/tasks/direct/rl_ur5/agents/PPO_skrl_waypoint_low_level.yaml
High-level PPO config: source/RL_UR5/RL_UR5/tasks/direct/rl_ur5/agents/PPO_skrl_hierarchical_gray_depth.yaml

Runtime Environment

If python scripts/skrl/train.py ... fails with ModuleNotFoundError: No module named 'isaacsim', activate the Isaac Lab environment and source the Isaac Sim setup script first:

conda activate env_isaaclab
source /path/to/isaacsim/setup_conda_env.sh

On this workstation, the verified setup was:

conda activate isaac
source /home/adi2440/isaacsim/setup_conda_env.sh

🎮 Evaluation

Playing the Default Trained Model

To visualize and evaluate the default trained hierarchical checkpoint, you can run inference with the following command:

python scripts/skrl/play.py \
    --task=UR5-Hierarchical-Depth-PPO \
    --num_envs 2 \
    --enable_cameras \
    --checkpoint logs/skrl/logs/skrl_hierarchical_depth/v1/checkpoints/best_agent.pt

Evaluation Arguments

Argument	Description	Default
`--checkpoint`	Path to trained model checkpoint	Required
`--num_envs`	Number of parallel evaluation environments	2
`--enable_cameras`	Enable camera rendering	False
`--video`	Record evaluation episodes	False
`--video_length`	Number of steps to record	200

Batch Evaluation

Evaluate multiple checkpoints or conditions:

# Evaluate with different environment counts
for n in 1 2 4 8; do
    python scripts/skrl/play.py \
        --task=UR5-Hierarchical-Depth-PPO \
        --num_envs $n \
        --enable_cameras \
        --checkpoint logs/skrl/logs/skrl_hierarchical_depth/v1/checkpoints/best_agent.pt
done

⚙️ Configuration

WandB Integration

This project includes Weights & Biases (WandB) integration for experiment tracking. Configuration is located at:

source/RL_UR5/RL_UR5/tasks/direct/rl_ur5/agents/PPO_skrl_hierarchical_gray_depth.yaml

To enable WandB logging, modify the configuration:

# In PPO_skrl_hierarchical_gray_depth.yaml
agent:
  experiment:
    wandb: true
    wandb_kwargs:
      project: "ur5-manipulation"
      entity: "your-wandb-username"

Environment Configuration

Customize task parameters in the environment configuration files:

# Example configuration structure
sim:
  dt: 0.01                    # Simulation timestep
  substeps: 1                  # Physics substeps
  
env:
  num_envs: 2048              # Number of parallel environments
  episode_length_s: 10.0      # Episode duration in seconds
  
robot:
  controller:
    type: "joint_position"    # Controller type
    stiffness: 800.0
    damping: 40.0

📊 Results

Our trained models achieve:

Success Rate: 90%+ on arm avoidance tasks
Training Time: ~10 hours on RTX 3080 (128 environments with Tiled Camera Data)
Sim-to-Real Gap: Minimal with proper domain randomization

Visualizations

Training progress and evaluation videos are automatically saved to the logs directory. View them with:

# TensorBoard visualization
tensorboard --logdir logs/skrl/logs/

# Video playback
python scripts/visualize_results.py --log_dir logs/skrl/logs/<experiment_name>

🔧 Troubleshooting

Common Issues

1. CUDA Out of Memory

# Reduce number of environments
python scripts/skrl/train.py --task=UR5-Hierarchical-Depth-PPO --num_envs 1 --enable_cameras

2. Camera not rendering

Ensure --enable_cameras flag is set
Check GPU drivers support RTX rendering
Try running without --headless for debugging

3. Module not found errors

# Ensure conda environment is activated
conda activate env_isaaclab
source /path/to/isaacsim/setup_conda_env.sh

# Reinstall project dependencies
pip install -e source/RL_UR5 --force-reinstall

Getting Help

📚 Check the Isaac Lab documentation
💬 Open an issue on our GitHub repository

🚧 Coming Soon: Real Robot Deployment

We're actively working on deploying our trained policies to physical UR5 robots! The upcoming release will include:

Planned Features

🤖 Real UR5 Integration: Direct deployment pipeline from simulation to physical UR5 arm
📦 Pre-trained Checkpoints: Battle-tested models ready for real-world deployment
🔌 ROS2 Bridge: Seamless integration with ROS2 for robot control and sensor data
📷 Camera Calibration: Automated tools for camera-robot calibration
🛡️ Safety Layers: Built-in collision detection and emergency stop mechanisms
📊 Real-time Monitoring: Live visualization of robot state and vision input

📝 Citation

If you use this work in your research, please cite:

@software{ur5_isaac_manipulation,
  author = {Aditya Parameshwaran},
  title = {Vision-Based UR5 Manipulation in Isaac Lab},
  year = {2024},
  publisher = {GitHub},
  url = {https://github.com/yourusername/ur5-isaac-lab}
}

📄 License

This project is released under the BSD-3-Clause License. See the LICENSE file for full details.

Third-Party Licenses

This project incorporates code from:

Isaac Lab: BSD-3-Clause License
NVIDIA Isaac Sim: Subject to NVIDIA EULA
Python Dependencies: Various licenses (see requirements.txt)

For a complete list of third-party licenses, please refer to the docs/licenses/ directory.

🙏 Acknowledgments

This work builds upon:

NVIDIA Isaac Lab for the simulation framework
SKRL for reinforcement learning algorithms
The robotics research community for continuous inspiration

Happy Training! 🚀

FilesExpand file tree

README.md

Latest commit

History