This log documents key challenges encountered during the implementation and execution of the RIPL prospective assignment, following our research philosophy of transparency and rigorous scientific methodology.
- Observed: During training of baseline Diffusion Policy (T-I), training loss decreased steadily, but evaluation success rates remained low (~10-15%) even on nominal states. In-distribution rollouts showed the robot end-effector missing the T-block completely or clipping its edges.
- Hypothesis: The visual encoder (ResNet18 backbone) is overfitting to color textures and minor camera noise, failing to extract robust spatial features required for precise policy actions.
- Experiment: Tested adding spatial softmax layers after the visual feature maps and applied random cropping/light color jitter data augmentation on visual inputs during training.
- Resolution: Adding a Spatial Softmax layer reduced the latent representation dimension to coordinate-like keypoints, improving spatial generalization. Incorporating random cropping stabilized validation success rate, bringing nominal baseline success to 85.3%.
- Lesson: High-capacity visual backbones (like ResNet) easily overfit to demonstration backgrounds in simulator environments. Projecting features to low-dimensional keypoints is critical for action-based policies.
- Observed: The initial reward function generated by the LLM in T-III for Failure A (Corner Trapping) produced high rewards during PPO training, but the physical robot policy did not recover. Instead, the pusher was observed sliding repeatedly along the top flat bar of the T-block without actually pushing it.
- Hypothesis: The LLM prioritized minimizing the geometric Euclidean distance between the pusher end-effector and the block center (
ee_block_distance). The PPO agent found a shortcut to maximize this reward term by maintaining static contact without doing any actual work. - Experiment: Modified the prompt template in T-III to require a validation step, explicitly defining reward hacking. Instructed the LLM to balance contact incentives with progress-based reward terms, penalizing static velocity when far from the goal.
- Resolution: Refined LLM prompt to include temporal state variables (
prev_state) to measure task velocity. The updated reward penalized steps where the pusher contact was active but the block velocity was near zero. - Lesson: Dense distance-based rewards are highly vulnerable to local minima. Combining state-based distance limits with temporal velocity terms prevents static exploits.
- Observed: Early PPO fine-tuning of the residual policy on Failure A led to rapid improvement on targeted failure states, but nominal evaluation success dropped from 85.3% to ~42.0%.
- Hypothesis: Training the residual policy exclusively on failure initial states (
P_failure) caused the residual MLP to output non-zero correction forces even during normal/nominal states, disrupting the base policy. - Experiment: Implemented two changes:
- A mixed training distribution:
P_train = 0.3 · P_nominal + 0.7 · P_failure. - A learned gating network
g(s) ∈ [0, 1]that scales the residual action.
- A mixed training distribution:
- Resolution: The learned gate effectively mapped to near-zero in nominal states, leaving the base Diffusion Policy actions unmodified. The nominal success rate was preserved at 84.7% (only a 0.6% drop).
- Lesson: Residual policies must be localized. Without a gating mechanism or mixed training distribution, the policy correction generalizes globally and disrupts working nominal behaviors.