From Imitation to Refinement -- Residual RL for Precise Assembly (2407.16677v4)

Abstract: Recent advances in Behavior Cloning (BC) have made it easy to teach robots new tasks. However, we find that the ease of teaching comes at the cost of unreliable performance that saturates with increasing data for tasks requiring precision. The performance saturation can be attributed to two critical factors: (a) distribution shift resulting from the use of offline data and (b) the lack of closed-loop corrective control caused by action chucking (predicting a set of future actions executed open-loop) critical for BC performance. Our key insight is that by predicting action chunks, BC policies function more like trajectory "planners" than closed-loop controllers necessary for reliable execution. To address these challenges, we devise a simple yet effective method, ResiP (Residual for Precise Manipulation), that overcomes the reliability problem while retaining BC's ease of teaching and long-horizon capabilities. ResiP augments a frozen, chunked BC model with a fully closed-loop residual policy trained with reinforcement learning (RL) that addresses distribution shifts and introduces closed-loop corrections over open-loop execution of action chunks predicted by the BC trajectory planner. Videos, code, and data: https://residual-assembly.github.io.

Residual RL for Precise Visual Assembly: An Expert Overview

The paper under analysis explores the integration of Behavior Cloning (BC) and Reinforcement Learning (RL), with a particular focus on enhancing task performance in complex visual assembly tasks, specifically multi-part robotic assembly from RGB images.

Problem Context and Approach

BC has shown promise in robotic manipulation due to its simplicity in using human demonstrations to learn control policies. However, it lacks robustness in scenarios requiring corrective behaviors beyond the demonstrated actions. The paper identifies these scenarios, like multi-part assembly, as areas where BC frequently fails due to its dependence on the fixed strategies learned from demonstrations, which cannot adapt to deviations occurring in real-time deployments.

The authors propose a novel pipeline leveraging RL's capability to learn corrective actions via exploration and sparse rewards, thereby complementing the base BC policies. The proposed method, termed Residual RL for Precise Manipulation (ResiP), introduces residual policies that learn corrective action layers on top of BC-trained diffusion models.

Methodological Innovations

ResiP is characterized by the following components:

1. Action Chunking and Diffusion Models: The paper advocates for the use of advanced policy architectures like action chunking and diffusion models to increase initial success rates, enabling RL fine-tuning to leverage non-zero success starts effectively. The action chunking helps the model handle temporal dependencies better than single-step approaches.
2. Residual Policies: A core contribution of this work is training residual policies using PPO, which operate on top of already trained BC models. By predicting corrective actions, these residual models circumvent the instability typically encountered when directly fine-tuning complex models with RL. This approach removes the intrinsic architecture-based complexity that makes RL fine-tuning challenging.
3. Teacher-Student Distillation Pipeline: The pipeline also incorporates a distillation process where the corrected behaviors obtained through residual learning in simulation are distilled into high-quality RGB-based datasets. These datasets, enhanced with visual domain randomization, are used to train the real-world operational policy, bridging the sim-to-real gap effectively.

Empirical Findings and Implications

The paper conducts comprehensive experiments on a set of tasks from FurnitureBench. It conclusively demonstrates that residual RL significantly improves success rates over standalone BC policies and other RL fine-tuning methods. ResiP achieves up to 95% success rates on certain tasks under lower initial randomness settings. However, performance does saturate under higher complexity task settings. These findings assert the necessity of starting with a competent BC policy that the residual can build upon.

Additionally, the paper underscores the utility of generating and leveraging large synthetic datasets, observing marked improvements in vision-based policy training over smaller real-world demonstration sets. Nonetheless, a gap remains between the performance of the RL-trained policies and their distilled counterparts, which invite further exploration into bridging this divide.

Conclusion and Future Directions

The approach depicted in this work advances the application of RL in robotic assembly, demonstrating that a hybrid of BC with RL via residual learning can overcome numerous challenges faced by either approach in isolation. The utilization of residual policies enables learning systems that are not only adaptable and precise but also feasible to deploy directly in real-world settings with limited domain-specific tuning.

The implications of this work open several avenues for future research. Developing more sophisticated methods to minimize the performance gaps between synthetic and real-world tasks, improving robustness to macro-level deviations, and exploring enhanced sim-to-real transfer methods constitute natural progressions. Additionally, integrating more refined exploration strategies and state representation learning could further optimize such hybridized pipelines for broader robotic applications.

In sum, this paper provides valuable insights into the intersection of BC and RL, detailing a structured method to harness their combined strengths to advance robotic assembly accuracy and adaptiveness in dynamic, real-world conditions.