TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correction (2405.10315v3)

Abstract: Learning in simulation and transferring the learned policy to the real world has the potential to enable generalist robots. The key challenge of this approach is to address simulation-to-reality (sim-to-real) gaps. Previous methods often require domain-specific knowledge a priori. We argue that a straightforward way to obtain such knowledge is by asking humans to observe and assist robot policy execution in the real world. The robots can then learn from humans to close various sim-to-real gaps. We propose TRANSIC, a data-driven approach to enable successful sim-to-real transfer based on a human-in-the-loop framework. TRANSIC allows humans to augment simulation policies to overcome various unmodeled sim-to-real gaps holistically through intervention and online correction. Residual policies can be learned from human corrections and integrated with simulation policies for autonomous execution. We show that our approach can achieve successful sim-to-real transfer in complex and contact-rich manipulation tasks such as furniture assembly. Through synergistic integration of policies learned in simulation and from humans, TRANSIC is effective as a holistic approach to addressing various, often coexisting sim-to-real gaps. It displays attractive properties such as scaling with human effort. Videos and code are available at https://transic-robot.github.io/

Transferring Robot Policies from Simulation to Reality with Human-in-the-Loop Learning

Introduction

Transferring robot control policies from simulation to the real world can enable the development of versatile robots. However, the shift from simulation to reality (sim-to-real) often poses significant challenges due to various discrepancies. In this work, the authors propose an approach that incorporates human intervention to bridge these sim-to-real gaps. Instead of relying on domain-specific knowledge, the approach leverages human assistance to correct robot policies in real-time during their execution in the real world.

Key Ideas

The core idea behind this work is a human-in-the-loop framework where humans can observe and intervene during robot execution. When the robot encounters difficulties or errors, human operators provide corrections via teleoperation. These corrections are then used to train additional policies that can be combined with the original simulation-trained policies. This method aims to close the sim-to-real gap holistically, addressing several types of observed gaps through human interaction.

Method Overview

The proposed approach consists of several stages:

1. Simulation Training: Robots are initially trained in a simulated environment using reinforcement learning (RL). This stage allows for extensive data generation without the need for physical robots.
2. Human Intervention: Once the base policies are trained, they are deployed on real robots. Human operators monitor these executions and intervene when necessary, providing corrections via teleoperation.
3. Learning Residual Policies: The corrections made by human operators are collected to learn residual policies. These residual policies aim to correct the potential errors that occur due to the sim-to-real gap.
4. Policy Integration: Both the original simulation policies and the residual policies learned from human corrections are integrated to achieve high-quality performance in real-world tasks.

Strong Numerical Results

In their experiments, the authors demonstrate that their approach yields superior performance compared to traditional methods for sim-to-real transfer. Some highlights include:

• Stabilizing Tasks: Achieved 100% success rate in stabilizing a tabletop as compared to 55% with the best traditional method.
• Complex Manipulations: Managed an 85% success rate in screwing a light bulb, which significantly surpasses other baselines.
• Efficiency: Required significantly fewer real-robot trajectories to achieve enhanced performance.

Implications

Practical Implications

The practical implications of this method are substantial. By effectively utilizing human intervention, this approach can:

• Reduce the dependency on exact simulation models which are often complex and resource-intensive to create.
• Enable safer and more reliable deployment of robots in real-world settings, especially in intricate manipulation tasks such as furniture assembly.

Theoretical Implications

From a theoretical standpoint, this shows that human feedback mechanisms can address complex, multifaceted issues such as sim-to-real gaps in a comprehensive manner. This finding encourages further exploration into human-in-the-loop systems and their broader applicability in other robotic domains.

Future Directions

While this research shows promising results, several future directions can be considered:

• Scalability: Exploring the scalability of this approach to more complex or varied environments and tasks.
• Automation: Developing methods to automate parts of the human intervention process, potentially reducing the need for continuous human oversight.
• Embedding Human Knowledge: Expanding the framework to embed more inherent human-like decision-making processes directly into robot policies.

Conclusion

The human-in-the-loop approach proposed in this paper effectively bridges the sim-to-real gap in robot manipulation tasks. By integrating human corrections into simulation-trained policies, robots can perform complex tasks with higher success rates and safety. This work opens new possibilities for deploying versatile robots in real-world applications, enhancing their capabilities through synergistic human-robot collaboration.