Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model (1610.03518v1)

Abstract: Developing control policies in simulation is often more practical and safer than directly running experiments in the real world. This applies to policies obtained from planning and optimization, and even more so to policies obtained from reinforcement learning, which is often very data demanding. However, a policy that succeeds in simulation often doesn't work when deployed on a real robot. Nevertheless, often the overall gist of what the policy does in simulation remains valid in the real world. In this paper we investigate such settings, where the sequence of states traversed in simulation remains reasonable for the real world, even if the details of the controls are not, as could be the case when the key differences lie in detailed friction, contact, mass and geometry properties. During execution, at each time step our approach computes what the simulation-based control policy would do, but then, rather than executing these controls on the real robot, our approach computes what the simulation expects the resulting next state(s) will be, and then relies on a learned deep inverse dynamics model to decide which real-world action is most suitable to achieve those next states. Deep models are only as good as their training data, and we also propose an approach for data collection to (incrementally) learn the deep inverse dynamics model. Our experiments shows our approach compares favorably with various baselines that have been developed for dealing with simulation to real world model discrepancy, including output error control and Gaussian dynamics adaptation.

Overview of Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model

This paper addresses the significant challenge in robotics and autonomous systems of transferring learned control policies from simulation environments to real-world applications. The proposed approach leverages the concept of learning a deep inverse dynamics model to bridge the discrepancy between simulated and real-world dynamics. This technique allows for the effective deployment of policies optimized in simulations onto physical robots without the necessity for inordinately detailed and computationally expensive simulations that mirror every aspect of real-world physics.

Key Contributions

The core contribution of this work is the development and implementation of a framework that utilizes deep inverse dynamics to adapt actions derived from a simulation-based control policy when applied in real-world settings. The approach resolves the barrier wherein simulation-based policies fail due to differences in friction, mass, and other dynamic properties between simulation and reality. The primary components of the approach are as follows:

1. Inverse Dynamics Model: The inverse dynamics model is essential for predicting the required real-world actions to achieve desired outcomes based on simulations. Learning this model allows the adaptation mechanism to map the simulated control outputs to actionable real-world commands.
2. Data Collection and Model Training: The paper presents a methodology for incrementally collecting data to train the inverse dynamics model effectively. This involves initial training with a poor model and iterative improvements based on the collected experiences which closely emulate test scenarios.
3. Evaluation Through Experiments: Two families of experiments are conducted: transferring policies between two simulated environments (Sim1 to Sim2) and transferring from a simulated to a real-world environment utilizing a Fetch robot. The experiments reveal a significant advantage over existing techniques such as output error control and Gaussian dynamics adaptation, particularly in environments with complex dynamics like contact and collision.

Implications and Future Directions

The results demonstrate the potential of using learned inverse dynamics models for effective cross-environment policy transfer, which has profound implications for robotics where sim-to-real challenges are prevalent. By addressing the discrepancies between simulated and actual environments, this work advances the field toward more robust, adaptable robotic systems capable of managing unexpected real-world conditions.

Theoretically, this approach supports the concept of generalization in control policies beyond predefined settings or narrowly tailored simulators. Practically, it enables a more efficient use of computational resources by reducing the reliance on precise and potentially expensive simulation environments.

Looking forward, future research could focus on several promising avenues:

• State and Observation Adaptation: While this paper addresses action adaptation, future work could explore methods for adapting states and observations between domains, particularly for sensors like cameras or LIDAR which are challenging to simulate accurately.
• Scaling and Real-World Applications: Further exploration could assess scalability to more complex tasks or domains such as autonomous driving, where only partial action observations might be available.

By bridging simulation and reality through learned inverse dynamics models, this research contributes a significant step toward the integration of simulated learning methodologies in practical, real-world automation and robotic applications.