Preparing for the Unknown: Learning a Universal Policy with Online System Identification (1702.02453v3)

Abstract: We present a new method of learning control policies that successfully operate under unknown dynamic models. We create such policies by leveraging a large number of training examples that are generated using a physical simulator. Our system is made of two components: a Universal Policy (UP) and a function for Online System Identification (OSI). We describe our control policy as universal because it is trained over a wide array of dynamic models. These variations in the dynamic model may include differences in mass and inertia of the robots' components, variable friction coefficients, or unknown mass of an object to be manipulated. By training the Universal Policy with this variation, the control policy is prepared for a wider array of possible conditions when executed in an unknown environment. The second part of our system uses the recent state and action history of the system to predict the dynamics model parameters mu. The value of mu from the Online System Identification is then provided as input to the control policy (along with the system state). Together, UP-OSI is a robust control policy that can be used across a wide range of dynamic models, and that is also responsive to sudden changes in the environment. We have evaluated the performance of this system on a variety of tasks, including the problem of cart-pole swing-up, the double inverted pendulum, locomotion of a hopper, and block-throwing of a manipulator. UP-OSI is effective at these tasks across a wide range of dynamic models. Moreover, when tested with dynamic models outside of the training range, UP-OSI outperforms the Universal Policy alone, even when UP is given the actual value of the model dynamics. In addition to the benefits of creating more robust controllers, UP-OSI also holds out promise of narrowing the Reality Gap between simulated and real physical systems.

• The paper proposes integrating a Universal Policy with Online System Identification for real-time adaptation in dynamic robotic environments.
• It leverages extensive simulation to explore a wide range of dynamic models, bridging the simulation-to-reality gap in control policy deployment.
• The evaluation shows UP-OSI outperforming baseline methods in tasks like cart-pole swing-up and locomotion under varied conditions.

Overview of Learning a Universal Policy with Online System Identification

The paper "Preparing for the Unknown: Learning a Universal Policy with Online System Identification" presents an innovative approach to robust control policies by integrating a Universal Policy (UP) with Online System Identification (OSI). The motivation of this paper is to address the Reality Gap that often arises when transitioning control policies learned in simulation to real-world robotic environments. The authors propose an architecture that incorporates an extensive exploration of the dynamic model space through simulation, aiming to cover the range of possible real-world conditions.

Methodology

The proposed system consists of two main components: a Universal Policy (UP) and an Online System Identification (OSI) module. The UP is designed to be adaptable across a variety of dynamic models, by inputting both the system state and the model parameters directly into the control policy. This approach deviates from traditional methods that typically rely on a hypothesized fixed model or require extensive real-world trials to fit and optimize control policies. By leveraging a wide parameter space in training, UP prepares the policy for deployment under diverse conditions without prior detail of the dynamic system.

Complementarily, the OSI module aims to dynamically identify these model parameters during operation. It uses recent state-action histories as input to predict model parameters, thereby allowing the UP to adapt in real-time to the varying dynamics. The training for both components relies entirely on simulation, bypassing the necessity for real-world experiments until deployment.

Evaluation and Results

The evaluation of the UP-OSI framework spanned several classical dynamic control tasks such as cart-pole swing-up and inverted pendulum stabilization, as well as more advanced tasks involving locomotion and manipulation. Notably, UP-OSI displayed robust performance against baseline policies, demonstrating its capability to handle tasks both within and outside the range of trained model dynamics. Crucially, the system was able to outperform policies equipped with true model parameter information (UP-true) in some instances when tested on dynamics outside the training distribution.

The results illustrate that UP-OSI not only closes the gap between simulated learning and real-world dynamics but also offers a substantial improvement over previous attempts to port simulation-trained policies directly to physical systems. Additionally, the capability of UP-OSI to adapt to environmental changes, such as varying friction conditions in locomotion tasks, underscores its potential for real-world applicability.

Implications and Future Directions

This research provides a substantial contribution to bridging simulation-based learning and practical applications in robotics. By minimizing the initial demand for expensive and time-consuming real-world trials, UP-OSI has broad implications for accelerating the deployment of robotic systems in complex, dynamic environments. It opens avenues for further extending the expressiveness and adaptability of control policies through more sophisticated parameterizations and potentially incorporating stochastic elements to address higher-order uncertainties.

For future developments, testing the UP-OSI framework in real robots will be a critical step. The transfer from simulation to reality, while accounting for the unmodeled aspects of real-world dynamics, will test the practical boundaries of the presented approach. Moreover, exploring the impact of different neural network architectures or hyperparameters on the learning efficiency and generalization capacity of UP-OSI represents a promising area for optimization.

In summation, the integration of Universal Policies with Online System Identification offers a compelling methodological advance for robotic control policy learning, enabling robust and adaptable performance across varied and unforeseen dynamic environments.