Robust Recovery Controller for a Quadrupedal Robot using Deep Reinforcement Learning (1901.07517v1)

Abstract: The ability to recover from a fall is an essential feature for a legged robot to navigate in challenging environments robustly. Until today, there has been very little progress on this topic. Current solutions mostly build upon (heuristically) predefined trajectories, resulting in unnatural behaviors and requiring considerable effort in engineering system-specific components. In this paper, we present an approach based on model-free Deep Reinforcement Learning (RL) to control recovery maneuvers of quadrupedal robots using a hierarchical behavior-based controller. The controller consists of four neural network policies including three behaviors and one behavior selector to coordinate them. Each of them is trained individually in simulation and deployed directly on a real system. We experimentally validate our approach on the quadrupedal robot ANYmal, which is a dog-sized quadrupedal system with 12 degrees of freedom. With our method, ANYmal manifests dynamic and reactive recovery behaviors to recover from an arbitrary fall configuration within less than 5 seconds. We tested the recovery maneuver more than 100 times, and the success rate was higher than 97 %.

• The paper presents a novel hierarchical deep reinforcement learning framework that enables quadrupedal robots to recover from falls using behavior-specific policies and a behavior selector.
• The method decomposes recovery into self-righting, standing, and locomotion tasks, achieving a success rate exceeding 97% in trials on the ANYmal robot.
• The approach leverages TRPO with GAE and high-fidelity simulation-to-reality transfer, offering a flexible solution for autonomous recovery in dynamic environments.

Overview of Robust Recovery Controller for Quadrupedal Robot Using Deep Reinforcement Learning

The paper, "Robust Recovery Controller for a Quadrupedal Robot using Deep Reinforcement Learning," addresses the challenge of enabling quadrupedal robots to autonomously recover from falls. The capacity to recover from falls is an essential aspect of navigating complex environments, and the authors have introduced a novel approach through a model-free Deep Reinforcement Learning (RL)-based hierarchical controller. This controller enables quadrupedal robots to perform recovery maneuvers with high success rates and can be directly deployed from simulation to real-world environments.

Methodology and Experimental Validation

The core contribution of the paper lies in the development of a hierarchical behavior-based controller, consisting of four neural network policies: three behavior policies and one behavior selector. The behavior policies—self-righting, standing up, and locomotion—are trained individually in simulation to achieve distinct tasks. The behavior selector coordinates these behaviors, allowing for adaptive transitions based on the current situation rather than adhering to rigid predefined sequences.

The experimental validation leverages the quadrupedal robot ANYmal, which possesses 12 degrees of freedom, to validate the efficacy of the proposed controller. ANYmal successfully demonstrated recovery from various fall configurations within five seconds in 100 trials with a success rate exceeding 97%. This result underscores the robustness of the RL-based controller in handling different corner cases that former solutions struggled with, such as entrapment of the robot's legs under the base.

Technical Details and Implementation

In terms of implementation, the training process utilizes Trust Region Policy Optimization (TRPO) and Generalized Advantage Estimation (GAE) to develop control policies efficiently and effectively within a simulated environment. The authors emphasize the importance of using high-fidelity simulations paired with a simulation-to-reality transfer as crucial components in overcoming the reality gap between simulated and real conditions. Neural networks are employed to estimate dynamic states, such as the base height during degenerate contact conditions, to ensure accurate state observation and operation consistency.

The paper further discusses the benefits of decomposing the control task into multiple behaviors, simplifying the overall implementation and refining the cost function design, which results in more natural and effective autonomous recovery maneuvers. By avoiding complex state modeling and contact sequence predefinitions required by optimization-based methods, the RL-based approach offers enhanced flexibility and adaptability.

Implications and Future Work

The paper hints at broader implications for legged robotics, particularly in harsh environments where failure recovery is crucial. The proposed technique holds promise for expanding the autonomy and robustness of quadrupedal robots. Despite the promising outcomes, the current implementation is limited to flat terrain, which may not adequately represent real-world challenges such as inclined or uneven terrain. Addressing these limitations would necessitate enhanced training environments featuring randomized terrain properties in future work.

The implications for AI and robotics extend to creating systems that can operate more autonomously in dynamically changing environments, which can benefit applications ranging from logistics and search-and-rescue missions to planetary exploration.

Conclusion

Overall, the paper successfully proposes a novel RL-based controller architecture capable of robust fall recovery for quadrupedal robots. The findings and methods outlined pave the way for future iterations on more complex terrains, contributing to the continuous advancement of autonomous legged robot resilience and operational adaptability. The utilization of model-free Deep RL offers notable advantages in bypassing the constraints of traditional optimization-based methods, presenting a significant step forward in this domain.