Learning Symmetric and Low-energy Locomotion (1801.08093v3)

Abstract: Learning locomotion skills is a challenging problem. To generate realistic and smooth locomotion, existing methods use motion capture, finite state machines or morphology-specific knowledge to guide the motion generation algorithms. Deep reinforcement learning (DRL) is a promising approach for the automatic creation of locomotion control. Indeed, a standard benchmark for DRL is to automatically create a running controller for a biped character from a simple reward function. Although several different DRL algorithms can successfully create a running controller, the resulting motions usually look nothing like a real runner. This paper takes a minimalist learning approach to the locomotion problem, without the use of motion examples, finite state machines, or morphology-specific knowledge. We introduce two modifications to the DRL approach that, when used together, produce locomotion behaviors that are symmetric, low-energy, and much closer to that of a real person. First, we introduce a new term to the loss function (not the reward function) that encourages symmetric actions. Second, we introduce a new curriculum learning method that provides modulated physical assistance to help the character with left/right balance and forward movement. The algorithm automatically computes appropriate assistance to the character and gradually relaxes this assistance, so that eventually the character learns to move entirely without help. Because our method does not make use of motion capture data, it can be applied to a variety of character morphologies. We demonstrate locomotion controllers for the lower half of a biped, a full humanoid, a quadruped, and a hexapod. Our results show that learned policies are able to produce symmetric, low-energy gaits. In addition, speed-appropriate gait patterns emerge without any guidance from motion examples or contact planning.

• The paper introduces mirror symmetry loss in the DRL framework, ensuring learned gait patterns mimic natural bilateral symmetry.
• It employs curriculum learning with modulated physical assistance to gradually reduce energy consumption while improving locomotion control.
• Experimental results demonstrate significant improvements over baseline DRL methods across diverse morphologies.

Overview of "Learning Symmetric and Low-Energy Locomotion"

The paper "Learning Symmetric and Low-Energy Locomotion," authored by Wenhao Yu, Greg Turk, and C. Karen Liu, presents a novel approach to developing locomotion control policies through deep reinforcement learning (DRL) by emphasizing both the symmetry and energy efficiency of generated movements. Contrary to many conventional techniques, which heavily rely on pre-recorded motion capture data, finite state machines, or morphology-specific adjustments, the authors propose a minimalist approach focusing on DRL to address specific challenges inherent in locomotion tasks.

Core Contributions

The paper lays out two important modifications to the DRL framework that significantly enhance the quality and realism of synthesized locomotion:

1. Mirror Symmetry Loss: This component is introduced not within the reward function but directly in the loss function of the neural network policy being trained. It penalizes asymmetrical actions, thereby ensuring that learned gait patterns are symmetric, akin to those found in real-world bipeds.
2. Curriculum Learning with Modulated Physical Assistance: A curriculum learning strategy is employed, which provides temporary physical assistance during the early phases of training. The assistance is gradually reduced as the character becomes adept, ensuring that energy efficiency is prioritized without hindering the convergence rate. This method facilitates handling diverse morphologies, enhancing generalization across different character structures such as bipeds, quadrupeds, and hexapods.

Strong Numerical Results

Experimental evaluations demonstrated that the proposed learning methodology produces symmetric and low-energy locomotion patterns across varied settings. The approach successfully yielded speed-appropriate gaits without motion examples, resulting in outcomes that mimic natural locomotor activities observed in biological entities. Importantly, the results indicated substantial improvements over baseline DRL methods, whether focusing on learning efficiency or the quality of the formed control policies.

Implications and Future Directions

In terms of implications, this research bolsters the potential of DRL in automating the generation of naturalistic character animations without the need for extensive motion data, unlocking opportunities in fields such as computer graphics and robotics. It suggests that principles adhered to in biomechanics—specifically regarding energy efficiency and symmetry—can be effectively integrated into learning algorithms for improved motor control.

Theoretically, this work compels further exploration of DRL techniques that transcend standard L2-norm loss metrics to embrace domain-specific principles that shape the expected outcome. Practically, the presented methods can significantly reduce the design effort in creating control algorithms for new robotic morphologies in simulation environments, facilitating more cohesive and adaptive control systems.

Future developments might explore enhancing the proposed framework by directly integrating biomechanical constraints into the DRL training process or extending the techniques to other complex environments, including variable terrain and tasks beyond straightforward locomotion. Furthermore, understanding the interplay between environmental adaptations and symmetry constraints could yield even more robust locomotion strategies applicable to real-world robotics applications.

In sum, this research serves as a compelling illustration of the capabilities and advancements in DRL for robotic and animatronic locomotion, setting the stage for subsequent inquiries within this dynamic field.