OPT-Mimic: Imitation of Optimized Trajectories for Dynamic Quadruped Behaviors (2210.01247v3)

Abstract: Reinforcement Learning (RL) has seen many recent successes for quadruped robot control. The imitation of reference motions provides a simple and powerful prior for guiding solutions towards desired solutions without the need for meticulous reward design. While much work uses motion capture data or hand-crafted trajectories as the reference motion, relatively little work has explored the use of reference motions coming from model-based trajectory optimization. In this work, we investigate several design considerations that arise with such a framework, as demonstrated through four dynamic behaviours: trot, front hop, 180 backflip, and biped stepping. These are trained in simulation and transferred to a physical Solo 8 quadruped robot without further adaptation. In particular, we explore the space of feed-forward designs afforded by the trajectory optimizer to understand its impact on RL learning efficiency and sim-to-real transfer. These findings contribute to the long standing goal of producing robot controllers that combine the interpretability and precision of model-based optimization with the robustness that model-free RL-based controllers offer.

• The paper presents a novel framework combining offline trajectory optimization with imitation-based reinforcement learning for enhanced quadruped dynamic control.
• It employs a simplified single rigid body model to generate reference trajectories that reduce computation times compared to traditional RL-only approaches.
• The study demonstrates effective sim-to-real transfer for maneuvers like trots and backflips, while noting challenges in biped stepping due to robot design constraints.

Analyzing "OPT-Mimic: Imitation of Optimized Trajectories for Dynamic Quadruped Behaviors"

The paper "OPT-Mimic: Imitation of Optimized Trajectories for Dynamic Quadruped Behaviors" by Yuni Fuchioka, Zhaoming Xie, and Michiel van de Panne presents a comprehensive paper into the integration of trajectory optimization and reinforcement learning (RL) for enhancing dynamic behaviors in quadrupedal robots. This research investigates the effectiveness of using trajectory optimization as a reference for RL, addressing several considerations salient to this integration, focusing on efficiency and sim-to-real transfer for various dynamic behaviors.

Methodological Framework

The paper leverages trajectory optimization to derive reference motions from a simplified dynamics model, subsequently employing an imitation-based RL framework to develop a closed-loop controller that mimics these motions on a more complex, full-order robot model. This approach is evaluated on the Solo 8 quadruped robot, employing diverse dynamic behaviors—trot, front hop, 180 backflip, and biped stepping.

Trajectory Optimization and RL Integration

Trajectory Optimization: As a core component, trajectory optimization is employed to produce kinematically and dynamically feasible motions offline. The authors opt for a Single Rigid Body (SRB) model due to its balance between simplicity and flexibility, facilitating the exploration of complex maneuvers like backflips and dynamic hopping. The trajectory optimization framework effectively generates open-loop trajectories accommodating the dynamic constraints and contact conditions inherent in legged locomotion.

Reinforcement Learning: The RL component builds on Proximal Policy Optimization (PPO), chosen for robustness against hyperparameter sensitivity. RL training leverages the optimized trajectories as a reference, aiming to bridge the model simplicity of the SRB and the intricacies of a full-order dynamic model necessary for real-world application. The design includes specific considerations for feedforward configurations—considering target velocities and torques—and their implications on learning efficiency and transferability to physical robots.

Results and Observations

The authors report efficient generation of feasible trajectories via the SRB model, with significantly reduced solve times compared to conventional RL-only approaches. The results indicate successful sim-to-real transfers for the majority of dynamic tasks, particularly the trot, front-hop, and 180-backflip defaults. However, it is noted that the biped-step motion faced transfer challenges due to the robot's design constraints—specifically lacking abduction DOFs and the presence of compliance in the lower leg material, which diverges from the idealized simulation model.

The results demonstrate that while feedforward torques for velocity and torque improved the overall RL training efficacy, excessive reliance or misconfiguration could degrade transfer performance, particularly due to increased joint stiffness and resultant torque spikes during physical execution.

Implications and Future Work

Combining trajectory optimization with RL offers a nuanced approach to quadruped control, capitalizing on the speed and predictability of the former while harnessing the adaptive robustness characteristic of the latter. The implications extend beyond immediate control applications, suggesting broader applications in areas where quick adaptation to optimized trajectories can be beneficial, such as in handling unstructured environments or when deploying robots outside controlled laboratory settings.

Future inquiries could focus on enhancing adaptability and robustness, potentially incorporating flexible timing strategies or refining the RL training to mitigate discrepancies between real-world dynamics and simplified models further. Additionally, scaling this integrated approach to more complex robot architectures or multi-agent systems could yield insights into the generalization capabilities and scalability of combining trajectory optimization and imitation-based RL.

In conclusion, this paper illustrates the potential for integrating foundational model-based techniques with advanced RL methodologies to advance dynamic motion capabilities in quadrupedal robotics—a promising direction for both theoretical inquiry and practical application in the robotics domain.