Learning Multimodal Bipedal Locomotion and Implicit Transitions: A Versatile Policy Approach (2303.05711v2)

Abstract: In this paper, we propose a novel framework for synthesizing a single multimodal control policy capable of generating diverse behaviors (or modes) and emergent inherent transition maneuvers for bipedal locomotion. In our method, we first learn efficient latent encodings for each behavior by training an autoencoder from a dataset of rough reference motions. These latent encodings are used as commands to train a multimodal policy through an adaptive sampling of modes and transitions to ensure consistent performance across different behaviors. We validate the policy performance in simulation for various distinct locomotion modes such as walking, leaping, jumping on a block, standing idle, and all possible combinations of inter-mode transitions. Finally, we integrate a task-based planner to rapidly generate open-loop mode plans for the trained multimodal policy to solve high-level tasks like reaching a goal position on a challenging terrain. Complex parkour-like motions by smoothly combining the discrete locomotion modes were generated in 3 min. to traverse tracks with a gap of width 0.45 m, a plateau of height 0.2 m, and a block of height 0.4 m, which are all significant compared to the dimensions of our mini-biped platform.

• The paper presents a framework that learns a single RL policy for multimodal bipedal locomotion using latent encodings to command diverse movements.
• It employs an autoencoder to extract compact latent representations from reference motions, enabling adaptive sampling to overcome mode collapse and aliasing.
• The integrated task-based planner leverages implicit transitions to facilitate agile navigation over complex terrains, paving the way for real-world robotic applications.

Multimodal Bipedal Locomotion and Implicit Transitions: A Versatile Policy Framework

The paper at hand presents a framework for learning diverse locomotion behaviors in bipedal robots by synthesizing a single multimodal control policy. This method addresses the challenge of enabling bipedal robots to perform multiple locomotion modes and to manage smooth transitions between these modes without explicit transition demonstrations—a notable issue in reinforcement learning (RL) for robotic locomotion. The authors, Lokesh Krishna and Quan Nguyen, leverage an autoencoder to train effective latent encodings from reference motions, which are then used as commands to train the RL policy.

Key Components and Methodology

1. Latent Space Encoding: The authors utilize an autoencoder model to derive a low-dimensional latent space from reference motions. This latent space represents a compact encoding of the complete trajectory, allowing the RL policy to access temporal features of the desired motion without necessitating a direct correspondence with the robot's states. This method facilitates rapid realization of behaviors and simplifies the task of planning complex sequences of actions, as the latent commands encapsulate the essential characteristics of the locomotion mode.
2. Multimodal Policy with Implicit Transitions: The paper proposes training a single policy conditioned on these latent vectors using model-free RL, specifically PPO (Proximal Policy Optimization), to imitate the commanded locomotion modes. The policy is designed to adaptively sample modes and transitions during training, thus learning emergent transition maneuvers implicitly. The adaptive sampling of modes ensures that the policy remains robust against overfitting to specific modes or transitions, known as mode collapse and aliasing. This is achieved by skewing the sampling probabilities towards modes or transitions where the policy's performance is sub-optimal, dynamically encouraging learning where it is most needed.
3. Task-Based Mode Planning: A task-based planner is integrated to exploit the trained multimodal policy for solving high-level tasks like traversing a complex terrain. The planner generates mode plans based on simple open-loop time-dependent transitions using tabular Q-learning to identify optimal sequences that guide the robot through the task. Such an approach circumvents the computational burden and complexity of planning in real time, emphasizing the effectiveness of controlling the robot with a preconditioned policy that guarantees smooth inter-mode transitions.

Numerical Results and Evaluation

The authors showcase the framework's capacity to simulate complex bipedal behaviors like walking, leaping, and executing parkour-like maneuvers over non-trivial terrain features (gaps, blocks, plateaus). In quantitative terms, they present normalized mean returns across modes and transitions, underscoring an improvement with adaptive sampling over uniform sampling. Further analyses reveal distinct behavioral modes when encoded with the proposed latent vectors, unlike one-hot encodings that resulted in indistinguishable actions, thereby avoiding mode aliasing. The realized motions exhibit agility by successfully navigating challenging obstacles relative to the robot's dimensions, indicative of the multimodal policy's effectiveness.

Implications and Future Directions

This work contributes significantly to the field of RL-based robotic locomotion, offering a novel strategy for training versatile multimodal policies capable of adaptive behavior modulation and inherent transitions. It alleviates traditional challenges of catastrophic forgetting and the complexities tied with behavior-specific policies. Future developments could involve extending the framework with more sophisticated planners to tackle real-time loco-manipulation tasks and adapting to unforeseen terrains with dynamic environments. Incorporating more extensive demonstrations and exploring transferability to hardware remain essential for bridging the gap between simulation and real-world deployments. As RL matures, integrating holistic frameworks like the one presented herein could push the boundaries of autonomous robotics, paving the way for robots that exhibit fluid, adaptable movement across a wider range of tasks and scenarios.