Robust Imitation of Diverse Behaviors (1707.02747v2)

Abstract: Deep generative models have recently shown great promise in imitation learning for motor control. Given enough data, even supervised approaches can do one-shot imitation learning; however, they are vulnerable to cascading failures when the agent trajectory diverges from the demonstrations. Compared to purely supervised methods, Generative Adversarial Imitation Learning (GAIL) can learn more robust controllers from fewer demonstrations, but is inherently mode-seeking and more difficult to train. In this paper, we show how to combine the favourable aspects of these two approaches. The base of our model is a new type of variational autoencoder on demonstration trajectories that learns semantic policy embeddings. We show that these embeddings can be learned on a 9 DoF Jaco robot arm in reaching tasks, and then smoothly interpolated with a resulting smooth interpolation of reaching behavior. Leveraging these policy representations, we develop a new version of GAIL that (1) is much more robust than the purely-supervised controller, especially with few demonstrations, and (2) avoids mode collapse, capturing many diverse behaviors when GAIL on its own does not. We demonstrate our approach on learning diverse gaits from demonstration on a 2D biped and a 62 DoF 3D humanoid in the MuJoCo physics environment.

Robust Imitation of Diverse Behaviors: Insights and Implications

The paper "Robust Imitation of Diverse Behaviors" introduces a novel approach to address the challenges of imitation learning in diverse behavior imitation for embodied agents, utilizing a combination of deep generative models. The research context emphasizes the limitations of purely supervised strategies in achieving robust imitation learning, particularly when the agent's trajectory diverges from seen demonstrations. Generative Adversarial Imitation Learning (GAIL), while adept in certain areas, suffers from mode collapse and significant training complexities. This paper successfully integrates Variational Autoencoders (VAEs) with GAIL to form a robust methodology that overcomes these challenges.

Methodological Innovations

The core innovation in this paper is the development of a novel procedure that leverages both VAEs and GAIL to harness their respective strengths. The approach begins with constructing a VAE on demonstration trajectories, which learns semantic policy embeddings. These embeddings allow for smooth interpolation and meaningful representation of behavior, elegantly addressing the need for capturing diversity without sacrificing robustness. The paper describes a two-fold decoder architecture within the VAE: an MLP policy for generating actions and a dynamics model that incorporates WaveNet-like autoregressive properties to capture state transitions.

On the adversarial front, the authors devise a refined version of GAIL that conditions the discriminator on VAE-derived latent embeddings. This conditioning mitigates mode collapse, a notorious issue in GAN-based learning, by ensuring diverse mode coverage. The policy optimization via Trust Region Policy Optimization (TRPO) further enhances the robustness of the trained models.

Experimental Results

The experiments, conducted using the MuJoCo physics engine, showcase the method's efficacy in various tasks, including robotic arm maneuvering and humanoid locomotion with a large number of degrees of freedom. Key findings include:

• Smooth interpolation in the VAE's latent space translates to interpretable transitions in task space, evident in the Jaco arm experiments.
• The method facilitates significant improvements in imitating diverse bipedal walking patterns, outperforming baseline approaches such as standalone BC and GAIL.
• For high-dimensional humanoid control, the method offers stable and semantically organized trajectory execution, with the adversarial component proving essential for robustness.

Implications and Future Directions

The synthesis of VAEs and GAIL in this work highlights the critical balance between diversity and robustness in imitation learning. This balance is crucial for applications requiring embodied agents to perform under varying conditions and execute a wide range of behaviors effectively. The developed method opens new avenues for advancing the state-of-the-art in both robotics and animated avatars, particularly in fields demanding flexible and adaptive motor control solutions.

Future research directions could explore the expansion of this methodology to incorporate more nuanced environmental interactions or to scale across broader task domains. Additionally, further refinement of the conditional aspect of the adversarial model may yield even richer representations of the latent behavior space, paving the way for more sophisticated hierarchical control models.

In conclusion, this paper contributes significant advancements to imitation learning, presenting a robust framework that effectively integrates deep generative approaches to address the long-standing challenge of diversity in behavior imitation.