MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies (1905.09808v1)

Abstract: Humans are able to perform a myriad of sophisticated tasks by drawing upon skills acquired through prior experience. For autonomous agents to have this capability, they must be able to extract reusable skills from past experience that can be recombined in new ways for subsequent tasks. Furthermore, when controlling complex high-dimensional morphologies, such as humanoid bodies, tasks often require coordination of multiple skills simultaneously. Learning discrete primitives for every combination of skills quickly becomes prohibitive. Composable primitives that can be recombined to create a large variety of behaviors can be more suitable for modeling this combinatorial explosion. In this work, we propose multiplicative compositional policies (MCP), a method for learning reusable motor skills that can be composed to produce a range of complex behaviors. Our method factorizes an agent's skills into a collection of primitives, where multiple primitives can be activated simultaneously via multiplicative composition. This flexibility allows the primitives to be transferred and recombined to elicit new behaviors as necessary for novel tasks. We demonstrate that MCP is able to extract composable skills for highly complex simulated characters from pre-training tasks, such as motion imitation, and then reuse these skills to solve challenging continuous control tasks, such as dribbling a soccer ball to a goal, and picking up an object and transporting it to a target location.

• The paper presents MCP as a novel approach that enhances composability by multiplicatively combining concurrent primitives for complex control tasks.
• It employs a multi-task RL pre-training stage to learn diverse motion primitives that can be flexibly transferred to high-dimensional action tasks.
• Experimental results on bipeds, humanoids, and a T-Rex demonstrate that MCP outperforms traditional methods in dynamic and intricate motor control scenarios.

Introduction to Multiplicative Compositional Policies

The paper "MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies" introduces a novel framework for enhancing the composability and flexibility of hierarchical control systems in reinforcement learning (RL). The focus of this paper is on the development and application of Multiplicative Compositional Policies (MCP) to learn reusable and adaptable motor skills for complex, high-dimensional tasks typically encountered by humanoid agents.

In addressing the limitations of traditional additive methods for skill composition, the authors propose a multiplicative model that combines primitives simultaneously, thereby enabling a broader and more flexible skill set. This approach circumvents the typical bottleneck of hierarchical policies restricted to temporal sequencing by promoting spatial composition instead.

Formulation and Implementation

The articulation of MCPs is underpinned by a multi-task RL framework that facilitates skill transfer through a comprehensive pre-training stage. Here, the system learns a repository of primitives by mimicking a variety of motion patterns, which are then applied to more complex transfer tasks. Notably, MCPs differ from standard methods by allowing multiple primitives to influence the policy at any given time through a probabilistic model that creates a multiplicative composition of primitive action distributions.

This formulation leads to a composite policy that is better suited for addressing the combinatorial explosion of skill sets needed in high-degree-of-freedom systems, such as those mimicking complex biological morphologies. The crux of this method lies in its ability to integrate several primitives, turning the space of potential actions into a rich blend of behaviors that increase the versatility and efficiency of the policy.

The experimental portion of the paper rigorously examines the MCP framework using simulated characters including bipeds, humanoids, and a T-Rex, each tasked with intricate locomotion and manipulation challenges. The results demonstrably highlight the advantages of MCPs over traditional methods, particularly under conditions of increased complexity and high-dimensional action spaces.

Comparative Analysis and Performance Insights

A key strength of this research lies in its comprehensive comparative analysis against well-established methodologies, including mixture-of-experts models, traditional hierarchical strategies, and continuous latent space approaches. MCPs showcase superior performance metrics, particularly in scenarios involving complex motor controls and long-horizon tasks, affirming their efficacy in producing coherent and specialized exploration strategies.

In notable cases like the humanoid dribbling a soccer ball and the T-Rex performing a similar task, MCPs significantly outperform other methods, demonstrating their robustness and adaptability in dynamic scenarios. The numerical results substantiate MCPs’ ability to facilitate learning in tasks where standard transfer methods fail or require extensive readjustment.

Theoretical and Practical Implications

The theoretical implications of MCPs underscore a paradigm shift in skill composition viewable not merely through temporal chaining but also as a spatial composition of concurrent skills. Practically, MCPs pave the way for more nuanced and sophisticated control strategies in artificial intelligence applications requiring real-time adaptation and contextual skill reuse. This holds considerable potential for advancing robotics, animation, and other domains necessitating complex motor function emulation.

One speculative direction for future research could involve expanding MCPs with temporal abstractions that, when combined with spatial composition, could result in even richer behavioral repertoires. Furthermore, enhancing the efficiency of pre-training with automated motion repertoire generation could significantly boost the applicability of MCP frameworks.

In summary, the development of MCP provides a robust mechanism that enhances skill flexibility and reusability in RL, demonstrably advancing the field towards more sophisticated and human-like autonomous agents capable of navigating a diversity of complex control environments.