Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks (2110.03655v3)

Abstract: Realistic manipulation tasks require a robot to interact with an environment with a prolonged sequence of motor actions. While deep reinforcement learning methods have recently emerged as a promising paradigm for automating manipulation behaviors, they usually fall short in long-horizon tasks due to the exploration burden. This work introduces Manipulation Primitive-augmented reinforcement Learning (MAPLE), a learning framework that augments standard reinforcement learning algorithms with a pre-defined library of behavior primitives. These behavior primitives are robust functional modules specialized in achieving manipulation goals, such as grasping and pushing. To use these heterogeneous primitives, we develop a hierarchical policy that involves the primitives and instantiates their executions with input parameters. We demonstrate that MAPLE outperforms baseline approaches by a significant margin on a suite of simulated manipulation tasks. We also quantify the compositional structure of the learned behaviors and highlight our method's ability to transfer policies to new task variants and to physical hardware. Videos and code are available at https://ut-austin-rpl.github.io/maple

Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks

The paper introduces MAPLE—a reinforcement learning (RL) framework that aims to enhance the efficacy of robotic manipulation tasks by integrating predefined behavior primitives into the learning process. The framework is designed to address the challenges faced by deep reinforcement learning (DRL) methods in handling long-horizon tasks, primarily due to the exploration burden inherent in DRL techniques.

Overview of the Methodology

At the core of MAPLE lies the use of behavior primitives: these are high-level, predefined modules that perform specific manipulation actions such as grasping, reaching, and pushing. By endowing the RL agent with a library of such primitives, MAPLE enables the composition of complex behaviors from simpler, robust modules, contrasting with standard DRL approaches that rely solely on low-level atomic actions. The model is implemented within the Parameterized Action MDP (PAMDP) framework, where actions are executed through a combination of primitive selection and parameter specification.

MAPLE employs a hierarchical policy structure comprising a high-level task policy to choose the primitive type and a low-level parameter policy to determine the execution parameters of the chosen primitive. This hierarchical approach allows the system to efficiently manage the heterogeneity of primitive parameters and temporal resolutions, suiting the diverse nature of manipulation tasks.

Significant Results

Empirically, MAPLE demonstrated substantial improvements over baseline approaches on multiple manipulation tasks within the robosuite simulation framework. Specifically, it showed robust task performance and a $70\%$ increase in task success rate relative to traditional DRL methods that rely purely on atomic actions. This clearly indicates the effectiveness of incorporating behavior primitives into the learning paradigm.

The framework was evaluated on eight manipulation tasks of varying complexities, which showcased its adaptability and the value of qualitative compositional structures within the learned behaviors. Furthermore, MAPLE's framework facilitated the transfer of learned policies to physical hardware, highlighting its practical utility.

Implications and Future Directions

The use of behavior primitives within an RL framework has far-reaching implications. It not only improves the exploration efficiency of RL algorithms but also retains the algorithm's flexibility when additional low-level actions are required. The robustness and reusability of behavior primitives also allow for scalability across different manipulation tasks without the necessity for intricate manual engineering.

The research advances the potential for future investigations into dynamic primitive learning, where the RL agent could autonomously discover and incorporate new primitives based on task requirements. Another exciting development could involve integrating data-driven models to refine affordances—thereby improving exploratory aspects of policy learning.

Overall, MAPLE presents a significant advancement in robotic manipulation tasks via reinforcement learning, highlighting the promising intersections between predefined functional modules and data-driven decision-making strategies. As the field progresses, there's substantial potential for optimizing and expanding this framework to tackle increasingly complex environments and tasks. The research lays a foundation for future developments in efficient, adaptable, and interpretable RL systems for robotics.