Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning (2206.08686v2)

Abstract: Achieving human-level dexterity is an important open problem in robotics. However, tasks of dexterous hand manipulation, even at the baby level, are challenging to solve through reinforcement learning (RL). The difficulty lies in the high degrees of freedom and the required cooperation among heterogeneous agents (e.g., joints of fingers). In this study, we propose the Bimanual Dexterous Hands Benchmark (Bi-DexHands), a simulator that involves two dexterous hands with tens of bimanual manipulation tasks and thousands of target objects. Specifically, tasks in Bi-DexHands are designed to match different levels of human motor skills according to cognitive science literature. We built Bi-DexHands in the Issac Gym; this enables highly efficient RL training, reaching 30,000+ FPS by only one single NVIDIA RTX 3090. We provide a comprehensive benchmark for popular RL algorithms under different settings; this includes Single-agent/Multi-agent RL, Offline RL, Multi-task RL, and Meta RL. Our results show that the PPO type of on-policy algorithms can master simple manipulation tasks that are equivalent up to 48-month human babies (e.g., catching a flying object, opening a bottle), while multi-agent RL can further help to master manipulations that require skilled bimanual cooperation (e.g., lifting a pot, stacking blocks). Despite the success on each single task, when it comes to acquiring multiple manipulation skills, existing RL algorithms fail to work in most of the multi-task and the few-shot learning settings, which calls for more substantial development from the RL community. Our project is open sourced at https://github.com/PKU-MARL/DexterousHands.

• The paper introduces Bi-DexHands, a simulator that achieves high efficiency and sample throughput via Isaac Gym-enabled reinforcement learning.
• It demonstrates that on-policy RL algorithms master simpler tasks while multi-agent methods excel in complex bimanual coordination.
• Key results highlight challenges in multi-task learning and task generalization, setting a foundation for future advancements in robotic dexterity.

Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning

The paper "Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning" addresses the complex challenge of achieving human-level dexterity in robotic manipulation. The focus is on bimanual dexterous tasks, demanding intricate coordination between two robotic hands. This paper introduces the Bimanual Dexterous Hands Benchmark (Bi-DexHands), a simulator designed in Isaac Gym, tailored for reinforcement learning (RL) algorithms.

Bimanual Dexterous Hands Benchmark

Bi-DexHands is differentiated by its simulation of two robotic hands handling a diverse array of tasks and objects, aligning with human motor skill development. Constructed in Isaac Gym, the simulator achieves over 30,000 frames per second on a single NVIDIA RTX 3090 GPU, underscoring its efficiency. The benchmark encompasses various RL paradigms such as Single-agent, Multi-agent (MARL), Offline RL, Multi-task RL, and Meta RL, providing a comprehensive platform for algorithm evaluation.

Key Contributions

1. High Efficiency: Utilizing Isaac Gym allows Bi-DexHands to simulate thousands of environments concurrently. This capability enhances the sample efficiency crucial for RL training in complex tasks.
2. Comprehensive Benchmarking: The benchmark includes evaluations across popular RL algorithms, examining their performance under different settings. The experiments indicate that while on-policy algorithms like PPO can master simple tasks linked to younger developmental stages, multi-agent algorithms are more effective for tasks requiring intricate bimanual cooperation.
3. Heterogeneous Cooperation: The agents within Bi-DexHands (representing different parts of the hand) exhibit heterogeneity, offering a distinct challenge compared to environments where agents share parameters.
4. Task Generalization and Cognition: A variety of tasks were introduced to test algorithms on their ability to generalize dexterous skills. The tasks are inspired by cognitive science literature related to human motor development, facilitating comparative studies in robotic skill learning.

Results and Observations

The paper finds that single-task mastery is feasible with current RL algorithms, but multi-task and few-shot learning remain hurdles. Notably, numerical results suggest existing algorithms are limited in acquiring multiple manipulation skills simultaneously. The insight drawn is that multi-task RL algorithms face significant challenges in generalizing across the full spectrum of tasks provided by Bi-DexHands.

Implications and Future Directions

The research underscores the gap between current robotic capabilities and human-level dexterity, particularly in the context of task generalization and meta-learning. Practically, the development of more sophisticated RL algorithms capable of managing multiple tasks efficiently is essential. Theoretically, the benchmark serves as a foundational step for studies attempting to bridge the gap between human and robotic dexterity.

Future work suggested involves enhancing simulated environments to include deformable object manipulation, extending the paper to incorporate visual observations for better sim-to-real transfer, and focusing on algorithms that improve task generalization.

In summary, "Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning" presents substantial advancements and challenges in robotic dexterous manipulation aligned with cognitive developmental stages, offering a robust platform for progressing RL-based robotic control systems.