RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation (2410.07864v1)

Abstract: Bimanual manipulation is essential in robotics, yet developing foundation models is extremely challenging due to the inherent complexity of coordinating two robot arms (leading to multi-modal action distributions) and the scarcity of training data. In this paper, we present the Robotics Diffusion Transformer (RDT), a pioneering diffusion foundation model for bimanual manipulation. RDT builds on diffusion models to effectively represent multi-modality, with innovative designs of a scalable Transformer to deal with the heterogeneity of multi-modal inputs and to capture the nonlinearity and high frequency of robotic data. To address data scarcity, we further introduce a Physically Interpretable Unified Action Space, which can unify the action representations of various robots while preserving the physical meanings of original actions, facilitating learning transferrable physical knowledge. With these designs, we managed to pre-train RDT on the largest collection of multi-robot datasets to date and scaled it up to 1.2B parameters, which is the largest diffusion-based foundation model for robotic manipulation. We finally fine-tuned RDT on a self-created multi-task bimanual dataset with over 6K+ episodes to refine its manipulation capabilities. Experiments on real robots demonstrate that RDT significantly outperforms existing methods. It exhibits zero-shot generalization to unseen objects and scenes, understands and follows language instructions, learns new skills with just 1~5 demonstrations, and effectively handles complex, dexterous tasks. We refer to https://rdt-robotics.github.io/rdt-robotics/ for the code and videos.

• The paper demonstrates a novel diffusion model scaled to 1.2 billion parameters, advancing coordination in dual-arm robotic systems.
• It employs a Physically Interpretable Unified Action Space to standardize multi-modal representations and enable transferable knowledge across diverse robots.
• Experimental results validate zero-shot generalization, precise language instruction following, and effective learning from minimal demonstrations.

Overview of "RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation"

The paper presents the Robotics Diffusion Transformer (RDT), a novel diffusion-based foundation model designed to enhance bimanual manipulation in robotic systems. Addressing inherent challenges of coordinating dual-arm robots and limited training data, RDT leverages diffusion models to represent multi-modal action distributions efficiently. This work stands out by scaling the model to 1.2 billion parameters, making it the largest diffusion foundation model for robotic tasks to date.

The authors introduce a Physically Interpretable Unified Action Space, which standardizes action representations across different robotic systems while maintaining their physical significance. This innovation aids in learning transferrable knowledge, allowing effective pre-training on extensive multi-robot datasets. The fine-tuning phase is bolstered by a self-created dataset with over 6,000 bimanual task episodes, leading to significant improvements in manipulation abilities.

Challenges and Approach

Bimanual manipulation requires mastering complex tasks involving intricate coordination between two robot arms. Previous methods either relied on task-specific primitives or focused on small-scale models, leading to limited generalization capabilities. RDT's architecture incorporates diffusion models, which excel in modeling complex distributions, thus addressing the challenge of multi-modality in action spaces.

The architecture employs Transformers capable of handling diverse modalities, including vision and language, crucial for understanding and executing complex tasks. Specific architectural enhancements, such as MLP decoding and improved normalization, cater to the unique dynamics and nonlinearities of robotic data, positioning RDT as an expressive and scalable solution for bimanual manipulation.

Data Utilization

The scarcity of dual-arm robot data is a well-documented obstacle. The researchers overcome this by adopting a pre-training and fine-tuning methodology. Pre-training leverages the vast multi-robot datasets, expanding the dataset size to three orders of magnitude. For fine-tuning on bimanual tasks, they collect a diverse and comprehensive dataset that comprises a wide range of manipulation tasks in various environments, thus enhancing RDT's applicability to practical scenarios.

Experimental Validation

Experiments conducted on real robotic systems demonstrate RDT's superior capabilities compared to existing methods. Notably, RDT achieves zero-shot generalization to unseen objects and environments, follows language instructions with precision, and learns new skills from as few as 1 to 5 demonstrations. These capabilities underscore RDT's robustness and flexibility in handling diverse and complex tasks.

Implications and Future Work

The implementation of RDT signifies a step forward in developing generalizable robotic systems that are not constrained by traditional limitations of data and model expressiveness. Practically, this work can facilitate more adaptive and capable robotic systems for real-world applications across varied domains. Theoretically, it opens avenues for further exploration into using foundation models in robotics, particularly in multi-modal and multi-task settings.

Future developments could focus on refining the model's efficiency and exploring its application in even more varied robotic configurations and environments, potentially integrating more nuanced sensory data. Additionally, enhancing the model's real-time performance could further elevate its use in immediate, hands-on applications.

Overall, the paper makes crucial strides towards comprehensive robotic manipulation models, providing a solid foundation for subsequent research in advanced autonomous systems.