Open X-Embodiment: Robotic Learning Datasets and RT-X Models (2310.08864v8)

Abstract: Large, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, this has led to a consolidation of pretrained models, with general pretrained backbones serving as a starting point for many applications. Can such a consolidation happen in robotics? Conventionally, robotic learning methods train a separate model for every application, every robot, and even every environment. Can we instead train generalist X-robot policy that can be adapted efficiently to new robots, tasks, and environments? In this paper, we provide datasets in standardized data formats and models to make it possible to explore this possibility in the context of robotic manipulation, alongside experimental results that provide an example of effective X-robot policies. We assemble a dataset from 22 different robots collected through a collaboration between 21 institutions, demonstrating 527 skills (160266 tasks). We show that a high-capacity model trained on this data, which we call RT-X, exhibits positive transfer and improves the capabilities of multiple robots by leveraging experience from other platforms. More details can be found on the project website https://robotics-transformer-x.github.io.

Open X-Embodiment: Robotic Learning Datasets and RT-X Models

The paper presents an innovative exploration into the field of robotic learning, focusing on the assemblage and utilization of large-scale, diverse datasets to train what are referred to as "generalist" robotic policies. This initiative, termed Open X-Embodiment, serves as a promising step towards unifying robotic learning across various platforms through data sharing and collaborative experiments.

Overview

The paper outlines the creation of a comprehensive dataset derived from 22 different robots, gathered collaboratively across 21 institutions. This dataset encompasses 527 skills and over 160,000 tasks, providing a robust foundation for training and evaluating generalized robotic policies. The primary inquiry guiding this research is whether robotics can benefit from large-scale, general-purpose pretrained models, akin to recent advancements in NLP and computer vision domains.

RT-X Models

Central to this paper are the RT-X models, specifically RT-1-X and RT-2-X, which leverage Transformer-based architectures to facilitate cross-platform learning. The paper details how RT-1-X, an adaptation of the RT-1 architecture, and RT-2-X, building on a vision-LLM (VLM) approach, are trained on this diverse dataset. The results demonstrate significant positive transfer, with RT-1-X outperforming previous specialized methods by an average of 50% in success rate.

Experimental Insights

The paper conducts extensive evaluations across various small and large dataset domains, deploying RT-1-X and RT-2-X models to assess in-distribution performance and generalization capabilities to novel tasks. Notably, RT-2-X shows remarkable generalization and emergent skill capabilities, leveraging its substantial capacity and pre-trained VLM foundations.

1. Small-Scale Dataset Domains: RT-1-X showed marked improvements over specialized models, indicating positive transfer from large, diverse datasets.
2. Large-Scale Dataset Domains: The RT-2-X model, due to its immense capacity and VLM pre-training, successfully outperformed specific domain models, especially in emergent skill tasks.

Implications and Future Work

This endeavor illustrates a pivotal leap towards achieving generalist robot policies, emphasizing the importance of collaborative and cross-embodiment data utilization in robotics. The authors propose that significant strides can be made with continued exploration into transfer across differing robot modalities and generalization to unseen robotic configurations.

Future research could delve into diversifying the sensory modalities and robotic architectures involved, aiming for broader applicability. Moreover, exploring decision criteria for realizing positive transfer, as well as scaling up dataset diversity, could further catalyze advancements in this domain.

In conclusion, the Open X-Embodiment initiative not only pushes the boundaries of robotic learning but also provides valuable datasets and model architectures for the broader academic community. By laying the groundwork for X-embodiment learning, this paper sets the stage for future developments that may redefine the capabilities and reach of robotic systems in dynamic environments.