Learning Generalizable Manipulation Policies with Object-Centric 3D Representations (2310.14386v1)

Abstract: We introduce GROOT, an imitation learning method for learning robust policies with object-centric and 3D priors. GROOT builds policies that generalize beyond their initial training conditions for vision-based manipulation. It constructs object-centric 3D representations that are robust toward background changes and camera views and reason over these representations using a transformer-based policy. Furthermore, we introduce a segmentation correspondence model that allows policies to generalize to new objects at test time. Through comprehensive experiments, we validate the robustness of GROOT policies against perceptual variations in simulated and real-world environments. GROOT's performance excels in generalization over background changes, camera viewpoint shifts, and the presence of new object instances, whereas both state-of-the-art end-to-end learning methods and object proposal-based approaches fall short. We also extensively evaluate GROOT policies on real robots, where we demonstrate the efficacy under very wild changes in setup. More videos and model details can be found in the appendix and the project website: https://ut-austin-rpl.github.io/GROOT .

• The paper introduces GROOT, which uses imitation learning with object-centric 3D representations to achieve robust manipulation policies.
• It employs interactive segmentation and temporal tracking via a VOS model to convert annotations into 3D point clouds for enhanced spatial reasoning.
• Empirical results show that GROOT outperforms existing methods in varied visual conditions, enabling reliable policy transfer to novel objects.

An Overview of GROOT: Learning Generalizable Manipulation Policies with Object-Centric 3D Representations

The paper introduces GROOT, a novel imitation learning (IL) framework designed to enhance the generalization capabilities of vision-based manipulation policies. GROOT effectively incorporates object-centric and 3D representations to develop robust policies that extend beyond their initial training environments, tackling challenges such as varied backgrounds, differing camera angles, and new objects. This representational strategy aims to mitigate the limitations of existing IL methods that struggle with perceptual variations.

The core innovation of GROOT lies in its use of object-centric 3D representations, which are pivotal for improving policy robustness against visual disturbances. The approach is structured into several key steps, which include interactive segmentation for annotating task-relevant objects, temporal tracking of these objects using a Video Object Segmentation (VOS) model, and the conversion of segmentation data into 3D point clouds. This process allows GROOT to maintain focus on essential visual features while discarding irrelevant background distractions, thus enhancing its resilience against changes in visual conditions.

GROOT also leverages a transformer-based policy model that benefits from the segmentation of point clouds into clusters, a method inspired by Point-MAE, which facilitates improved spatial reasoning and attention dynamics for manipulation tasks. This architecture is equipped to handle temporal information, further addressing challenges associated with partial observability in dynamic tasks.

A distinguishing feature of GROOT is its segmentation correspondence model, designed to ensure that GROOT policies can manipulate new object instances at deployment time. This model utilizes an open-vocabulary segmentation model (SAM) alongside a pretrained semantic feature model (DINOv2) to align new object instances with those seen during training, enabling seamless policy transfer to novel but semantically related objects.

The empirical evaluation of GROOT encompasses comprehensive experimental setups in both simulated and real-world environments, spanning multiple manipulation tasks. These experiments demonstrate the superior generalization performance of GROOT compared to existing IL methods like BC-RNN, VIOLA, and MAE-Policy. Particularly notable is GROOT's ability to maintain high success rates under challenging visual conditions that involve background changes and varying camera viewpoints.

In terms of practical implications, GROOT's framework can significantly extend the operational flexibility of manipulation policies, reducing the dependency on specific training configurations. This advancement has important ramifications for real-world robotic applications, wherein safety, cost-efficiency, and robustness in unstructured environments are paramount.

Theoretically, GROOT underscores the importance of structured visual representations and modular perception strategies in enhancing policy generalization abilities. Future developments in AI could build upon GROOT's combination of open-world visual recognition models and imitation learning, potentially extending its versatility to more complex multi-object tasks and diverse robotic morphologies.

In conclusion, GROOT represents a meaningful step forward in the quest for generalizable robotic control policies. While the current framework is primarily tested on a single manipulator type, further research may unlock additional cross-domain applicability, broadening GROOT’s impact across a spectrum of robotic tasks and settings.