RT-H: Action Hierarchies Using Language (2403.01823v2)

Abstract: Language provides a way to break down complex concepts into digestible pieces. Recent works in robot imitation learning use language-conditioned policies that predict actions given visual observations and the high-level task specified in language. These methods leverage the structure of natural language to share data between semantically similar tasks (e.g., "pick coke can" and "pick an apple") in multi-task datasets. However, as tasks become more semantically diverse (e.g., "pick coke can" and "pour cup"), sharing data between tasks becomes harder, so learning to map high-level tasks to actions requires much more demonstration data. To bridge tasks and actions, our insight is to teach the robot the language of actions, describing low-level motions with more fine-grained phrases like "move arm forward". Predicting these language motions as an intermediate step between tasks and actions forces the policy to learn the shared structure of low-level motions across seemingly disparate tasks. Furthermore, a policy that is conditioned on language motions can easily be corrected during execution through human-specified language motions. This enables a new paradigm for flexible policies that can learn from human intervention in language. Our method RT-H builds an action hierarchy using language motions: it first learns to predict language motions, and conditioned on this and the high-level task, it predicts actions, using visual context at all stages. We show that RT-H leverages this language-action hierarchy to learn policies that are more robust and flexible by effectively tapping into multi-task datasets. We show that these policies not only allow for responding to language interventions, but can also learn from such interventions and outperform methods that learn from teleoperated interventions. Our website and videos are found at https://rt-hierarchy.github.io.

Introducing RT-H: A Novel Framework for Robotic Policies Using Language-Conditioned Action Hierarchies

Leveraging Language for Robotic Task Learning

The ability to understand and perform a wide range of tasks with minimal supervision is a coveted goal in robotics. A promising approach to this problem involves teaching robots to understand tasks through the lens of natural language. Recent advancements have seen robots being instructed using high-level task descriptions, benefiting from the inherent structure and adaptability language offers. However, a significant hurdle emerges as the diversity of these tasks increases, making the direct mapping from task descriptions to actions less effective due to the need for substantially more demonstration data.

RT-H: Bridging the Gap with Language Motions

To address this challenge, we introduce a novel framework, RT-H (Robot Transformer with Action Hierarchies), which enhances the robot's understanding of tasks by incorporating an intermediate layer of "language motions." These fine-grained phrases, like “move arm forward” or “close gripper,” serve as stepping stones between high-level tasks and the actual robot actions, facilitating a more robust learning process.

The RT-H framework operates through two main phases:

1. Language Motion Prediction: The model predicts the next language motion based on the current visual observations and the high-level task description.
2. Action Prediction: Conditioned on the visual context, the high-level task, and the inferred language motion, the model predicts the precise actions to execute.

This hierarchical approach, grounded in language, not only leads to better performance on diverse tasks by leveraging shared low-level motions across tasks but also allows for more intuitive human-robot interactions. Humans can easily correct or guide the robot using language motions, providing a pathway for rapid learning and adaptability.

Robust Experimental Validation

RT-H's efficacy is demonstrated through rigorous experimentation. The framework shows a substantial improvement in policy performance on a composite dataset consisting of multiple tasks, outperforming the flat model RT-2 by a notable margin. Additionally, RT-H exhibits outstanding flexibility and contextuality in handling language motions, effectively responding to corrections and adapting its behavior according to the task and scene context. This adaptability extends to unseen tasks and conditions, where RT-H, with minimal human intervention, demonstrates promising success rates, highlighting its potential for generalization.

Future Directions: Beyond the Current State

Despite these achievements, RT-H's journey is far from complete. Future research directions include exploring varying levels of abstraction within the action hierarchy, extending the methodology to integrate multiple steps of action reasoning, and enhancing the model's ability to learn from human videos with actions described solely in language. Additionally, incorporating RT-H's compressed action space into reinforcement learning methods could pave the way for more efficient policy exploration and learning.

Conclusion

RT-H sets a new standard in robot learning, illustrating the profound impact of intertwining language with robotic action prediction. By fostering a deeper connection between language and actions, RT-H not only advances our ability to teach robots a diverse array of tasks but also enhances the intuitiveness and effectiveness of human-robot interaction. As we continue to explore this promising avenue, the future of robotic task learning looks ever more auspicious.