Yell At Your Robot: Improving On-the-Fly from Language Corrections (2403.12910v1)

Abstract: Hierarchical policies that combine language and low-level control have been shown to perform impressively long-horizon robotic tasks, by leveraging either zero-shot high-level planners like pretrained language and vision-LLMs (LLMs/VLMs) or models trained on annotated robotic demonstrations. However, for complex and dexterous skills, attaining high success rates on long-horizon tasks still represents a major challenge -- the longer the task is, the more likely it is that some stage will fail. Can humans help the robot to continuously improve its long-horizon task performance through intuitive and natural feedback? In this paper, we make the following observation: high-level policies that index into sufficiently rich and expressive low-level language-conditioned skills can be readily supervised with human feedback in the form of language corrections. We show that even fine-grained corrections, such as small movements ("move a bit to the left"), can be effectively incorporated into high-level policies, and that such corrections can be readily obtained from humans observing the robot and making occasional suggestions. This framework enables robots not only to rapidly adapt to real-time language feedback, but also incorporate this feedback into an iterative training scheme that improves the high-level policy's ability to correct errors in both low-level execution and high-level decision-making purely from verbal feedback. Our evaluation on real hardware shows that this leads to significant performance improvement in long-horizon, dexterous manipulation tasks without the need for any additional teleoperation. Videos and code are available at https://yay-robot.github.io/.

Language-Driven Robot Learning and Adaptation: A New Framework for Improving Robotic Task Performance

Introduction

Robotics research has long pursued the capability for robots to perform complex tasks that involve multiple stages and precise maneuvers. Traditionally, the development of high-level policies for orchestrating such tasks has been hindered by the challenge of obtaining scalable, high-quality training data. In their recent contribution, Lucy Xiaoyang Shi \textit{et al.} introduce a novel framework, Yell At Your Robot (YAY Robot), aimed at leveraging natural language as both a medium for human-robot interaction and a mechanism for learning. Their framework is particularly designed to improve robots' performance on long-horizon tasks through the incorporation of language corrections, enabling on-the-fly adaptation and continuous improvement based purely on verbal feedback.

Approach Overview

The paper proposes a hierarchical policy structure where a high-level policy generates language instructions interpreted and executed by a lower-level policy. This setup leverages the expressive power of natural language to bridge the gap between user expectations and robot actions. A key innovation of their approach is its capacity to harness verbal corrections from human observers to refine the robot's behavior in real-time and iteratively improve the high-level decision-making policy.

The efficacy of this framework is showcased in three bi-manual manipulation tasks: bag packing, trail mix preparation, and plate cleaning. These tasks are selected for their relevance to practical applications and their requirement for delicate manipulations and precise control.

Implementational Details

At the core of their system is a Language-Conditioned Behavior Cloning (LCBC) policy learning from a dataset annotated with verbal instructions. The high-level policy is responsible for generating these language instructions based on the robot's observations, while the low-level policy translates these instructions into actionable commands. Human-provided corrections directly intervene in the high-level policy's outputs, offering a straightforward path for real-time adjustments.

One of the noteworthy aspects of their implementation is the efficiency in data annotation, facilitated by a live-narration method where operators speak the instructions synchronously with teleoperating the robot. This method not only increases the volume of obtainable data but also enriches the diversity of scenarios and corrections the robot can learn from.

Experimental Insights

The evaluation of YAY Robot on real-world tasks presented significant findings. With the inclusion of language corrections, task success rates saw improvements ranging from 15\% to 50\% across different task stages, underscoring the value of verbal feedback in enhancing robotic performance. Moreover, the iterative finetuning of the high-level policy with corrective feedback progressively reduced the necessity for human intervention.

Comparative analysis against non-hierarchical imitation learning methods demonstrated the superiority of the hierarchical approach, particularly in handling complex tasks with multiple stages and potential points of failure.

Future Directions and Limitations

While the framework showcases promising results, the reliance on a sophisticated low-level policy capable of interpreting a wide range of language instructions underscores a notable limitation. Future research directions may include enhancing the flexibility and robustness of the low-level policy and exploring the integration of non-verbal communication forms such as gestures for richer human-robot interactions.

Final Thoughts

YAY Robot represents a significant step towards more interactive and adaptable robotic systems, where natural language serves as the bridge between human intuition and robotic action. Through innovative data annotation techniques and hierarchical policy design, this work paves the way for robots to not only perform complex tasks more effectively but also evolve through interaction with their human users.