Instructing Hierarchical Tasks to Robots by Verbal Commands (2311.18329v1)

Abstract: Natural language is an effective tool for communication, as information can be expressed in different ways and at different levels of complexity. Verbal commands, utilized for instructing robot tasks, can therefor replace traditional robot programming techniques, and provide a more expressive means to assign actions and enable collaboration. However, the challenge of utilizing speech for robot programming is how actions and targets can be grounded to physical entities in the world. In addition, to be time-efficient, a balance needs to be found between fine- and course-grained commands and natural language phrases. In this work we provide a framework for instructing tasks to robots by verbal commands. The framework includes functionalities for single commands to actions and targets, as well as longer-term sequences of actions, thereby providing a hierarchical structure to the robot tasks. Experimental evaluation demonstrates the functionalities of the framework by human collaboration with a robot in different tasks, with different levels of complexity. The tools are provided open-source at https://petim44.github.io/voice-jogger/

Abstract

This paper presents a framework designed to enable the instruction of robots using verbal commands. Natural language's expressiveness and flexibility make it suitable for replacing conventional robot programming techniques, potentially streamlining the assignment of tasks and facilitating human-robot collaboration.

Introduction

The use of speech for interacting with robots is a desirable goal due to its ubiquity and efficiency in human communication. Despite longstanding efforts to achieve fluent human-robot dialogue, significant challenges remain, particularly in grounding the instructions within physical reality—connecting spoken words to concrete robot actions and associated targets.

Approach

The framework outlined in the paper proposes a dual-tiered system of instructing robots: basic and hierarchical commands. It bridges the gap between speech recognition and the tangible execution of robot operations without relying on complex planning algorithms.

• For basic commands, it encompasses initiating system operation, specific movements, saving positions, and gripper actions.
• Hierarchical commands focus on sequencing actions, like recording series of movements or task repetition based on verbal instructions, which additive commands can build upon to create more complex actions.

Experimental Results

The framework was tested on a robotic system displaying a high degree of functionality in response to verbal instructions. Commands were effectively grounded to actions and targets, demonstrating the technology's potential across different task complexities. The tools and their developments were validated through numerous experiments, including long-term tasks, and the framework was made publicly accessible for further research and development.

Conclusion

The results confirm that speech can effectively instruct both immediate robot actions and complex, hierarchical tasks. This innovation provides a more natural and adaptable means of human-robot interaction. Future enhancements may involve leveraging LLMs to interpret more complex or abstract instructions, potentially enabling even more sophisticated human-robot collaborations. The work also underscores the importance of multimodal sensory integration to overcome the current reliance on specific verbal instructions.