Conversational Language Models for Human-in-the-Loop Multi-Robot Coordination

Abstract: With the increasing prevalence and diversity of robots interacting in the real world, there is need for flexible, on-the-fly planning and cooperation. LLMs are starting to be explored in a multimodal setup for communication, coordination, and planning in robotics. Existing approaches generally use a single agent building a plan, or have multiple homogeneous agents coordinating for a simple task. We present a decentralised, dialogical approach in which a team of agents with different abilities plans solutions through peer-to-peer and human-robot discussion. We suggest that argument-style dialogues are an effective way to facilitate adaptive use of each agent's abilities within a cooperative team. Two robots discuss how to solve a cleaning problem set by a human, define roles, and agree on paths they each take. Each step can be interrupted by a human advisor and agents check their plans with the human. Agents then execute this plan in the real world, collecting rubbish from people in each room. Our implementation uses text at every step, maintaining transparency and effective human-multi-robot interaction.

• The paper introduces a decentralized dialogical framework leveraging conversational language models for human-in-the-loop multi-robot coordination.
• It employs techniques like agent identity initialization and argument-style dialogues to effectively assign tasks and adapt plans in dynamic environments.
• Demonstrations with TurtleBot3 robots validate the system’s ability to manage cleaning tasks, highlighting its flexibility and potential for scaling complex applications.

Conversational LLMs for Human-in-the-Loop Multi-Robot Coordination

Introduction

The paper "Conversational LLMs for Human-in-the-Loop Multi-Robot Coordination" (2402.19166) addresses the increasing demand for flexible and adaptive robotic systems in real-world environments. With the anticipated growth in household robotics, there is a pressing need for systems that can seamlessly adapt to new tasks without extensive prior training. This paper explores the integration of LLMs in a multimodal framework to facilitate communication and coordination among robots and human operators.

The primary contribution of this research is a decentralized dialogical approach where heterogeneous agents with varying capabilities collaboratively plan solutions through peer-to-peer and human-robot discussions. The methodology contrasts with existing approaches that often rely on a single agent or homogeneous agent teams for simple task coordination. By leveraging argument-style dialogues, the proposed system effectively utilizes each agent's abilities within a cooperative team framework.

Demonstration

The demonstration section illustrates a practical implementation of the proposed system, involving a human-robot interactive scenario where robots collaboratively solve a cleaning problem. The system employs text-based exchanges at each step, ensuring transparency and effective human-multi-robot interaction. The demonstration highlights several key stages:

1. Agent Identity and Environment Modeling: Agents are initialized with predefined identities and a flowchart-style environment model (Figure 1). This setup allows agents to understand spatial relationships within the task environment.
2. Task Assignment and Discussion: Human supervisors assign tasks, triggering agent discussions on task execution. The agents independently define roles, collaborate on problem-solving strategies, and sequentially validate their plans with human oversight.
3. Execution and Replanning: The agreed-upon plans are executed by the robotic agents. Agents can autonomously detect execution challenges and request human intervention to replan, showcasing the system's adaptive planning capability.

   Figure 1: An example dialogue where the rooms are extracted.

The hardware implementation involves two TurtleBot3 robots equipped with LIDAR for collision avoidance and optical cameras for environmental detection. The robots demonstrate their capabilities by collecting rubbish as part of a simulated waste collection task.

Implications and Future Directions

The integration of pre-trained LLMs for decentralized mission planning introduces significant advantages in terms of flexibility and human readability. The system's ability to interface directly with human operators enhances its applicability in dynamic and uncertain environments, where human input may be crucial for refining objectives and correcting errors.

The proposed framework sets a precedent for future explorations into language-based robotic coordination, potentially facilitating the development of generalist robotics that can leverage LLMs for diverse tasks. While the current proof-of-concept demonstrates feasibility, further research is required to scale these systems for complex, real-world applications involving larger teams of varied robots and more intricate task environments.

Conclusion

This paper presents a novel approach to multi-robot coordination using conversational LLMs. By enabling robots to engage in argument-based dialogues, the proposed system harnesses the interpretative and reasoning strengths of LLMs to plan, coordinate, and execute tasks with human oversight. The demonstration reinforces the potential of this language-based interaction framework to revolutionize multi-agent coordination in robotics. Future research could explore expanding these systems for more complex tasks and larger, more diverse teams of agents.