EMPOWER: Embodied Multi-role Open-vocabulary Planning with Online Grounding and Execution

Abstract: Task planning for robots in real-life settings presents significant challenges. These challenges stem from three primary issues: the difficulty in identifying grounded sequences of steps to achieve a goal; the lack of a standardized mapping between high-level actions and low-level commands; and the challenge of maintaining low computational overhead given the limited resources of robotic hardware. We introduce EMPOWER, a framework designed for open-vocabulary online grounding and planning for embodied agents aimed at addressing these issues. By leveraging efficient pre-trained foundation models and a multi-role mechanism, EMPOWER demonstrates notable improvements in grounded planning and execution. Quantitative results highlight the effectiveness of our approach, achieving an average success rate of 0.73 across six different real-life scenarios using a TIAGo robot.

• The paper introduces EMPOWER, an embodied multi-role open-vocabulary planning framework for robots that uses foundation models to improve task planning and execution.
• EMPOWER employs a hierarchical method with distinct agent roles (semantic, grounded, planning) and models like YOLO-World for robust open-vocabulary grounding of actions in complex scenes.
• Empirical results using the TIAGo robot demonstrate EMPOWER's capability to achieve a 0.73 success rate across challenging real-world scenarios, enabling more adaptable autonomous systems.

Overview of EMPOWER: Embodied Multi-role Open-vocabulary Planning with Online Grounding and Execution

In the field of robotic task planning, EMPOWER represents a notable effort focused on optimizing the interaction between robots and complex real-world environments. Authored by Francesco Argenziano and colleagues, this framework addresses significant obstacles faced by robots in task execution, specifically the challenges of grounding actions to execute a planned sequence of tasks efficiently. These challenges stem from aligning high-level actions—described using open-vocabulary constructs—with low-level robotic commands, all while managing computational overhead intrinsic to robotic hardware.

EMPOWER leverages advanced Foundation Models (FMs), noted for their ability to incorporate commonsense knowledge, enabling multi-role mechanisms and stipulating open-vocabulary online grounding. This improves the planning and execution processes in embodied agents, exemplified through practical experiments using the TIAGo robot.

Methodological Approach

The EMPOWER framework introduces a hierarchical strategy that employs pre-trained FMs and cloud services to create an open-vocabulary representation of the environment. This approach facilitates the grounding of actions and enables comprehension of complex scenes, courtesy of robust multi-role prompting techniques. Key components include:

1. Multi-role Planning: The framework utilizes three distinct roles within its architecture to achieve high-level planning and grounding. These roles are assigned to agents that specialize in semantic knowledge extraction (SMK), grounded knowledge extraction (GMK), and planning (P). This multi-role strategy enhances the robustness and accuracy of the task planning process.
2. Open-vocabulary Grounding: An integral component of EMPOWER's architecture is its capacity to perform open-vocabulary grounding, facilitated by NLP techniques and the employment of models such as YOLO-World and EfficientViT-SAM. These models ensure effective object detection and scene understanding, critical for grounding tasks.
3. Action Execution: The framework emphasizes mapping high-level actions to low-level executable actions. This ensures that the robotic agent's tasks are achievable within real-world settings. The TIAGo robot demonstrates the efficacy of EMPOWER by executing actions derived from multi-role planning in real-life scenarios.

Empirical Results

Quantitatively, EMPOWER showcases significant efficacy with a reported success rate of 0.73 across six distinct experimental scenarios involving the TIAGo robot. These scenarios posed complex task execution challenges, necessitating a nuanced understanding of the environment and the tasks. The success rates underscore the capability of EMPOWER to effectively interpret and interact within real-world contexts with minimal computational constraints.

Implications and Future Directions

The practical implications of EMPOWER suggest broader applications for AI and robotics, particularly emphasizing adaptability in real-world environments where predefined vocabularies are ineffective. This framework's integration of open-vocabulary models creates prospects for enhanced autonomous robotic systems capable of handling unexpected scenarios with a degree of semantic understanding previously unattainable.

From a theoretical perspective, the approach instigated by EMPOWER may inspire further discourse on the convergence of robotics and advanced LLMs. It highlights the potential for LLMs to contribute substantively to embodied AI, paving the way for their use in planning, interaction, and decision-making contexts that resemble human-like reasoning and adaptability.

Continued exploration in this domain could refine the balance between computational efficiency and the complexity of tasks undertaken by robots, fostering the development of more autonomous systems capable of overcoming the inherent limitations currently faced in dynamic environments.