PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks (2411.00081v1)

Abstract: We present a benchmark for Planning And Reasoning Tasks in humaN-Robot collaboration (PARTNR) designed to study human-robot coordination in household activities. PARTNR tasks exhibit characteristics of everyday tasks, such as spatial, temporal, and heterogeneous agent capability constraints. We employ a semi-automated task generation pipeline using LLMs, incorporating simulation in the loop for grounding and verification. PARTNR stands as the largest benchmark of its kind, comprising 100,000 natural language tasks, spanning 60 houses and 5,819 unique objects. We analyze state-of-the-art LLMs on PARTNR tasks, across the axes of planning, perception and skill execution. The analysis reveals significant limitations in SoTA models, such as poor coordination and failures in task tracking and recovery from errors. When LLMs are paired with real humans, they require 1.5x as many steps as two humans collaborating and 1.1x more steps than a single human, underscoring the potential for improvement in these models. We further show that fine-tuning smaller LLMs with planning data can achieve performance on par with models 9 times larger, while being 8.6x faster at inference. Overall, PARTNR highlights significant challenges facing collaborative embodied agents and aims to drive research in this direction.

• The paper presents PARTNR, a benchmark that assesses embodied AI planning and reasoning over 100,000 tasks in detailed household environments.
• It employs a novel LLM-driven, simulation-in-the-loop approach to generate diverse, realistic tasks and evaluation functions while reducing common errors.
• Findings indicate that state-of-the-art LLMs underperform compared to human-avatar collaborations, suggesting that fine-tuned smaller models offer efficient alternatives.

Evaluation of PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-Agent Tasks

The paper "PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-Agent Tasks" presents a comprehensive benchmark aimed at evaluating and advancing the capabilities of embodied AI agents in collaborative human-robot scenarios. PARTNR stands as a significant contribution to the field by providing a diverse and large-scale dataset designed to test the planning and reasoning abilities of AI agents over a range of household tasks that require both spatial and temporal understanding.

The benchmark consists of 100,000 unique natural language tasks set within 60 intricately modeled houses, containing 5,819 distinct objects, thereby offering a rich environment for studying embodied AI. It categorizes tasks into four main types: constraint-free, spatial, temporal, and heterogeneous, with a combination of these characteristics leading to complex task scenarios that necessitate effective collaboration between agents. The diverse task set challenges agents to reason beyond simple navigation and object manipulation, requiring dynamic interaction and collaboration in partially observable environments.

The authors have introduced a novel method for large-scale task generation using LLMs, integrated with a simulation-in-the-loop mechanism for grounding. This approach facilitates the reduction of errors such as hallucinations or conceptually infeasible tasks, paving the way for efficient generation of realistic yet challenging task instructions that call for creative problem-solving from AI agents. The evaluation functions accompanying these tasks are similarly generated, leveraging LLMs to ensure they capture the complexity and nuances of real-world task success criteria.

One of the significant findings of this paper lies in the limitations revealed in the performance of current state-of-the-art LLMs on collaborative tasks. Despite recent advances in natural language processing and AI, these models struggle considerably with planning in decentralized multi-agent settings, often requiring more time than their human counterparts or even single-agent systems to complete tasks. Specifically, the authors report that human-avatar collaborations solve 93% of tasks efficiently, whereas LLMs successfully complete only 30% under non-privileged conditions due to challenges in coordination, task tracking, and error recovery.

This paper also discusses the implication of model size on performance, demonstrating that smaller fine-tuned models can achieve performance on par with larger, non-tuned models while remaining substantially more efficient in terms of inference speed. This highlights a promising direction for future research wherein fine-tuning may allow for smaller, less resource-intensive models to approach the efficacy of their larger counterparts.

Practically, the PARTNR benchmark is poised to drive further studies into improving collaborative dynamics in embodied agents. It highlights critical areas such as perception, task division, and error recovery as central challenges that need addressing. The benchmark poses a call to action to the AI community, urging the development of models capable of reasoning through complex sequences of actions autonomously, thus embodying the nuanced capabilities required for genuine human-robot collaboration.

In conclusion, the PARTNR benchmark, through its robust design and comprehensive approach, is expected to stimulate significant progress in AI research focused on embodied multi-agent systems. The necessity for effective collaboration across diverse settings and task types underscores the benchmark’s potential as a tool for spurring innovation and improving the integration of AI in everyday human environments. Future research should exploit this benchmark to explore new paradigms of multi-agent coordination and reasoning, ultimately bridging the gap towards more autonomous and seamless human-robot interactions in the real world.