Vision-Language Interpreter for Robot Task Planning (2311.00967v2)

Abstract: LLMs are accelerating the development of language-guided robot planners. Meanwhile, symbolic planners offer the advantage of interpretability. This paper proposes a new task that bridges these two trends, namely, multimodal planning problem specification. The aim is to generate a problem description (PD), a machine-readable file used by the planners to find a plan. By generating PDs from language instruction and scene observation, we can drive symbolic planners in a language-guided framework. We propose a Vision-Language Interpreter (ViLaIn), a new framework that generates PDs using state-of-the-art LLM and vision-LLMs. ViLaIn can refine generated PDs via error message feedback from the symbolic planner. Our aim is to answer the question: How accurately can ViLaIn and the symbolic planner generate valid robot plans? To evaluate ViLaIn, we introduce a novel dataset called the problem description generation (ProDG) dataset. The framework is evaluated with four new evaluation metrics. Experimental results show that ViLaIn can generate syntactically correct problems with more than 99\% accuracy and valid plans with more than 58\% accuracy. Our code and dataset are available at https://github.com/omron-sinicx/ViLaIn.

Vision-Language Interpreter for Robot Task Planning

This paper focuses on the development of a Vision-Language Interpreter (ViLaIn) for enhancing the interface between symbolic planners and language-guided robotic task planning. At the core of this innovation is the capacity to process linguistic instructions and scene observations to generate Problem Descriptions (PDs) that are syntactically and semantically coherent for robotic operation. The main motivation of this research is to bridge the gap between the ease-of-use offered by LLMs and the interpretability of symbolic planners, thereby advancing the reliability and accessibility of robotic systems.

ViLaIn operates by integrating state-of-the-art LLMs and vision-LLMs to interpret multimodal input and produce PDs. This framework allows symbolic planners to utilize these generated PDs for executing valid robot plans. To evaluate the effectiveness of ViLaIn, the authors introduce a novel dataset known as the Problem Description Generation (ProDG) dataset. This dataset spans across three domains—cooking, Blocksworld, and Hanoi—each offering unique challenges for task planning and execution.

One of the compelling aspects of ViLaIn is its corrective re-prompting capability, which leverages error feedback from symbolic planners to refine PDs iteratively. This process is further enhanced through the application of Chain-of-Thought (CoT) prompting, a method that improves decision-making accuracy by modeling the reasoning process of the LLM. The framework demonstrates a high rate of syntactic correctness (R > 99%) and the creation of valid plans (R > 58%) according to the experimental evaluation using ProDG dataset.

The paper's methodology also highlights the division of labor between the object estimator, initial state estimator, and goal estimator. Through the combined strength of these modules, ViLaIn efficiently tackles the transformation of linguistic and visual data into a machine-readable format. Importantly, the ProDG dataset and the accompanying metrics broaden the scope for quantitative analysis of the framework's output, distinguishing between syntactic and semantic correctness.

The implications of this research span across both practical and theoretical dimensions. Practically, the proposed framework can significantly reduce the barrier for non-experts to engage with and instruct robotic systems. Theoretically, it provides a novel approach to integrating LLMs with symbolic action planners, marrying interpretability with flexibility. The introduction of error-driven refinement and CoT prompts indicates a path forward for enhancing model robustness in dynamic environments.

Looking ahead, the paper suggests several promising directions for future developments in AI and robotics. These include integrating ViLaIn with operational robotics systems to execute instructions in real-world contexts, utilizing real-time feedback from robotic failures for PD refinement, and automating certain components to minimize human intervention when adapting to new task domains.

Overall, the paper presents a well-founded and comprehensive exploration of vision-language techniques applied to robotic planning, pointing toward new opportunities for AI-driven innovation in interactive and autonomous systems.