Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld (2311.16714v2)

Abstract: While LLMs excel in a simulated world of texts, they struggle to interact with the more realistic world without perceptions of other modalities such as visual or audio signals. Although vision-LLMs (VLMs) integrate LLM modules (1) aligned with static image features, and (2) may possess prior knowledge of world dynamics (as demonstrated in the text world), they have not been trained in an embodied visual world and thus cannot align with its dynamics. On the other hand, training an embodied agent in a noisy visual world without expert guidance is often challenging and inefficient. In this paper, we train a VLM agent living in a visual world using an LLM agent excelling in a parallel text world. Specifically, we distill LLM's reflection outcomes (improved actions by analyzing mistakes) in a text world's tasks to finetune the VLM on the same tasks of the visual world, resulting in an Embodied Multi-Modal Agent (EMMA) quickly adapting to the visual world dynamics. Such cross-modality imitation learning between the two parallel worlds is achieved by a novel DAgger-DPO algorithm, enabling EMMA to generalize to a broad scope of new tasks without any further guidance from the LLM expert. Extensive evaluations on the ALFWorld benchmark's diverse tasks highlight EMMA's superior performance to SOTA VLM-based agents, e.g., 20%-70% improvement in the success rate.

Evaluation of "Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld"

The paper presents the Embodied Multi-Modal Agent (EMMA), a novel endeavor that merges LLMs and vision-LLMs (VLMs) to create agents capable of functioning effectively in both textual and visual environments. This innovative approach addresses several longstanding challenges in the pursuit of developing AGI, particularly in terms of embodying multi-modal agents that can perceive and act based on dynamic interactions in their environments.

Overview

The paper highlights an intrinsic limitation with conventional LLMs and VLMs. While LLMs have shown outstanding proficiency in understanding and interacting with textual information, their implementations have not been adequately extended to visual or embodied environments. Similarly, VLMs, despite their utility in aligning verbal and visual data, often demonstrate suboptimal performance when tasked with operating as embodied agents in dynamic visual landscapes.

EMMA offers a sophisticated approach to overcoming these challenges by employing an interactive imitation learning strategy called DAgger-DPO. This tactic leverages cross-modality learning in parallel text and visual worlds to refine the VLM agent based on the expertise of a superior-performing LLM agent. By doing so, EMMA effectively absorbs and incorporates the world knowledge that the LLM has constructed within its textual environment.

Methodology

The core methodology involves a bidirectional learning process where insights from tasks completed in a textual environment are translated into a visual modality, enabling EMMA to better grasp and align with visual world dynamics. This is accomplished by distilling LLM's reflection outcomes—such as improved actions derived from mistake analyses in a text environment—to hone EMMA's capabilities in visual tasks.

The training framework draws upon a rule-based expert system that provides a foundation for these interactions, with the DAgger-DPO algorithm substantially enhancing task adaptability and success rate. Significant attention is given to integrating expert-generated feedback and leveraging a carefully structured learning environment, enhancing EMMA’s ability to generalize and excel in previously unseen and diverse tasks.

Results

The outcomes illustrated in the paper effectively showcase EMMA’s superiority over existing VLM-based agents, demonstrating improvement metrics between 20% and 70% in success rates on the ALFWorld benchmark, a simulation environment that unifies text and visual challenges. These quantifiable results underscore a pivotal advancement in enhancing agent performance through cross-modal learning and retrospective reflection processes.

Implications

This research holds vast implications for the broader AI community. Practically, it enhances the potential for developing autonomous systems that can multitask and adapt across varying environments within a consistent framework. Theoretically, it offers insights into the seamless integration of distinct AI modules, promoting further exploration and refinement of multi-modal learning techniques. EMMA's success positions it as a benchmark for AI models transcending singular modalities.

Future Directions

Potential future directions include the expansion of EMMA’s adaptability in more intricate and less structured real-world scenarios, further refining the cross-modal learning framework and exploring deeper integration with real-time environmental feedback mechanisms. Additionally, future research could look into scaling EMMA’s architecture to handle long-horizon planning tasks that are representative of more complex, real-world challenges.

EMMA presents a significant model for understanding and addressing the dynamic needs of multi-modal AI systems, indicating promising avenues for future AI development and multi-modal agent training.