How to Build a Pre-trained Multimodal model for Simultaneously Chatting and Decision-making? (2410.15885v1)

Abstract: Existing large pre-trained models typically map text input to text output in an end-to-end manner, such as ChatGPT, or map a segment of text input to a hierarchy of action decisions, such as OpenVLA. However, humans can simultaneously generate text and actions when receiving specific input signals. For example, a driver can make precise driving decisions while conversing with a friend in the passenger seat. Motivated by this observation, we consider the following question in this work: is it possible to construct a pre-trained model that can provide both language interaction and precise decision-making capabilities in dynamic open scenarios. We provide a definitive answer to this question by developing a new model architecture termed Visual Language Action model for Chatting and Decision Making (VLA4CD), and further demonstrating its performance in challenging autonomous driving tasks. Specifically, we leverage LoRA to fine-tune a pre-trained LLM with data of multiple modalities covering language, visual, and action. Unlike the existing LoRA operations used for LLM fine-tuning, we have designed new computational modules and training cost functions for VLA4CD. These designs enable VLA4CD to provide continuous-valued action decisions while outputting text responses. In contrast, existing LLMs can only output text responses, and current VLA models can only output action decisions. Moreover, these VLA models handle action data by discretizing and then tokenizing the discretized actions, a method unsuitable for complex decision-making tasks involving high-dimensional continuous-valued action vectors, such as autonomous driving. The experimental results on CARLA validate that: (1) our proposed model construction method is effective; (2) compared to the SOTA VLA model, VLA4CD can provide more accurate real-time decision-making while retaining the text interaction capability inherent to LLMs.

• The paper presents the novel VLA4CD model that simultaneously produces continuous-valued actions and text responses.
• It employs a transformer-based architecture with LoRA fine-tuning to directly handle continuous action spaces without discretization.
• Experiments on the CARLA platform demonstrate improved driving accuracy, safety, and enhanced interactive dialogue capability.

Overview of Visual Language Action Models for Chatting and Decision Making

The paper, "How to Build a Pre-trained Multimodal Model for Simultaneously Chatting and Decision-making?", presents a novel approach to developing large pre-trained models that can perform both language interaction and decision-making tasks concurrently. Unlike typical models that are limited to single-modal output, the proposed Visual Language Action model for Chatting and Decision Making (VLA4CD) can provide both continuous-valued action decisions and text responses in an end-to-end manner.

Introduction

LLMs such as GPT-3.5 and GPT-4 possess impressive zero-shot generalization and reasoning capabilities. However, their application has been predominantly in text-based tasks. This paper explores how the capabilities of LLMs can be integrated into decision-making tasks that occur in dynamic environments, a scenario not well-handled by previous models.

Methodology

The VLA4CD architecture builds on the transformer-based LLM architecture, enhanced with LoRA for fine-tuning pre-trained models on multimodal data. Key improvements over existing Visual Language Action (VLA) models include:

1. Synchronous Text and Action Generation: VLA4CD allows simultaneous generation of action decisions and text responses, unlike traditional models which process these outputs serially. This capability draws inspiration from human multitasking, such as driving while conversing.
2. Handling Continuous Action Spaces: Previous VLA models required discretization of continuous action spaces, which is unsuitable for complex tasks like autonomous driving. VLA4CD processes continuous actions directly, improving the model's applicability to real-world scenarios.

Experiments and Results

The efficacy of VLA4CD was demonstrated through extensive experiments using the CARLA autonomous driving platform. The following results highlight its capabilities:

• Improved Decision Accuracy: VLA4CD achieved significantly better driving scores, safe driving distances, and lower collision rates than state-of-the-art models.
• Enhanced Dialogue Ability: Beyond decision-making, VLA4CD showed superior performance in generating and maintaining coherent text interactions during autonomous driving tasks.

Implications and Future Directions

This research opens new avenues for developing multifunctional autonomous systems capable of interacting with humans while performing complex tasks. In practice, VLA4CD could drive advancements in areas such as robotics, autonomous navigation, and human-computer interaction.

Future development may focus on optimizing the integration of multimodal inputs and outputs and expanding the model's capabilities across other tasks and domains. As the model's core idea is broadly applicable, these advancements might further enhance robustness and adaptability in diverse real-world applications.

Conclusion

VLA4CD demonstrates that integrating LLMs with multimodal inputs can effectively address the challenges of simultaneous language interaction and action decision-making in dynamic environments. The approach marks a step forward in constructing comprehensive AI systems with expanded utility in various sophisticated tasks.