OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Driving (2412.15208v2)

Abstract: Since the advent of Multimodal LLMs (MLLMs), they have made a significant impact across a wide range of real-world applications, particularly in Autonomous Driving (AD). Their ability to process complex visual data and reason about intricate driving scenarios has paved the way for a new paradigm in end-to-end AD systems. However, the progress of developing end-to-end models for AD has been slow, as existing fine-tuning methods demand substantial resources, including extensive computational power, large-scale datasets, and significant funding. Drawing inspiration from recent advancements in inference computing, we propose OpenEMMA, an open-source end-to-end framework based on MLLMs. By incorporating the Chain-of-Thought reasoning process, OpenEMMA achieves significant improvements compared to the baseline when leveraging a diverse range of MLLMs. Furthermore, OpenEMMA demonstrates effectiveness, generalizability, and robustness across a variety of challenging driving scenarios, offering a more efficient and effective approach to autonomous driving. We release all the codes in https://github.com/taco-group/OpenEMMA.

• The paper introduces OpenEMMA, an open-source end-to-end autonomous driving framework leveraging Multimodal Large Language Models (MLLMs).
• OpenEMMA enhances planning accuracy and interpretability by integrating Chain-of-Thought reasoning and an optimized YOLO3D model for object detection.
• Evaluated on nuScenes, OpenEMMA outperforms zero-shot baselines in trajectory planning and is released open-source to foster community development.

OpenEMMA: An Open-Source Framework for Autonomous Driving

The paper introduces OpenEMMA, an open-source end-to-end autonomous driving framework, which leverages Multimodal LLMs (MLLMs) to address the complexities of autonomous driving in a computationally efficient manner. The framework integrates a Chain-of-Thought reasoning process and advanced object detection to improve trajectory planning in real-world scenarios.

Motivation and Approach

The motivation for OpenEMMA arises from the challenges associated with traditional autonomous driving systems, which often rely on modular architectures. These systems can suffer from communication errors between components and adaptation issues when confronted with novel or unforeseen conditions. OpenEMMA aims to circumvent these challenges by employing an end-to-end learning approach, which allows for holistic optimization of driving tasks.

The framework processes visual inputs from front-facing cameras in conjunction with historical data on vehicle dynamics. By framing driving tasks within the field of Visual Question Answering (VQA), OpenEMMA can harness the robust reasoning capabilities of MLLMs. The framework performs trajectory planning by predicting a series of future speed and curvature vectors, which are then integrated to produce the final path of the vehicle.

Methodology

OpenEMMA’s methodology is built around several novel components:

1. Reasoning and Planning: OpenEMMA employs the Chain-of-Thought reasoning process, a technique known for eliciting deep reasoning capabilities in large models. The system generates human-interpretable intermediate representations—speed and curvature vectors—to aid in more accurate and context-aware trajectory planning.
2. Object Detection: The framework addresses the limitation of MLLMs in spatial reasoning and object detection by integrating an optimized YOLO3D model. This component enhances the system’s accuracy in determining the positions and dimensions of objects within a driving scene, crucial for making informed driving decisions.
3. Open-Source Access and Experimentation: By releasing the entire codebase along with datasets and model weights, the research community is provided with the necessary tools to further develop, test, and refine the framework, paving the way for continued advancements in the field of autonomous driving.

Experimental Evaluation

The paper presents a comprehensive set of experiments conducted on the nuScenes dataset. The experiments evaluate the trajectory planning capabilities of OpenEMMA using multiple pre-trained MLLMs, including LLaVA-1.6, Llama-3.2, and Qwen2-VL models. Results show that OpenEMMA consistently outperforms a zero-shot baseline, achieving lower L2 norm errors and reduced failure rates. Notably, OpenEMMA demonstrates significant improvements in challenging driving scenarios, highlighting its robustness and adaptability.

Implications and Future Directions

The introduction of OpenEMMA presents several implications for the development of autonomous driving systems. Practically, the framework offers an open and accessible tool for researchers to explore the integration of MLLMs in end-to-end autonomous driving. Theoretically, it underscores the potential of Chain-of-Thought reasoning and the integration of visual specialists in enhancing both the decision-making accuracy and interpretability of autonomous systems.

Future research could explore the extension of OpenEMMA's reasoning capabilities by incorporating more advanced inference-time techniques, such as CoT-SC and ToT. Additionally, future developments in MLLM capacities for spatial reasoning and object detection might eventually eliminate the need for external object detection models, offering a more unified solution for autonomous driving.

In summary, OpenEMMA represents a notable step towards efficient and effective autonomous driving frameworks, balancing the need for high accuracy with computational cost considerations, while encouraging open scientific inquiry through the provision of its open-source resources.