DriveLMM-o1: A Step-by-Step Reasoning Dataset and Large Multimodal Model for Driving Scenario Understanding (2503.10621v1)

Abstract: While large multimodal models (LMMs) have demonstrated strong performance across various Visual Question Answering (VQA) tasks, certain challenges require complex multi-step reasoning to reach accurate answers. One particularly challenging task is autonomous driving, which demands thorough cognitive processing before decisions can be made. In this domain, a sequential and interpretive understanding of visual cues is essential for effective perception, prediction, and planning. Nevertheless, common VQA benchmarks often focus on the accuracy of the final answer while overlooking the reasoning process that enables the generation of accurate responses. Moreover, existing methods lack a comprehensive framework for evaluating step-by-step reasoning in realistic driving scenarios. To address this gap, we propose DriveLMM-o1, a new dataset and benchmark specifically designed to advance step-wise visual reasoning for autonomous driving. Our benchmark features over 18k VQA examples in the training set and more than 4k in the test set, covering diverse questions on perception, prediction, and planning, each enriched with step-by-step reasoning to ensure logical inference in autonomous driving scenarios. We further introduce a large multimodal model that is fine-tuned on our reasoning dataset, demonstrating robust performance in complex driving scenarios. In addition, we benchmark various open-source and closed-source methods on our proposed dataset, systematically comparing their reasoning capabilities for autonomous driving tasks. Our model achieves a +7.49% gain in final answer accuracy, along with a 3.62% improvement in reasoning score over the previous best open-source model. Our framework, dataset, and model are available at https://github.com/ayesha-ishaq/DriveLMM-o1.

Overview of DriveLMM-o1: A Step-by-Step Reasoning Dataset and Model for Autonomous Driving

This paper introduces DriveLMM-o1, a novel approach targeting the autonomous driving domain by developing a comprehensive dataset and large multimodal model specifically designed for evaluating and enhancing step-by-step reasoning capabilities in complex driving scenarios. The authors identify critical gaps in existing Visual Question Answering (VQA) benchmarks that primarily focus on end-answer accuracy without adequately addressing the intricacies of reasoning processes necessary for autonomous driving tasks, such as perception, prediction, and planning.

Contributions

The paper makes several significant contributions to the field:

1. Dataset Creation and Benchmark: The authors present a new dataset specifically crafted for autonomous driving scenarios. The training set comprises over 18,000 VQA examples, with an additional 4,000 examples included in the test set. These examples are rich in detail and emphasize step-by-step reasoning, covering core tasks required for effective autonomous driving: perception, prediction, and planning.
2. Multimodal Model Development: A large multimodal model fine-tuned on the created dataset is introduced. This model integrates visual information from multiview images and LiDAR point clouds, allowing for a comprehensive understanding of the driving environment. The model is designed to improve upon existing models with a focus on reasoning process correctness and decision transparency.
3. Evaluation Metrics: The paper introduces novel evaluation metrics that focus on logical coherence and reasoning quality, rather than just final answer accuracy. This includes the introduction of specific metrics tailored to driving, such as risk assessment accuracy, traffic rule adherence, and scene awareness, among others.
4. Performance Results: The authors provide extensive benchmark comparisons, evaluating both open-source and closed-source models on the proposed dataset. The developed model achieves a notable increase of +7.49% in final answer accuracy and a 3.62% improvement in reasoning score compared to the previous best open-source model, highlighting its effectiveness.

Implications

The paper’s implications are manifold, impacting both theoretical understanding and practical application in autonomous driving.

• Theoretical Advances: By focusing on the reasoning process, this research deepens our understanding of AI models’ decision-making mechanisms in dynamic and uncertain environments. The incorporation of multimodal data enriches the cognitive process models follow, which is critical for developing trust in autonomous systems.
• Practical Contributions: For practitioners, the dataset and model proposed offer a new benchmark for developing and testing autonomous driving systems. The detailed, scene-specific questions push models to consider a breadth of scenarios, promoting enhanced safety and reliability in real-world applications.

Future Directions

The paper opens several avenues for future research. A primary area for development could be the extension of the dataset to include even more diverse and challenging scenarios, promoting broad generalization in different environmental contexts. Furthermore, the proposal of customized architectures or learning strategies that better integrate and leverage multimodal cues could further refine the reasoning accuracy and efficiency of such autonomous driving models.

In conclusion, "DriveLMM-o1: A Step-by-Step Reasoning Dataset and Large Multimodal Model for Driving Scenario Understanding" provides timely advancements by addressing critical issues in VQA tasks within autonomous driving. Through its development of a specialized dataset and model tailored for step-by-step reasoning, it sets a new standard for evaluating and improving the cognitive capabilities of autonomous systems.