LVLM-MPC Collaboration for Autonomous Driving: A Safety-Aware and Task-Scalable Control Architecture (2505.04980v1)

Abstract: This paper proposes a novel Large Vision-LLM (LVLM) and Model Predictive Control (MPC) integration framework that delivers both task scalability and safety for Autonomous Driving (AD). LVLMs excel at high-level task planning across diverse driving scenarios. However, since these foundation models are not specifically designed for driving and their reasoning is not consistent with the feasibility of low-level motion planning, concerns remain regarding safety and smooth task switching. This paper integrates LVLMs with MPC Builder, which automatically generates MPCs on demand, based on symbolic task commands generated by the LVLM, while ensuring optimality and safety. The generated MPCs can strongly assist the execution or rejection of LVLM-driven task switching by providing feedback on the feasibility of the given tasks and generating task-switching-aware MPCs. Our approach provides a safe, flexible, and adaptable control framework, bridging the gap between cutting-edge foundation models and reliable vehicle operation. We demonstrate the effectiveness of our approach through a simulation experiment, showing that our system can safely and effectively handle highway driving while maintaining the flexibility and adaptability of LVLMs.

LVLM-MPC Collaboration for Autonomous Driving: A Safety-Aware and Task-Scalable Control Architecture

The paper "LVLM-MPC Collaboration for Autonomous Driving: A Safety-Aware and Task-Scalable Control Architecture" presents an innovative framework for autonomous driving applications, integrating a Large Vision-LLM (LVLM) with Model Predictive Control (MPC). This integration aims to ensure both task scalability and safety in autonomous driving environments.

Overview of the Hybrid Architecture

The proposed framework leverages LVLMs to handle high-level task planning across diverse driving scenarios. LVLMs provide symbolic task commands that are processed by MPC Builder, which automatically generates the appropriate MPCs. This hybrid architecture is designed to address two critical challenges in autonomous driving: scalability of task planning and safety assurance during execution.

Scalability and Safety Assurance

Scalability: Traditional model-based planners like MPC often struggle with scalability when combined with the extensive task space of modern LVLMs. The paper suggests that hand-crafting controllers for a limited set of tasks restricts versatility. Hence, the integration of MPC Builder with LVLMs facilitates rapid adaptation to various tasks by synthesizing MPCs from a compact library of primitive design elements.

Safety Assurance: Existing frameworks typically follow a unidirectional pipeline where high-level commands succeed or fail silently without dynamic feedback. The proposed bidirectional communication between LVLMs and MPC ensures that infeasibility is handled effectively through automatic rejection and constructive replanning. The notion of intermediate Optimal Control Problems (iOCP) is introduced to ensure smooth task transitions, mitigating deadlocks and wasted computation.

Simulation and Results

The effectiveness of the LVLM-MPC collaboration is demonstrated through highway driving simulations. The results indicate superior safety and efficiency compared to baseline methods. LVLM-MPCBuilder consistently maintained a 100% success rate in simulations, highlighting its capability to navigate congested traffic safely while accommodating flexible and adaptable task execution.

Implications and Future Work

The integration of LVLMs and MPC presents significant implications for autonomous vehicle design. The framework efficiently bridges the gap between high-level semantic reasoning provided by foundation models and precise, reliable vehicle operation afforded by MPC. Future advancements may deliver more complex scenario handling and real-world deployment. Additionally, expanding this approach to encompass more diverse environmental conditions and task requirements poses intriguing avenues for further research.

In conclusion, this paper articulates a compelling case for the synergistic use of LVLMs and MPC in autonomous driving. The proposed architecture not only meets the demands of scalability and safety but also provides a robust platform for exploring next-generation autonomous vehicle capabilities.