Multi-Agent Reinforcement Learning-based Cooperative Autonomous Driving in Smart Intersections (2505.04231v1)

Abstract: Unsignalized intersections pose significant safety and efficiency challenges due to complex traffic flows. This paper proposes a novel roadside unit (RSU)-centric cooperative driving system leveraging global perception and vehicle-to-infrastructure (V2I) communication. The core of the system is an RSU-based decision-making module using a two-stage hybrid reinforcement learning (RL) framework. At first, policies are pre-trained offline using conservative Q-learning (CQL) combined with behavior cloning (BC) on collected dataset. Subsequently, these policies are fine-tuned in the simulation using multi-agent proximal policy optimization (MAPPO), aligned with a self-attention mechanism to effectively solve inter-agent dependencies. RSUs perform real-time inference based on the trained models to realize vehicle control via V2I communications. Extensive experiments in CARLA environment demonstrate high effectiveness of the proposed system, by: \textit{(i)} achieving failure rates below 0.03\% in coordinating three connected and autonomous vehicles (CAVs) through complex intersection scenarios, significantly outperforming the traditional Autoware control method, and \textit{(ii)} exhibiting strong robustness across varying numbers of controlled agents and shows promising generalization capabilities on other maps.

• The paper introduces a roadside unit (RSU)-centric multi-agent reinforcement learning (MARL) framework for cooperative autonomous driving at unsignalized intersections.
• The methodology uses a two-stage hybrid RL framework combining offline Conservative Q-Learning with Behavior Cloning and online Multi-Agent Proximal Policy Optimization with self-attention.
• Experiments in CARLA simulation showed the system achieved a failure rate below 0.03% when coordinating up to three vehicles, outperforming a conventional benchmark.

Multi-Agent Reinforcement Learning-based Cooperative Autonomous Driving in Smart Intersections

The research addresses the critical challenge of managing unsignalized urban intersections, which are notorious for accidents and inefficiencies due to their complex traffic dynamics. The paper introduces a roadside unit (RSU)-centric cooperative driving framework that uses multi-agent reinforcement learning (MARL) to coordinate connected and autonomous vehicles (CAVs) through these intersections.

The decision-making core of this system is a two-stage hybrid reinforcement learning (RL) framework. Initially, policies are developed offline using Conservative Q-Learning (CQL) combined with Behavior Cloning (BC), allowing the system to establish a strong foundational understanding of driving behaviors and interactions from pre-existing datasets. This is followed by online refinement in a simulated environment utilizing Multi-Agent Proximal Policy Optimization (MAPPO), which incorporates self-attention mechanisms. This component effectively models inter-agent dependencies and adapts to varying traffic participant numbers, enhancing robustness and safety.

The effectiveness of this approach was demonstrated through extensive experiments conducted in the CARLA simulation environment. The proposed system achieved a failure rate below 0.03% when coordinating up to three CAVs, which is notably lower than the failure rate encountered by the conventional Autoware benchmark system. The system also showed promising generalization capabilities across various intersection scenarios, suggesting its potential for broader application in intelligent transportation systems (ITS).

Additionally, the research highlights the advantage of using RSUs equipped with LiDAR for global monitoring, which overcomes the limitations of individual vehicle perception by providing a comprehensive overview of the traffic environment. This centralized perception enables more effective decision-making, enhancing collective safety and traffic throughput. The use of a hybrid RL framework accelerates learning convergence and simplifies the computational requirements for the CAVs by offloading intensive processing to the RSU.

Implications of this research are extensive both in terms of practical and theoretical advancements in autonomous vehicle systems. The development of role-specific policy networks tailored to different driving maneuvers (e.g., left, right, straight) contributes to the precision of vehicle coordination at intersections. Moreover, the integration of self-attention mechanisms in MARL frameworks could propel future AI systems towards achieving higher levels of autonomy and adaptability, optimizing interactions in complex traffic scenarios.

Future research directions suggested in the paper include the practical validation of the framework in real-world environments, refining the two-stage learning process, and scaling the system to accommodate intersections with more dynamic and varying conditions. This paper provides a solid foundation for further exploration and development of smart intersections, advancing the goal of seamless integration of autonomous driving technology into everyday urban traffic systems.