Learning to Control and Coordinate Mixed Traffic Through Robot Vehicles at Complex and Unsignalized Intersections

Abstract: Intersections are essential road infrastructures for traffic in modern metropolises. However, they can also be the bottleneck of traffic flows as a result of traffic incidents or the absence of traffic coordination mechanisms such as traffic lights. Recently, various control and coordination mechanisms that are beyond traditional control methods have been proposed to improve the efficiency of intersection traffic by leveraging the ability of autonomous vehicles. Amongst these methods, the control of foreseeable mixed traffic that consists of human-driven vehicles (HVs) and robot vehicles (RVs) has emerged. We propose a decentralized multi-agent reinforcement learning approach for the control and coordination of mixed traffic by RVs at real-world, complex intersections -- an open challenge to date. We design comprehensive experiments to evaluate the effectiveness, robustness, generalizablility, and adaptability of our approach. In particular, our method can prevent congestion formation via merely 5% RVs under a real-world traffic demand of 700 vehicles per hour. In contrast, without RVs, congestion will form when the traffic demand reaches as low as 200 vehicles per hour. Moreover, when the RV penetration rate exceeds 60%, our method starts to outperform traffic signal control in terms of the average waiting time of all vehicles. Our method is not only robust against blackout events, sudden RV percentage drops, and V2V communication error, but also enjoys excellent generalizablility, evidenced by its successful deployment in five unseen intersections. Lastly, our method performs well under various traffic rules, demonstrating its adaptability to diverse scenarios. Videos and code of our work are available at https://sites.google.com/view/mixedtrafficcontrol

• The paper demonstrates that a decentralized multi-agent RL strategy can prevent congestion with just 5% RV penetration, outperforming traditional traffic signals.
• It utilizes the Rainbow DQN architecture for centralized training and decentralized execution, effectively encoding traffic states into actionable vector representations.
• The approach is robust across varying intersection layouts and blackout scenarios, achieving up to an 81% reduction in extreme waiting times.

Summary of "Learning to Control and Coordinate Mixed Traffic Through Robot Vehicles at Complex and Unsignalized Intersections" (2301.05294)

Introduction

The paper presents a novel approach to addressing traffic congestion at intersections, which are critical infrastructures yet often bottlenecks due to incidents or lack of coordination mechanisms. The authors propose a decentralized multi-agent reinforcement learning strategy to optimize the flow of mixed traffic, consisting of human-driven vehicles (HVs) and robot vehicles (RVs), at complex unsignalized intersections. This innovative approach is explored through comprehensive experiments that demonstrate its effectiveness in preventing congestion with relatively few RVs in the traffic mix and achieving superior performance compared to traditional traffic signal systems.

Methodology

The core of the paper's methodology lies in leveraging reinforcement learning (RL) for decentralized traffic control. Each RV makes autonomous decisions based on a collective traffic condition representation, decentralized execution of the RL policy, and ensuring maneuver safety. The RL architecture employed, Rainbow DQN, is trained centrally, while execution remains decentralized. Key elements include:

• Traffic Representation: Encoding of intersection traffic states into fixed-length vectors representing queue lengths, waiting times, and occupancy maps along different traffic streams.
• Conflict-Free Operation: A fail-safe mechanism to avoid collisions by strategically managing RV entry priorities based on the observed traffic states.
• Reward Design: A conflict-aware reward function encouraging RVs to optimize traffic flow while penalizing moves that would result in conflicts.

Results

Experiments are conducted using high-fidelity traffic simulations based on real-world data from intersections in Colorado Springs, CO, USA. Key findings include:

Figure 1: Our reward timely reflects congestion forming at the intersection.

• Preventing Congestion: With just 5% RV penetration, the method can prevent congestion under typical traffic demands (700 vehicles/hour), outperforming scenarios devoid of RV coordination, where congestion appears at only 200 vehicles/hour.
• Performance vs. Traffic Lights: With 60% or more RVs, the method consistently reduces average waiting time compared to intersections controlled by traffic signals, offering up to 81% reduction in extreme cases.

  Figure 2: Traffic congestion levels at intersection 229 under different control mechanisms.

• Robustness and Generalizability: The approach remains effective in blackout situations, illustrating robustness against both control failures and sudden reductions in RV rates. Testing on previously unseen intersections shows good adaptability without further training, indicating the approach's generalizability across different intersection topologies.

Discussion

The implications of this research are substantial for enhancing urban traffic systems. The model-free RL approach offers a feasible path towards real-world deployment of mixed traffic controls, utilizing RVs as pivotal elements in traffic system efficiency without relying on infrastructure changes. The flexibility of RVs over static systems, such as traditional traffic signals, provides a new avenue for dynamic adaptation to fluctuating traffic states.

Figure 3: Average waiting time reduction at intersection 229.

Furthermore, the paper's strong numerical results suggest potential for significant reductions in congestion and waiting times as RV penetration increases, highlighting both practical and logistical advantages for future urban planning.

Conclusion

The study establishes a significant precedent in traffic management research by shifting focus from infrastructure-centric solutions to autonomous vehicle-centric strategies. The model demonstrates robust efficiency gains and flexibility, encouraging further exploration of decentralized intersection control using reinforcement learning frameworks. Future work includes exploring hierarchical learning designs for finer control outputs and integrating predictive traffic flow models to optimize large-scale traffic coordination. The promising results advocate for continued research and potential real-world application leveraging advancements in robot vehicles.