Multi-Agent Deep Reinforcement Learning for Large-scale Traffic Signal Control

Abstract: Reinforcement learning (RL) is a promising data-driven approach for adaptive traffic signal control (ATSC) in complex urban traffic networks, and deep neural networks further enhance its learning power. However, centralized RL is infeasible for large-scale ATSC due to the extremely high dimension of the joint action space. Multi-agent RL (MARL) overcomes the scalability issue by distributing the global control to each local RL agent, but it introduces new challenges: now the environment becomes partially observable from the viewpoint of each local agent due to limited communication among agents. Most existing studies in MARL focus on designing efficient communication and coordination among traditional Q-learning agents. This paper presents, for the first time, a fully scalable and decentralized MARL algorithm for the state-of-the-art deep RL agent: advantage actor critic (A2C), within the context of ATSC. In particular, two methods are proposed to stabilize the learning procedure, by improving the observability and reducing the learning difficulty of each local agent. The proposed multi-agent A2C is compared against independent A2C and independent Q-learning algorithms, in both a large synthetic traffic grid and a large real-world traffic network of Monaco city, under simulated peak-hour traffic dynamics. Results demonstrate its optimality, robustness, and sample efficiency over other state-of-the-art decentralized MARL algorithms.

• The paper proposes a novel decentralized multi-agent A2C algorithm for managing traffic signals efficiently through local and neighbor observations.
• The method leverages spatial discounting and fingerprinting to mitigate partial observability and reduce non-local noise.
• Experiments on synthetic grids and Monaco’s network show improved queue lengths, delays, and vehicle speeds under high traffic loads.

Multi-Agent Deep Reinforcement Learning for Large-scale Traffic Signal Control

Introduction

The paper "Multi-Agent Deep Reinforcement Learning for Large-scale Traffic Signal Control" (1903.04527) addresses the complexities of adaptive traffic signal control (ATSC) in urban environments that are facing increasing congestion levels. It proposes the use of multi-agent reinforcement learning (MARL) to manage traffic signals efficiently by distributing the control efforts across local deep reinforcement learning (DRL) agents. This approach is designed to improve scalability and optimality in managing traffic networks, particularly through the use of advantage actor-critic (A2C) methods.

Methodology

The primary challenge tackled by this paper is the feasibility of deploying centralized RL models for large-scale traffic networks due to the high dimensionality of the joint action space. MARL is suggested as the solution, where each traffic intersection is governed by a local agent, thereby promoting decentralization. The paper introduces a scalable decentralized algorithm using A2C for each agent, optimized through two methods: enhancing observability and reducing learning difficulty. The inclusion of neighbor observation and fingerprinting information helps mitigate the partial observability issue, while spatial discounting of non-local rewards and state data reduces noise and improves local performance.

Experimental Results

The proposed multi-agent A2C algorithm was compared against independent A2C and Q-learning agents in simulated environments: a synthetic traffic grid and a real-world network in Monaco. Experiments demonstrated that the multi-agent A2C achieved robust, optimal, and sample-efficient results, outperforming other decentralized MARL algorithms. Key performance metrics such as queue length, intersection delay, and vehicle speed favored A2C.

Implications and Future Work

This paper provides significant implications for practical traffic management systems. Not only does the MA2C exhibit superior handling of traffic signals under high loads, but it also suggests that incorporating local observations with neighborhood communication is crucial for stability. Future work must address deployment challenges like more realistic traffic simulations, robustness to sensor noise, and efficient communication pipelines. Practical implementations could enhance urban mobility, minimize congestion, and potentially integrate broader smart city applications.

Conclusion

In conclusion, this paper paves the way for applying advanced MARL techniques to urban traffic control systems. Its successful implementation demonstrates the potential for significant improvements in traffic signal management, leading to enhanced efficiency and reduced congestion within complex urban networks. As cities continue to grow, the need for intelligent ATSC strategies such as those proposed in this paper will become increasingly essential.