CoordLight: Decentralized Traffic Control

• CoordLight is a decentralized multi-agent reinforcement learning framework that leverages queue-dynamic state encoding (QDSE) and neighbor-aware policy optimization (NAPO) for adaptive traffic signal control.
• The framework formulates traffic control as a Dec-POMDP, using per-lane queue dynamics and local information exchange to reduce both local and downstream congestion.
• Integrating GRU-based temporal modeling and attention mechanisms, CoordLight demonstrates enhanced training stability and scalability over large networks with measurable improvements in average travel times.

CoordLight is a decentralized multi-agent reinforcement learning framework for adaptive traffic signal control that combines a queue-dynamics grounded state representation, Queue Dynamic State Encoding (QDSE), with a neighbor-aware policy optimization algorithm, Neighbor-aware Policy Optimization (NAPO). It models traffic signal control over a road network as a Dec-POMDP, targets stable, scalable coordination among intersections, and is designed to improve intra-neighborhood traffic by enhancing decision-making at individual junctions as well as coordination with neighboring agents, thereby scaling up to network-level traffic optimization (Zhang et al., 25 Mar 2026).

1. Problem formulation and control objective

CoordLight formulates adaptive traffic signal control over a road network $G=(V,E)$ as a Dec-POMDP in which each intersection $i\in V$ is an agent selecting a signal phase at discrete decision times $t$, with a fixed control step of $5$ seconds and an enforced yellow of $2$ seconds on phase changes in CityFlow (Zhang et al., 25 Mar 2026). Neighbors are defined by spatial adjacency through road connectivity, so that $j\in N(i)$ if there is a link between the outgoing lanes of $i$ and incoming lanes of $j$.

Partial observability is explicit in the formulation. The global state $s^t\in S$ is the joint traffic configuration, whereas each agent receives a local observation $z_i^t$ comprising the local state $i\in V$0 and the states of its immediate neighbors $i\in V$1: $i\in V$2 In CoordLight, $i\in V$3 is the QDSE vector and $i\in V$4 are neighbors’ QDSEs obtained via local information exchange.

The action space $i\in V$5 is a finite set of collision-free signal phases $i\in V$6 that can be activated without a fixed cycle. All agents pick phases synchronously and enact them for $i\in V$7 seconds. The environment transition is governed by microscopic traffic dynamics in CityFlow coupled with signal-induced service. CoordLight uses a per-lane queue dynamics model,

$i\in V$8

where $i\in V$9 is the stopped vehicle count on incoming lane $t$0, $t$1 counts moving vehicles that will join the queue within the next control step, and $t$2 counts departures served by green.

The reward is regional and cooperative rather than purely ego-centric. The per-agent reward at time $t$3 is

$t$4

and the network-level return is

$t$5

This reward is defined so that local decisions reduce both local congestion and downstream congestion experienced by neighbors. Execution is fully decentralized, with agent $t$6 sampling $t$7. Training is on-policy with parameter sharing across agents: a single actor $t$8 and critic $t$9 are updated using trajectories collected from all agents. CoordLight does not rely on centralized critics or mixing networks; it uses a privileged local critic that ingests only local and immediate neighbor information.

2. Queue Dynamic State Encoding

QDSE is the framework’s state representation and is built from per-lane queue dynamics and short-term forecasts derived from vehicle queuing models (Zhang et al., 25 Mar 2026). Its starting point is the discrete-time queue update

$5$0

where $5$1 is stopped-vehicle count, $5$2 is arrivals-to-queue within the control step, and $5$3 is departures. Departures depend on the phase and effective green time $5$4. With saturation flow $5$5 and green time $5$6,

$5$7

subject to upstream and downstream constraints such as spillback. In CityFlow, service is capped by a throughput $5$8 per green step, so $5$9 when the movement from $2$0 is authorized in phase $2$1.

Arrivals are approximated using IDM prediction. Let $2$2 be moving vehicles currently on lane $2$3, and let $2$4 be the distance from vehicle $2$5 to the tail of the queue. With current speed $2$6 and acceleration $2$7, the predicted travel distance during step $2$8 is

$2$9

Arrivals-to-queue are then approximated as

$j\in N(i)$0

The paper notes that this underestimates arrivals when queues grow during the step, but the error is small for short steps of $j\in N(i)$1 seconds.

For an intersection $j\in N(i)$2, the final QDSE vector is

$j\in N(i)$3

Per incoming lane $j\in N(i)$4, these six features are: $j\in N(i)$5, stopped vehicle count; $j\in N(i)$6, number of entering vehicles; $j\in N(i)$7, number of leaving vehicles; $j\in N(i)$8, number of moving vehicles on lane $j\in N(i)$9; $i$0, number of vehicles closely following the foremost moving vehicle; and $i$1, distance from the queue tail to the foremost moving vehicle. The design encodes current congestion, imminent congestion, and partial movement authorization. Features are aggregated in fixed order across lanes to build $i$2.

In the neural architecture, features are normalized per lane and per intersection before embedding, and temporal dependencies are modeled via a GRU to extract trends in QDSE sequences. CoordLight does not explicitly encode turning ratios, but spillback risk is stated to be implicitly captured via $i$3 and $i$4 relative to $i$5, while waiting time correlates with $i$6 and $i$7 history via the GRU. This suggests that QDSE is intended not only as a descriptive state but also as a short-horizon predictive representation.

3. Neighbor-aware Policy Optimization

NAPO augments independent PPO with neighbor-aware attention and advantage estimation (Zhang et al., 25 Mar 2026). The coordination graph is the adjacency graph $i$8, where an edge joins intersections with direct road connections, and each agent has up to four neighbors $i$9. Observations include neighbors’ QDSEs, and the critic additionally uses neighbors’ current actions.

The actor and critic both use attention over neighbors. Let $j$0 be the embedded feature of $j$1, and $j$2 the embeddings of neighbor states. For one attention head,

$j$3

$j$4

$j$5

The context $j$6 is fused with the ego feature using a residual connection and passed through a GRU. Multi-head attention concatenates head-specific contexts.

The critic includes an action-aware attention module. Neighbors’ current actions $j$7 are embedded into $j$8, after which

$j$9

$s^t\in S$0

This action context is fused with the state representation through a residual connection and GRU to produce the value and an auxiliary queue-prediction head. Attention masks handle missing neighbors.

Advantage estimation is neighbor-aware. The temporal-difference residual is

$s^t\in S$1

where

$s^t\in S$2

Generalized advantage estimation is then

$s^t\in S$3

The paper states that using a baseline not conditioned on $s^t\in S$4 avoids bias in policy gradients, while attention-derived neighbor weighting and temporal GRU states improve robustness.

Policy optimization uses PPO with parameter sharing. The actor loss is

$s^t\in S$5

where

$s^t\in S$6

The entropy term is

$s^t\in S$7

The critic loss uses the TD target

$s^t\in S$8

with

$s^t\in S$9

Auxiliary queue prediction heads are supervised by

$z_i^t$0

The total optimization objectives are

$z_i^t$1

$z_i^t$2

4. Network architecture, communication, and training procedure

The actor takes, for agent $z_i^t$3 at time $z_i^t$4, the concatenated observation $z_i^t$5 together with one-hot positional embeddings of agent indices, forming inputs embedded to $z_i^t$6 for ego plus four neighbors (Zhang et al., 25 Mar 2026). Spatial attention uses ego queries and neighbor keys and values to compute $z_i^t$7 and context $z_i^t$8; residual fusion yields $z_i^t$9; a GRU produces $i\in V$00; and the outputs are a softmax policy over phases and an auxiliary head predicting the next-step queue vector.

The critic shares the same state embedding and spatial attention structure, producing a state representation $i\in V$01 and query $i\in V$02 for the state-action decoder. Neighbor actions are embedded and fed to the action-aware attention block to yield $i\in V$03 and $i\in V$04. After residual fusion and GRU, the critic outputs the scalar value $i\in V$05 and auxiliary queue prediction $i\in V$06.

Communication is strictly local. At each control step, agents locally read neighbors’ QDSE $i\in V$07 and neighbors’ current actions $i\in V$08 for the critic. The paper emphasizes that attention is purely learned weighting and that there is no explicit message passing beyond sharing local state and action vectors within neighborhoods. With fixed $i\in V$09, multi-head attention has per-step complexity $i\in V$10 per agent, and total complexity per update is $i\in V$11 across the network.

Training is decentralized with parameter sharing and on-policy PPO. The hyperparameters are: batch size $i\in V$12; actor learning rate $i\in V$13; critic learning rate $i\in V$14; Adam optimizer; value loss coefficient $i\in V$15; entropy coefficient $i\in V$16; prediction loss coefficient $i\in V$17; discount $i\in V$18; GAE $i\in V$19; PPO clip $i\in V$20; $i\in V$21 PPO epochs per batch; hidden dimension $i\in V$22 for linear, attention, and GRU layers. Episode length is $i\in V$23 seconds. The decision step is $i\in V$24 seconds; if the same phase repeats, it extends $i\in V$25 seconds, otherwise a $i\in V$26-second yellow and a $i\in V$27-second green are inserted to keep the $i\in V$28-second decision interval.

A simplified training loop initializes shared actor and critic parameters, resets GRU hidden states at the start of each episode, collects trajectories $i\in V$29, computes $i\in V$30 and $i\in V$31 via neighbor-aware TD and GAE, forms TD targets, and then recomputes policies and queue predictions over stored trajectories for $i\in V$32 PPO epochs before updating $i\in V$33 and $i\in V$34 with Adam.

5. Empirical evaluation

Experiments use CityFlow and three real-world traffic networks: Jinan, China, with $i\in V$35 intersections; Hangzhou, China, with $i\in V$36; and New York, USA, with $i\in V$37 intersections (Zhang et al., 25 Mar 2026). Each intersection has four incoming roads and three lanes per road. Traffic demand is evaluated over multiple flows: Jinan $i\in V$38, $i\in V$39, $i\in V$40; Hangzhou $i\in V$41, $i\in V$42; and New York $i\in V$43, $i\in V$44. Evaluation uses $i\in V$45 episodes with different seeds per trained policy. The primary metric is average travel time,

$i\in V$46

with truncation at $i\in V$47 seconds for vehicles that do not exit.

The baseline set includes conventional methods—Fixed-Time, MaxPressure, Advanced-MP—and MARL methods—CoLight, MPLight, Advanced-CoLight, Advanced-MPLight, DenseLight, and SocialLight. All methods are evaluated under a consistent control step and homogeneous policies.

The reported quantitative results are as follows.

Network / flow	CoordLight average travel time (s)	Comparison stated in the paper
Jinan $i\in V$48	199.24	vs SocialLight: 217.92
Jinan $i\in V$49	198.21	vs SocialLight: 211.75
Jinan $i\in V$50	191.05	vs SocialLight: 210.46
Hangzhou $i\in V$51	248.45	vs DenseLight*: 248.43
Hangzhou $i\in V$52	250.87	vs DenseLight*: 272.27
New York $i\in V$53	748.32	vs SocialLight: 771.92; vs CoLight: 1221.77
New York $i\in V$54	1039.15	vs SocialLight: 1106.69; vs CoLight: 1476.18

For Jinan, the paper reports improvements over SocialLight of approximately $i\in V$55 to $i\in V$56. For Hangzhou $i\in V$57, the comparison against DenseLight is described as notably $i\in V$58. For New York, unpaired t-tests versus SocialLight across seven experiments yield $i\in V$59-values $i\in V$60, significant after Bonferroni correction.

The ablations separate the contributions of representation and optimization. On $i\in V$61, QDSE is compared with VC, GP, EP, ATS, and DTSE. The paper states that QDSE reduces average queue length and its variance, increases average speed, and lowers travel time; DTSE yields smoother training but similar final performance. NAPO ablations show that removing the spatio-temporal network or the critic’s state-action decoder degrades performance and stability; CoordLight-Base, described as fully connected IPPO, lags significantly. Including neighbor actions in the critic accelerates and stabilizes training.

Robustness to sensing noise is also quantified. Adding Gaussian noise to $i\in V$62 with $i\in V$63 meters increases travel time by up to approximately $i\in V$64 on $i\in V$65. This is presented as evidence of resilience to realistic sensing imperfections.

6. Scalability, robustness, limitations, and positioning

CoordLight’s scalability is attributed to three design choices: parameter sharing across agents, local attention over constant-size neighbor sets with $i\in V$66, and fully decentralized execution without global mixers or centralized critics (Zhang et al., 25 Mar 2026). Compute per agent per step scales as $i\in V$67, while memory scales linearly with $i\in V$68 for storing local hidden states. The reported evaluation up to $i\in V$69 intersections is used to demonstrate network-level scaling.

The framework addresses several classical MARL difficulties in traffic control. Partial observability is mitigated by fusing neighbors’ QDSE through attention and encoding temporal dependencies with the GRU. Non-stationarity is addressed through parameter sharing and attention-driven neighbor weighting, which the paper states reduce variance and improve stationarity in learning targets. Demand shifts are probed across low- and high-demand settings, with performance described as stable, including travel times around $i\in V$70 seconds on high-demand Hangzhou while other baselines degrade.

The limitations are also explicit. Queue prediction under severe spillback or gridlock can be biased because IDM-based arrivals may underestimate $i\in V$71 when queues grow rapidly. Long yellow or all-red periods, or unsignalized turns, may violate service assumptions. Homogeneous policies may under-serve minor approaches under extremely skewed demand, and the paper notes that reward shaping could incorporate fairness terms. Future work is identified in heterogeneous networks and asynchronous control, continuous phase durations, explicit spillback modeling, priority vehicles, accident and closure handling, and broader domain randomization.

A plausible implication is that CoordLight’s contribution is less a single architectural novelty than a coordinated decomposition of the control problem into queue-centric local forecasting and neighbor-aware decentralized optimization. Within the scope evaluated in CityFlow, the framework is positioned as a scalable alternative to both independent-agent baselines and more communication-heavy traffic MARL schemes, while preserving decentralized execution and avoiding centralized critics or mixing networks (Zhang et al., 25 Mar 2026).