Momentum Based Reward Design for Low Emission Traffic Signal Control

Abstract: Urban traffic congestion is a growing global issue contributing significantly to long commute times and environmental pollution. Traditional traffic signal control systems often fail to adapt to dynamic traffic conditions. Adaptive traffic signal control can improve urban traffic without changing road infrastructure. Deep Reinforcement Learning (DRL) has shown strong performance for this task, but existing delay and queue-based rewards often produce short-sighted or unstable policies. This paper proposes a Momentum-Based Reward Function (MBRF) that encourages vehicles to keep moving rather than penalizing congestion alone. The method is evaluated in SUMO (Simulation of Urban MObility) using standard traffic metrics such as waiting time, queue length, throughput, and CO2 emissions. Results show that the proposed reward produces better throughput-emission trade-offs and more stable learning behavior than delay or queue-based rewards, as well as classical controllers such as Max Pressure and LQF.

• The paper introduces a momentum-based reward that incentivizes continuous vehicle motion and minimizes deceleration in high-mass vehicles.
• It formulates the traffic signal control problem as an MDP and employs a DQN algorithm to balance throughput, stability, and emission reduction.
• Experimental results demonstrate improvements in waiting times, queue lengths, throughput, and a 49% reduction in CO2 emissions over classical baselines.

Momentum-Based Reward Design for Low Emission Traffic Signal Control: An Analytical Essay

Background and Motivation

Urban traffic congestion remains a primary agent of increased commute times and environmental pollution, with traditional signal control strategies often failing in dynamic, stochastic urban settings. The paper "Momentum Based Reward Design for Low Emission Traffic Signal Control" (2605.29693) presents a novel reward function for adaptive traffic signal control using Deep Reinforcement Learning (DRL). Classical reward signals, such as delay or queue-length, are frequently responsible for unstable phase switching and fail to incentivize continuous traffic flow, leading to increased emissions and reduced intersection capacity utilization.

The momentum-based reward function (MBRF) directly operates on the physical principle of vehicle momentum, incentivizing sustained motion and penalizing unnecessary decelerations, especially in high-mass vehicles. The approach aims to promote stable phase persistence proportional to discharge efficiency, implicitly balancing throughput and emission without explicit environmental optimization terms.

Problem Formulation and Methodological Details

The traffic signal control problem is formalized as an MDP $(\mathcal{S}, \mathcal{A}, \mathcal{P}, R, \gamma)$, with the agent observing state vectors composed of phase, minimum green satisfaction, lane densities, and queue lengths. Actions correspond to signal phase selections with explicit minimum/maximum green and yellow transition constraints.

The reward design is centered around momentum:

$R_t = \frac{1}{M} \sum_{i=1}^{M} m_i v_i$

where $m_i$ is vehicle mass, $v_i$ is speed, and $M$ is the vehicle count within the intersection scope at timestep $t$. In homogeneous settings ($m_i=1$), the system incentivizes speed; in heterogeneous scenarios, real vehicle masses are used (cars, trucks, buses, motorcycles), penalizing deceleration of heavy vehicles disproportionately—leading to emission reductions.

A DQN learning algorithm is employed with standard stabilization techniques (experience replay, target network), and evaluation is conducted in SUMO under randomized traffic demands, strictly controlling for hyperparameter parity across baselines.

Figure 1: The traffic signal control problem structured as an MDP, with agent state observation, phase action selection, and momentum-based reward feedback via SUMO/TraCI.

The environment considered is a two-way single intersection scenario with multimodal road users.

Figure 2: Two-way single intersection with lane-based through, left, and right turning options.

Experimental Evaluation

Experiments systematically compare MBRF against classical (Max Pressure, LQF) and DRL baselines (waiting-time, queue-length, differential waiting rewards). Metrics include average waiting time, queue length, throughput, travel time, and total CO$_2$ emissions, computed using SUMO's emission model.

Traffic Efficiency and Congestion Dynamics

The waiting-time reward and classical controllers yield high cumulative waiting times and severe queue accumulation, with negligible congestion mitigation under high demand. In contrast, queue-based and differential waiting rewards attain lower waiting times but exhibit substantial instability and increased emissions. MBRF achieves a moderate waiting time (16.7\,s), the lowest average queue length (13.0 vehicles), and superior throughput (664 vehicles), explicitly outperforming all baselines in stability and capacity utilization.

Figure 3: Average waiting time evolution, indicating superior stability and moderate congestion under MBRF.

Figure 4: Average queue length, with MBRF exhibiting the lowest and most stable queue trajectories.

Figure 5: Throughput comparisons, MBRF achieving highest vehicle completion rates.

Travel Time and Emission Analysis

While delay-minimizing rewards produce minimal average travel time, resulting policies are unstable and environmentally detrimental. MBRF delivers significantly reduced CO$_2$ emissions (14337\,g), a 49\% decrease relative to delay-based approaches, and competitive travel times (41.9\,s), indicating successful implicit trade-off optimization.

Figure 6: Average travel time comparison, highlighting multi-objective balancing by MBRF.

Figure 7: Cumulative CO$_2$ emissions, MBRF consistently demonstrating low emissions across stochastic realizations.

Robustness and Heterogeneous Traffic

MBRF demonstrates low variance across stochastic seeds, indicating robust policy synthesis and resilience to initialization and demand fluctuation. In heterogeneous settings—where vehicle masses are directly integrated—the approach achieves highest throughput (519), lowest emissions (12545\,g), and consistent performance across all traffic metrics.

Implications and Future Directions

The results highlight the importance of reward design in DRL-based traffic signal control. MBRF's motion-centric approach implicitly enables multi-objective optimization, achieving efficiency, stability, and emission reduction without hand-tuned weights. Theoretical implications include a shift away from penalty-based reward shaping toward alignment with physical flow dynamics—thereby offering resilience against reward sparsity and instability common in classical formulations.

Practically, MBRF is compatible with both homogeneous and heterogeneous traffic, requires only vehicle speed and mass (available in most ITS deployments), and outperforms classical and DRL-based baseline controllers in real-world-relevant metrics. The approach obviates the need for explicit emission estimation models or complex reward weighting schemes.

Future developments should address scalability to multi-intersection and partially observable networks, integration of communication latency, and application to fleet management scenarios. Evaluation with model-free RL approaches (PPO, actor-critic) and broader network structures is warranted to confirm generalizability.

Conclusion

The momentum-based reward design proposed in "Momentum Based Reward Design for Low Emission Traffic Signal Control" delivers significant improvements in throughput, queue minimization, emission reduction, and training stability for adaptive traffic signal control. By prioritizing continuous vehicle movement—especially for high-mass vehicles—this reward formulation addresses key limitations of delay and queue-based objectives and advances the field toward practical, robust, and environmentally sustainable urban traffic management.