Traffic expertise meets residual RL: Knowledge-informed model-based residual reinforcement learning for CAV trajectory control

Abstract: Model-based reinforcement learning (RL) is anticipated to exhibit higher sample efficiency compared to model-free RL by utilizing a virtual environment model. However, it is challenging to obtain sufficiently accurate representations of the environmental dynamics due to uncertainties in complex systems and environments. An inaccurate environment model may degrade the sample efficiency and performance of model-based RL. Furthermore, while model-based RL can improve sample efficiency, it often still requires substantial training time to learn from scratch, potentially limiting its advantages over model-free approaches. To address these challenges, this paper introduces a knowledge-informed model-based residual reinforcement learning framework aimed at enhancing learning efficiency by infusing established expert knowledge into the learning process and avoiding the issue of beginning from zero. Our approach integrates traffic expert knowledge into a virtual environment model, employing the Intelligent Driver Model (IDM) for basic dynamics and neural networks for residual dynamics, thus ensuring adaptability to complex scenarios. We propose a novel strategy that combines traditional control methods with residual RL, facilitating efficient learning and policy optimization without the need to learn from scratch. The proposed approach is applied to CAV trajectory control tasks for the dissipation of stop-and-go waves in mixed traffic flow. Experimental results demonstrate that our proposed approach enables the CAV agent to achieve superior performance in trajectory control compared to the baseline agents in terms of sample efficiency, traffic flow smoothness and traffic mobility. The source code and supplementary materials are available at: https://zihaosheng.github.io/traffic-expertise-RL/.

• The paper presents a novel RL framework that blends traffic expertise with model-based residual learning for efficient CAV trajectory control.
• It leverages a dual-environment strategy using IDM modeling and neural network residuals to boost learning efficiency and stability.
• Experimental results show improved sample efficiency, faster convergence, and reduced stop-and-go waves in simulated mixed traffic scenarios.

Knowledge-informed Model-based Residual Reinforcement Learning for CAV Trajectory Control

Introduction

The paper introduces a novel reinforcement learning (RL) framework blending traffic expertise and model-based residual learning for controlling connected automated vehicles (CAVs) trajectory. The framework aims to enhance the learning efficiency by infusing established traffic expert knowledge and addresses challenges such as the need to learn entirely from scratch, low sample efficiency, and inaccuracies in model-based RL dynamics.

Methodology

The core of this paper is the knowledge-informed model-based residual reinforcement learning (RRL) framework. This approach combines traditional control methods with model-based RL, where the following key components are highlighted:

1. Virtual Environment Modeling: The virtual environment utilizes a traffic-informed model incorporating the Intelligent Driver Model (IDM) for basic dynamics and neural networks (NNs) for residual dynamics addressing complex, uncertain dynamics not covered by the IDM. This fosters adaptability to more varied traffic scenarios.
2. Residual Learning Strategy: The framework integrates a conventional controller as a baseline, augmented by a residual RL agent learning corrective actions for trajectory control, ensuring data efficiency and policy optimization.
3. Model-based RL with Residuals: The learning process leverages dual-environment interactions (actual and virtual) to update both policy and value functions iteratively.

The paper discusses its theoretical underpinnings, proving convergence toward optimal policies and model efficiency in policy transfer from virtual to real environments.

Implementation Details

The proposed system's implementation capitalizes on the SUMO and Flow simulation environments for constructing experimental settings. It includes tasks that simulate real CAV applications using varying mixed traffic models where CAVs and human-driven vehicles co-exist. Parameters of IDM, PI controllers, and TRPO are customized to enhance the applicability and efficiency of the RL framework.

Experimental Results

The proposed approach demonstrates a significant improvement in terms of reward, sample efficiency, and stability over baseline RL methods like SAC, PPO, and TRPO across various traffic scenarios, including ring roads, figure-eight roads, and merges.

• Performance Metrics: The paper highlights superior CAV performance across key metrics: sample efficiency, convergence rate, and stabilized traffic flow.
• Visual Insights: Space-time trajectories and velocity heat maps are provided, showcasing noticeable reduction in stop-and-go waves under the proposed strategy.

Figure 1: Space-time trajectories and velocity heat map under different CAV control strategies (IDM and ours) in three scenarios.

Computing and Training Efficiency

The computational analysis suggests a trade-off between initial computational cost and operational efficiency. Although training incurs additional upfront computational expense, the framework's offline nature ensures real-time decision-making efficiency without the continuous need for intense computational resources.

Conclusion

The knowledge-informed model-based residual RL framework presents an efficient, adaptive method for CAV control, leveraging established domain expertise and enhancing reinforcement learning with residual dynamics and model-based strategies. Future work could explore meta-learning extensions, real-world validations, and multi-agent learning environments.