Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
194 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

SECRM-2D: RL-Based Efficient and Comfortable Route-Following Autonomous Driving with Analytic Safety Guarantees (2407.16857v1)

Published 23 Jul 2024 in cs.RO, cs.LG, and stat.ML

Abstract: Over the last decade, there has been increasing interest in autonomous driving systems. Reinforcement Learning (RL) shows great promise for training autonomous driving controllers, being able to directly optimize a combination of criteria such as efficiency comfort, and stability. However, RL- based controllers typically offer no safety guarantees, making their readiness for real deployment questionable. In this paper, we propose SECRM-2D (the Safe, Efficient and Comfortable RL- based driving Model with Lane-Changing), an RL autonomous driving controller (both longitudinal and lateral) that balances optimization of efficiency and comfort and follows a fixed route, while being subject to hard analytic safety constraints. The aforementioned safety constraints are derived from the criterion that the follower vehicle must have sufficient headway to be able to avoid a crash if the leader vehicle brakes suddenly. We evaluate SECRM-2D against several learning and non-learning baselines in simulated test scenarios, including freeway driving, exiting, merging, and emergency braking. Our results confirm that representative previously-published RL AV controllers may crash in both training and testing, even if they are optimizing a safety objective. By contrast, our controller SECRM-2D is successful in avoiding crashes during both training and testing, improves over the baselines in measures of efficiency and comfort, and is more faithful in following the prescribed route. In addition, we achieve a good theoretical understanding of the longitudinal steady-state of a collection of SECRM-2D vehicles.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (24)
  1. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, 2015. [Online]. Available: https://doi.org/10.1038/nature14236
  2. N. Lazic, C. Boutilier, T. Lu, E. Wong, B. Roy, M. Ryu, and G. Imwalle, “Data center cooling using model-predictive control,” Advances in Neural Information Processing Systems, vol. 31, 2018.
  3. G. Li, Y. Yang, S. Li, X. Qu, N. Lyu, and S. E. Li, “Decision making of autonomous vehicles in lane change scenarios: Deep reinforcement learning approaches with risk awareness,” Transportation Research Part C: Emerging Technologies, vol. 134, p. 103452, 2022.
  4. O. Elsamadisy, T. Shi, I. Smirnov, and B. Abdulhai, “Safe, efficient and comfortable reinforcement-learning-based car-following for AVs with analytic safety guarantee and dynamic target speed,” Transportation Research Record, vol. 2678(1), pp. 643–661, 2024.
  5. S. Zhang, H. Peng, S. Nageshrao, and E. Tseng, “Discretionary lane change decision making using reinforcement learning with model-based exploration,” in 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA).   IEEE, 2019, pp. 844–850.
  6. C. Xi, T. Shi, Y. Wu, and L. Sun, “Efficient motion planning for automated lane change based on imitation learning and mixed-integer optimization,” in 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC).   IEEE, 2020, pp. 1–6.
  7. T. Pan, W. H. Lam, A. Sumalee, and R. Zhong, “Modeling the impacts of mandatory and discretionary lane-changing maneuvers,” Transportation research part C: emerging technologies, vol. 68, pp. 403–424, 2016.
  8. T. Shi, P. Wang, X. Cheng, C.-Y. Chan, and D. Huang, “Driving decision and control for automated lane change behavior based on deep reinforcement learning,” in 2019 IEEE intelligent transportation systems conference (ITSC).   IEEE, 2019, pp. 2895–2900.
  9. C.-J. Hoel, K. Driggs-Campbell, K. Wolff, L. Laine, and M. J. Kochenderfer, “Combining planning and deep reinforcement learning in tactical decision making for autonomous driving,” IEEE transactions on intelligent vehicles, vol. 5, no. 2, pp. 294–305, 2019.
  10. Z. Cao, D. Yang, S. Xu, H. Peng, B. Li, S. Feng, and D. Zhao, “Highway exiting planner for automated vehicles using reinforcement learning,” IEEE Transactions on Intelligent Transportation Systems, vol. 22, no. 2, pp. 990–1000, 2020.
  11. F. Ye, X. Cheng, P. Wang, C.-Y. Chan, and J. Zhang, “Automated lane change strategy using proximal policy optimization-based deep reinforcement learning,” in 2020 IEEE Intelligent Vehicles Symposium (IV).   IEEE, 2020, pp. 1746–1752.
  12. S. Udatha, Y. Lyu, and J. Dolan, “Reinforcement learning with probabilistically safe control barrier functions for ramp merging,” in 2023 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2023, pp. 5625–5630.
  13. B. Vanholme, D. Gruyer, B. Lusetti, S. Glaser, and S. Mammar, “Highly automated driving on highways based on legal safety,” IEEE Transactions on Intelligent Transportation Systems, vol. 14, no. 1, pp. 333–347, 2013.
  14. M. Zhu, Y. Wang, Z. Pu, J. Hu, X. Wang, and R. Ke, “Safe, efficient, and comfortable velocity control based on reinforcement learning for autonomous driving,” Transportation Research Part C: Emerging Technologies, vol. 117, p. 102662, 2020.
  15. Y.-T. Yen, J.-J. Chou, C.-S. Shih, C.-W. Chen, and P.-K. Tsung, “Proactive car-following using deep-reinforcement learning,” in 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC).   IEEE, 2020, pp. 1–6.
  16. T. Shi, Y. Ai, O. ElSamadisy, and B. Abdulhai, “Bilateral deep reinforcement learning approach for better-than-human car-following,” in 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC).   IEEE, 2022, pp. 3986–3992.
  17. P. Gipps, “A behavioural car-following model for computer simulation,” Transportation Research Part B: Methodological, vol. 15, no. 2, pp. 105–111, 1981.
  18. W. Zhao, T. He, R. Chen, T. Wei, and C. Liu, “State-wise safe reinforcement learning: A survey,” arXiv preprint arXiv:2302.03122, 2023.
  19. T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” in 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, Y. Bengio and Y. LeCun, Eds., 2016. [Online]. Available: http://arxiv.org/abs/1509.02971
  20. P. Á. López, M. Behrisch, L. Bieker-Walz, J. Erdmann, Y.-P. Flötteröd, R. Hilbrich, L. Lücken, J. Rummel, P. Wagner, and E. WieBner, “Microscopic traffic simulation using SUMO,” 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pp. 2575–2582, 2018.
  21. Treiber, Hennecke, and Helbing, “Congested traffic states in empirical observations and microscopic simulations,” Physical Review E: Statistical Physics, Plasmas, Fluids, and Related Interdisciplinary Topics, vol. 62 2 Pt A, pp. 1805–24, 2000.
  22. A. Kesting, M. Treiber, and D. Helbing, “General lane-changing model MOBIL for car-following models,” Transportation Research Record, vol. 1999(1), pp. 86–94, 2007.
  23. J. Erdmann, “SUMO’s lane-changing model,” in Modeling Mobility with Open Data: 2nd SUMO Conference 2014 Berlin, Germany, May 15-16, 2014.   Springer, 2015, pp. 105–123.
  24. T. Shi, “SECRM-2D demo,” 2023, https://youtu.be/J5UyJXGAGX0, Last accessed May 20, 2023.

Summary

  • The paper introduces an RL-based framework that integrates hard analytic safety constraints for both lateral and longitudinal vehicle control.
  • It unifies discretionary and mandatory lane changes into a single optimization objective to enhance efficiency and comfort.
  • Simulation results in SUMO demonstrate robust performance with a 0% crash rate and improved route adherence under varied traffic conditions.

SECRM-2D: RL-Based Efficient and Comfortable Autonomous Driving with Analytical Safety Guarantees

The task of developing autonomous vehicles (AVs) that can drive safely and efficiently in dynamic environments poses significant challenges. Among various techniques, Reinforcement Learning (RL) has shown considerable potential for optimizing multiple performance metrics of AVs, such as efficiency, comfort, and stability. However, the lack of explicit safety guarantees often hinders the deployment of RL-based controllers in real-world scenarios. This paper presents SECRM-2D, a robust RL-based model that aims to bridge this gap by integrating hard analytic safety constraints applicable to both longitudinal and lateral vehicle control.

Core Contributions

SECRM-2D introduces a decision-making framework for autonomous vehicles that simultaneously optimizes efficiency, comfort, and route adherence, all while ensuring compliance with strict safety protocols. Key innovations include:

  • Analytic Safety Constraints: The paper extends previous longitudinal safety constraints to lateral (lane-changing) scenarios, establishing a model that supports both discretionary and mandatory lane changes. Safety constraints are derived from principles akin to the Vienna convention on road traffic, providing explicit certifiable safety guarantees.
  • Unified Lane Change Framework: The model uniquely handles discretionary and mandatory lane changes within a single optimization objective, enhancing its applicability in diverse traffic scenarios.
  • Steady-State Platooning Analysis: The authors present a theoretical analysis of a platoon of SECRM-2D controlled vehicles reaching a steady-state, thus offering insights into long-term behavior under varying traffic conditions. The researchers derive explicit formulas linking the gap between vehicles to platoon speed, implicating broader traffic flow implications.

Experimental Evaluation and Insights

The evaluation comprises simulated environments modeled in the SUMO microscopic traffic simulator. The operational scope tested includes both a simple loop network and a more complex freeway structure based on real-world geometries.

  • Discretionary Lane-Change Scenarios: In a basic loop network, SECRM-2D demonstrated notable improvements in both efficiency and comfort over baseline models like IDM+MOBIL, Gipps with Greedy Lane Selection, and a PPO-based lane-change algorithm. SECRM-2D consistently maintained a 0% crash rate, illustrating its robustness in diverse traffic conditions, including emergency braking.
  • Route-Following Scenarios: SECRM-2D navigated a freeway interchange with high efficacy, showcasing the framework's capability to generalize route adherence across varied network conditions. The controller leveraged its RL architecture to learn effective bypassing tactics in congested traffic, thus outperforming several baselines in both speed stability and safety adherence.

Implications for Autonomous Driving

The paper's results reinforce the viability of combining RL with explicit safety guarantees in training AV controllers. By addressing both the efficiency-comfort trade-off and the critical issue of safety, SECRM-2D contributes to unlocking possibilities for safer real-world deployment of autonomous vehicles.

Future Directions

The broader implications of SECRM-2D suggest several intriguing avenues for future research. While the current model effectively utilizes high-level analytic safety constraints, further work could explore more granular safety improvements, potentially addressing cases with uncertain sensor readings or variable reaction times among mixed traffic participants. Additionally, extending theoretical analyses to cover more complex interaction scenarios, such as multi-agent systems or mixed human and autonomous traffic environments, would provide even more comprehensive insights.

Combining RL frameworks with explicit safety constraints offers a promising path toward the practical realization of autonomous driving. SECRM-2D exemplifies how rigorous safety protocols can be harmonized with advanced RL strategies to create robust, deployable AV systems.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube