Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
157 tokens/sec
GPT-4o
43 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Parameterized Decision-making with Multi-modal Perception for Autonomous Driving (2312.11935v1)

Published 19 Dec 2023 in cs.AI

Abstract: Autonomous driving is an emerging technology that has advanced rapidly over the last decade. Modern transportation is expected to benefit greatly from a wise decision-making framework of autonomous vehicles, including the improvement of mobility and the minimization of risks and travel time. However, existing methods either ignore the complexity of environments only fitting straight roads, or ignore the impact on surrounding vehicles during optimization phases, leading to weak environmental adaptability and incomplete optimization objectives. To address these limitations, we propose a parameterized decision-making framework with multi-modal perception based on deep reinforcement learning, called AUTO. We conduct a comprehensive perception to capture the state features of various traffic participants around the autonomous vehicle, based on which we design a graph-based model to learn a state representation of the multi-modal semantic features. To distinguish between lane-following and lane-changing, we decompose an action of the autonomous vehicle into a parameterized action structure that first decides whether to change lanes and then computes an exact action to execute. A hybrid reward function takes into account aspects of safety, traffic efficiency, passenger comfort, and impact to guide the framework to generate optimal actions. In addition, we design a regularization term and a multi-worker paradigm to enhance the training. Extensive experiments offer evidence that AUTO can advance state-of-the-art in terms of both macroscopic and microscopic effectiveness.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (67)
  1. D. Schrank, L. Albert, B. Eisele, and T. Lomax, “2021 urban mobility report,” Texas A&M Transportation Institute, Tech. Rep., 2021.
  2. M. Won, T. Park, and S. H. Son, “Toward mitigating phantom jam using vehicle-to-vehicle communication,” IEEE transactions on intelligent transportation systems, pp. 1313–1324, 2016.
  3. X. Qu, Y. Yu, M. Zhou, C.-T. Lin, and X. Wang, “Jointly dampening traffic oscillations and improving energy consumption with electric, connected and automated vehicles: a reinforcement learning based approach,” Applied Energy, p. 114030, 2020.
  4. S. E. Merriman, K. L. Plant, K. M. Revell, and N. A. Stanton, “Challenges for automated vehicle driver training: a thematic analysis from manual and automated driving,” Transportation research part F: traffic psychology and behaviour, pp. 238–268, 2021.
  5. B. R. Kiran, I. Sobh, V. Talpaert, P. Mannion, A. A. Al Sallab, S. Yogamani, and P. Pérez, “Deep reinforcement learning for autonomous driving: A survey,” IEEE Transactions on Intelligent Transportation Systems, 2021.
  6. E. Leurent and J. Mercat, “Social attention for autonomous decision-making in dense traffic,” arXiv preprint arXiv:1911.12250, 2019.
  7. S. Aradi, “Survey of deep reinforcement learning for motion planning of autonomous vehicles,” IEEE Transactions on Intelligent Transportation Systems, 2020.
  8. S. Nageshrao, H. E. Tseng, and D. Filev, “Autonomous highway driving using deep reinforcement learning,” in IEEE SMC, 2019, pp. 2326–2331.
  9. L. Xiao, M. Wang, and B. Van Arem, “Realistic car-following models for microscopic simulation of adaptive and cooperative adaptive cruise control vehicles,” Transportation Research Record, pp. 1–9, 2017.
  10. J. Erdmann, “Sumo’s lane-changing model,” in Modeling Mobility with Open Data, 2015, pp. 105–123.
  11. V. Milanés and S. E. Shladover, “Modeling cooperative and autonomous adaptive cruise control dynamic responses using experimental data,” Transportation Research Part C: Emerging Technologies, pp. 285–300, 2014.
  12. A. Prakash, K. Chitta, and A. Geiger, “Multi-modal fusion transformer for end-to-end autonomous driving,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 7077–7087.
  13. J. Chen, B. Yuan, and M. Tomizuka, “Deep imitation learning for autonomous driving in generic urban scenarios with enhanced safety,” in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019, pp. 2884–2890.
  14. D. A. Tedjopurnomo, Z. Bao, B. Zheng, F. M. Choudhury, and A. K. Qin, “A survey on modern deep neural network for traffic prediction: Trends, methods and challenges,” IEEE Transactions on Knowledge and Data Engineering (TKDE), pp. 1544–1561, 2020.
  15. W. Fu, Y. Li, Z. Ye, and Q. Liu, “Decision making for autonomous driving via multimodal transformer and deep reinforcement learning,” in 2022 IEEE International Conference on Real-time Computing and Robotics (RCAR).   IEEE, 2022, pp. 481–486.
  16. S. Liu, Y. Xia, C. Xu, J. Xie, H. Su, and K. Zheng, “Impact-aware maneuver decision with enhanced perception for autonomous vehicle,” in 2023 IEEE 39th International Conference on Data Engineering (ICDE), 2023.
  17. Ó. Pérez-Gil, R. Barea, E. López-Guillén, L. M. Bergasa, C. Gómez-Huélamo, R. Gutiérrez, and A. Díaz-Díaz, “Deep reinforcement learning based control for autonomous vehicles in carla,” Multimedia Tools and Applications, pp. 3553–3576, 2022.
  18. M. Zhu, Y. Wang, Z. Pu, J. Hu, X. Wang, and R. Ke, “Safe, efficient, and comfortable velocity control based on reinforcement learning for autonomous driving,” Transportation Research Part C: Emerging Technologies, p. 102662, 2020.
  19. Z. Xu, S. Liu, Z. Wu, X. Chen, K. Zeng, K. Zheng, and H. Su, “Patrol: A velocity control framework for autonomous vehicle via spatial-temporal reinforcement learning,” in Proceedings of the 30th ACM International Conference on Information & Knowledge Management (CIKM), 2021, pp. 2271–2280.
  20. A. Talebpour and H. S. Mahmassani, “Influence of connected and autonomous vehicles on traffic flow stability and throughput,” Transportation Research Part C: Emerging Technologies, pp. 143–163, 2016.
  21. R. E. Stern, S. Cui, M. L. Delle Monache, R. Bhadani, M. Bunting, M. Churchill, N. Hamilton, H. Pohlmann, F. Wu, B. Piccoli et al., “Dissipation of stop-and-go waves via control of autonomous vehicles: Field experiments,” Transportation Research Part C: Emerging Technologies, pp. 205–221, 2018.
  22. C. Gómez-Huélamo, A. Diaz-Diaz, J. Araluce, M. E. Ortiz, R. Gutiérrez, F. Arango, Á. Llamazares, and L. M. Bergasa, “How to build and validate a safe and reliable autonomous driving stack? a ros based software modular architecture baseline,” in 2022 IEEE Intelligent Vehicles Symposium (IV), 2022, pp. 1282–1289.
  23. D. Feng, C. Haase-Schütz, L. Rosenbaum, H. Hertlein, C. Glaeser, F. Timm, W. Wiesbeck, and K. Dietmayer, “Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges,” IEEE Transactions on Intelligent Transportation Systems, pp. 1341–1360, 2020.
  24. C. Gómez-Huélamo, J. D. Egido, L. M. Bergasa, R. Barea, E. López-Guillén, F. Arango, J. Araluce, and J. López, “Train here, drive there: Simulating real-world use cases with fully-autonomous driving architecture in carla simulator,” in Workshop of Physical Agents.   Springer, 2020, pp. 44–59.
  25. M.-F. Chang, J. Lambert, P. Sangkloy, J. Singh, S. Bak, A. Hartnett, D. Wang, P. Carr, S. Lucey, D. Ramanan et al., “Argoverse: 3d tracking and forecasting with rich maps,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 8748–8757.
  26. R. Gutiérrez, E. López-Guillén, L. M. Bergasa, R. Barea, Ó. Pérez, C. Gómez-Huélamo, F. Arango, J. Del Egido, and J. López-Fernández, “A waypoint tracking controller for autonomous road vehicles using ros framework,” Sensors, p. 4062, 2020.
  27. Y. Liu, Y. Gao, Q. Zhang, D. Ding, and D. Zhao, “Multi-task safe reinforcement learning for navigating intersections in dense traffic,” Journal of the Franklin Institute, 2022.
  28. Y. Chen, C. Dong, P. Palanisamy, P. Mudalige, K. Muelling, and J. M. Dolan, “Attention-based hierarchical deep reinforcement learning for lane change behaviors in autonomous driving,” in CVPR Workshops, 2019, pp. 0–0.
  29. K. Zhu and T. Zhang, “Deep reinforcement learning based mobile robot navigation: A review,” Tsinghua Science and Technology, pp. 674–691, 2021.
  30. D. Horgan, J. Quan, D. Budden, G. Barth-Maron, M. Hessel, H. Van Hasselt, and D. Silver, “Distributed prioritized experience replay,” arXiv preprint arXiv:1803.00933, 2018.
  31. M. I. Ribeiro, “Kalman and extended kalman filters: Concept, derivation and properties,” Institute for Systems and Robotics, p. 46, 2004.
  32. A. Diaz-Diaz, M. Ocaña, Á. Llamazares, C. Gómez-Huélamo, P. Revenga, and L. M. Bergasa, “Hd maps: Exploiting opendrive potential for path planning and map monitoring,” in 2022 IEEE Intelligent Vehicles Symposium (IV), 2022, pp. 1211–1217.
  33. Z. Lari, A. Habib, and E. Kwak, “An adaptive approach for segmentation of 3d laser point cloud,” International archives of the photogrammetry, remote sensing and spatial information sciences, vol. 38, no. 5, p. W12, 2011.
  34. C.-Y. Wang, A. Bochkovskiy, and H.-Y. M. Liao, “Yolov7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 7464–7475.
  35. S. Liu, H. Su, Y. Zhao, K. Zeng, and K. Zheng, “Lane change scheduling for autonomous vehicle: A prediction-and-search framework,” in Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021, pp. 3343–3353.
  36. M. Fu, T. Zhang, W. Song, Y. Yang, and M. Wang, “Trajectory prediction-based local spatio-temporal navigation map for autonomous driving in dynamic highway environments,” IEEE Transactions on Intelligent Transportation Systems, 2021.
  37. H. Cui, V. Radosavljevic, F.-C. Chou, T.-H. Lin, T. Nguyen, T.-K. Huang, J. Schneider, and N. Djuric, “Multimodal trajectory predictions for autonomous driving using deep convolutional networks,” in 2019 International Conference on Robotics and Automation (ICRA), 2019, pp. 2090–2096.
  38. W. Zeng, C. Lin, K. Liu, J. Lin, and A. K. Tung, “Modeling spatial nonstationarity via deformable convolutions for deep traffic flow prediction,” IEEE Transactions on Knowledge and Data Engineering (TKDE), 2021.
  39. J. Gao, C. Sun, H. Zhao, Y. Shen, D. Anguelov, C. Li, and C. Schmid, “Vectornet: Encoding hd maps and agent dynamics from vectorized representation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 11 525–11 533.
  40. B. Varadarajan, A. Hefny, A. Srivastava, K. S. Refaat, N. Nayakanti, A. Cornman, K. Chen, B. Douillard, C. P. Lam, D. Anguelov et al., “Multipath++: Efficient information fusion and trajectory aggregation for behavior prediction,” in 2022 International Conference on Robotics and Automation (ICRA), 2022, pp. 7814–7821.
  41. M. Liang, B. Yang, R. Hu, Y. Chen, R. Liao, S. Feng, and R. Urtasun, “Learning lane graph representations for motion forecasting,” in European Conference on Computer Vision, 2020, pp. 541–556.
  42. S. Morad, R. Kortvelesy, M. Bettini, S. Liwicki, and A. Prorok, “Popgym: Benchmarking partially observable reinforcement learning,” in The Eleventh International Conference on Learning Representations.
  43. J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Empirical evaluation of gated recurrent neural networks on sequence modeling,” arXiv preprint arXiv:1412.3555, 2014.
  44. X. Chen, H. Zhang, F. Zhao, Y. Cai, H. Wang, and Q. Ye, “Vehicle trajectory prediction based on intention-aware non-autoregressive transformer with multi-attention learning for internet of vehicles,” IEEE Transactions on Instrumentation and Measurement, pp. 1–12, 2022.
  45. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al., “Human-level control through deep reinforcement learning,” nature, pp. 529–533, 2015.
  46. C. Pérez-D’Arpino, C. Liu, P. Goebel, R. Martín-Martín, and S. Savarese, “Robot navigation in constrained pedestrian environments using reinforcement learning,” in IEEE ICRA, 2021, pp. 1140–1146.
  47. X. Xiao, B. Liu, G. Warnell, and P. Stone, “Motion planning and control for mobile robot navigation using machine learning: a survey,” Autonomous Robots, pp. 569–597, 2022.
  48. C. J. Watkins and P. Dayan, “Q-learning,” Machine learning, pp. 279–292, 1992.
  49. T. Shang, G. Lian, Y. Zhao, X. Liu, and W. Wang, “Off-ramp vehicle mandatory lane-changing duration in small spacing section of tunnel-interchange section based on survival analysis,” Journal of Advanced Transportation, 2022.
  50. Z. Huang, Z. Zhang, H. Li, L. Qin, and J. Rong, “Determining appropriate lane-changing spacing for off-ramp areas of urban expressways,” Sustainability, p. 2087, 2019.
  51. Y. Xia, S. Liu, X. Chen, Z. Xu, K. Zheng, and H. Su, “Rise: A velocity control framework with minimal impacts based on reinforcement learning,” in Proceedings of the 31st ACM International Conference on Information & Knowledge Management (CIKM), 2022, pp. 2210–2219.
  52. S. Fujimoto and S. S. Gu, “A minimalist approach to offline reinforcement learning,” Advances in neural information processing systems, vol. 34, pp. 20 132–20 145, 2021.
  53. D. Brandfonbrener, W. Whitney, R. Ranganath, and J. Bruna, “Offline rl without off-policy evaluation,” Advances in neural information processing systems, vol. 34, pp. 4933–4946, 2021.
  54. I. Kostrikov, A. Nair, and S. Levine, “Offline reinforcement learning with implicit q-learning,” arXiv preprint arXiv:2110.06169, 2021.
  55. T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” arXiv preprint arXiv:1509.02971, 2015.
  56. A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “CARLA: An open urban driving simulator,” in Proceedings of the 1st Annual Conference on Robot Learning, 2017, pp. 1–16.
  57. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  58. N. Djuric, V. Radosavljevic, H. Cui, T. Nguyen, F.-C. Chou, T.-H. Lin, and J. Schneider, “Short-term motion prediction of traffic actors for autonomous driving using deep convolutional networks,” arXiv preprint arXiv:1808.05819, p. 6, 2018.
  59. W. Masson, P. Ranchod, and G. Konidaris, “Reinforcement learning with parameterized actions,” in Thirtieth AAAI Conference on Artificial Intelligence, 2016.
  60. M. Hausknecht and P. Stone, “Deep reinforcement learning in parameterized action space,” arXiv preprint arXiv:1511.04143, 2015.
  61. I. Syarif, A. Prugel-Bennett, and G. Wills, “Svm parameter optimization using grid search and genetic algorithm to improve classification performance,” TELKOMNIKA (Telecommunication Computing Electronics and Control), vol. 14, no. 4, pp. 1502–1509, 2016.
  62. X. Jia, P. Wu, L. Chen, J. Xie, C. He, J. Yan, and H. Li, “Think twice before driving: Towards scalable decoders for end-to-end autonomous driving,” in CVPR, 2023.
  63. D. Coelho and M. Oliveira, “A review of end-to-end autonomous driving in urban environments,” IEEE Access, vol. 10, pp. 75 296–75 311, 2022.
  64. P. Wu, X. Jia, L. Chen, J. Yan, H. Li, and Y. Qiao, “Trajectory-guided control prediction for end-to-end autonomous driving: A simple yet strong baseline,” Advances in Neural Information Processing Systems, vol. 35, pp. 6119–6132, 2022.
  65. J. Wang, Q. Zhang, D. Zhao, and Y. Chen, “Lane change decision-making through deep reinforcement learning with rule-based constraints,” in 2019 International Joint Conference on Neural Networks (IJCNN).   IEEE, 2019, pp. 1–6.
  66. H. Krasowski, X. Wang, and M. Althoff, “Safe reinforcement learning for autonomous lane changing using set-based prediction,” in 2020 IEEE 23rd international conference on Intelligent Transportation Systems (ITSC).   IEEE, 2020, pp. 1–7.
  67. K. Muhammad, A. Ullah, J. Lloret, J. Del Ser, and V. H. C. de Albuquerque, “Deep learning for safe autonomous driving: Current challenges and future directions,” IEEE Transactions on Intelligent Transportation Systems, vol. 22, no. 7, pp. 4316–4336, 2020.
Citations (40)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now