Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

SocialFormer: Social Interaction Modeling with Edge-enhanced Heterogeneous Graph Transformers for Trajectory Prediction (2405.03809v1)

Published 6 May 2024 in cs.AI

Abstract: Accurate trajectory prediction is crucial for ensuring safe and efficient autonomous driving. However, most existing methods overlook complex interactions between traffic participants that often govern their future trajectories. In this paper, we propose SocialFormer, an agent interaction-aware trajectory prediction method that leverages the semantic relationship between the target vehicle and surrounding vehicles by making use of the road topology. We also introduce an edge-enhanced heterogeneous graph transformer (EHGT) as the aggregator in a graph neural network (GNN) to encode the semantic and spatial agent interaction information. Additionally, we introduce a temporal encoder based on gated recurrent units (GRU) to model the temporal social behavior of agent movements. Finally, we present an information fusion framework that integrates agent encoding, lane encoding, and agent interaction encoding for a holistic representation of the traffic scene. We evaluate SocialFormer for the trajectory prediction task on the popular nuScenes benchmark and achieve state-of-the-art performance.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. Y. Huang, J. Du, Z. Yang, Z. Zhou, L. Zhang, and H. Chen, “A survey on trajectory-prediction methods for autonomous driving,” IEEE Transactions on Intelligent Vehicles, vol. 7, pp. 652–674, 2022.
  2. Z. Ding and H. Zhao, “Incorporating driving knowledge in deep learning based vehicle trajectory prediction: A survey,” IEEE Transactions on Intelligent Vehicles, vol. 8, pp. 3996–4015, 2023.
  3. Y. Hu, A. Nakhaei, M. Tomizuka, and K. Fujimura, “Interaction-aware decision making with adaptive strategies under merging scenarios,” in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2019, pp. 151–158.
  4. L. Sur, C. Tang, Y. Niu, E. Sachdeva, C. Choi, T. Misu, M. Tomizuka, and W. Zhan, “Domain knowledge driven pseudo labels for interpretable goal-conditioned interactive trajectory prediction,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022.
  5. S. Kumar, Y. Gu, J. Hoang, G. C. Haynes, and M. Marchetti-Bowick, “Interaction-based trajectory prediction over a hybrid traffic graph,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2021, pp. 5530–5535.
  6. E. M. Rella, J.-N. Zaech, A. Liniger, and L. Van Gool, “Decoder fusion rnn: Context and interaction aware decoders for trajectory prediction,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2021, pp. 5937–5943.
  7. X. Jia, P. Wu, L. Chen, Y. Liu, H. Li, and J. Yan, “Hdgt: Heterogeneous driving graph transformer for multi-agent trajectory prediction via scene encoding,” IEEE transactions on pattern analysis and machine intelligence, 2023.
  8. H. Cui, V. Radosavljevic, F.-C. Chou, T.-H. Lin, T. Nguyen, T.-K. Huang, J. Schneider, and N. Djuric, “Multimodal trajectory predictions for autonomous driving using deep convolutional networks,” in Int. Conference on Robotics and Automation (ICRA).   IEEE, 2019.
  9. Y. Chai, B. Sapp, M. Bansal, and D. Anguelov, “Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction,” in Conference on Robot Learning, 2019.
  10. X. Mo, Y. Xing, and C. Lv, “Recog: A deep learning framework with heterogeneous graph for interaction-aware trajectory prediction,” CoRR, vol. abs/2012.05032, 2020.
  11. Y. Yuan, X. Weng, Y. Ou, and K. M. Kitani, “Agentformer: Agent-aware transformers for socio-temporal multi-agent forecasting,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 9813–9823.
  12. J. Gao, C. Sun, H. Zhao, Y. Shen, D. Anguelov, C. Li, and C. Schmid, “Vectornet: Encoding hd maps and agent dynamics from vectorized representation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 11 525–11 533.
  13. M. Liang, B. Yang, R. Hu, Y. Chen, R. Liao, S. Feng, and R. Urtasun, “Learning lane graph representations for motion forecasting,” in Computer Vision–ECCV: 16th European Conference, Glasgow, Proceedings, Part II 16.   Springer, 2020.
  14. N. Deo, E. Wolff, and O. Beijbom, “Multimodal trajectory prediction conditioned on lane-graph traversals,” in Conference on Robot Learning.   PMLR, 2022, pp. 203–212.
  15. X. Li, X. Ying, and M. C. Chuah, “Grip: Graph-based interaction-aware trajectory prediction,” in 2019 IEEE Intelligent Transportation Systems Conference (ITSC).   IEEE, 2019, pp. 3960–3966.
  16. H. Jeon, J. Choi, and D. Kum, “Scale-net: Scalable vehicle trajectory prediction network under random number of interacting vehicles via edge-enhanced graph convolutional neural network,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2020, pp. 2095–2102.
  17. H. Zhao, J. Gao, T. Lan, C. Sun, B. Sapp, B. Varadarajan, Y. Shen, Y. Shen, Y. Chai, C. Schmid et al., “Tnt: Target-driven trajectory prediction,” in Conference on Robot Learning.   PMLR, 2021, pp. 895–904.
  18. X. Mo, Z. Huang, Y. Xing, and C. Lv, “Multi-agent trajectory prediction with heterogeneous edge-enhanced graph attention network,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 7, pp. 9554–9567, 2022.
  19. M. Zipfl, F. Hertlein, A. Rettinger, S. Thoma, L. Halilaj, J. Luettin, S. Schmid, and C. Henson, “Relation-based motion prediction using traffic scene graphs,” in 25th International Conference on Intelligent Transportation Systems (ITSC).   IEEE, 2022.
  20. H. Berkemeyer, R. Franceschini, T. Tran, L. Che, and G. Pipa, “Feasible and adaptive multimodal trajectory prediction with semantic maneuver fusion,” in International Conference on Robotics and Automation (ICRA).   IEEE, 2021.
  21. M. Bansal, A. Krizhevsky, and A. Ogale, “Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst,” arXiv preprint arXiv:1812.03079, 2018.
  22. J. Hong, B. Sapp, and J. Philbin, “Rules of the road: Predicting driving behavior with a convolutional model of semantic interactions,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 8454–8462.
  23. L. Halilaj, I. Dindorkar, J. Lüttin, and S. Rothermel, “A knowledge graph-based approach for situation comprehension in driving scenarios,” in The Semantic Web: 18th International Conference, ESWC Proceedings.   Springer, 2021.
  24. J. Gu, C. Sun, and H. Zhao, “Densetnt: End-to-end trajectory prediction from dense goal sets,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
  25. Z. Huang, X. Mo, and C. Lv, “Multi-modal motion prediction with transformer-based neural network for autonomous driving,” in Int. Conference on Robotics and Automation (ICRA).   IEEE, 2022.
  26. M. Liu, H. Cheng, L. Chen, H. Broszio, J. Li, R. Zhao, M. Sester, and M. Y. Yang, “Laformer: Trajectory prediction for autonomous driving with lane-aware scene constraints,” CoRR, vol. abs/2302.13933, 2023.
  27. D. Park, H. Ryu, Y. Yang, J. Cho, J. Kim, and K. Yoon, “Leveraging future relationship reasoning for vehicle trajectory prediction,” in The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023.   OpenReview.net, 2023.
  28. P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio, “Graph attention networks,” arXiv preprint arXiv:1710.10903, 2017.
  29. D. Grimm, P. Schörner, M. Dreßler, and J.-M. Zöllner, “Holistic graph-based motion prediction,” in International Conference on Robotics and Automation (ICRA).   IEEE, 2023.
  30. Z. Sheng, Y. Xu, S. Xue, and D. Li, “Graph-based spatial-temporal convolutional network for vehicle trajectory prediction in autonomous driving,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 10, pp. 17 654–17 665, 2022.
  31. D. Cao, J. Li, H. Ma, and M. Tomizuka, “Spectral temporal graph neural network for trajectory prediction,” in International Conference on Robotics and Automation (ICRA).   IEEE, 2021.
  32. D. Grimm, M. Zipfl, F. Hertlein, A. Naumann, J. Lüttin, S. Thoma, S. Schmid, L. Halilaj, A. Rettinger, and J. M. Zöllner, “Heterogeneous graph-based trajectory prediction using local map context and social interactions,” 2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC), pp. 2901–2907, 2023.
  33. M. Zipfl and J. M. Zöllner, “Towards traffic scene description: The semantic scene graph,” in 25th International Conference on Intelligent Transportation Systems (ITSC).   IEEE, 2022, pp. 3748–3755.
  34. L. Mlodzian, Z. Sun, H. Berkemeyer, S. Monka, Z. Wang, S. Dietze, L. Halilaj, and J. Luettin, “nuscenes knowledge graph-a comprehensive semantic representation of traffic scenes for trajectory prediction,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 42–52.
  35. Z. Hu, Y. Dong, K. Wang, and Y. Sun, “Heterogeneous graph transformer,” in Proceedings of the Web Conference, 2020, pp. 2704–2710.
  36. H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, “nuscenes: A multimodal dataset for autonomous driving,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 11 621–11 631.
  37. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al., “Pytorch: An imperative style, high-performance deep learning library,” Advances in neural information processing systems, vol. 32, 2019.
  38. M. Fey and J. E. Lenssen, “Fast graph representation learning with PyTorch Geometric,” in ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
  39. I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in International Conference on Learning Representations, 2017.
  40. T. Phan-Minh, E. C. Grigore, F. A. Boulton, O. Beijbom, and E. M. Wolff, “Covernet: Multimodal behavior prediction using trajectory sets,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 14 074–14 083.
  41. N. Deo and M. M. Trivedi, “Trajectory forecasts in unknown environments conditioned on grid-based plans,” arXiv preprint arXiv:2001.00735, 2020.
  42. T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone, “Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data,” in Computer Vision–ECCV: 16th European Conference, Glasgow, Proceedings, Part XVIII 16.   Springer, 2020.
  43. R. Girgis, F. Golemo, F. Codevilla, M. Weiss, J. A. D’Souza, S. E. Kahou, F. Heide, and C. Pal, “Latent variable sequential set transformers for joint multi-agent motion prediction,” in International Conference on Learning Representations, ICLR 2022, 2022.
  44. C. Wang, Y. Wang, M. Xu, and D. J. Crandall, “Stepwise goal-driven networks for trajectory prediction,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 2716–2723, 2022.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets