Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Deciphering Movement: Unified Trajectory Generation Model for Multi-Agent (2405.17680v1)

Published 27 May 2024 in cs.CV

Abstract: Understanding multi-agent behavior is critical across various fields. The conventional approach involves analyzing agent movements through three primary tasks: trajectory prediction, imputation, and spatial-temporal recovery. Considering the unique input formulation and constraint of these tasks, most existing methods are tailored to address only one specific task. However, in real-world applications, these scenarios frequently occur simultaneously. Consequently, methods designed for one task often fail to adapt to others, resulting in performance drops. To overcome this limitation, we propose a Unified Trajectory Generation model, UniTraj, that processes arbitrary trajectories as masked inputs, adaptable to diverse scenarios. Specifically, we introduce a Ghost Spatial Masking (GSM) module embedded within a Transformer encoder for spatial feature extraction. We further extend recent successful State Space Models (SSMs), particularly the Mamba model, into a Bidirectional Temporal Mamba to effectively capture temporal dependencies. Additionally, we incorporate a Bidirectional Temporal Scaled (BTS) module to comprehensively scan trajectories while maintaining the temporal missing relationships within the sequence. We curate and benchmark three practical sports game datasets, Basketball-U, Football-U, and Soccer-U, for evaluation. Extensive experiments demonstrate the superior performance of our model. To the best of our knowledge, this is the first work that addresses this unified problem through a versatile generative framework, thereby enhancing our understanding of multi-agent movement. Our datasets, code, and model weights are available at https://github.com/colorfulfuture/UniTraj-pytorch.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (97)
  1. The treatment of missing values and its effect on classifier accuracy. In Classification, Clustering, and Data Mining Applications, pages 639–647. 2004.
  2. Social lstm: Human trajectory prediction in crowded spaces. In CVPR, pages 961–971, 2016.
  3. Diffusion-based time series imputation and forecasting with structured state space models. Transactions on Machine Learning Research, 2022.
  4. Social ways: Learning multi-modal distributions of pedestrian trajectories with GANs. In CVPR Workshop, pages 2964–2972, 2019.
  5. On the estimation of arima models with missing values. Lecture Notes in Statistics, page 9.
  6. Adapt: Efficient multi-agent trajectory prediction with adaptation. In ICCV, pages 8295–8305, 2023.
  7. Learning pedestrian group representations for multi-modal trajectory prediction. In ECCV, pages 270–289, 2022.
  8. Eigentrajectory: Low-rank descriptors for multi-modal trajectory forecasting. In ICCV, pages 10017–10029, 2023.
  9. Mambamixer: Efficient selective state space models with dual token and channel selection. arXiv preprint arXiv:2403.19888, 2024.
  10. Nearest neighbor imputation algorithms: a critical evaluation. BMC Medical Informatics and Decision Making, 16(3):197–208, 2016.
  11. Brits: bidirectional recurrent imputation for time series. In NeurIPS, pages 6776–6786, 2018.
  12. Unsupervised sampling promoting for stochastic human trajectory prediction. In CVPR, pages 17874–17884, 2023a.
  13. Traj-mae: Masked autoencoders for trajectory prediction. In ICCV, pages 8351–8362, 2023b.
  14. Temporal disentangled contrastive diffusion model for spatiotemporal imputation. arXiv preprint arXiv:2402.11558, 2024.
  15. Rntrajrec: Road network enhanced trajectory recovery with spatial-temporal transformer. In ICDE, pages 829–842, 2023c.
  16. Discovering popular routes from trajectories. In ICDE, pages 900–911, 2011.
  17. Interpreting trajectories from multiple views: A hierarchical self-attention network for estimating the time of arrival. In KDD, pages 2771–2779, 2022.
  18. Adamsformer for spatial action localization in the future. In CVPR, pages 17885–17895, 2023.
  19. Exploring the limitations of behavior cloning for autonomous driving. In CVPR, pages 9329–9338, 2019.
  20. Toward abnormal trajectory and event detection in video surveillance. IEEE Transactions on Circuits and Systems for Video Technology, 27(3):683–695, 2016.
  21. Human behavior analysis in video surveillance: A social signal processing perspective. Neurocomputing, 100:86–97, 2013.
  22. Eta prediction with graph neural networks in google maps. In CIKM, pages 3767–3776, 2021.
  23. P. Kingma Diederik and Ba Jimmy. Adam: A method for stochastic optimization. In ICLR, 2015.
  24. Supervised learning from incomplete data via an em approach. In NeurIPS, 1993.
  25. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.
  26. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021a.
  27. Combining recurrent, convolutional, and continuous-time models with linear state space layers. In NeurIPS, pages 572–585, 2021b.
  28. Vip3d: End-to-end visual trajectory prediction via 3d agent queries. In CVPR, pages 5496–5506, 2023.
  29. Stochastic trajectory prediction via motion indeterminacy diffusion. In CVPR, pages 17113–17122, 2022.
  30. Social GAN: Socially acceptable trajectories with generative adversarial networks. In CVPR, pages 2255–2264, 2018.
  31. A graph-based approach for trajectory similarity computation in spatial networks. In KDD, pages 556–564, 2021.
  32. Importance is in your attention: agent importance prediction for autonomous driving. In CVPR, pages 2532–2535, 2022.
  33. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997.
  34. Collaborative motion prediction via neural motion message passing. In CVPR, pages 6319–6328, 2020.
  35. The trajectron: Probabilistic multi-agent trajectory modeling with dynamic spatiotemporal graphs. In ICCV, pages 2375–2384, 2019.
  36. Motiondiffuser: Controllable multi-agent motion prediction using diffusion. In CVPR, pages 9644–9653, 2023.
  37. Social-bigat: Multimodal trajectory forecasting using bicycle-gan and graph attention networks. In NeurIPS, pages 137–146, 2019.
  38. Traj-llm: A new exploration for empowering trajectory prediction with pre-trained large language models. arXiv preprint arXiv:2405.04909, 2024.
  39. Scene informer: Anchor-based occlusion inference and trajectory prediction in partially observable environments. In ICRA, 2023.
  40. Conditional generative neural system for probabilistic trajectory prediction. In IROS, 2019.
  41. Graph-based spatial transformer with memory replay for multi-future pedestrian trajectory prediction. In CVPR, pages 2231–2241, 2022.
  42. Vmamba: Visual state space model. arXiv preprint arXiv:2401.10166, 2024.
  43. Naomi: Non-autoregressive multiresolution sequence imputation. In NeurIPS, 2019.
  44. Multiple object tracking: A literature review. Artificial intelligence, 293:103448, 2021.
  45. Multivariate time series imputation with generative adversarial networks. In NeurIPS, pages 1603–1614, 2018.
  46. E2gan: End-to-end generative adversarial network for multivariate time series imputation. In IJCAI, pages 3094–3100, 2019.
  47. It is not the journey but the destination: Endpoint conditioned trajectory prediction. In ECCV, pages 759–776, 2020.
  48. Leapfrog diffusion model for stochastic trajectory prediction. In CVPR, pages 5517–5526, 2023.
  49. Generative semi-supervised learning for multivariate time series imputation. In AAAI, pages 8983–8991, 2021.
  50. Social-STGCNN: A social spatio-temporal graph convolutional neural network for human trajectory prediction. In CVPR, pages 14424–14432, 2020.
  51. Daniel S Nagin. Group-based trajectory modeling: an overview. Handbook of quantitative criminology, pages 53–67, 2010.
  52. Missing data: A comparison of neural network and expectation maximization techniques. Current Science, pages 1514–1521, 2007.
  53. Time-series imputation of temporally-occluded multiagent trajectories. arXiv preprint arXiv:2106.04219, 2021.
  54. Leveraging future relationship reasoning for vehicle trajectory prediction. In ICLR, 2023.
  55. Improving transferability for cross-domain trajectory prediction via neural stochastic differential equation. In AAAI, 2024.
  56. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019.
  57. Efficientvmamba: Atrous selective scan for light weight visual mamba. arXiv preprint arXiv:2403.09977, 2024.
  58. Imitative non-autoregressive modeling for trajectory forecasting and imputation. In CVPR, pages 12736–12745, 2020.
  59. Multiple-level point embedding for solving human trajectory imputation with prediction. ACM Transactions on Spatial Algorithms and Systems, 9(2):1–22, 2023.
  60. Fjmp: Factorized joint multi-agent motion prediction over learned directed acyclic interaction graphs. In CVPR, pages 13745–13755, 2023.
  61. SoPhie: An attentive GAN for predicting paths compliant to social and physical constraints. In CVPR, pages 1349–1358, 2019.
  62. Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data. In ECCV, pages 683–700, 2020.
  63. Soccertrack: A dataset and tracking algorithm for soccer with fish-eye and drone videos. In CVPR, pages 3569–3579, 2022.
  64. Motionlm: Multi-agent motion forecasting as language modeling. In ICCV, pages 8579–8590, 2023.
  65. Sgcn: Sparse graph convolution network for pedestrian trajectory prediction. In CVPR, pages 8994–9003, 2021.
  66. Trajectory unified transformer for pedestrian trajectory prediction. In ICCV, pages 9675–9684, 2023.
  67. Recursive social behavior graph for trajectory prediction. In CVPR, pages 660–669, 2020.
  68. Csdi: Conditional score-based diffusion models for probabilistic time series imputation. In NeurIPS, pages 24804–24816, 2021.
  69. Missing value estimation methods for dna microarrays. Bioinformatics, 17(6):520–525, 2001.
  70. Game plan: What ai can do for football, and what football can do for ai. Artificial Intelligence Research, 71:41–88, 2021.
  71. Attention is all you need. In NeurIPS, 2017.
  72. Social attention: Modeling attention in human crowds. In ICRA, pages 1–7, 2018.
  73. Large window-based mamba unet for medical image segmentation: Beyond convolution and self-attention. arXiv preprint arXiv:2403.07332, 2024a.
  74. Tacticai: an ai assistant for football tactics. Nature communications, 15(1):1–13, 2024b.
  75. Is mamba effective for time series forecasting? arXiv preprint arXiv:2403.11144, 2024c.
  76. Constructing popular routes from uncertain trajectories. In KDD, pages 195–203, 2012.
  77. Micro-macro spatial-temporal graph-based encoder-decoder for map-constrained trajectory recovery. IEEE Transactions on Knowledge and Data Engineering, 2024.
  78. Diffimpute: Tabular data imputation with denoising diffusion probabilistic model. arXiv preprint arXiv:2403.13863, 2024.
  79. Groupnet: Multiscale hypergraph neural networks for trajectory prediction with relational reasoning. In CVPR, pages 6498–6507, 2022a.
  80. Remember intentions: Retrospective-memory-based trajectory prediction. In CVPR, pages 6488–6497, 2022b.
  81. Socialvae: Human trajectory prediction using timewise latents. In ECCV, 2022c.
  82. Yi Xu and Yun Fu. Adapting to length shift: Flexilength network for trajectory prediction. arXiv preprint arXiv:2404.00742, 2024.
  83. Cf-lstm: Cascaded feature-based long short-term networks for predicting pedestrian trajectory. In AAAI, volume 34, pages 12541–12548, 2020.
  84. Tra2tra: Trajectory-to-trajectory prediction with a global social spatial-temporal attentive neural network. IEEE Robotics and Automation Letters, 6(2):1574–1581, 2021.
  85. Uncovering the missing pattern: Unified framework towards trajectory imputation and prediction. In CVPR, pages 9632–9643, 2023.
  86. Deep learning for person re-identification: A survey and outlook. IEEE transactions on pattern analysis and machine intelligence, 44(6):2872–2893, 2021.
  87. Gain: Missing data imputation using generative adversarial nets. In ICML, pages 5689–5698, 2018a.
  88. Estimating missing data in temporal data streams using multi-directional recurrent neural networks. IEEE Transactions on Biomedical Engineering, 66(5):1477–1490, 2018b.
  89. Diffusion-ts: Interpretable diffusion for general time series generation. arXiv preprint arXiv:2403.01742, 2024.
  90. Generating multi-agent trajectories using programmatic weak supervision. In Proceedings of the International Conference on Learning Representations, 2018.
  91. Learning calibratable policies using programmatic style-consistency. In ICML, pages 11001–11011. PMLR, 2020.
  92. SR-LSTM: State refinement for LSTM towards pedestrian trajectory prediction. In CVPR, pages 12085–12094, 2019.
  93. Look more but care less in video recognition. In NeurIPS, pages 30813–30825, 2022.
  94. Motion mamba: Efficient and long sequence motion generation with hierarchical and bidirectional selective ssm. arXiv preprint arXiv:2403.07487, 2024.
  95. Trajgat: A map-embedded graph attention network for real-time vehicle trajectory imputation of roadside perception. Transportation research part C: emerging technologies, 142:103787, 2022.
  96. Query-centric trajectory prediction. In CVPR, pages 17863–17873, 2023.
  97. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417, 2024.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - colorfulfuture/UniTraj-pytorch (4 stars)
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets