Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
133 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Action-slot: Visual Action-centric Representations for Multi-label Atomic Activity Recognition in Traffic Scenes (2311.17948v2)

Published 29 Nov 2023 in cs.CV and cs.LG

Abstract: In this paper, we study multi-label atomic activity recognition. Despite the notable progress in action recognition, it is still challenging to recognize atomic activities due to a deficiency in a holistic understanding of both multiple road users' motions and their contextual information. In this paper, we introduce Action-slot, a slot attention-based approach that learns visual action-centric representations, capturing both motion and contextual information. Our key idea is to design action slots that are capable of paying attention to regions where atomic activities occur, without the need for explicit perception guidance. To further enhance slot attention, we introduce a background slot that competes with action slots, aiding the training process in avoiding unnecessary focus on background regions devoid of activities. Yet, the imbalanced class distribution in the existing dataset hampers the assessment of rare activities. To address the limitation, we collect a synthetic dataset called TACO, which is four times larger than OATS and features a balanced distribution of atomic activities. To validate the effectiveness of our method, we conduct comprehensive experiments and ablation studies against various action recognition baselines. We also show that the performance of multi-label atomic activity recognition on real-world datasets can be improved by pretraining representations on TACO. We will release our source code and dataset. See the videos of visualization on the project page: https://hcis-lab.github.io/Action-slot/

Definition Search Book Streamline Icon: https://streamlinehq.com
References (82)
  1. CARLA Autonomous Driving Challenge. https://carlachallenge.org/, 2022.
  2. ScenarioRunner for CARLA. https://github.com/carla-simulator/scenario_runner, 2023.
  3. Ordered atomic activity for fine-grained interactive traffic scenario understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 8624–8636, 2023.
  4. ViViT: A Video Vision Transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6836–6846, 2021.
  5. Discovering Objects that Can Move. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11789–11798, 2022.
  6. Object Level Visual Reasoning in Videos. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
  7. Is space-time attention all you need for video understanding? In International Conference on Machine Learning (ICML), 2021.
  8. Invariant Slot Attention: Object Discovery with Slot-Centric Reference Frames. International Conference on Machine Learning (ICML), 2023.
  9. nuscenes: A Multimodal Dataset for Autonomous Driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  10. AdvDO: Realistic Adversarial Attacks for Trajectory Prediction. In European Conference on Computer Vision (ECCV). Springer, 2022.
  11. End-to-end Object Detection with Transformers. In European Conference on Computer Vision (ECCV), pages 213–229. Springer, 2020.
  12. Quo vadis, Action Recognition? A New Model and the Kinetics Dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6299–6308, 2017.
  13. Atomic Scenes for Scalable Traffic Scene Recognition in Monocular Videos. In IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1–9, 2016.
  14. Learning from all vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17222–17231, 2022.
  15. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 801–818, 2018.
  16. Neat: Neural attention fields for end-to-end autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15793–15803, 2021.
  17. Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014.
  18. What are they doing?: Collective activity classification using spatio-temporal relationship among people. In IEEE International Conference on Computer Vision Workshops (ICCV-W), pages 1282–1289, 2009.
  19. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9268–9277, 2019.
  20. Imagenet: A Large-scale Hierarchical Image Database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248–255, 2009.
  21. Causalaf: causal autoregressive flow for safety-critical driving scenario generation. In Conference on Robot Learning, pages 812–823. PMLR, 2023.
  22. CARLA: An Open Urban Driving Simulator. In Proceedings of the 1st Annual Conference on Robot Learning, pages 1–16, 2017.
  23. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations (ICLR), 2021.
  24. SAVi++: Towards End-to-End Object-centric Learning from Real-world Videos. Conference on Neural Information Processing Systems (NeurIPS), 2022.
  25. PyTorchVideo: A Deep Learning Library for Video Understanding. In Proceedings of the 29th ACM International Conference on Multimedia, 2021a. https://pytorchvideo.org/.
  26. Multiscale vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6824–6835, 2021b.
  27. Christoph Feichtenhofer. X3D: Expanding Architectures for Efficient Video Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 203–213, 2020.
  28. SlowFast Networks for Video Recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6202–6211, 2019.
  29. Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
  30. AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6047–6056, 2018.
  31. Vip3d: End-to-end visual trajectory prediction via 3d agent queries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5496–5506, 2023.
  32. King: Generating safety-critical driving scenarios for robust imitation via kinematics gradients. In European Conference on Computer Vision, pages 335–352. Springer, 2022.
  33. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  34. Mask R-CNN. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2961–2969, 2017.
  35. Planning-oriented autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
  36. A hierarchical deep temporal model for group activity recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1971–1980, 2016.
  37. Car that Knows Before You Do: Anticipating Maneuvers via Learning Temporal Driving Models. In IEEE International Conference on Computer Vision (ICCV), 2015.
  38. THUMOS challenge: Action recognition with a large number of classes. http://crcv.ucf.edu/THUMOS14/, 2014.
  39. CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2901–2910, 2017.
  40. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950, 2017.
  41. Conditional Object-Centric Learning from Video. In International Conference on Learning Representations (ICLR), 2022.
  42. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.
  43. HMDB: A Large Video Database for Human Motion Recognition. In 2011 International Conference on Computer Vision (ICCV), pages 2556–2563, 2011.
  44. Riskbench: A scenario-based benchmark for risk identification. arXiv preprint arXiv:2312.01659, 2023.
  45. DBUS: Human Driving Behavior Understanding System. In ICCV Workshops, pages 2436–2444, 2019.
  46. Visual Semantic Search: Retrieving Videos via Complex Textual Queries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2014.
  47. Spatiotemporal relationship reasoning for pedestrian intent prediction. pages 3485–3492, 2020.
  48. Object-centric Learning with Slot Attention. Advances in Neural Information Processing Systems (NeurIPS), 33:11525–11538, 2020.
  49. Decoupled Weight Decay Regularization. In International Conference on Learning Representations, 2019.
  50. Deep OC-SORT: Multi-Pedestrian Tracking by Adaptive Re-Identification. arXiv preprint arXiv:2302.11813, 2023.
  51. TITAN: Future Forecast using Action Priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11186–11196, 2020.
  52. Social-stgcnn: A social spatio-temporal graph convolutional neural network for human trajectory prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14424–14432, 2020.
  53. Egoenv: Human-centric environment representations from egocentric video. arXiv preprint arXiv:2207.11365, 2022.
  54. The 7th AI City Challenge. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2023.
  55. Learning situational driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  56. Toward Driving Scene Understanding: A Dataset for Learning Driver Behavior and Causal Reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7699–7707, 2018.
  57. PIE: A Large-Scale Dataset and Models for Pedestrian Intention Estimation and Trajectory Prediction. In The IEEE International Conference on Computer Vision (ICCV), 2019.
  58. Generating useful accident-prone driving scenarios via a learned traffic prior. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  59. Trace and pace: Controllable pedestrian animation via guided trajectory diffusion. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  60. Plant: Explainable planning transformers via object-level representations. arXiv preprint arXiv:2210.14222, 2022.
  61. Object scene representation transformer. In Advances in Neural Information Processing Systems, pages 9512–9524. Curran Associates, Inc., 2022.
  62. Universal Embeddings for Spatio-Temporal Tagging of Self-Driving Logs. In Conference on Robot Learning (CoRL), 2020.
  63. Reasonnet: End-to-end driving with temporal and global reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13723–13733, 2023.
  64. Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding. In Proceedings of the European Conference on Computer Vision (ECCV), pages 510–526. Springer, 2016.
  65. ROAD: The ROad event Awareness Dataset for Autonomous Driving. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1):1036–1054, 2022.
  66. Scalability in Perception for Autonomous Driving: Waymo Open Dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
  67. Shift: a synthetic driving dataset for continuous multi-task domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21371–21382, 2022.
  68. Boosting Standard Classification Architectures Through a Ranking Regularizer. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2020.
  69. Egodistill: Egocentric head motion distillation for efficient video understanding. arXiv preprint arXiv:2301.02217, 2023.
  70. VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training. In Advances in Neural Information Processing Systems, 2022.
  71. Video Classification with Channel-separated Convolutional Networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5552–5561, 2019.
  72. Attention is all you need. Conference on Neural Information Processing Systems, 30, 2017.
  73. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7794–7803, 2018.
  74. Learning Actor Relation Graphs for Group Activity Recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
  75. Distribution-balanced loss for multi-label classification in long-tailed datasets. In European Conference on Computer Vision (ECCV), 2020.
  76. Learning Road Scene-level Representations via Semantic Region Prediction. In Conference on Robot Learning (CoRL), 2022.
  77. Safebench: A benchmarking platform for safety evaluation of autonomous vehicles. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
  78. Temporal pyramid network for action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 591–600, 2020.
  79. Every moment counts: Dense detailed labeling of actions in complex videos. International Journal of Computer Vision, 2017.
  80. Coaching a teachable student. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7805–7815, 2023.
  81. Slot-VPS: Object-centric Representation Learning for Video Panoptic Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3093–3103, 2022.
  82. Query-centric trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17863–17873, 2023.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets