Papers
Topics
Authors
Recent
Search
2000 character limit reached

Towards More Practical Group Activity Detection: A New Benchmark and Model

Published 5 Dec 2023 in cs.CV | (2312.02878v2)

Abstract: Group activity detection (GAD) is the task of identifying members of each group and classifying the activity of the group at the same time in a video. While GAD has been studied recently, there is still much room for improvement in both dataset and methodology due to their limited capability to address practical GAD scenarios. To resolve these issues, we first present a new dataset, dubbed Caf\'e. Unlike existing datasets, Caf\'e is constructed primarily for GAD and presents more practical scenarios and metrics, as well as being large-scale and providing rich annotations. Along with the dataset, we propose a new GAD model that deals with an unknown number of groups and latent group members efficiently and effectively. We evaluated our model on three datasets including Caf\'e, where it outperformed previous work in terms of both accuracy and inference speed.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. Convolutional relational machine for group activity recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7892–7901, 2019.
  2. Social scene understanding: End-to-end multi-person action localization and collective activity recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4315–4324, 2017.
  3. End-to-end object detection with transformers. In Proc. European Conference on Computer Vision (ECCV), pages 213–229. Springer, 2020.
  4. Quo vadis, action recognition? a new model and the kinetics dataset. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  5. What are they doing?: Collective activity classification using spatio-temporal relationship among people. In Proc. IEEE International Conference on Computer Vision (ICCV) Workshops, pages 1282–1289. IEEE, 2009.
  6. Discovering groups of people in images. In Proc. European Conference on Computer Vision (ECCV), pages 417–433. Springer, 2014.
  7. Motchallenge: A benchmark for single-camera multiple target tracking. International Journal of Computer Vision (IJCV), 129(4):845–881, 2021.
  8. Imagenet: A large-scale hierarchical image database. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248–255, 2009.
  9. Structure inference machines: Recurrent neural networks for analyzing relations in group activity recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4772–4781, 2016.
  10. An image is worth 16x16 words: Transformers for image recognition at scale. In Proc. International Conference on Learning Representations (ICLR), 2021.
  11. Joint learning of social groups, individuals action and sub-group activities in videos. In Proc. European Conference on Computer Vision (ECCV), pages 177–195. Springer, 2020.
  12. Jrdb-act: A large-scale dataset for spatio-temporal action, social group and activity detection. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 20983–20992, 2022.
  13. A mobile vision system for robust multi-person tracking. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1–8. IEEE, 2008.
  14. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision (IJCV), 111:98–136, 2015.
  15. Actor-transformers for group activity recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 839–848, 2020.
  16. Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv:2107.08430, 2021.
  17. Panoramic human activity recognition. In Proc. European Conference on Computer Vision (ECCV), pages 244–261. Springer, 2022.
  18. Deep residual learning for image recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  19. Mask r-cnn. In Proc. IEEE International Conference on Computer Vision (ICCV), pages 2961–2969, 2017.
  20. Progressive relation learning for group activity recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 980–989, 2020.
  21. Hierarchical relational networks for group activity recognition and retrieval. In Proc. European Conference on Computer Vision (ECCV), pages 721–736, 2018.
  22. A hierarchical deep temporal model for group activity recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1971–1980, 2016.
  23. Detector-free weakly supervised group activity recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 20083–20093, 2022.
  24. Adam: A method for stochastic optimization. In Proc. International Conference on Learning Representations (ICLR), 2015.
  25. Semi-supervised classification with graph convolutional networks. Proc. International Conference on Learning Representations (ICLR), 2017.
  26. Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83–97, 1955.
  27. Dn-detr: Accelerate detr training by introducing query denoising. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 13619–13627, 2022a.
  28. Self-supervised social relation representation for human group detection. In Proc. European Conference on Computer Vision (ECCV), pages 142–159. Springer, 2022b.
  29. Groupformer: Group activity recognition with clustered spatial-temporal transformer. In Proc. IEEE International Conference on Computer Vision (ICCV), pages 13668–13677, 2021.
  30. Learning action-guided spatio-temporal transformer for group activity recognition. In Proceedings of the 30th ACM International Conference on Multimedia, pages 2051–2060, 2022c.
  31. Sbgar: Semantics based group activity recognition. In Proc. IEEE International Conference on Computer Vision (ICCV), pages 2876–2885, 2017.
  32. Jrdb: A dataset and benchmark of egocentric robot visual perception of humans in built environments. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2021.
  33. On spectral clustering: Analysis and an algorithm. Proc. Neural Information Processing Systems (NeurIPS), 14, 2001.
  34. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.
  35. Empowering relational network by self-attention augmented conditional random fields for group activity recognition. In Proc. European Conference on Computer Vision (ECCV), pages 71–90. Springer, 2020.
  36. stagnet: An attentive semantic rnn for group activity recognition. In Proc. European Conference on Computer Vision (ECCV), pages 101–117, 2018.
  37. Public life in public space (plps): A multi-task, multi-group video dataset for public life research. In Proc. IEEE International Conference on Computer Vision (ICCV) Workshops, pages 3618–3627, 2021.
  38. Generalized intersection over union: A metric and a loss for bounding box regression. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 658–666, 2019.
  39. Crowdhuman: A benchmark for detecting human in a crowd. arXiv preprint arXiv:1805.00123, 2018.
  40. Cern: confidence-energy recurrent network for group activity recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5523–5531, 2017.
  41. Hunting group clues with transformers for social group activity recognition. In Proc. European Conference on Computer Vision (ECCV), 2022.
  42. Attention is all you need. In Proc. Neural Information Processing Systems (NeurIPS), pages 5998–6008, 2017.
  43. Temporal segment networks: Towards good practices for deep action recognition. In Proc. European Conference on Computer Vision (ECCV), pages 20–36. Springer, 2016.
  44. Recurrent modeling of interaction context for collective activity recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3048–3056, 2017.
  45. Learning actor relation graphs for group activity recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9964–9974, 2019.
  46. Participation-contributed temporal dynamic model for group activity recognition. In Proc. ACM Multimedia Conference (ACMMM), pages 1292–1300, 2018.
  47. Higcin: hierarchical graph-based cross inference network for group activity recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2020a.
  48. Social adaptive module for weakly-supervised group activity recognition. In Proc. European Conference on Computer Vision (ECCV), pages 208–224. Springer, 2020b.
  49. Learning visual context for group activity recognition. In Proc. AAAI Conference on Artificial Intelligence (AAAI), pages 3261–3269, 2021.
  50. Spatio-temporal dynamic inference network for group activity recognition. In Proc. IEEE International Conference on Computer Vision (ICCV), pages 7476–7485, 2021.
  51. Self-tuning spectral clustering. Proc. Neural Information Processing Systems (NeurIPS), 17, 2004.
  52. Citypersons: A diverse dataset for pedestrian detection. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3213–3221, 2017.
  53. Bytetrack: Multi-object tracking by associating every detection box. In Proc. European Conference on Computer Vision (ECCV), 2022.
  54. Deformable detr: Deformable transformers for end-to-end object detection. In Proc. International Conference on Learning Representations (ICLR), 2021.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up