Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Weakly-Supervised Temporal Action Localization by Inferring Salient Snippet-Feature (2303.12332v3)

Published 22 Mar 2023 in cs.CV

Abstract: Weakly-supervised temporal action localization aims to locate action regions and identify action categories in untrimmed videos simultaneously by taking only video-level labels as the supervision. Pseudo label generation is a promising strategy to solve the challenging problem, but the current methods ignore the natural temporal structure of the video that can provide rich information to assist such a generation process. In this paper, we propose a novel weakly-supervised temporal action localization method by inferring salient snippet-feature. First, we design a saliency inference module that exploits the variation relationship between temporal neighbor snippets to discover salient snippet-features, which can reflect the significant dynamic change in the video. Secondly, we introduce a boundary refinement module that enhances salient snippet-features through the information interaction unit. Then, a discrimination enhancement module is introduced to enhance the discriminative nature of snippet-features. Finally, we adopt the refined snippet-features to produce high-fidelity pseudo labels, which could be used to supervise the training of the action localization network. Extensive experiments on two publicly available datasets, i.e., THUMOS14 and ActivityNet v1.3, demonstrate our proposed method achieves significant improvements compared to the state-of-the-art methods.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (49)
  1. Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4724–4733.
  2. Rethinking the Faster R-CNN Architecture for Temporal Action Localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1130–1139.
  3. Dual-Evidential Learning for Weakly-supervised Temporal Action Localization. In Proceedings of the European Conference on Computer Vision, 192–208.
  4. ASM-Loc: Action-Aware Segment Modeling for Weakly-Supervised Temporal Action Localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13925–13935.
  5. Momentum Contrast for Unsupervised Visual Representation Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9729–9738.
  6. ActivityNet: A large-scale video benchmark for human activity understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 961–970.
  7. Relational Prototypical Network for Weakly Supervised Temporal Action Localization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, 11053–11060.
  8. Foreground-Action Consistency Network for Weakly Supervised Temporal Action Localization. In Proceedings of the IEEE/CVF Conference on Computer Vision, 8002–8011.
  9. Weakly Supervised Temporal Action Localization via Representative Snippet Knowledge Propagation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3272–3281.
  10. THUMOS challenge: Action recognition with a large number of classes. https://www.crcv.ucf.edu/THUMOS14/.
  11. The Kinetics Human Action Video Dataset. CoRR, abs/1705.06950.
  12. Adam: A Method for Stochastic Optimization. In Proceedings of the International Conference on Learning Representations.
  13. Background Suppression Network for Weakly-Supervised Temporal Action Localization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, 11320–11327.
  14. Weakly-supervised Temporal Action Localization by Uncertainty Modeling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 1854–1862.
  15. Exploring Denoised Cross-Video Contrast for Weakly-Supervised Temporal Action Localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 19914–19924.
  16. Learning Salient Boundary Feature for Anchor-free Temporal Action Localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3320–3329.
  17. Single Shot Temporal Action Detection. In Proceedings of the 25th ACM International Conference on Multimedia, 988–996.
  18. BSN: Boundary Sensitive Network for Temporal Action Proposal Generation. In Proceedings of the European Conference on Computer Vision, 3–19.
  19. Completeness Modeling and Context Separation for Weakly Supervised Temporal Action Localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1298–1307.
  20. T-C3D: Temporal Convolutional 3D Network for Real-time Action Recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 7138–7145.
  21. Large-scale Vehicle Re-identification in Urban Surveillance Videos. In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME), 1–6. IEEE.
  22. The Blessings of Unlabeled Background in Untrimmed Videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6176–6185.
  23. Gaussian Temporal Awareness Networks for Action Localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 344–353.
  24. Action Unit Memory Network for Weakly Supervised Temporal Action Localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9969–9979.
  25. Weakly-Supervised Action Localization with Expectation-Maximization Multi-Instance Learning. In Proceedings of the European Conference on Computer Vision, 729–745.
  26. Adversarial Background-Aware Loss for Weakly-Supervised Temporal Activity Localization. In Proceedings of the European Conference on Computer Vision, 283–299.
  27. Collaborative Foreground, Background, and Action Modeling Network for Weakly Supervised Temporal Action Localization. IEEE Transactions on Circuits and Systems for Video Technology, 33(11): 6939–6951.
  28. Moon, T. 1996. The expectation-maximization algorithm. IEEE Signal Processing Magazine, 13(6): 47–60.
  29. D2-Net: Weakly-Supervised Action Localization via Discriminative Embeddings and Denoised Activations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 13608–13617.
  30. 3C-Net: Category Count and Center Loss for Weakly-Supervised Action Localization. In Proceedings of the IEEE/CVF Conference on Computer Vision, 8679–8687.
  31. Weakly Supervised Action Localization by Sparse Temporal Pooling Network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6752–6761.
  32. RefineLoc: Iterative Refinement for Weakly-Supervised Action Localization. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 3319–3328.
  33. W-TALC: Weakly-supervised Temporal Activity Localization and Classification. In Proceedings of the European Conference on Computer Vision, 563–579.
  34. Attentive Relational Networks for Mapping Images to Scene Graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3957–3966.
  35. stagNet: An Attentive Semantic RNN for Group Activity Recognition. In Proceedings of the European Conference on Computer Vision (ECCV), 104–120.
  36. Semantics-Aware Spatial-Temporal Binaries for Cross-Modal Video Retrieval. IEEE Transactions on Image Processing, 30: 2989–3004.
  37. STC-GAN: Spatio-Temporally Coupled Generative Adversarial Networks for Predictive Scene Parsing. IEEE Transactions on Image Processing, 29: 5420–5430.
  38. ACM-Net: Action Context Modeling Network for Weakly-Supervised Temporal Action Localization. arXiv preprint arXiv:2104.02967.
  39. Weakly-Supervised Action Localization by Generative Attention Modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1009–1019.
  40. Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1049–1058.
  41. UntrimmedNets for Weakly Supervised Action Recognition and Detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4325–4334.
  42. Segregated Temporal Assembly Recurrent Networks for Weakly Supervised Multiple Action Detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, 9070–9078.
  43. Revisiting Anchor Mechanisms for Temporal Action Localization. IEEE Transactions on Image Processing, 29: 8535–8548.
  44. Uncertainty Guided Collaborative Training for Weakly Supervised Temporal Action Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 53–63.
  45. Graph Convolutional Networks for Temporal Action Localization. In Proceedings of the IEEE/CVF Conference on Computer Vision, 7094–7103.
  46. Two-Stream Consensus Network for Weakly-Supervised Temporal ActionLocalization. In Proceedings of the European Conference on Computer Vision, 37–54.
  47. Adaptive Two-Stream Consensus Network for Weakly-Supervised Temporal Action Localization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4): 4136–4151.
  48. CoLA: Weakly-Supervised Temporal Action Localization With Snippet Contrastive Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 16010–16019.
  49. Temporal Action Detection with Structured Segment Networks. In 2017 IEEE International Conference on Computer Vision, 2914–2923.
Citations (7)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now