Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Efficient Movie Scene Detection using State-Space Transformers (2212.14427v2)

Published 29 Dec 2022 in cs.CV

Abstract: The ability to distinguish between different movie scenes is critical for understanding the storyline of a movie. However, accurately detecting movie scenes is often challenging as it requires the ability to reason over very long movie segments. This is in contrast to most existing video recognition models, which are typically designed for short-range video analysis. This work proposes a State-Space Transformer model that can efficiently capture dependencies in long movie videos for accurate movie scene detection. Our model, dubbed TranS4mer, is built using a novel S4A building block, which combines the strengths of structured state-space sequence (S4) and self-attention (A) layers. Given a sequence of frames divided into movie shots (uninterrupted periods where the camera position does not change), the S4A block first applies self-attention to capture short-range intra-shot dependencies. Afterward, the state-space operation in the S4A block is used to aggregate long-range inter-shot cues. The final TranS4mer model, which can be trained end-to-end, is obtained by stacking the S4A blocks one after the other multiple times. Our proposed TranS4mer outperforms all prior methods in three movie scene detection datasets, including MovieNet, BBC, and OVSD, while also being $2\times$ faster and requiring $3\times$ less GPU memory than standard Transformer models. We will release our code and models.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (52)
  1. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv: 1409.0473, 2014.
  2. Analysis and re-use of videos in educational digital libraries with automatic scene detection. In Italian Research Conference on Digital Libraries, pages 155–164. Springer, 2015.
  3. A deep siamese network for scene detection in broadcast videos. In Proceedings of the 23rd ACM international conference on Multimedia, pages 1199–1202, 2015.
  4. Shot and scene detection via hierarchical clustering for re-using broadcast video. In International conference on computer analysis of images and patterns, pages 801–811. Springer, 2015.
  5. BBC. Planet earth. https://www.bbc.co.uk/programmes/b006mywy.
  6. Is space-time attention all you need for video understanding? In Proceedings of the International Conference on Machine Learning (ICML), July 2021.
  7. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  8. End-to-end object detection with transformers. In European conference on computer vision, pages 213–229. Springer, 2020.
  9. Brandon Castellano. Pyscenedetect: Intelligent scene cut detection and video splitting tool, 2018.
  10. Scene detection in videos using shot clustering and sequence alignment. IEEE transactions on multimedia, 11(1):89–100, 2008.
  11. Shot contrastive self-supervised learning for scene boundary detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9796–9805, 2021.
  12. Rethinking attention with performers. arXiv preprint arXiv:2009.14794, 2020.
  13. Video shot detection and condensed representation. a review. IEEE signal processing magazine, 23(2):28–37, 2006.
  14. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860, 2019.
  15. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  16. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  17. It’s raw! audio generation with state-space models. International Conference on Machine Learning (ICML), 2022.
  18. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021.
  19. Combining recurrent, convolutional, and continuous-time models with linear state space layers. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
  20. Ankit Gupta. Diagonal state spaces are as effective as structured state spaces. arXiv preprint arXiv:2203.14343, 2022.
  21. Video scene segmentation using a novel boundary evaluation criterion and dynamic programming. In 2011 IEEE International conference on multimedia and expo, pages 1–6. IEEE, 2011.
  22. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  23. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, abs/1606.08415, 3, 2016.
  24. Transformer quality in linear time. In International Conference on Machine Learning, pages 9099–9117. PMLR, 2022.
  25. Movienet: A holistic dataset for movie understanding. In European Conference on Computer Vision, pages 709–727. Springer, 2020.
  26. Transformers are rnns: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pages 5156–5165. PMLR, 2020.
  27. Ephraim Katz. The film encyclopedia. Thomas Y. Crowell, 1979.
  28. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  29. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451, 2020.
  30. The language of actions: Recovering the syntax and semantics of goal-directed human activities. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 780–787, 2014.
  31. Learning to recognize procedural activities with distant supervision. arXiv preprint arXiv:2201.10990, 2022.
  32. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
  33. Long range language modeling via gated state spaces. arXiv preprint arXiv:2206.13947, 2022.
  34. Long movie clip classification with state-space video models. European Conference on Computer Vision (ECCV), 2022.
  35. Boundary-aware self-supervised learning for video scene segmentation. arXiv preprint arXiv:2201.05277, 2022.
  36. Keeping your eye on the ball: Trajectory attention in video transformers. Advances in neural information processing systems, 34:12493–12506, 2021.
  37. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.
  38. A local-to-global approach to multi-modal movie scene segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10146–10155, 2020.
  39. Detection and representation of scenes in videos. IEEE transactions on Multimedia, 7(6):1097–1105, 2005.
  40. Robust and efficient video scene detection using optimal sequential grouping. In 2016 IEEE international symposium on multimedia (ISM), pages 275–280. IEEE, 2016.
  41. Constructing table-of-content for videos. Multimedia systems, 7(5):359–368, 1999.
  42. Temporal video segmentation to scenes using high-level audiovisual features. IEEE Transactions on Circuits and Systems for Video Technology, 21(8):1163–1177, 2011.
  43. Coin: A large-scale dataset for comprehensive instructional video analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1207–1216, 2019.
  44. Comprehensive instructional video analysis: The coin dataset and performance evaluation. IEEE transactions on pattern analysis and machine intelligence, 43(9):3138–3153, 2020.
  45. Storygraphs: visualizing character interactions as a timeline. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 827–834, 2014.
  46. Robert sklar. film, an international history of the medium, 1993;; kristin thompson, david bordwell. film history, an introduction, 1994. 1895, revue d’histoire du cinéma, 17(1):170–170, 1994.
  47. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  48. Towards long-form video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1884–1894, 2021.
  49. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
  50. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33:17283–17297, 2020.
  51. Graph-based high-order relation modeling for long-term action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8984–8993, 2021.
  52. Constructing table-of-content for videos. In Exploration of Visual Data, pages 53–73. Springer, 2003.
Citations (36)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now