2000 character limit reached
Using Saliency and Cropping to Improve Video Memorability (2309.11881v1)
Published 21 Sep 2023 in cs.CV and cs.MM
Abstract: Video memorability is a measure of how likely a particular video is to be remembered by a viewer when that viewer has no emotional connection with the video content. It is an important characteristic as videos that are more memorable are more likely to be shared, viewed, and discussed. This paper presents results of a series of experiments where we improved the memorability of a video by selectively cropping frames based on image saliency. We present results of a basic fixed cropping as well as the results from dynamic cropping where both the size of the crop and the position of the crop within the frame, move as the video is played and saliency is tracked. Our results indicate that especially for videos of low initial memorability, the memorability score can be improved.
- Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet, April 2015. arXiv:1411.1045.
- Anonymised. Predicting Media Memorability: Comparing Visual, Textual and Auditory Features. In Proceedings of the MediaEval 2021 Workshop, December 2021.
- Anonymised. Analysing the Memorability of a Procedural Crime-Drama TV Series, CSI. In Proceedings of the 19th International Conference on Content-Based Multimedia Indexing, CBMI ’22, page 174–180, New York, NY, USA, 2022. Association for Computing Machinery.
- Predicting media memorability using ensemble models. In Proceedings of MediaEval 2019, Sophia Antipolis, France. CEUR Workshop Proceedings, October 2019.
- Memorability: A stimulus-driven perceptual neural signature distinctive from memory. NeuroImage, 149:141–152, 2017.
- VideoMem: Constructing, analyzing, predicting short-term and long-term video memorability. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2531–2540, 2019.
- Review of visual saliency detection with comprehensive information. IEEE Transactions on Circuits and Systems for Video Technology, 29(10):2941–2959, 2018.
- Predicting human eye fixations via an LSTM-based saliency attentive model. IEEE Transactions on Image Processing, 27(10):5142–5154, 2018.
- Overview of MediaEval 2020 Predicting Media Memorability Task: What Makes a Video Memorable? In MediaEval Multimedia Benchmark Workshop Working Notes, 2020.
- What makes an object memorable? In Proceedings of the IEEE International Conference on Computer Vision, pages 1089–1097, 2015.
- Understanding the intrinsic memorability of images. Advances in Neural Information Processing Systems, 24, 2011.
- Deep Video Inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
- Understanding Low- and High-Level Contributions to Fixation Prediction. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 4799–4808, Venice, October 2017. IEEE.
- DeepGaze IIE: Calibrated prediction in and out-of-domain for state-of-the-art saliency modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12919–12928, 2021.
- Learning trajectory-aware transformer for video super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5687–5696, 2022.
- Memorability of natural scenes: The role of attention. In 2013 IEEE international conference on image processing, pages 196–200. IEEE, 2013.
- Multimodal memorability: Modeling effects of semantics and decay on video memorability. In European Conference on Computer Vision, pages 223–240. Springer, 2020.
- Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
- Predicting media memorability with audio, video, and text representations. In Proceedings of the MediaEval 2020 Workshop, 12 2020.
- Overview of The MediaEval 2021 Predicting Media Memorability Task. In CEUR Workshop Proceedings, volume 3181, 2021.
- The influence of audio on video memorability with an audio gestalt regulated video memorability system. In 2021 International Conference on Content-Based Multimedia Indexing (CBMI), pages 1–6. IEEE, 6 2021.
- Overview of the MediaEval 2022 predicting video memorability task. In Proceedings of MediaEval 2022, 2022.
- Leveraging audio gestalt to predict media memorability. In MediaEval Multimedia Benchmark Workshop Working Notes, arXiv preprint arXiv:2012.15635, 2020.
- Diffusing surrogate dreams of video scenes to predict video memorability. In Proceedings of the MediaEval 2022 Workshop, December 2022.
- Image saliency: From intrinsic to extrinsic context. In CVPR 2011, pages 417–424. IEEE, 2011.