Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
162 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Investigating Event-Based Cameras for Video Frame Interpolation in Sports (2407.02370v2)

Published 2 Jul 2024 in cs.CV

Abstract: Slow-motion replays provide a thrilling perspective on pivotal moments within sports games, offering a fresh and captivating visual experience. However, capturing slow-motion footage typically demands high-tech, expensive cameras and infrastructures. Deep learning Video Frame Interpolation (VFI) techniques have emerged as a promising avenue, capable of generating high-speed footage from regular camera feeds. Moreover, the utilization of event-based cameras has recently gathered attention as they provide valuable motion information between frames, further enhancing the VFI performances. In this work, we present a first investigation of event-based VFI models for generating sports slow-motion videos. Particularly, we design and implement a bi-camera recording setup, including an RGB and an event-based camera to capture sports videos, to temporally align and spatially register both cameras. Our experimental validation demonstrates that TimeLens, an off-the-shelf event-based VFI model, can effectively generate slow-motion footage for sports videos. This first investigation underscores the practical utility of event-based cameras in producing sports slow-motion content and lays the groundwork for future research endeavors in this domain.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (46)
  1. S. Tulyakov, D. Gehrig, S. Georgoulis, J. Erbach, M. Gehrig, Y. Li, and D. Scaramuzza, “Time Lens: Event-based video frame interpolation,” in IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), (Nashville, TN, USA), pp. 16150–16159, Inst. Electr. Electron. Eng. (IEEE), Jun. 2021.
  2. A. S. Parihar, D. Varshney, K. Pandya, and A. Aggarwal, “A comprehensive survey on video frame interpolation techniques,” Vis. Comput., vol. 38, pp. 295–319, Jan. 2021.
  3. S. Niklaus, L. Mai, and F. Liu, “Video frame interpolation via adaptive separable convolution,” in IEEE Int. Conf. Comput. Vis. (ICCV), (Venice, Italy), pp. 261–270, Inst. Electr. Electron. Eng. (IEEE), Oct. 2017.
  4. S. Meyer, O. Wang, H. Zimmer, M. Grosse, and A. Sorkine-Hornung, “Phase-based frame interpolation for video,” in IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), (Boston, MA, USA), pp. 1410–1418, Inst. Electr. Electron. Eng. (IEEE), Jun. 2015.
  5. S. Niklaus and F. Liu, “Context-aware synthesis for video frame interpolation,” in IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), (Salt Lake City, UT, USA), pp. 1701–1710, Inst. Electr. Electron. Eng. (IEEE), Jun. 2018.
  6. J. van Amersfoort, W. Shi, A. Acosta, F. Massa, J. Totz, Z. Wang, and J. Caballero, “Frame interpolation with multi-scale deep loss functions and generative adversarial networks,” arXiv, vol. abs/1711.06045, 2017.
  7. EVS Broadcast Equipment, “XtraMotion.” https://evs.com/products/live-replays-storytelling/xtramotion.
  8. T. Xue, B. Chen, J. Wu, D. Wei, and W. T. Freeman, “Video enhancement with task-oriented flow,” Int. J. Comput. Vis., vol. 127, pp. 1106–1125, Feb. 2019.
  9. K. Soomro, A. R. Zamir, and M. Shah, “UCF101: A dataset of 101 human actions classes from videos in the wild,” arXiv, vol. abs/1212.0402, 2012.
  10. H. Sim, J. Oh, and M. Kim, “XVFI: eXtreme video frame interpolation,” in IEEE/CVF Int. Conf. Comput. Vis. (ICCV), (Montréal, Can.), pp. 14469–14478, Inst. Electr. Electron. Eng. (IEEE), Oct. 2021.
  11. S. Baker, S. Roth, D. Scharstein, M. J. Black, J. Lewis, and R. Szeliski, “A database and evaluation methodology for optical flow,” in IEEE Int. Conf. Comput. Vis. (ICCV), (Rio de Janeiro, Brazil), pp. 1–8, Inst. Electr. Electron. Eng. (IEEE), 2007.
  12. M. Choi, H. Kim, B. Han, N. Xu, and K. M. Lee, “Channel attention is all you need for video frame interpolation,” in AAAI Conf. Artif. Intell., vol. 34, pp. 10663–10671, Association for the Advancement of Artificial Intelligence (AAAI), Apr. 2020.
  13. J. Chen and H. Jiang, “SportsSloMo: A new benchmark and baselines for human-centric video frame interpolation,” arXiv, vol. abs/2308.16876, 2023.
  14. X. Zheng, Y. Liu, Y. Lu, T. Hua, T. Pan, W. Zhang, D. Tao, and L. Wang, “Deep learning for event-based vision: A comprehensive survey and benchmarks,” arXiv, vol. abs/2302.08890, 2023.
  15. S. Tulyakov, A. Bochicchio, D. Gehrig, S. Georgoulis, Y. Li, and D. Scaramuzza, “Time Lens++: Event-based frame interpolation with parametric nonlinear flow and multi-scale fusion,” in IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), (New Orleans, LA, USA), pp. 17734–17743, Inst. Electr. Electron. Eng. (IEEE), Jun. 2022.
  16. Y. Gao, S. Li, Y. Li, Y. Guo, and Q. Dai, “SuperFast: 200×\times× video frame interpolation via event camera,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 45, pp. 7764–7780, Jun. 2023.
  17. Springer, 2014.
  18. B. T. Naik, M. F. Hashmi, N. D. Bokde, and Z. M. Yaseen, “A comprehensive review of computer vision in sports: Open issues, future trends and research directions,” Appl. Sci., vol. 12, pp. 1–49, Apr. 2022.
  19. C. K. Ingwersen, C. Møller Mikkelstrup, J. N. Jensen, M. Rieger Hannemose, and A. B. Dahl, “SportsPose - a dynamic 3D sports pose dataset,” in IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), (Vancouver, Can.), pp. 5219–5228, Inst. Electr. Electron. Eng. (IEEE), Jun. 2023.
  20. A. Scott, I. Uchida, M. Onishi, Y. Kameda, K. Fukui, and K. Fujii, “SoccerTrack: A dataset and tracking algorithm for soccer with fish-eye and drone videos,” in IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), (New Orleans, LA, USA), pp. 3568–3578, Inst. Electr. Electron. Eng. (IEEE), Jun. 2022.
  21. G. Van Zandycke, V. Somers, M. Istasse, C. D. Don, and D. Zambrano, “DeepSportradar-v1: Computer vision dataset for sports understanding with high quality annotations,” in Int. ACM Work. Multimedia Content Anal. Sports (MMSports), (Lisbon, Port.), pp. 1–8, ACM, Oct. 2022.
  22. M. Istasse, V. Somers, P. Elancheliyan, J. De, and D. Zambrano, “DeepSportradar-v2: A multi-sport computer vision dataset for sport understandings,” in Int. ACM Work. Multimedia Content Anal. Sports (MMSports), (Ottawa, Ontario, Can.), pp. 23–29, ACM, Oct. 2023.
  23. S. Giancola, M. Amine, T. Dghaily, and B. Ghanem, “SoccerNet: A scalable dataset for action spotting in soccer videos,” in IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), (Salt Lake City, UT, USA), pp. 1792–179210, Inst. Electr. Electron. Eng. (IEEE), Jun. 2018.
  24. A. Deliège, A. Cioppa, S. Giancola, M. J. Seikavandi, J. V. Dueholm, K. Nasrollahi, B. Ghanem, T. B. Moeslund, and M. Van Droogenbroeck, “SoccerNet-v2: A dataset and benchmarks for holistic understanding of broadcast soccer videos,” in IEEE Int. Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), CVsports, (Nashville, TN, USA), pp. 4508–4519, Jun. 2021.
  25. A. Cioppa, A. Deliège, S. Giancola, B. Ghanem, and M. Van Droogenbroeck, “Scaling up SoccerNet with multi-view spatial localization and re-identification,” Sci. Data, vol. 9, pp. 1–9, Jun. 2022.
  26. A. Cioppa, S. Giancola, A. Deliege, L. Kang, X. Zhou, Z. Cheng, B. Ghanem, and M. Van Droogenbroeck, “SoccerNet-tracking: Multiple object tracking dataset and benchmark in soccer videos,” in IEEE Int. Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), CVsports, (New Orleans, LA, USA), pp. 3490–3501, Inst. Electr. Electron. Eng. (IEEE), Jun. 2022.
  27. J. Held, A. Cioppa, S. Giancola, A. Hamdi, B. Ghanem, and M. Van Droogenbroeck, “VARS: Video assistant referee system for automated soccer decision making from multiple views,” in IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), (Vancouver, Can.), pp. 5086–5097, Inst. Electr. Electron. Eng. (IEEE), Jun. 2023.
  28. H. Mkhallati, A. Cioppa, S. Giancola, B. Ghanem, and M. Van Droogenbroeck, “SoccerNet-caption: Dense video captioning for soccer broadcasts commentaries,” in IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), (Vancouver, Can.), pp. 5074–5085, Inst. Electr. Electron. Eng. (IEEE), Jun. 2023.
  29. R. Vandeghen, A. Cioppa, and M. Van Droogenbroeck, “Semi-supervised training to improve player and ball detection in soccer,” in IEEE Int. Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), CVsports, (New Orleans, LA, USA), pp. 3480–3489, Inst. Electr. Electron. Eng. (IEEE), Jun. 2022.
  30. K. Seweryn, G. Cheć, S. Łukasik, and A. Wróblewska, “Improving object detection quality in football through super-resolution techniques,” arXiv, vol. abs/2402.00163, 2024.
  31. M. Boeker and C. Midoglu, “Soccer athlete data visualization and analysis with an interactive dashboard,” in Int. Conf. Multimedia Retr., vol. 13833 of Lect. Notes Comput. Sci., pp. 565–576, Springer Int. Publ., 2023.
  32. A. Cioppa, A. Deliege, M. Istasse, C. De Vleeschouwer, and M. Van Droogenbroeck, “ARTHuS: Adaptive real-time human segmentation in sports through online distillation,” in IEEE Int. Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), CVsports, (Long Beach, CA, USA), pp. 2505–2514, Inst. Electr. Electron. Eng. (IEEE), Jun. 2019.
  33. A. Maglo, A. Orcesi, and Q.-C. Pham, “Efficient tracking of team sport players with few game-specific annotations,” in IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), (New Orleans, LA, USA), pp. 3460–3470, Inst. Electr. Electron. Eng. (IEEE), Jun. 2022.
  34. A. Arbués Sangüesa, A. Martín, J. Fernández, C. Ballester, and G. Haro, “Using player’s body-orientation to model pass feasibility in soccer,” in IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), (Seattle, WA, USA), pp. 3875–3884, Inst. Electr. Electron. Eng. (IEEE), Jun. 2020.
  35. A. Cioppa, A. Deliège, and M. Van Droogenbroeck, “A bottom-up approach based on semantics for the interpretation of the main camera stream in soccer games,” in IEEE Int. Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), CVsports, (Salt Lake City, UT, USA), pp. 1846–1855, Jun. 2018.
  36. J. V. B. Soares, A. Shah, and T. Biswas, “Temporally precise action spotting in soccer videos using dense detection anchors,” in IEEE Int. Conf. Image Process. (ICIP), (Bordeaux, France), pp. 2796–2800, Inst. Electr. Electron. Eng. (IEEE), Oct. 2022.
  37. J. Hong, H. Zhang, M. Gharbi, M. Fisher, and K. Fatahalian, “Spotting temporally precise, fine-grained events in video,” arXiv, vol. abs/2207.10213, 2022.
  38. L. Wu, Z. Yang, Q. Wang, M. Jian, B. Zhao, J. Yan, and C. W. Chen, “Fusing motion patterns and key visual information for semantic event recognition in basketball videos,” arXiv, vol. abs/2007.06288, 2020.
  39. J. Denize, M. Liashuha, J. Rabarisoa, A. Orcesi, and R. Hérault, “COMEDIAN: Self-supervised learning and knowledge distillation for action spotting using transformers,” arXiv, vol. abs/2309.01270, 2023.
  40. K. Seweryn, A. Wróblewska, and S. Łukasik, “Survey of action recognition, spotting and spatio-temporal localization in soccer – current trends and research perspectives,” arXiv, vol. abs/2309.12067, 2023.
  41. A. Cioppa, A. Deliège, S. Giancola, F. Magera, O. Barnich, B. Ghanem, and M. Van Droogenbroeck, “Camera calibration and player localization in SoccerNet-v2 and investigation of their representations for action spotting,” in IEEE Int. Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), CVsports, (Nashville, TN, USA), pp. 4532–4541, Jun. 2021.
  42. A. Cioppa, A. Deliège, S. Giancola, B. Ghanem, M. Van Droogenbroeck, R. Gade, and T. B. Moeslund, “A context-aware loss function for action spotting in soccer videos,” in IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), (Seattle, WA, USA), pp. 13123–13133, Inst. Electr. Electron. Eng. (IEEE), Jun. 2020.
  43. S. Gautam, C. Midoglu, S. S. Sabet, D. B. K. Kshatri, and P. Halvorsen, “Assisting soccer game summarization via audio intensity analysis of game highlights,” in IOE Graduate Conference, vol. 12, pp. 25–32, Oct. 2022.
  44. C. Midoglu, S. Hicks, V. Thambawita, T. Kupka, and P. Halvorsen, “MMSys’22 grand challenge on AI-based video production for soccer,” in ACM Multimedia Systems Conference (MMSys), (Athlone, Ireland), pp. 1–6, Jun. 2022.
  45. M. H. Sarkhoosh, S. M. M. Dorcheh, C. Midoglu, S. S. Sabet, T. Kupka, D. Johansen, M. A. Riegler, and P. Halvorsen, “AI-based cropping of soccer videos for different social media representations,” in Int. Conf. Multimedia Retr., vol. 14557 of Lect. Notes Comput. Sci., pp. 279–287, Springer Nat. Switz., 2024.
  46. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Trans. Image Process., vol. 13, pp. 600–612, Apr. 2004.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now