SportsSloMo: A New Benchmark and Baselines for Human-centric Video Frame Interpolation (2308.16876v2)

Abstract: Human-centric video frame interpolation has great potential for improving people's entertainment experiences and finding commercial applications in the sports analysis industry, e.g., synthesizing slow-motion videos. Although there are multiple benchmark datasets available in the community, none of them is dedicated for human-centric scenarios. To bridge this gap, we introduce SportsSloMo, a benchmark consisting of more than 130K video clips and 1M video frames of high-resolution ($\geq$720p) slow-motion sports videos crawled from YouTube. We re-train several state-of-the-art methods on our benchmark, and the results show a decrease in their accuracy compared to other datasets. It highlights the difficulty of our benchmark and suggests that it poses significant challenges even for the best-performing methods, as human bodies are highly deformable and occlusions are frequent in sports videos. To improve the accuracy, we introduce two loss terms considering the human-aware priors, where we add auxiliary supervision to panoptic segmentation and human keypoints detection, respectively. The loss terms are model agnostic and can be easily plugged into any video frame interpolation approaches. Experimental results validate the effectiveness of our proposed loss terms, leading to consistent performance improvement over 5 existing models, which establish strong baseline models on our benchmark. The dataset and code can be found at: https://neu-vi.github.io/SportsSlomo/.

• The paper introduces the SportsSloMo dataset with over 130K high-resolution sports clips, providing a rich resource for human-centric video frame interpolation.
• The paper demonstrates that existing VFI techniques struggle with complex human movements and occlusions in sports, leading to notable performance drops.
• The paper proposes innovative human-aware loss terms that consistently improve interpolation performance as measured by PSNR, SSIM, and IE metrics.

An Expert Review of "SportsSloMo: A New Benchmark and Baselines for Human-centric Video Frame Interpolation"

The paper "SportsSloMo: A New Benchmark and Baselines for Human-centric Video Frame Interpolation" presents a significant contribution to the field of computer vision and specifically to the domain of video frame interpolation (VFI) by introducing a novel benchmark dataset named SportsSloMo. This work focuses on enhancing human-centric video frame interpolation by addressing the complex challenges posed by sports videos, which involve deformable human bodies and frequent occlusions.

Summary of Contributions

1. Introduction of SportsSloMo Dataset: The authors have curated SportsSloMo, a large benchmark featuring over 130,000 high-resolution ($\geq$720p) slow-motion sports video clips sourced from YouTube. This dataset comprises more than 1 million frames, providing a diverse range of sports scenarios that are absent in existing benchmarks. The dataset's primary focus is human-centric video content, which is crucial given the increasing consumer interest in slow-motion videos for enhanced entertainment and sports analysis.
2. Challenges in Human-centric Scenarios: The paper highlights the difficulties faced by state-of-the-art VFI techniques when applied to this dataset. Human-centric scenarios in sports inherently involve highly deformable structures and occlusions, making it challenging for existing methods to maintain their performance levels observed in general-purpose datasets. The authors demonstrate a noticeable decrease in accuracy for several re-trained models on SportsSloMo.
3. Proposed Human-aware Loss Terms: To address the intricacies of human motion and occlusion, the authors introduce novel human-aware loss terms. These terms incorporate auxiliary supervision via human segmentation in the panoptic setting and keypoint detection. The human-aware losses are model-agnostic, allowing them to be integrated into existing video interpolation methodologies straightforwardly.
4. Empirical Validation and Enhancement of VFI Models: Extensive experiments validate that these loss terms lead to consistent performance improvement across various models. Notably, they employ these loss terms on several flow-based and flow-agnostic VFI models, demonstrating enhancements in PSNR, SSIM, and IE metrics on the SportsSloMo dataset.

Implications and Future Directions

Practically, this research advances the capabilities of video frame interpolation systems, which can now better handle sporting scenes involving humans. This enhancement has implications in areas such as sports broadcasting and coaching, where detailed frame-by-frame analysis of athletic performance is valuable.

Theoretically, the introduction of human-aware loss functions invites future explorations into more sophisticated context-aware loss designs. Additionally, the nuances captured by the dataset could encourage the development of more robust motion representation strategies, potentially using 3D cognition of scenes. This paper also implicitly points towards the need for integrating higher-order motion prediction models that account for complex occlusions and non-linear motion in future work.

The SportsSloMo benchmark and associated findings offer a solid foundation for subsequent innovations in video processing for human-centric applications. As the benchmark dataset is made publicly accessible, it is expected to stimulate a wave of research in fields paralleling human-centric visual interpolation and adjacent domains like video super-resolution and dynamic activity recognition in crowded sports scenes.