RAPIQUE: Rapid and Accurate Video Quality Prediction of User Generated Content (2101.10955v2)

Abstract: Blind or no-reference video quality assessment of user-generated content (UGC) has become a trending, challenging, heretofore unsolved problem. Accurate and efficient video quality predictors suitable for this content are thus in great demand to achieve more intelligent analysis and processing of UGC videos. Previous studies have shown that natural scene statistics and deep learning features are both sufficient to capture spatial distortions, which contribute to a significant aspect of UGC video quality issues. However, these models are either incapable or inefficient for predicting the quality of complex and diverse UGC videos in practical applications. Here we introduce an effective and efficient video quality model for UGC content, which we dub the Rapid and Accurate Video Quality Evaluator (RAPIQUE), which we show performs comparably to state-of-the-art (SOTA) models but with orders-of-magnitude faster runtime. RAPIQUE combines and leverages the advantages of both quality-aware scene statistics features and semantics-aware deep convolutional features, allowing us to design the first general and efficient spatial and temporal (space-time) bandpass statistics model for video quality modeling. Our experimental results on recent large-scale UGC video quality databases show that RAPIQUE delivers top performances on all the datasets at a considerably lower computational expense. We hope this work promotes and inspires further efforts towards practical modeling of video quality problems for potential real-time and low-latency applications. To promote public usage, an implementation of RAPIQUE has been made freely available online: \url{https://github.com/vztu/RAPIQUE}.

• The paper proposes a dual-branch model integrating natural scene statistics and deep CNN features for rapid UGC video quality prediction.
• It achieves state-of-the-art accuracy on benchmarks like KoNViD-1k while offering a 20x speedup on Full HD videos.
• The study's findings have practical implications for real-time video streaming, compression optimization, and enhancing quality assessment methods.

RAPIQUE: Rapid and Accurate Video Quality Prediction of User Generated Content

The proliferation of user-generated content (UGC) on platforms such as YouTube and Facebook demands advances in video quality assessment methodologies to address the diverse and complex distortions present in such videos. RAPIQUE, proposed by Tu et al., offers a promising solution by providing both rapid and accurate video quality evaluations, comparable to state-of-the-art methodologies, but with significantly improved computational efficiency.

Overview of RAPIQUE

RAPIQUE combines techniques from spatial and temporal domain analyses alongside deep convolutional neural network (CNN) features to create an effective video quality model. It employs a two-branch framework: one branch captures quality-aware features using natural scene statistics (NSS) from spatial and temporal data, and the other extracts semantics-aware features via a deep CNN. This dual approach is designed to efficiently evaluate the quality of UGC videos by successfully leveraging both low-level quality cues and high-level semantic information.

Experiment Results and Discussions

Evaluation of RAPIQUE reveals inspiring results across multiple UGC video datasets: KoNViD-1k, LIVE-VQC, and YouTube-UGC. Its performance demonstrates consistency and robustness, showcasing superior correlation with subjective video quality scores. Notably, RAPIQUE leads the pack on the KoNViD-1k database and the composite All-Combined dataset composed of multiple sources. It achieves competitive results on other databases, reflecting its potent applicability irrespective of the underlying data or distortion types present.

One intriguing aspect of RAPIQUE's functionality is its computational advantage over other sophisticated video quality assessment models. Compared to previous methods like TLVQM and VIDEVAL, RAPIQUE operates with a relative execution speed-up of 20x on Full HD (1080p) videos. Its computational demand scales effectively even when faced with increasing video resolutions—a critical requirement for real-time applications.

Implications and Future Directions

RAPIQUE introduces a novel and efficient approach to video quality assessment that combines spatial and temporal statistical analyses with semantic feature extraction. This methodology bears implications for improving automation in video compression, streaming optimization, and real-time content assessment. Its lightweight yet potent architecture opens avenues for facilitating fast quality evaluations without compromising accuracy, thus advantageous for both online platforms managing vast quantities of UGC and research efforts in video processing.

Looking ahead, RAPIQUE's design suggests further enhancements could be focused on extending its adaptability and precision in varied application scenarios such as virtual reality video and high dynamic range imaging. Its module synergy provides a promising baseline for developing adaptive models capable of leveraging sophisticated datasets and AI technologies, paving the way for the evolution of video quality assessment in the context of UGC.

In conclusion, RAPIQUE stands out as an efficient and effective model for rapid video quality assessment, utilizing a blend of statistical and deep learning features to meet the challenges posed by user-generated content. The methodologies and results presented in this paper are expected to inspire and propel further advancements and applications in this domain.