Neighbourhood Representative Sampling for Efficient End-to-end Video Quality Assessment (2210.05357v1)

Abstract: The increased resolution of real-world videos presents a dilemma between efficiency and accuracy for deep Video Quality Assessment (VQA). On the one hand, keeping the original resolution will lead to unacceptable computational costs. On the other hand, existing practices, such as resizing and cropping, will change the quality of original videos due to the loss of details and contents, and are therefore harmful to quality assessment. With the obtained insight from the study of spatial-temporal redundancy in the human visual system and visual coding theory, we observe that quality information around a neighbourhood is typically similar, motivating us to investigate an effective quality-sensitive neighbourhood representatives scheme for VQA. In this work, we propose a unified scheme, spatial-temporal grid mini-cube sampling (St-GMS) to get a novel type of sample, named fragments. Full-resolution videos are first divided into mini-cubes with preset spatial-temporal grids, then the temporal-aligned quality representatives are sampled to compose the fragments that serve as inputs for VQA. In addition, we design the Fragment Attention Network (FANet), a network architecture tailored specifically for fragments. With fragments and FANet, the proposed efficient end-to-end FAST-VQA and FasterVQA achieve significantly better performance than existing approaches on all VQA benchmarks while requiring only 1/1612 FLOPs compared to the current state-of-the-art. Codes, models and demos are available at https://github.com/timothyhtimothy/FAST-VQA-and-FasterVQA.

• The paper presents a novel quality-sensitive neighbourhood sampling paradigm that significantly reduces computational load in high-resolution video quality assessment.
• It introduces the Spatial-temporal Grid Mini-cube Sampling (St-GMS) scheme and Fragment Attention Network (FANet) to effectively capture both local and global video quality details.
• Empirical results show that the FAST-VQA model achieves up to 1612x speed improvement over existing methods, underscoring the potential of semantic pre-training in VQA.

Neighbourhood Representative Sampling for Efficient End-to-end Video Quality Assessment

In the paper titled "Neighbourhood Representative Sampling for Efficient End-to-end Video Quality Assessment", the authors address the computational difficulties imposed by high-resolution video processing on Video Quality Assessment (VQA) systems. They propose a novel sampling paradigm known as quality-sensitive neighbourhood representatives, which effectively balances computational efficiency with the preservation of critical quality information.

The underlying challenge is that high-resolution videos demand vast computational resources, making classical and deep VQA methods inefficient or infeasible for end-to-end processing. Traditional methods that rely on reducing resolution or cropping frames impair the video's quality, thus distorting the assessment. By leveraging insights from the human visual system and visual coding theories, the authors present a pioneering spatial-temporal sampling scheme called Spatial-temporal Grid Mini-cube Sampling (St-GMS). This framework maintains necessary video quality details while ensuring computational feasibility.

The St-GMS scheme operates by spatially partitioning video frames into uniform grids and temporally segmenting videos into equally-sized fragments. Mini-cubes are sampled from these spatial-temporal segments, representing global and local quality information effectively. The authors accompany this with the Fragment Attention Network (FANet), tailored to interpret the sampled fragments without misinterpreting artificial discontinuities as defects. FANet employs Gated Relative Position Biases (GRPB) and a unique Intra-Patch Non-Linear Regression (IP-NLR) head to improve accuracy, culminating in the creation of FAST-VQA and FasterVQA models.

These methodological advancements result in remarkable improvements in both computational efficiency and assessment accuracy, outperforming existing VQA methods by a considerable margin. Notably, the proposed FAST-VQA achieves an astonishing efficiency of up to 1612 times compared to the latest state-of-the-art method, all while retaining high accuracy levels.

The paper further evaluates the robustness of these models through comprehensive benchmark comparisons and detailed ablation studies. It also explores the implications of different sampling bearings and network configurations. The introduction of Adaptive Multi-scale Inference (AMI) demonstrates the versatility and adaptability of these models across various video resolutions and scales.

An important theoretical implication is the identification of a significant role for semantic pre-training in VQA. The empirical results highlight that features obtained through semantic pre-training contribute profoundly to the model's performance. This suggests a potential upper bound on the efficacy of semantic-blind VQA methods, perhaps advocating for further integration between semantic recognition and quality assessment techniques.

Practically, the breakthrough is significant for the implementation of deep VQA methodologies in real-world applications, particularly those involving mobile devices and cloud-based solutions where computational resources may be constrained. With unprecedented efficiency and performance, these models can feasibly process videos of various lengths and resolutions, expanding the potential application of VQA in digital media industries, telecommunications, and user experience research.

In summary, this work represents a significant step forward in VQA research, enabling high-dimensional data processing to be both effective and efficient. The proposed sampling paradigm and network architecture set new standards and open promising avenues for subsequent research, potentially influencing adjacent domains in video processing and beyond. Future developments may explore further optimizations of sampling techniques and network enhancements or investigate the paradigm's applicability in related tasks such as video recognition and object detection.