Multi-Frame Quality Enhancement for Compressed Video (1803.04680v4)

Abstract: The past few years have witnessed great success in applying deep learning to enhance the quality of compressed image/video. The existing approaches mainly focus on enhancing the quality of a single frame, ignoring the similarity between consecutive frames. In this paper, we investigate that heavy quality fluctuation exists across compressed video frames, and thus low quality frames can be enhanced using the neighboring high quality frames, seen as Multi-Frame Quality Enhancement (MFQE). Accordingly, this paper proposes an MFQE approach for compressed video, as a first attempt in this direction. In our approach, we firstly develop a Support Vector Machine (SVM) based detector to locate Peak Quality Frames (PQFs) in compressed video. Then, a novel Multi-Frame Convolutional Neural Network (MF-CNN) is designed to enhance the quality of compressed video, in which the non-PQF and its nearest two PQFs are as the input. The MF-CNN compensates motion between the non-PQF and PQFs through the Motion Compensation subnet (MC-subnet). Subsequently, the Quality Enhancement subnet (QE-subnet) reduces compression artifacts of the non-PQF with the help of its nearest PQFs. Finally, the experiments validate the effectiveness and generality of our MFQE approach in advancing the state-of-the-art quality enhancement of compressed video. The code of our MFQE approach is available at https://github.com/ryangBUAA/MFQE.git

• The paper introduces a novel Multi-Frame Quality Enhancement (MFQE) method that improves compressed video quality by leveraging information from high-quality Peak Quality Frames (PQFs).
• It uses an SVM to detect Peak Quality Frames (PQFs) and a novel Multi-Frame CNN combining motion compensation and quality enhancement.
• Experimental results demonstrate MFQE outperforms state-of-the-art methods in PSNR and effectively reduces quality fluctuations in compressed sequences.

Multi-Frame Quality Enhancement for Compressed Video

In the domain of video compression and quality enhancement, the paper titled "Multi-Frame Quality Enhancement for Compressed Video" presents a noteworthy approach that seeks to capitalize on multi-frame information to enhance video quality. Video compression, while reducing the required storage and bandwidth, often results in noticeable quality degradation. The authors address this by introducing the concept of Multi-Frame Quality Enhancement (MFQE), leveraging the similarities between consecutively compressed frames.

Key Contributions and Methodology

The authors highlight the prevalent issue of quality fluctuation across consecutive video frames in compressed sequences. Notably, they propose an MFQE method that utilizes high-quality frames—termed Peak Quality Frames (PQFs)—to enhance neighboring low-quality frames. This methodology stands as the first initiative in this direction, differentiating itself from existing single-frame enhancement techniques.

The proposed approach involves several critical components:

1. PQF Detection: A Support Vector Machine (SVM)-based detector is trained to identify PQFs within a video sequence. The detector's efficacy is measured by precision, recall, and the F1-score, demonstrating its robustness in recognizing frames with superior quality relative to their neighbors.
2. MF-CNN Architecture: The authors design a novel Multi-Frame Convolutional Neural Network (MF-CNN) that comprises two sub-networks:
   • Motion Compensation Subnet (MC-subnet): This component estimates and compensates for motion between PQFs and the non-PQF, thereby aligning frames optimally for further processing.
   • Quality Enhancement Subnet (QE-subnet): Utilizing both temporal and spatial information from PQFs and the current frame, the QE-subnet significantly reduces artifacts and enhances quality.
3. Training Regimen: The MF-CNN is trained jointly, with an emphasis on accurately compensating for motion and making effective quality corrections.

Results and Evaluation

The experimental validations of the MFQE approach provide compelling evidence of its performance. The MFQE consistently outperforms state-of-the-art methods such as AR-CNN, DnCNN, Li et al.'s method, DCAD, and DS-CNN in terms of PSNR improvement. Notably, the MFQE approach achieves an impressive PSNR enhancement, particularly for non-PQFs, thereby mitigating quality fluctuations inherent in compressed sequences. These fluctuations, often detrimental to the viewer's Quality of Experience (QoE), are effectively reduced by the proposed methodology.

Furthermore, by fine-tuning the MF-CNN model for different compression standards, such as H.264, the authors demonstrate the generalizability and adaptability of their approach, maintaining high quality improvements across different codecs.

Implications and Future Directions

From a theoretical standpoint, this research introduces a new perspective in the field of video quality enhancement, emphasizing the potential benefits of multi-frame analysis. Practically, it offers a robust framework for improving video quality post-compression, which could be integrated into streaming services and video editing software to enhance the consumer viewing experience significantly.

Future investigations could explore expanding this mechanism by integrating advanced motion estimation techniques or incorporating additional data from past and future frames to refine the quality enhancement further. As compression techniques evolve and video content continues to proliferate, approaches like MFQE are critical for maintaining high content quality while minimizing bandwidth and storage costs.