AnimeSR: Learning Real-World Super-Resolution Models for Animation Videos (2206.07038v3)

Abstract: This paper studies the problem of real-world video super-resolution (VSR) for animation videos, and reveals three key improvements for practical animation VSR. First, recent real-world super-resolution approaches typically rely on degradation simulation using basic operators without any learning capability, such as blur, noise, and compression. In this work, we propose to learn such basic operators from real low-quality animation videos, and incorporate the learned ones into the degradation generation pipeline. Such neural-network-based basic operators could help to better capture the distribution of real degradations. Second, a large-scale high-quality animation video dataset, AVC, is built to facilitate comprehensive training and evaluations for animation VSR. Third, we further investigate an efficient multi-scale network structure. It takes advantage of the efficiency of unidirectional recurrent networks and the effectiveness of sliding-window-based methods. Thanks to the above delicate designs, our method, AnimeSR, is capable of restoring real-world low-quality animation videos effectively and efficiently, achieving superior performance to previous state-of-the-art methods. Codes and models are available at https://github.com/TencentARC/AnimeSR.

• The paper introduces learnable degradation operators that capture authentic degradations from low-quality animation videos.
• It constructs the AVC dataset with 553 high-quality, diverse animation clips to enhance model training and evaluation.
• It develops an efficient multi-scale network architecture that improves processing speed and reduces artifacts in super-resolved animations.

Overview of "AnimeSR: Learning Real-World Super-Resolution Models for Animation Videos"

This paper addresses the challenge of real-world video super-resolution (VSR) specifically for animation content. It presents a novel approach called AnimeSR, which focuses on three critical improvements over existing methods: using learnable degradation models, constructing a comprehensive dataset, and developing an efficient multi-scale network structure.

Key Contributions

1. Learnable Degradation Operators: Traditional real-world super-resolution techniques utilize basic, non-learnable operators like blur and noise, attempting to simulate degradation. This paper introduces an innovative method where basic degradation operators are learned from real low-quality animation videos using tiny neural networks. These neural-network-based operators are integrated within the degradation pipeline, providing a nuanced capture of real-world degradation distributions.
2. AVC Dataset: A large, high-quality animation video dataset, named AVC (Animation Video Clip), has been constructed to support effective training and evaluation processes. The dataset comprises 553 high-quality clips with diverse animation styles. Such a dataset is crucial for comprehensive model training and evaluation, overcoming the limitations posed by previous datasets with only single images or low-quality frames.
3. Efficient Multi-Scale Network Structure: The paper proposes a network architecture leveraging the efficiency of unidirectional recurrent networks and the effectiveness of sliding-window-based methods. The network consists of multi-scale design elements within recurrent blocks, enhancing its ability to fuse features across different scales, a proficiency particularly beneficial for animation content that often features simple lines and smooth color regions.

Methodology

The proposed AnimeSR architecture makes practical improvements in animation VSR by intelligently synthesizing degradation processes. The integration of learnable operators is facilitated by an "input-rescaling strategy," ensuring that even without paired LR-HR data, effective models can be trained. This strategy is inspired by the unique characteristics of animation videos, notably the prominence of lines and smooth color fields.

Moreover, the dataset development (AVC) undertakes rigorous selection criteria, incorporating high-quality clips and diverse styles, further augmented by manual and algorithmic curation. This ensures the training data's relevance and diversity to generalize well to real-world scenarios.

Experimental Results

AnimeSR was evaluated against several existing methods, including Real-ESRGAN and RealBasicVSR. The results suggest that AnimeSR provides superior performance with cleaner outputs and fewer artifacts. Notably, AnimeSR maintains higher efficiency, processing animations significantly faster than its predecessors while using fewer parameters.

Evaluations were primarily performed on the AVC-RealLQ test set using no-reference image quality assessment (NR-IQA) metrics such as NIQE and MANIQA scores. The MANIQA score, in particular, aligns well with perceptual quality, reinforcing the improved visual quality achieved by AnimeSR.

Implications and Future Directions

The implications of this work are substantial for both the practical enhancement of existing animation content and theoretical advancements in understanding degradation processes. By innovating the integration of neural networks into the degradation synthesis process, AnimeSR sets a precedence for more dynamic and adaptable super-resolution techniques.

Future research could explore broader applications of neural-based degradation operators beyond animation, testing the adaptability of such models to various types of content. Additionally, there's potential in expanding the dataset pool with more diverse animation styles or attempting a similar methodological approach within different video content domains to bridge the gap between simulated and real-world degradations.

In summary, "AnimeSR: Learning Real-World Super-Resolution Models for Animation Videos" is a significant contribution to the field of VSR, offering a sophisticated approach to tackling the nuances of animation video content, thus enhancing the viewer's experience in the high-definition era.