Deep Learning for Image Super-resolution: A Survey (1902.06068v2)

Abstract: Image Super-Resolution (SR) is an important class of image processing techniques to enhance the resolution of images and videos in computer vision. Recent years have witnessed remarkable progress of image super-resolution using deep learning techniques. This article aims to provide a comprehensive survey on recent advances of image super-resolution using deep learning approaches. In general, we can roughly group the existing studies of SR techniques into three major categories: supervised SR, unsupervised SR, and domain-specific SR. In addition, we also cover some other important issues, such as publicly available benchmark datasets and performance evaluation metrics. Finally, we conclude this survey by highlighting several future directions and open issues which should be further addressed by the community in the future.

• The paper presents a comprehensive review of deep learning methods for image super-resolution, categorizing techniques into supervised, unsupervised, and domain-specific applications.
• It details innovative network architectures and training strategies, including iterative upsampling, residual learning, and varied loss functions that enhance model performance.
• The survey evaluates standard datasets and metrics like PSNR and SSIM, outlining future directions focused on network efficiency and tailored domain improvements.

Deep Learning for Image Super-resolution: A Survey

Introduction

The paper "Deep Learning for Image Super-resolution: A Survey" by Zhihao Wang, Jian Chen, and Steven C.H. Hoi provides a rigorous review of recent advancements in image super-resolution (SR) achieved using deep learning techniques. This comprehensive survey categorizes the methods into three primary groups: supervised SR, unsupervised SR, and domain-specific SR applications, with additional emphasis on publicly available benchmark datasets and performance evaluation metrics.

Problem Setting and Key Terminology

Problem Definition: The paper defines image super-resolution as the task of recovering a high-resolution (HR) image from a low-resolution (LR) image. The difficulty of SR arises from the ill-posed nature of the problem, as multiple HR images can map to a single LR image.

Datasets and Metrics: Several datasets, such as DIV2K, Set5, and Urban100, have become standard for benchmarking SR algorithms. The paper reviews evaluation metrics including Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and more recent learning-based perceptual metrics.

Supervised Super-resolution

Frameworks:

1. Pre-upsampling SR – Initial LR image is upsampled using conventional methods before being processed by CNNs.
2. Post-upsampling SR – CNNs first extract features from the LR image, after which learnable upscaling layers are applied at the end.
3. Progressive Upsampling SR – A series of CNNs progressively upsamples the image, which is particularly appropriate for large scaling factors.
4. Iterative Up-and-down Sampling SR – Using iterative refinements via back-projection to better capture LR-HR dependencies.

Upsampling Methods: The survey contrasts traditional interpolation-based upsampling (bicubic and bilinear) with more modern learnable methods like transposed convolution and sub-pixel convolution layers, emphasizing the importance of end-to-end learnability and computational efficiency.

Network Designs: Innovative network designs like residual learning, recursive learning, dense connections, and attention mechanisms have significantly enhanced SR performance. Residual learning, notably employed in ResNet architecture, mitigates the vanishing gradient problem and improves convergence rates.

Learning Strategies: The paper discusses the role of varied loss functions (e.g., content loss, adversarial loss, texture loss) and techniques such as batch normalization, curriculum learning, and multi-supervision, all of which facilitate efficient and effective training of SR models.

Unsupervised Super-resolution

The survey underscores the relevance of unsupervised SR, especially given the difficulty of collecting paired LR-HR datasets. Methods like Zero-shot Super-resolution (ZSSR) and Cycle-in-Cycle GAN (CinCGAN) indicate promising avenues where models can generalize better to real-world scenarios with diverse degradation processes.

Domain-specific SR Applications

Applications of SR span several domains:

• Depth Map SR: Using HR RGB images for guiding the SR process of depth maps.
• Face Image SR: Incorporating facial priors significantly boosts the performance of face-specific SR models.
• Hyperspectral Image SR: Enhances spatial resolution of hyperspectral images using high-resolution panchromatic images.
• Video SR: Utilizes temporal coherence across video frames to improve the quality of SR, with methods like VSRnet and FRVSR combining both spatial and temporal information.

Challenges and Future Directions

The survey identifies several challenges remaining in the field:

• Network Efficiency: While current models offer state-of-the-art performance, their resource demands remain high. Future models should balance performance with computational efficiency.
• Better Loss Functions: Developing more nuanced loss functions that align with human perceptual quality remains crucial.
• Unsupervised Learning: Enhancing unsupervised methods will make SR applicable to a broader range of real-world settings where paired datasets are not available.
• Domain-specific Improvements: Tailoring SR techniques to specific application domains like medical imaging and video processing will further push the boundaries of the field.

Conclusion

This survey provides a valuable consolidation of how deep learning techniques have revolutionized image super-resolution over recent years. By offering detailed insights into methodologies, applications, and future directions, it serves as a foundational reference for researchers pursuing advancements in SR and related areas.