"Zero-Shot" Super-Resolution using Deep Internal Learning (1712.06087v1)

Abstract: Deep Learning has led to a dramatic leap in Super-Resolution (SR) performance in the past few years. However, being supervised, these SR methods are restricted to specific training data, where the acquisition of the low-resolution (LR) images from their high-resolution (HR) counterparts is predetermined (e.g., bicubic downscaling), without any distracting artifacts (e.g., sensor noise, image compression, non-ideal PSF, etc). Real LR images, however, rarely obey these restrictions, resulting in poor SR results by SotA (State of the Art) methods. In this paper we introduce "Zero-Shot" SR, which exploits the power of Deep Learning, but does not rely on prior training. We exploit the internal recurrence of information inside a single image, and train a small image-specific CNN at test time, on examples extracted solely from the input image itself. As such, it can adapt itself to different settings per image. This allows to perform SR of real old photos, noisy images, biological data, and other images where the acquisition process is unknown or non-ideal. On such images, our method outperforms SotA CNN-based SR methods, as well as previous unsupervised SR methods. To the best of our knowledge, this is the first unsupervised CNN-based SR method.

• The paper introduces a novel deep internal learning approach that trains an image-specific CNN using only the test image.
• The method leverages the internal recurrence of image features by using down-scaled versions of the input to reconstruct high-resolution details.
• Experimental results show that the approach outperforms traditional supervised methods, particularly for images with unknown degradation.

Deep Internal Learning for Image Super-Resolution: An Overview

The paper under review introduces a novel approach called "Deep Internal Learning" (DIL), with a specific application to the problem of single image super-resolution (SR). Traditional deep learning-based SR methods generally operate under a supervised learning framework, necessitating large datasets for training and often struggle when applied to real-world images suffering from unknown or non-ideal acquisition processes. The proposed DIL method distinguishes itself by utilizing a single image at test time in an unsupervised manner, leveraging the internal recurrence of image features.

Key Methodological Contributions

The core contribution of this research is the implementation of an image-specific convolutional neural network (CNN) that is trained on the test data itself, instead of relying on pre-trained datasets. By exploiting the low entropy information present within a single image, the method offers adaptability to various image settings and features unique to the test image. Essentially, this approach turns the single image into its own training set by down-scaling it and then using these down-scaled versions to teach the network to reconstruct the high-resolution image.

Experimental Results

Empirical validation demonstrates that the proposed method performs robust super-resolution on diverse images, including old photographs, noisy images, and biological datasets, where the degradation process is often unknown. Notably, the DIL method outperforms state-of-the-art (SotA) CNN-based SR methods and previous unsupervised methods, especially in scenarios involving real-world artifacts like sensor noise and image compression. In ideal conditions with high-quality images and known blur kernels, the performance of DIL remains competitive with extensively trained supervised methods.

Implications and Future Work

The implications of this research are multifaceted:

1. Generalization and Robustness: The ability of DIL to generalize across various types of image degradation without needing a pre-defined training set marks a significant advancement in the field of image processing.
2. Practical Applications: By enabling robust SR for non-ideal acquisition conditions, the method has practical applications in fields such as medical imaging, historical image restoration, and general photography enhancement.

Proposed Extensions

Several future research directions are proposed, emphasizing the versatility and potential of the DIL methodology:

• Blind Super-Resolution using Coupled Autoencoders: Future work intends to integrate pairs of autoencoders for simultaneous SR and down-scaling method estimation, effectively addressing more complex, non-linear degradation effects.
• Enhancing SR with Hybrid Learning Models: A potential avenue is the fusion of internal learning with externally trained networks to optimize results based on the availability of external examples.
• Expansion to Other Image Enhancement Tasks: The principles of DIL could be extended to denoising, deblurring, dehazing, and correcting rolling shutter effects.
• High-Level Vision Tasks: Beyond low-level enhancements, the paper proposes applying DIL to high-level tasks such as edge-based image reconstruction and colorization, facilitating innovative style and domain-transfer techniques.
• Theoretic Modeling: The paper suggests investigating the distribution of information in natural images through the lens of deep learning, potentially uncovering new aspects of internal and external data distributions.
• Video Spatial-Temporal Super-Resolution: The method's expansion into the temporal domain aims to achieve space-time SR for video sequences.

Conclusion

The approach detailed in this paper provides a significant shift from traditional supervised learning paradigms, offering an unsupervised, image-specific method for robust and adaptable super-resolution. The promising results and the outlined future work highlight the method's potential for broad applications and further research within the domain of image enhancement and beyond.