Blind Super-Resolution Kernel Estimation using an Internal-GAN (1909.06581v6)

Abstract: Super resolution (SR) methods typically assume that the low-resolution (LR) image was downscaled from the unknown high-resolution (HR) image by a fixed 'ideal' downscaling kernel (e.g. Bicubic downscaling). However, this is rarely the case in real LR images, in contrast to synthetically generated SR datasets. When the assumed downscaling kernel deviates from the true one, the performance of SR methods significantly deteriorates. This gave rise to Blind-SR - namely, SR when the downscaling kernel ("SR-kernel") is unknown. It was further shown that the true SR-kernel is the one that maximizes the recurrence of patches across scales of the LR image. In this paper we show how this powerful cross-scale recurrence property can be realized using Deep Internal Learning. We introduce "KernelGAN", an image-specific Internal-GAN, which trains solely on the LR test image at test time, and learns its internal distribution of patches. Its Generator is trained to produce a downscaled version of the LR test image, such that its Discriminator cannot distinguish between the patch distribution of the downscaled image, and the patch distribution of the original LR image. The Generator, once trained, constitutes the downscaling operation with the correct image-specific SR-kernel. KernelGAN is fully unsupervised, requires no training data other than the input image itself, and leads to state-of-the-art results in Blind-SR when plugged into existing SR algorithms.

• The paper introduces KernelGAN, an unsupervised method that leverages an internal GAN to learn the unknown downscaling kernel from a single low-resolution image.
• It employs cross-scale patch recurrence with a deep linear network to improve convergence and enhance kernel estimation accuracy.
• KernelGAN achieves up to 1 dB improvement for 2× scaling and 0.47 dB for 4×, outperforming current state-of-the-art methods.

Blind Super-Resolution Kernel Estimation using an Internal-GAN

The paper presents a method for blind super-resolution (SR) kernel estimation, addressing the issue of unknown downscaling kernels in real-world low-resolution (LR) images. Traditional SR methods, typically trained on synthetically downscaled datasets using known kernels like bicubic, often falter when applied to LR images derived from unknown, non-ideal downscaling processes. The authors propose "KernelGAN," an unsupervised technique leveraging a single internal Generative Adversarial Network (GAN) that trains on the specific LR image in question during the test phase, learning the internal patch distribution of the image. This approach positions the method well within the broader context of internal learning frameworks, emphasizing its independence from external datasets.

Methodological Insights

The method exploits the property of cross-scale patch recurrence inherent in natural images. This concept, initially observed by Michaeli and Irani, states that the SR kernel maximizing patch similarity across image scales is likely the true downscaling kernel. KernelGAN implements this through an image-specific Internal-GAN composed of a generator $G$ and discriminator $D$, crafted to be fully convolutional. Here, $G$ learns a downscaled version of the LR image, which $D$ finds indistinguishable from the original LR image in terms of patch distribution. Crucially, $G$ is constructed as a deep linear network, an architectural choice driven by findings in optimization theory suggesting improved convergence benefits over single-layer representations.

Numerical Performance and Contrasts with SotA

Empirically, KernelGAN demonstrates significant improvements in blind-SR performance when integrated with contemporary non-blind SR algorithms. The paper details evaluations using the DIV2KRK dataset, a synthetic collection with realistic LR images derived from random kernel applications. The results indicate that KernelGAN, particularly when coupled with the Zero-Shot Super-Resolution (ZSSR) algorithm, surpasses existing state-of-the-art (SotA) methods by considerable margins: 1 dB for scale factor $\times 2$ and 0.47 dB for $\times 4$. Such enhancements underscore the utility of accurate SR-kernel estimation in preserving image detail and reducing artifacts inherent in non-blind approaches that fail to account for kernel variances.

Implications and Future Directions

The paper's contributions extend beyond empirical benchmarks to theoretically ground the benefits of deep linear networks in learning precise SR kernels, aligning with theoretical understandings from deep learning research. This work galvanizes interest in unsupervised and self-supervised learning paradigms for image processing tasks, emphasizing the viability of GAN-based internal models. In practical terms, KernelGAN opens avenues for super-resolution in diverse application domains—particularly where real-world degradation processes are poorly understood or uncharacterized.

In terms of future exploration, the intersections of KernelGAN's mechanisms with other self-supervised learning paradigms present intriguing prospects. Further, the methodology presents potential for adaptation in domains beyond super-resolution, such as unsupervised restoration in compressed or corrupted images, where kernel estimation remains a challenge. As AI progresses toward deploying robust models for such tasks in the wild, understanding and improving the dynamics of these internal learning methods will be pivotal.

The paper serves as a bellwether for future generative approaches to low-level vision tasks and encourages the exploration of internal learning across a spectrum of computer vision challenges.