Cross-Scale Internal Graph Neural Network for Image Super-Resolution (2006.16673v2)

Abstract: Non-local self-similarity in natural images has been well studied as an effective prior in image restoration. However, for single image super-resolution (SISR), most existing deep non-local methods (e.g., non-local neural networks) only exploit similar patches within the same scale of the low-resolution (LR) input image. Consequently, the restoration is limited to using the same-scale information while neglecting potential high-resolution (HR) cues from other scales. In this paper, we explore the cross-scale patch recurrence property of a natural image, i.e., similar patches tend to recur many times across different scales. This is achieved using a novel cross-scale internal graph neural network (IGNN). Specifically, we dynamically construct a cross-scale graph by searching k-nearest neighboring patches in the downsampled LR image for each query patch in the LR image. We then obtain the corresponding k HR neighboring patches in the LR image and aggregate them adaptively in accordance to the edge label of the constructed graph. In this way, the HR information can be passed from k HR neighboring patches to the LR query patch to help it recover more detailed textures. Besides, these internal image-specific LR/HR exemplars are also significant complements to the external information learned from the training dataset. Extensive experiments demonstrate the effectiveness of IGNN against the state-of-the-art SISR methods including existing non-local networks on standard benchmarks.

• The paper proposes a novel Cross-Scale Internal Graph Neural Network (IGNN) for single image super-resolution (SISR) by exploiting non-local self-similarity across different image scales.
• The method uses a Cross-Scale Graph Aggregation module to construct a graph of cross-scale correlations between image patches, aggregating corresponding high-resolution patch information to recover details.
• Extensive experiments show the IGNN outperforms state-of-the-art SISR models on standard benchmarks like Set5 and BSD100, demonstrating improved performance in PSNR and SSIM metrics.

Overview of Cross-Scale Internal Graph Neural Network for Image Super-Resolution

The paper "Cross-Scale Internal Graph Neural Network for Image Super-Resolution" proposes a novel approach to Single Image Super-Resolution (SISR) leveraging the non-local self-similarity property of images. Traditional methods often focus on exploiting internal image structures at the same scale to enhance resolution, but they fail to integrate cross-scale information effectively. This work proposes a cross-scale internal graph neural network (IGNN) that utilizes internal image-specific exemplars across different scales to achieve more detailed texture recovery in super-resolution tasks.

Introduction to the Approach

The authors introduce a Cross-Scale Graph Aggregation module (GraphAgg), which constructs a graph representing cross-scale correlations between patches in the image. The aim is to exploit the recurrence of similar patches across different scales in a natural image, which remains underutilized in existing methods. By dynamically constructing a graph with $k$-nearest neighbors and aggregating high-resolution (HR) information from the corresponding HR patches, the model helps recover more detailed features in the low-resolution (LR) query patch.

Methodology

The methodology revolves around constructing a cross-scale graph which consists of vertices representing patches and edges representing similarity-weighted connections between patches across different scales. The key operations in the GraphAgg module include:

1. Graph Construction: Queries for $k$-nearest neighbors in the downsampled LR image and identifies corresponding HR patches.
2. Patch Aggregation: Aggregates these HR patches using a weighted scheme based on edge properties determined by similarity metrics.

Adaptive mechanisms like Edge-Conditioned Convolution (ECC) and Adaptive Patch Normalization (AdaPN) help in robustly transferring the high-frequency details from HR patches to LR ones, enhancing the learned feature representations.

Numerical Results and Comparisons

Extensive experiments demonstrate the superiority of IGNN compared to state-of-the-art SISR models. The network is shown to outperform existing methods in benchmarks such as Set5, Set14, BSD100, Urban100, and Manga109, with notable improvements in PSNR and SSIM metrics. The results emphasize the efficiency of cross-scale patch recurrence exploitation, providing a significant competitive edge compared to traditional same-scale patch aggregation strategies.

Implications and Future Directions

The paper's approach to utilizing cross-scale patch recurrence has practical implications in advancing image restoration techniques, particularly in super-resolution tasks. The proposed model effectively alleviates the ill-posed nature of SISR by leveraging image-specific HR patch information, offering potential improvements in various applications like consumer electronics, surveillance systems, and autonomous vehicles, where enhanced image quality is pivotal.

Future developments could explore integrating more sophisticated graph-based aggregation strategies and adaptations in different domains such as video and facial recognition, where resolution enhancement continues to be of essence. Further investigation into scalability and integration with emerging AI technologies could unveil broader capabilities in image analysis and improvement.