DeSRA: Detect and Delete the Artifacts of GAN-based Real-World Super-Resolution Models (2307.02457v1)

Abstract: Image super-resolution (SR) with generative adversarial networks (GAN) has achieved great success in restoring realistic details. However, it is notorious that GAN-based SR models will inevitably produce unpleasant and undesirable artifacts, especially in practical scenarios. Previous works typically suppress artifacts with an extra loss penalty in the training phase. They only work for in-distribution artifact types generated during training. When applied in real-world scenarios, we observe that those improved methods still generate obviously annoying artifacts during inference. In this paper, we analyze the cause and characteristics of the GAN artifacts produced in unseen test data without ground-truths. We then develop a novel method, namely, DeSRA, to Detect and then Delete those SR Artifacts in practice. Specifically, we propose to measure a relative local variance distance from MSE-SR results and GAN-SR results, and locate the problematic areas based on the above distance and semantic-aware thresholds. After detecting the artifact regions, we develop a finetune procedure to improve GAN-based SR models with a few samples, so that they can deal with similar types of artifacts in more unseen real data. Equipped with our DeSRA, we can successfully eliminate artifacts from inference and improve the ability of SR models to be applied in real-world scenarios. The code will be available at https://github.com/TencentARC/DeSRA.

• The paper introduces DeSRA, a novel method that detects and removes GAN-induced artifacts by comparing MSE-SR outputs with GAN-SR results.
• It employs a two-step process combining local variance analysis and iterative fine-tuning to construct pseudo ground-truth for artifact correction.
• Empirical validation demonstrates over 75% artifact reduction, high IoU, and improved perceptual quality in real-world super-resolution tasks.

An Analysis of DeSRA: Addressing GAN-Induced Artifacts in Image Super-Resolution

The paper "DeSRA: Detect and Delete the Artifacts of GAN-based Real-World Super-Resolution Models" by Xie et al. provides an in-depth paper and methodological advancement in addressing a critical issue in the domain of super-resolution (SR) using Generative Adversarial Networks (GANs). Despite their ability to generate visually appealing high-resolution images, GAN-based SR models often produce undesirable artifacts, especially when applied to real-world, unseen data. This paper presents a novel approach named DeSRA, which detects and removes such artifacts, enhancing the application potential of GAN-SR models in practical environments.

Overview of GAN-induced Artifacts in Image Super-Resolution

Single image super-resolution (SISR) seeks to generate high-resolution (HR) images from low-resolution (LR) counterparts. While non-GAN methods often fail to reproduce fine textures, GAN-based SR models excel at generating detailed images. However, they are prone to introducing perceptually unpleasant artifacts during both the training and inference stages. More problematically, these GAN-inference artifacts are typically out-of-distribution and emerge only during the processing of unseen real-world data.

Methodological Innovations in DeSRA

DeSRA addresses the GAN-inference artifacts through a two-step process. It begins with artifact detection by measuring the relative local variance distance between MSE-based SR results and GAN-SR outputs. Using a combination of local texture differences and semantic-aware thresholds, this method effectively identifies regions marred by artifacts. DeSRA then employs a fine-tuning strategy to iteratively improve the GAN-SR model using a limited dataset. By replacing artifact regions in GAN-SR outputs with MSE-SR results, a pseudo ground-truth set is constructed, enabling the model to generalize and reduce artifacts in unseen data.

Empirical Validation and Results

Experiments conducted with Real-ESRGAN and LDL models demonstrate the efficacy of DeSRA in artifact detection and reduction. The method achieves high Intersection over Union (IoU), precision, and recall metrics in detecting artifacts across various datasets. Significantly, visual and quantitative results indicate that DeSRA effectively reduces artifact presence, confirmed by both technical metrics and human perceptual studies. Post-processing with DeSRA leads to an artifact removal rate of over 75% on evaluated datasets, while simultaneously preventing additional artifacts.

Theoretical Implications and Future Research Directions

DeSRA's methodology of leveraging MSE-SR results as a reference brings a valuable perspective on addressing artifacts without actual ground-truth data, which is often unavailable in real-world scenarios. The work underlines the potential of adaptive fine-tuning methods in improving the robustness of GAN-based systems. Future research may extend DeSRA's concepts, potentially exploring more sophisticated unsupervised or self-supervised learning techniques to better handle diverse real-world degradations and unseen artifacts.

Conclusion

The research presented in this paper marks an important development in enhancing the operational efficiency of GAN-based super-resolution models in real-world applications. By systematically identifying and mitigating GAN-induced artifacts, DeSRA serves as a pivotal step forward, promoting the practical deployment of SR technology across diverse and complex environmental settings. As the field progresses, integrating continual learning paradigms with methodologies like DeSRA could further solidify the bridge between high-fidelity image generation and robust, artifact-free output in real-world applications.