One-Step Effective Diffusion Network for Real-World Image Super-Resolution (2406.08177v3)

Abstract: The pre-trained text-to-image diffusion models have been increasingly employed to tackle the real-world image super-resolution (Real-ISR) problem due to their powerful generative image priors. Most of the existing methods start from random noise to reconstruct the high-quality (HQ) image under the guidance of the given low-quality (LQ) image. While promising results have been achieved, such Real-ISR methods require multiple diffusion steps to reproduce the HQ image, increasing the computational cost. Meanwhile, the random noise introduces uncertainty in the output, which is unfriendly to image restoration tasks. To address these issues, we propose a one-step effective diffusion network, namely OSEDiff, for the Real-ISR problem. We argue that the LQ image contains rich information to restore its HQ counterpart, and hence the given LQ image can be directly taken as the starting point for diffusion, eliminating the uncertainty introduced by random noise sampling. We finetune the pre-trained diffusion network with trainable layers to adapt it to complex image degradations. To ensure that the one-step diffusion model could yield HQ Real-ISR output, we apply variational score distillation in the latent space to conduct KL-divergence regularization. As a result, our OSEDiff model can efficiently and effectively generate HQ images in just one diffusion step. Our experiments demonstrate that OSEDiff achieves comparable or even better Real-ISR results, in terms of both objective metrics and subjective evaluations, than previous diffusion model-based Real-ISR methods that require dozens or hundreds of steps. The source codes are released at https://github.com/cswry/OSEDiff.

• The paper proposes OSEDiff, a one-step diffusion method that restores high-quality images by directly utilizing information from low-quality inputs.
• It fine-tunes pre-trained Stable Diffusion models with LoRA layers and variational score distillation to align outputs with natural image distributions.
• OSEDiff achieves over 100-fold faster inference and outperforms multi-step methods on perceptual and no-reference quality metrics.

A One-Step Effective Diffusion Network for Real-World Image Super-Resolution

The development of Real-World Image Super-Resolution (Real-ISR) techniques has been hampered by challenges associated with unknown and complex degradation patterns in low-quality (LQ) images. Traditional methods often require multiple diffusion steps and introduce randomness by initializing from random noise. The paper proposes a novel solution, the One-Step Effective Diffusion network (OSEDiff), which leverages pre-trained text-to-image (T2I) diffusion models for Real-ISR with minimal computational cost.

Key Contributions

The paper introduces OSEDiff, a technique that starts the diffusion process directly from the LQ image without introducing random noise, thereby eliminating the intrinsic variability found in traditional methods. The researchers argue that the intrinsic information within the LQ image serves sufficiently for high-quality (HQ) image restoration.

Methodology

OSEDiff utilizes pre-trained Stable Diffusion (SD) models fortified with trainable LoRA layers to fine-tune the model for Real-ISR tasks. This is achieved without discarding the powerful image priors of the pre-trained models. A significant contribution is the adaptation of the variational score distillation (VSD) in latent space for KL-divergence regularization, ensuring that the output aligns closely with the distribution of natural HQ images.

The proposed architecture uses a UNet backbone without added random noise, allowing rich information extraction from LQ images to directly drive the restoration process. The authors meticulously design the training loss, combining Mean Squared Error (MSE) and LPIPS for data fidelity, and employ VSD as a regularizer to enhance naturalness and generalization capabilities of the generated images.

Experimental Results

Empirical analyses confirm OSEDiff's efficacy, demonstrating its superior performance against state-of-the-art methods across multiple benchmarks, including DIV2K-Val, DrealSR, and RealSR. The model achieves remarkable improvements in perceptual quality metrics like LPIPS, DISTS, and FID, and achieves high scores in no-reference visual quality measures such as CLIPIQA and MUSIQ. Interestingly, although OSEDiff performs only a single diffusion step, it outperforms multi-step methods in a variety of scenarios.

Computational Efficiency

One of the most compelling aspects of OSEDiff is its drastic reduction in computational demand. With only one diffusion step, OSEDiff showcases over a hundredfold reduction in inference time compared to traditional methods like StableSR. The use of LoRA significantly reduces the number of trainable parameters, enhancing the model's efficiency while maintaining output quality.

Implications and Future Directions

The introduction of OSEDiff opens up possibilities for more efficient and effective applications of diffusion models in Real-ISR. By highlighting the potential of fine-tuning pre-trained models with low-rank adaptations, this work encourages future exploration into reducing computational overheads while maximizing performance outcomes in similar image restoration tasks. Future research could explore enhancing detail generation and tackling cases with intricate structures such as scene text, which remain challenging for OSEDiff.

In conclusion, the paper provides a promising direction for refining Real-ISR methodologies, addressing critical computational limitations, and ensuring high-quality outputs without excessive resource demands. The insights gained from this paper not only contribute to the ISR community but also to broader applications in computer vision where efficient, detail-oriented image generation is paramount.