Blind Image Restoration via Fast Diffusion Inversion (2405.19572v2)

Abstract: Image Restoration (IR) methods based on a pre-trained diffusion model have demonstrated state-of-the-art performance. However, they have two fundamental limitations: 1) they often assume that the degradation operator is completely known and 2) they alter the diffusion sampling process, which may result in restored images that do not lie onto the data manifold. To address these issues, we propose Blind Image Restoration via fast Diffusion inversion (BIRD) a blind IR method that jointly optimizes for the degradation model parameters and the restored image. To ensure that the restored images lie onto the data manifold, we propose a novel sampling technique on a pre-trained diffusion model. A key idea in our method is not to modify the reverse sampling, i.e, not to alter all the intermediate latents, once an initial noise is sampled. This is ultimately equivalent to casting the IR task as an optimization problem in the space of the input noise. Moreover, to mitigate the computational cost associated with inverting a fully unrolled diffusion model, we leverage the inherent capability of these models to skip ahead in the forward diffusion process using large time steps. We experimentally validate BIRD on several image restoration tasks and show that it achieves state of the art performance on all of them. Our code is available at https://github.com/hamadichihaoui/BIRD.

• The paper introduces SHRED, a novel zero-shot image restoration method that optimizes the initial noise in diffusion inversion without altering intermediate latent variables to adhere strictly to the data manifold.
• SHRED improves computational efficiency via strategic time step adjustments in the forward diffusion process and is applicable to a wide range of blind and non-blind image restoration tasks like inpainting, super-resolution, and deconvolution.
• Experimental results demonstrate that SHRED matches or surpasses state-of-the-art zero-shot methods, achieving competitive fidelity and efficiency, exhibiting robust inversion, and having practical implications for diverse real-world degradations.

Zero-shot Image Restoration via Diffusion Inversion

The paper introduces a novel method for image restoration (IR) that leverages the capabilities of diffusion models without altering the intermediate latent variables during reverse sampling. Diffusion models have been a focal point in generative learning for tasks that require high-quality image synthesis. They offer smooth representations of learned data distributions which are invaluable for solving IR tasks. Previous methods often modify the diffusion model's reverse process to adhere to the constraints imposed by the corrupted image. However, this paper contends that such modifications can lead to suboptimal results and potential deviations from the natural data manifold.

Method: SHRED

The proposed method, termed SHRED (Zero-SHot image REstoration via Diffusion inversion), frames the IR task as an optimization problem concentrated on the diffusion input noise space. This strategy ensures that the generation paths abide strictly by the data manifold, as it refrains from altering intermediate latent variables post-initiation from a noise sample. By utilizing Denoising Diffusion Implicit Models (DDIM), SHRED adopts a deterministic path in the diffusion process.

Key innovations in SHRED include:

1. Latent Optimization: Casting the IR task as a latent optimization problem where only the initial noise is subject to optimization. This design choice upholds the integrity of the data manifold.
2. Efficiency in Computation: The method counters the computational cost associated with inverting a diffusion model through strategic time step adjustments in the forward diffusion process. This hyperparameter ($\delta t$) facilitates a balance between image quality and the computational efficiency of the procedure.
3. Versatility Across Sampling Rates: SHRED is applicable to both blind and non-blind IR tasks, including image inpainting, super-resolution at various scales, compressed sensing, and blind deconvolution.

Results

The paper details rigorous experimental validation across multiple benchmarks, demonstrating that SHRED either matches or surpasses the current state-of-the-art in zero-shot IR methods. Particularly noteworthy are the following observations:

• Fidelity and Efficiency: The method achieves competitive performance in LPIPS and FID scores, particularly excelling in high-sampling rates for compressed sensing tasks and maintaining fidelity across several levels of image degradation.
• Robust Inversion: Experiments indicate robustness to initial conditions, confirming SHRED's reliability for hitting the natural image manifold consistently.
• Practical Implications: By mitigating computational overhead while maintaining high perceptual quality, SHRED has pronounced applications in scenarios demanding robust image restoration without fine-tuned adjustments for each variant of image degradation.

Implications and Future Directions

The introduction of SHRED has significant implications for the field of AI-driven image restoration. Its reliance on latent space optimization within the diffusion model framework highlights a shift towards harnessing inherent model capabilities without extensive post-process minutiae. This not only paves the way for more efficient applications but also aligns with trends favoring universal, non-specific IR solutions capable of adapting to diverse real-world degradations.

Future work could explore further enhancements in latent space manipulation or integrate SHRED with other generative frameworks to widen its applicability across different domains. Potential research could also delve into its integration in video and three-dimensional data restoration, expanding the horizons of zero-shot capabilities in generative learning.

In conclusion, the SHRED method offers a compelling approach to zero-shot image restoration, relying on the structural capabilities of diffusion models while ensuring computational feasibility and efficacy across diverse image degradation challenges.