HYPIR: Diffusion-GAN Image Restoration

• HYPIR is a two-stage framework combining pretrained diffusion models with adversarial fine-tuning for restoring images with high fidelity and realistic detail.
• It leverages a LoRA-adapted U-Net and a ConvNeXt-based discriminator to ensure rapid convergence and stability in the restoration process.
• HYPIR achieves state-of-the-art performance, balancing quantitative metrics like LPIPS with robust user control over texture and detail during inference.

HYPIR refers to “HArnessing diffusion-Yielded score Priors for Image Restoration,” a two-stage deep learning framework designed to address persistent challenges in image restoration: removal of degradation, realistic detail generation, and pixel-level fidelity. HYPIR integrates pretrained diffusion models for initialization with adversarial (GAN) fine-tuning, providing state-of-the-art restoration quality with high computational efficiency and enhanced user controls (Lin et al., 28 Jul 2025).

1. Motivation and Context

Image restoration models target transformation from degraded image space to the manifold of natural images. Canonical model classes include MSE-based CNNs (e.g. SRCNN, EDSR) optimized for PSNR/SSIM but prone to over-smoothing, GAN-based approaches that augment perceptual and adversarial losses to boost realism but risk mode collapse and instabilities, and diffusion-based methods which achieve top-tier visual fidelity and diversity via iterative denoising, albeit at the cost of prohibitive inference latency and computational demand.

None of these approaches simultaneously achieves optimal (i) restoration fidelity, (ii) perceptually realistic detail, and (iii) fast, stable training and inference. HYPIR was developed to overcome these trade-offs by leveraging the mode coverage and semantic priors of diffusion models as an initialization for fast adversarial refinement (Lin et al., 28 Jul 2025).

2. Technical Pipeline

HYPIR comprises two main stages:

Stage I — Diffusion Model Initialization

The framework adopts a pretrained latent diffusion model (e.g., SD2, SDXL, SD3, or FLUX) equipped with a U-Net backbone and VAE encoder/decoder $\mathcal{V}_E$, $\mathcal{V}_D$. A diffusion model learns the time-dependent score $$s_\theta(x_t, t) \approx \nabla_{x_t} \log p_t(x_t)$$. A single-step restoration is effected by: $$\hat{x}_0 = \frac{1}{\sqrt{\bar{\alpha}(t)}} \left( x_t + (1-\bar{\alpha}(t)) s_\theta(x_t, t) \right)$$ Feeding a VAE-encoded latent $y = \mathcal{V}_{E_R}(I_{\mathrm{deg}})$ at a noise-level $t$ yields an approximate reconstruction situated close to the natural image manifold.

Stage II — Adversarial Fine-Tuning

The VAE (encoder $\mathcal{V}_{E_R}$, decoder $\mathcal{V}_D$) is fixed; the U-Net $\mathcal{U}_\theta$ is initialized at $\theta \leftarrow \theta_{\mathrm{Diff}}$ and trained jointly with a ConvNeXt-based discriminator $\mathcal{V}_D$0. The loss is a sum of a conditional GAN objective and a pixel- plus perceptual-reconstruction term: $\mathcal{V}_D$1 where

$\mathcal{V}_D$2

$\mathcal{V}_D$3

Only LoRA adapters in the U-Net and the discriminator are updated; the substantial pretrained backbone acts as a strong prior.

3. Theoretical Guarantees

HYPIR provides several theoretical properties that underpin its empirical stability and performance:

• Diffusion-to-Restoration Proximity: For mild score approximation error $\mathcal{V}_D$4 and small degradation–diffusion kernel mismatch $\mathcal{V}_D$5, the 2-Wasserstein distance between the initialized network's pushforward $\mathcal{V}_D$6 and the true data distribution satisfies:

$\mathcal{V}_D$7

• Small Initial Gradient: Plugging this initialization into the GAN loss yields generator gradient at $\mathcal{V}_D$8 bounded by $\mathcal{V}_D$9, providing numerical stability from the outset.
• Uniform Mode-Mass Coverage: All measurable image subspaces $$s_\theta(x_t, t) \approx \nabla_{x_t} \log p_t(x_t)$$0 satisfy $$s_\theta(x_t, t) \approx \nabla_{x_t} \log p_t(x_t)$$1, essentially ensuring absence of mode collapse.
• Accelerated Convergence: If the generator loss is locally smooth and strongly convex, gradient descent converges to within $$s_\theta(x_t, t) \approx \nabla_{x_t} \log p_t(x_t)$$2 of optimum in $$s_\theta(x_t, t) \approx \nabla_{x_t} \log p_t(x_t)$$3 steps, with empirical convergence in approximately 10,000 steps (vs $$s_\theta(x_t, t) \approx \nabla_{x_t} \log p_t(x_t)$$4 from scratch).

4. Implementation Details

• Diffusion Backbones: SD2 (0.8B), SDXL (2.6B), SD3 (8B), FLUX (12B).
• VAE: Encoder/decoder inherited from diffusion backbone.
• Restoration U-Net: LoRA-adapted; only LoRA weights (rank 64) are updated.
• Discriminator: ConvNeXt-base, pretrained.
• Degradation Pre-removal: Fine-tuning $$s_\theta(x_t, t) \approx \nabla_{x_t} \log p_t(x_t)$$5 via a VAE-based reconstruction loss.
• Loss Weights: $$s_\theta(x_t, t) \approx \nabla_{x_t} \log p_t(x_t)$$6, $$s_\theta(x_t, t) \approx \nabla_{x_t} \log p_t(x_t)$$7, $$s_\theta(x_t, t) \approx \nabla_{x_t} \log p_t(x_t)$$8.
• Optimization: AdamW, learning rate $$s_\theta(x_t, t) \approx \nabla_{x_t} \log p_t(x_t)$$9; batch size 384 (accumulated effective 1536); 10k + 10k steps.
• Hardware: 64 NVIDIA A6000 GPUs, EMA decay 0.999.
• Input Resolutions: $$\hat{x}_0 = \frac{1}{\sqrt{\bar{\alpha}(t)}} \left( x_t + (1-\bar{\alpha}(t)) s_\theta(x_t, t) \right)$$0 (SD2/SDXL), $$\hat{x}_0 = \frac{1}{\sqrt{\bar{\alpha}(t)}} \left( x_t + (1-\bar{\alpha}(t)) s_\theta(x_t, t) \right)$$1 (SD3/FLUX); patch-based inference for larger images.
• Inference: Single forward pass.

5. Capabilities and User Control

HYPIR leverages inherited capabilities of large-scale diffusion models:

• Text-Guided Restoration: Conditional on text embeddings via LLaVA-generated captions at both training and test time; enables prompt-driven imputation in missing regions.
• Texture-Richness Control: User-defined Laplacian-derived texture statistic $$\hat{x}_0 = \frac{1}{\sqrt{\bar{\alpha}(t)}} \left( x_t + (1-\bar{\alpha}(t)) s_\theta(x_t, t) \right)$$2 modulates sharpness/granularity during inference.
• Fidelity–Generativity Trade-off: Injecting Gaussian noise into latent $$\hat{x}_0 = \frac{1}{\sqrt{\bar{\alpha}(t)}} \left( x_t + (1-\bar{\alpha}(t)) s_\theta(x_t, t) \right)$$3 interpolates between high-fidelity (low noise) and more creative outputs (high noise).
• Random Sampling: Multiple plausible reconstructions generated via noise seed variation. This configurability augments both utility and scientific interpretability (Lin et al., 28 Jul 2025).

6. Experimental Results

• Qualitative: On DIV2K (synthetic) and RealPhoto60 (real), HYPIR(SD2) outperforms RealESRGAN, StableSR, DiffBIR, OSEDiff, and others—generating sharper textures and more faithful structures. HYPIR(FLUX) enables restoration of historical photographs at up to $$\hat{x}_0 = \frac{1}{\sqrt{\bar{\alpha}(t)}} \left( x_t + (1-\bar{\alpha}(t)) s_\theta(x_t, t) \right)$$4 resolution.
• Quantitative: On DIV2K/RealPhoto60,
  • LPIPS: HYPIR(FLUX) achieves LPIPS 0.2022 (lowest).
  • No-Reference Metrics: Consistently top-ranked or within 1% of the best for NIQE, MUSIQ, MANIQA, CLIP-IQA, DeQA.
  • User Studies: Highest mean perceptual scores in both “lightweight” and “large-scale” model cohorts (100 participants, 26 images).
• Speed: Converges in $$\hat{x}_0 = \frac{1}{\sqrt{\bar{\alpha}(t)}} \left( x_t + (1-\bar{\alpha}(t)) s_\theta(x_t, t) \right)$$5 steps (vs $$\hat{x}_0 = \frac{1}{\sqrt{\bar{\alpha}(t)}} \left( x_t + (1-\bar{\alpha}(t)) s_\theta(x_t, t) \right)$$6 from scratch); inference for $$\hat{x}_0 = \frac{1}{\sqrt{\bar{\alpha}(t)}} \left( x_t + (1-\bar{\alpha}(t)) s_\theta(x_t, t) \right)$$7 images takes 0.1–0.2 s (vs 20–120 s for multi-step diffusion).

7. Limitations and Future Directions

• Prompt Conditioning: Reliance on automated LLaVA captions restricts prompt expressivity; manual or improved prompt pipelines could enhance controllability.
• Severe Degradations: For structured/non-Gaussian artifacts, augmenting the likelihood with $$\hat{x}_0 = \frac{1}{\sqrt{\bar{\alpha}(t)}} \left( x_t + (1-\bar{\alpha}(t)) s_\theta(x_t, t) \right)$$8 (Eq. 6 in the original paper) may be necessary rather than purely unconditional denoising.
• Extension to Video: Time-series consistency modules would be required for video restoration.

A plausible implication is that diffusion-initialized GANs offer a general recipe for fast, high-quality image restoration across diverse degradations, provided the appropriate design and initialization regimes are respected (Lin et al., 28 Jul 2025).