Detail Restoration Diffusion Model

• Detail restoration diffusion models are diffusion-based image restoration methods that recover high-frequency textures, semantic details, and structural coherence under degradation constraints.
• They encompass various formulations including restoration backbones, training-time diffusion priors, hybrid residual detail generators, and efficient one-step approaches.
• These models balance the distortion–perception trade-off by combining reconstructive fidelity with enhanced perceptual realism across applications like dehazing, blind face restoration, and cultural heritage preservation.

Searching arXiv for recent and foundational papers on diffusion-based detail restoration and related image restoration formulations. A detail restoration diffusion model is best understood as an umbrella designation for diffusion-based image restoration methods whose objective is not only to suppress degradations, but also to recover realistic high-frequency structure, semantically faithful content, and structurally coherent local detail. In the recent literature, this designation covers several distinct formulations: diffusion models used directly as restoration backbones, diffusion priors used only during training or optimization, hybrid systems that reserve diffusion for residual-detail synthesis, and accelerated or one-step variants that retain diffusion’s perceptual benefits while reducing inference cost (Tan et al., 2024, Wang et al., 2023, Yue et al., 2024).

1. Definition and conceptual boundaries

The modern detail-restoration literature begins from a shared diagnosis: conventional low-level restoration objectives often produce outputs that are globally reasonable yet locally unsatisfactory. In difficult inverse problems such as dehazing, deraining, low-light enhancement, denoising, deblurring, super-resolution, and blind face restoration, standard $L_1/L_2$ supervision can favor over-smoothed averages, while VGG perceptual losses and GAN losses introduce their own limitations. The formulation in "DiffLoss" (Tan et al., 2024) makes this explicit: pixel-space regression does not constrain outputs to lie on the distribution of natural clean images, VGG features are optimized for recognition rather than low-level restoration, and adversarial training is unstable and often task-specific. "Reconstruct-and-Generate Diffusion Model" (Wang et al., 2023) frames the same problem through spectral bias: reconstructive denoisers fit low-frequency structure more easily than subtle texture, leading to smooth but visually bland outputs.

A detail restoration diffusion model therefore differs from a generic diffusion generator in its restoration target and from a conventional restorer in its use of a generative prior. Its purpose is not unrestricted image synthesis, but reconstruction under measurement, degradation, or conditioning constraints. The core tension is the distortion–perception trade-off: deterministic restoration tends to improve PSNR and SSIM but can erase texture, while stochastic diffusion sampling improves perceptual realism but can sacrifice data fidelity or hallucinate content. "Reconciling Diffusion Model in Dual" formalizes this conflict directly and treats it as the central problem of zero-shot restoration (Wang et al., 3 Mar 2025).

This suggests that the term denotes a class of methods rather than a single architecture. What unifies that class is the treatment of diffusion as a mechanism for restoring missing or weakened detail—texture, edges, thin structures, facial identity cues, semantic consistency, or spatial coherence—under explicit restoration constraints.

2. Recurrent architectural formulations

Across the literature, several recurrent formulations appear. One formulation uses diffusion as the restoration backbone itself. "Refusion" restores images directly through an IR-SDE-based diffusion process and extends that process into a U-Net-based latent diffusion formulation for very large images, including $6000 \times 4000 \times 3$ HR dehazing (Luo et al., 2023). "DiffBFR" uses cascaded conditional diffusion for identity restoration and an unconditional face prior for texture polishing in blind face restoration (Qiu et al., 2023). "Image Restoration via Diffusion Models with Dynamic Resolution" shifts restoration across $64 \times 64 \times 3$, $128 \times 128 \times 3$, and $256 \times 256 \times 3$ subspaces so that coarse structure is recovered early and high-frequency details are refined later (Zheng et al., 14 May 2026). "Efficient Diffusion Model for Image Restoration by Residual Shifting" replaces the usual HQ-to-noise trajectory with a residual-shifting HQ-to-LQ Markov chain, making four-step restoration feasible (Yue et al., 2024).

A second formulation uses diffusion as a training-time prior or constraint while keeping a conventional restorer for deployment. "DiffLoss" is the clearest example: the trainable model is any restoration network $g_\phi$, while a frozen unconditional ImageNet-pretrained DDPM from Dhariwal and Nichol provides a naturalness-oriented optimization space and an h-space semantic constraint during training; the diffusion model is discarded at test time, so inference speed and parameter count remain those of the original backbone (Tan et al., 2024).

A third formulation is hybrid detail generation. "Reconstruct-and-Generate Diffusion Model" first reconstructs a faithful base image and then uses diffusion only for the residual detail image $d_0 = y - r_\theta(x)$, so diffusion is assigned specifically to high-frequency correction rather than full-image synthesis (Wang et al., 2023). "Restoring Real-World Images with an Internal Detail Enhancement Diffusion Model" retains a pretrained Stable Diffusion backbone, trains only a ControlNet branch, and inserts Internal Image Detail Enhancement (IIDE) into denoising so that the reverse process remains detail-preserving under mixed real-world degradations (Xiao et al., 24 May 2025).

A fourth formulation emphasizes efficiency or near-feed-forward restoration. "Diffusion Once and Done" performs all-in-one restoration with only one-step sampling of Stable Diffusion, using degradation-aware conditional LoRA and a decoder-side High-fidelity Detail Enhancement module to compensate for detail loss caused by one-step generation (Tang et al., 5 Aug 2025). "Selective Hourglass Mapping for Universal Image Restoration Based on Diffusion Model" uses a shared-distribution term so that multiple degradation distributions contract toward an impure Gaussian bottleneck and then expand back to task-specific clean outputs in only three inference steps (Zheng et al., 2024).

These families differ operationally, but they share a common design impulse: diffusion is used where conventional restoration is weakest, namely in the recovery of realistic detail under severe ambiguity.

3. Priors, prompts, and structural control

The most important technical differentiator among detail restoration diffusion models is how they condition diffusion so that detail is realistic and faithful. Several works build this conditioning from semantic priors. "SSP-IR"—functionally identified in the paper as MRIR—combines an explicit semantic prior from LLaVA-7B with an implicit visual semantic prior from CLIP image features refined by a three-layer MLP, then injects them into Stable Diffusion through text cross-attention and image cross-attention (Zhang et al., 2024). "Diff-Restorer" uses a CLIP image encoder to derive a semantic prompt $P_S \in \mathbb{R}^{768}$ and a degradation prompt $P_D \in \mathbb{R}^{256}$, so that content guidance and degradation identification are separated rather than entangled inside one embedding (Zhang et al., 2024). "DiffLoss" uses the bottleneck representation of a frozen DDPM U-Net as a semantic space and matches h-space features between clean and restored images after equal forward diffusion (Tan et al., 2024).

Other works emphasize structure priors. In "SSP-IR", a Pixel-level Processor supervised by RGB and FFT losses extracts degradation-independent structure features, and ControlNet plus pixel attention inject these features into the denoising U-Net to prevent unreasonable artifacts (Zhang et al., 2024). "Diff-Restorer" introduces an Image-guided Control Module whose Degradation Modulation Blocks use $P_D$ to select relevant channels, while a Control Decoder reconstructs an auxiliary image during training to regularize structure and color faithfulness (Zhang et al., 2024). "DiffuMural" extracts faint mural contours inside damaged regions with $6000 \times 4000 \times 3$0-means clustering and uses them as visual control $6000 \times 4000 \times 3$1, reflecting the restoration-specific fact that missing mural regions often preserve weak but useful shape traces (Han et al., 13 Apr 2025). "Uni-DocDiff" constructs a Prior Pool containing Sobel and Canny high-frequency priors together with median, Gaussian, and DCT low-frequency priors, then adaptively selects among them with the Prior Fusion Module according to task identity and current content (Zhao et al., 6 Aug 2025).

A further line of work replaces externally defined degradation models with internal self-conditioning. In IIDE, the DDIM-estimated clean image $6000 \times 4000 \times 3$2 is decoded from the current latent state and reused as an internal condition, so the reverse transition conditioned on the degraded input is encouraged to match a reverse transition conditioned on the model’s own clean-image prediction (Xiao et al., 24 May 2025). This suggests a broader shift in the field: rather than conditioning diffusion only once at the input, recent methods increasingly condition the reverse process at semantic, structural, and intermediate-state levels.

4. Objectives and restoration dynamics

The mathematical objectives of detail restoration diffusion models differ according to where detail is supposed to enter the system. In "DiffLoss", the diffusion prior is not sampled at inference; instead it defines auxiliary losses during training. The complete loss is

$6000 \times 4000 \times 3$3

where the first diffusion terms match diffusion-space reconstructions and the h-space term matches bottleneck semantics (Tan et al., 2024). The design explicitly separates fidelity anchoring from diffusion-based naturalness and semantic preservation.

In residual-detail formulations, the decomposition is more explicit. "Reconstruct-and-Generate Diffusion Model" defines

$6000 \times 4000 \times 3$4

so the reconstructive network recovers the main signal and the diffusion model learns only the residual-detail distribution (Wang et al., 2023). The adaptive step controller then predicts a patch-wise diffusion step count from $6000 \times 4000 \times 3$5, so that flat regions receive little or no generated detail while textured regions receive more.

In restoration-backbone formulations, the diffusion trajectory itself is redesigned around detail recovery. "ResShift" defines

$6000 \times 4000 \times 3$6

so the forward chain shifts the HQ image toward the observed LQ image rather than toward pure Gaussian noise (Yue et al., 2024). This directly aligns the reverse process with residual detail completion. "RDMD" instead keeps a diffusion prior but places it into dual regularization: $6000 \times 4000 \times 3$7 where a single pre-trained diffusion model acts both as a stochastic sampler and as a deterministic denoiser-like regularizer, with $6000 \times 4000 \times 3$8 controlling the distortion–perception trade-off (Wang et al., 3 Mar 2025).

A separate strand studies restoration stability rather than new priors. "Enhancing Diffusion Model Stability for Image Restoration via Gradient Management" identifies conflict between prior and likelihood gradients and temporal fluctuation in likelihood guidance, then proposes progressive likelihood warm-up and adaptive directional momentum smoothing to stabilize the reverse process (Wu et al., 9 Jul 2025). This suggests that detail restoration quality depends not only on the choice of prior, but also on the geometry of posterior-guided sampling.

5. Empirical behavior and application domains

Empirically, these methods are strongest when evaluation moves beyond distortion-only metrics. "DiffLoss" reports moderate but consistent gains in PSNR and SSIM on Dense-Haze, Rain100H, and LOL—for example, EfDeRain on Rain100H improves from $6000 \times 4000 \times 3$9 to $64 \times 64 \times 3$0—but the sharper result is the FID change on Dense-Haze with GridDehazeNet, from $64 \times 64 \times 3$1 in setting (a) to $64 \times 64 \times 3$2 in the final setting, יחד with strong classification gains on degraded CUB data, such as VGG16 low-light accuracy improving from $64 \times 64 \times 3$3 to $64 \times 64 \times 3$4 and ResNet50 from $64 \times 64 \times 3$5 to $64 \times 64 \times 3$6 (Tan et al., 2024). The authors explicitly caution that the method “improves the naturalness of restored results, instead of substantially removing more degradation,” which is a concise statement of the detail-restoration agenda.

In blind face restoration, "DiffBFR" achieves the best FID, NIQE, and LPIPS on CelebA-Test—$64 \times 64 \times 3$7, $64 \times 64 \times 3$8, and $64 \times 64 \times 3$9, respectively—while its ablation from IRM-s to the full TEM-equipped model reduces FID from $128 \times 128 \times 3$0 to $128 \times 128 \times 3$1, showing that identity reconstruction and texture polishing contribute differently to facial detail recovery (Qiu et al., 2023). For real-world degraded photos and super-resolution, IIDE improves old-photo restoration from $128 \times 128 \times 3$2 to $128 \times 128 \times 3$3 in PSNR, SSIM, LPIPS, FID, CLIPIQA, and MUSIQ, respectively, and also improves text-guided colorization (Xiao et al., 24 May 2025).

Efficiency-oriented methods show that detail restoration is no longer tied to very long reverse chains. "ResShift" reports superior or comparable performance even only with four sampling steps on super-resolution, inpainting, and blind face restoration (Yue et al., 2024). "SubDAPS++" reports, for example, $128 \times 128 \times 3$4 PSNR, $128 \times 128 \times 3$5 SSIM, $128 \times 128 \times 3$6 LPIPS, and $128 \times 128 \times 3$7 FID on FFHQ inpainting, while maintaining lower latency and memory than a full-resolution variant (Zheng et al., 14 May 2026). "Diffusion Once and Done" reduces inference to one step and reports $128 \times 128 \times 3$8 s versus $128 \times 128 \times 3$9 s for DA-CLIP, while using decoder-side enhancement to restore structural and textural details (Tang et al., 5 Aug 2025).

The application range has also broadened. Document restoration is addressed by Uni-DocDiff, which unifies deblurring, deshadowing, illumination rectification, binarization, handwriting removal, and dewarping inside a dual-stream architecture (Zhao et al., 6 Aug 2025). Cultural-heritage restoration is addressed by DiffuMural, which focuses on large-area Dunhuang mural loss under style and seam constraints (Han et al., 13 Apr 2025). Medical restoration appears in RetinaRegen, which reports $256 \times 256 \times 3$0 PSNR, $256 \times 256 \times 3$1 SSIM, and $256 \times 256 \times 3$2 LPIPS for the readability labels of the optic disc region on SynFundus-1M (Tang et al., 26 Feb 2025). These examples indicate that detail restoration diffusion models are no longer confined to natural-image benchmarks.

6. Limitations, controversies, and open directions

The dominant limitation remains computational. Iterative diffusion restoration is expensive in pixel space, especially at large resolutions, and latent diffusion can introduce its own encode–decode overhead or representation mismatch (Luo et al., 2023). Training-time regularization approaches such as DiffLoss preserve cheap deployment, but require a large frozen diffusion model during optimization; in DiffLoss the auxiliary DDPM has $256 \times 256 \times 3$3M parameters, and the paper does not quantify the training overhead in FLOPs or wall-clock terms (Tan et al., 2024). One-step and few-step methods reduce this cost substantially, but often need compensatory detail modules, distillation, or custom diffusion formulations (Tang et al., 5 Aug 2025, Yue et al., 2024).

A second limitation is fidelity control. Many papers explicitly acknowledge that perceptual realism and texture synthesis can outrun measurement faithfulness. RDMD treats this as a tunable design choice rather than a solved problem (Wang et al., 3 Mar 2025). RnG introduces a reconstructive base and an adaptive step controller precisely because unrestricted diffusion detail generation can introduce undesirable texture (Wang et al., 2023). In cultural and medical domains, the stakes are sharper: DiffuMural emphasizes historical authenticity and expert judgment because large damaged regions lack factual grounding (Han et al., 13 Apr 2025), while RetinaRegen improves readability but does not provide a pathology-preservation study (Tang et al., 26 Feb 2025).

A third limitation is reproducibility and theoretical clarity. Several papers present strong empirical results while leaving parts of the mechanism heuristic or under-specified. DiffLoss does not provide a formal score-based derivation of its naturalness space and leaves timestep sampling unspecified (Tan et al., 2024). DiffBFR’s ELBO discussion is suggestive rather than fully rigorous, and some notation is inconsistent (Qiu et al., 2023). TDiR gives only a partial diffusion formalization and omits key runtime and scheduler details (Anwar et al., 25 Jun 2025). This suggests that the field is still partly engineering-driven: empirical evidence for better detail recovery is often stronger than the accompanying theory.

Open directions are therefore comparatively clear. One direction is better efficiency without decoder-induced detail loss, as explored by dynamic resolution, residual shifting, and one-step latent restoration (Zheng et al., 14 May 2026, Yue et al., 2024, Tang et al., 5 Aug 2025). Another is better internal control of semantics and structure, visible in the turn toward MLLMs, CLIP-derived visual prompts, bottleneck semantics, and degradation-aware fusion (Zhang et al., 2024, Zhang et al., 2024). A third is stability-aware restoration, where gradient interaction, task interference, or schedule design are treated as first-class issues rather than implementation details (Wu et al., 9 Jul 2025, Zhao et al., 6 Aug 2025). Taken together, these trajectories indicate that the detail restoration diffusion model is evolving from a slow generative restorer into a broader family of controllable, efficiency-aware, domain-adapted restoration systems.