Decoupled Residual Denoising Diffusion Models

• The framework decouples semantic translation from denoising by sequentially applying stochastic noise diffusion and deterministic residual mapping.
• DRDD leverages Gaussian noise for domain harmonization, aligning feature distributions to enable unified image restoration across multiple domains.
• It uses paired supervision for learning residual mappings and unpaired clean target images for denoising, achieving state-of-the-art performance in data-efficient settings.

Decoupled Residual Denoising Diffusion Models (DRDD) are a diffusion-based framework for image-to-image translation that separates semantic translation from denoising by factorizing the process into two sequential and independent stages: a stochastic noise diffusion for domain harmonization and manifold lifting, followed by a deterministic residual diffusion that learns the source-to-target semantic mapping entirely within a fixed-noise domain. The framework was introduced for unified and data-efficient image-to-image translation, especially in settings where a single model must span multiple restoration tasks or multiple domains under limited paired supervision. Its central claim is that injected Gaussian noise does more than support conventional diffusion; it also reduces cross-domain discrepancy, and this harmonization effect should be preserved until semantic translation is complete rather than being eroded by a single coupled reverse process (Lin et al., 31 May 2026).

1. Problem setting and motivating observation

DRDD is formulated for image-to-image translation from an input image $I_{in}$ to a target image $I_0$, with emphasis on two difficult regimes: unified translation, where one model must solve multiple tasks or operate across multiple domains, and data-efficient translation, where paired source-target examples are scarce. The motivating diagnosis is that many diffusion I2I systems, including methods that start from pure Gaussian noise and methods that initialize from a noise-carrying input image, still perform a single coupled reverse process in which noise removal and semantic translation happen simultaneously at every step. DRDD identifies this coupling itself as a limitation (Lin et al., 31 May 2026).

The framework is built around a stronger interpretation of injected Gaussian noise. In addition to manifold lifting, the paper argues that applying similar Gaussian perturbation to different domains brings their feature or distributional representations closer together; this effect is termed domain harmonization. The theoretical statement given is that if $P$ and $Q$ are two distributions and $P_\sigma, Q_\sigma$ are the corresponding distributions after adding Gaussian noise $\mathcal N(0,\sigma^2)$, then

$$D_{\mathrm{KL}}(P_\sigma \parallel Q_\sigma) < D_{\mathrm{KL}}(P \parallel Q).$$

This proposition formalizes the claim that Gaussian smoothing contracts distribution divergence. In DRDD, the consequence is architectural rather than merely analytic: semantic mapping should be carried out while the harmonizing noise is still present, rather than removing that noise step by step while translation is still incomplete (Lin et al., 31 May 2026).

A recurrent misconception addressed by the framework is to treat Gaussian noise as only a nuisance variable or only a manifold-lifting device. DRDD instead treats it as a representation-alignment mechanism that is particularly useful for unified translation across tasks and domains.

2. Two-stage factorization of diffusion and translation

The forward process is split into two stages, indexed by superscripts $(1)$ and $(2)$. The first stage is stochastic noise diffusion on the target image, and the second is deterministic residual diffusion driven by the residual between input and target. The notation used by the framework is

• $I_0^{(1)}$: clean target image,
• $I_0$0: state at timestep $I_0$1 in the noise diffusion stage,
• $I_0$2: state at timestep $I_0$3 in the residual diffusion stage,
• $I_0$4: total number of noise diffusion steps,
• $I_0$5: total number of residual diffusion steps,
• $I_0$6: residual between source and target,
• $I_0$7,
• $I_0$8 (Lin et al., 31 May 2026).

The first forward stage adds Gaussian noise to the clean target:

$I_0$9

where $P$0. At the end of this stage, the terminal noisy target becomes the initial condition of the second stage:

$P$1

The second forward stage performs deterministic residual diffusion:

$P$2

$P$3

When $P$4, the final state is

$P$5

The fully diffused state is therefore the input image plus fixed Gaussian noise, not pure noise. This endpoint is structurally important because it encodes the source image in a harmonized noisy domain rather than discarding it.

The factorization can be summarized as

$P$6

A plausible implication is that the model is not asked to learn denoising and semantic translation as a single entangled operator; it learns semantic transport in a domain where source and target have already been partially aligned by Gaussian perturbation.

3. Reverse process and decoupled learning dynamics

Reverse inference follows the opposite order. Starting from

$P$7

DRDD first removes the semantic residual while keeping the noise level fixed, and only afterward applies a denoising diffusion process to recover the clean target. This ordering is the operational meaning of the term “decoupled” in DRDD (Lin et al., 31 May 2026).

In the residual-removal stage, a residual network $P$8 predicts the residual in the noisy domain. The predicted clean state for stage 2 is

$P$9

The reverse residual transition is deterministic:

$Q$0

which yields the update

$Q$1

The variance-free reverse kernel makes the residual stage explicitly deterministic.

Once residual removal reaches $Q$2, DRDD runs a denoising diffusion stage using a separate denoising network $Q$3. The reverse transition is

$Q$4

with

$Q$5

where

$Q$6

The corresponding iterative update is

$Q$7

When $Q$8, this reduces to the deterministic DDIM-style form

$Q$9

This reverse ordering encodes the core intuition of DRDD: semantic translation should be completed inside the harmonized noisy domain, and denoising should be deferred until afterward. The result is not merely a different schedule, but a different factorization of the generative inverse problem.

4. Objectives, supervision split, and model architecture

The two stages are trained separately and under different supervision requirements. The residual-removal network is trained on paired source-target data using the simplified objective

$P_\sigma, Q_\sigma$0

The denoising network is trained only to predict Gaussian noise on target images:

$P_\sigma, Q_\sigma$1

This supervision split is the main source of DRDD’s data-efficiency claim: the residual stage requires paired data, while the denoising stage requires only target-domain clean images and can therefore exploit abundant, unpaired target data (Lin et al., 31 May 2026).

The residual model is a basic U-Net that receives $P_\sigma, Q_\sigma$2, timestep $P_\sigma, Q_\sigma$3, and conditioning input $P_\sigma, Q_\sigma$4, with channel depth $P_\sigma, Q_\sigma$5 and channel multiplier $P_\sigma, Q_\sigma$6. The denoising model is an ADM-style diffusion U-Net that receives $P_\sigma, Q_\sigma$7 and timestep $P_\sigma, Q_\sigma$8, with channel depth $P_\sigma, Q_\sigma$9 and channel multiplier $\mathcal N(0,\sigma^2)$0. The paper notes one inconsistency in the main text where architecture assignments appear partially swapped, while the appendix clarifies that the residual model is the smaller U-Net and the denoising model is the larger diffusion U-Net. Inference uses DDIM sampling with step size 2 for both stages in the main experiments, and ablations report little performance difference between 2 and 10 steps. The framework is also presented as broadly compatible with DDPM, DDIM, and score-based SDE models; an explicit SDE decoupling is instantiated on IR-SDE under the name De-IRSDE (Lin et al., 31 May 2026).

A practical consequence of the architecture split is that DRDD can reuse pretrained denoising diffusion models for the target domain, while concentrating scarce paired supervision on the residual network that learns the task-specific mapping. This division of labor is a structural feature of the method rather than a training heuristic.

5. Empirical evaluation and unified translation performance

The empirical program addresses three questions: unified multi-task restoration, single-task multi-domain translation, and data efficiency. Evaluation covers All-in-One-5, CDD-11, MNMD, CelebA-HQ, FFHQ, Rain100H/L, LoL, VE-LOL-L, and Edges2Handbags / Edges2Shoes. On All-in-One-5, DRDD achieves the best average across SSIM / LPIPS / FID among compared methods, with reported averages of $\mathcal N(0,\sigma^2)$1, compared with DFPIR at $\mathcal N(0,\sigma^2)$2, AdAIR at $\mathcal N(0,\sigma^2)$3, VLUNet at $\mathcal N(0,\sigma^2)$4, DiffuIR at $\mathcal N(0,\sigma^2)$5, and DA-CLIP at $\mathcal N(0,\sigma^2)$6 (Lin et al., 31 May 2026).

The following aggregate results are representative of the reported behavior.

Benchmark	Reference comparison	DRDD
All-in-One-5 average	DFPIR: $\mathcal N(0,\sigma^2)$7	$\mathcal N(0,\sigma^2)$8
MNMD average	VLUNet: $\mathcal N(0,\sigma^2)$9	$$D_{\mathrm{KL}}(P_\sigma \parallel Q_\sigma) < D_{\mathrm{KL}}(P \parallel Q).$$0
All-in-One-3, 25% data	AdAIR: $$D_{\mathrm{KL}}(P_\sigma \parallel Q_\sigma) < D_{\mathrm{KL}}(P \parallel Q).$$1	$$D_{\mathrm{KL}}(P_\sigma \parallel Q_\sigma) < D_{\mathrm{KL}}(P \parallel Q).$$2

On CDD-11, DRDD achieves the highest average SSIM and is described as especially strong in difficult mixed scenarios such as low+haze+snow and low+haze+rain. On MNMD, which spans natural, medical, and remote sensing denoising with Gaussian, salt-and-pepper, and Poisson noise, DRDD outperforms RDDM, IR-SDE, and VLUNet, with average SSIM $$D_{\mathrm{KL}}(P_\sigma \parallel Q_\sigma) < D_{\mathrm{KL}}(P \parallel Q).$$3 and LPIPS $$D_{\mathrm{KL}}(P_\sigma \parallel Q_\sigma) < D_{\mathrm{KL}}(P \parallel Q).$$4, compared with RDDM at $$D_{\mathrm{KL}}(P_\sigma \parallel Q_\sigma) < D_{\mathrm{KL}}(P \parallel Q).$$5, IR-SDE at $$D_{\mathrm{KL}}(P_\sigma \parallel Q_\sigma) < D_{\mathrm{KL}}(P \parallel Q).$$6, and VLUNet at $$D_{\mathrm{KL}}(P_\sigma \parallel Q_\sigma) < D_{\mathrm{KL}}(P \parallel Q).$$7. The paper interprets this as direct support for the claim that preserving noisy-domain harmonization helps cross-domain mapping (Lin et al., 31 May 2026).

In low-data regimes, DRDD degrades less than comparison methods. Appendix numbers on All-in-One-3 report, at 25% paired data, RDDM $$D_{\mathrm{KL}}(P_\sigma \parallel Q_\sigma) < D_{\mathrm{KL}}(P \parallel Q).$$8, AdAIR $$D_{\mathrm{KL}}(P_\sigma \parallel Q_\sigma) < D_{\mathrm{KL}}(P \parallel Q).$$9, and DRDD $(1)$0; at 100% paired data, RDDM $(1)$1, AdAIR $(1)$2, and DRDD $(1)$3. An ablation against a matched entangled baseline, a variant without denoising network, and a general denoiser trained on isolated datasets is reported as evidence that the gain is not simply due to extra parameters. Another analysis introduces two MMD-based quantities,

$(1)$4

and the criterion

$(1)$5

The reported best fixed noise level is around $(1)$6, with a robust range roughly $(1)$7 to $(1)$8. Too little noise does not harmonize enough; too much corrupts the input (Lin et al., 31 May 2026).

6. Relation to residual diffusion and hybrid refinement methods

DRDD sits within a broader residual-oriented diffusion landscape, but its decoupling is stricter than several neighboring formulations. The paper itself positions earlier diffusion I2I methods such as SR3, RDDM, I$(1)$9SB, IR-SDE, and related models as still relying on a single coupled reverse process, even when they initialize from a noise-carrying input image rather than pure Gaussian noise; DRDD’s novelty is the full separation of semantic residual transport and denoising (Lin et al., 31 May 2026).

Resfusion is closely related in spirit because it injects the degradation residual into the forward process and starts reverse restoration from noisy degraded images, but it is not fully decoupled: it combines Gaussian noise and a schedule-weighted residual term into a single prediction target called resnoise, and is described in the paper as a coupled residual-noise reparameterization rather than a strict factorization of residual and noise dynamics (Shi et al., 2023). TTRD3’s Residual Denoising Dual Diffusion Model is even closer structurally: it uses two independent schedules, predicts residual and noise separately, and in one variant employs two fully separate U-Nets; the paper characterizes this as functionally a decoupled residual-and-denoising diffusion model, although it is embedded in remote sensing super-resolution and further conditioned by texture-transfer modules (Liu et al., 17 Apr 2025). The Reconstruct-and-Generate Diffusion Model for denoising implements a different but recognizable split: a reconstructive denoising network produces a faithful base estimate, and a diffusion model generates only the residual high-frequency detail $(2)$0, yielding the additive decomposition $(2)$1 (Wang et al., 2023). Residual Prior Diffusion generalizes the same intuition to unconditional generation by assigning coarse structure to a latent-variable prior and training diffusion only on the prior-centered residual discrepancy $(2)$2, thereby moving the decoupling from I2I translation to prior-conditioned generation (Kutsuna, 25 Dec 2025).

These neighboring models show that “residual diffusion” is not a single design pattern. DRDD’s distinctive contribution is the fixed-noise-domain factorization motivated by domain harmonization, not merely residual-aware conditioning or additive base-plus-detail refinement.

7. Significance, strengths, and limitations

The principal strengths attributed to DRDD are better unified performance across tasks and domains, better data efficiency through target-only denoiser training, conceptual clarity in separating semantic translation from denoising, backbone compatibility with DDPM/DDIM/SDE frameworks, fast inference with only a few DDIM steps per stage, and strong perceptual quality in both unified and single-task settings. The framework is especially suited to all-in-one restoration, cross-domain medical or remote or natural image translation, and regimes where paired data are limited but clean target-domain images are abundant (Lin et al., 31 May 2026).

The framework also has explicit limitations. Performance depends on choosing an appropriate fixed noise level; the residual stage still requires paired data; two-stage modeling introduces architectural complexity relative to one-network baselines; and the theoretical analysis is suggestive rather than fully comprehensive, since the harmonization proposition is about general distribution smoothing rather than task-specific semantic guarantees. The paper additionally notes that some equations and appendix notation are imperfectly typeset and that implementation details are best resolved through the released code. There is computational overhead from using two networks, even though the reported runtime-quality tradeoff remains favorable (Lin et al., 31 May 2026).

The enduring significance of DRDD is the reinterpretation of diffusion noise as an active alignment mechanism. In DRDD, the translation pipeline can be read as

$(2)$3

That factorization shifts the role of diffusion in image-to-image translation: the model is not simply a denoiser conditioned on an input image, but a staged translator that preserves a useful noisy geometry until semantic mapping has been completed.