Decoupled Residual Denoising Diffusion Models for Unified and Data Efficient Image-to-Image Translation

Abstract: We propose Decoupled Residual Denoising Diffusion models (DRDD) for unified and data-efficient image-to-image (I2I) translation. While diffusion models have advanced I2I translation in terms of quality and diversity, we uncover a previously under-explored property in diffusion models. Crucially, beyond its conventional role of manifold lifting (i.e., moving data off low-dimensional manifolds), injecting Gaussian noise facilitates domain harmonization by implicitly aligning feature distributions across domains, a property particularly advantageous for unified I2I translation. However, existing diffusion models prematurely erode this harmonization effect, as noise and residuals are simultaneously removed in a single coupled diffusion process. To address this, DRDD decouples the diffusion process into two sequential and independent diffusion stages: (1) a stochastic noise diffusion for domain harmonization and manifold lifting, and (2) a deterministic residual diffusion that learns the core semantic mapping entirely within the fixed-noise domain. This decoupling preserves harmonization and manifold lifting effects throughout the transformation, substantially simplifying the learning of unified mappings across diverse tasks and domains. Notably, the noise diffusion stage is trained exclusively on abundant, unpaired target-domain images, greatly improving data efficiency. Comprehensive theoretical and empirical analysis demonstrates that DRDD is broadly compatible with mainstream diffusion models and consistently delivers robust, unified I2I translation, even under limited paired data. Our code is available at https://github.com/HKU-HealthAI/DRDD.

• The paper presents a DRDD framework that decouples noise and residual diffusion, enabling effective domain harmonization.
• It demonstrates improved data efficiency and state-of-the-art performance across unified, multi-domain image-to-image translation tasks.
• The study identifies an optimal noise regime that balances injected noise for robust harmonization without compromising reconstruction quality.

Decoupled Residual Denoising Diffusion Models for Unified and Data Efficient Image-to-Image Translation

Introduction

The paper "Decoupled Residual Denoising Diffusion Models for Unified and Data Efficient Image-to-Image Translation" (2606.01048) systematically re-examines the operational paradigms of diffusion models in Image-to-Image (I2I) translation. It challenges the entrenched view that denoising and semantic translation must be tightly coupled within diffusion-based frameworks, revealing a previously ignored property: injected Gaussian noise not only achieves manifold lifting but also harmonizes feature distributions across disparate domains. The principal contribution is the Decoupled Residual Denoising Diffusion (DRDD) framework, which isolates noise diffusion and residual diffusion into independent stages, preserving domain harmonization effects and markedly improving data efficiency. The approach demonstrates strong empirical results across unified, multi-domain, and data-limited scenarios, with broad compatibility across major diffusion architectures.

Problem Motivation and Related Work

Contemporary diffusion-based I2I models, including RDDM and IR-SDE, perform denoising and residual (domain) translation concurrently in a single, coupled reverse process. This design choice reflects an implicit assumption—that denoising can encompass both removal of stochastic noise and the deterministic domain shift mapping. However, this coupling leads to a rapid erosion of the harmonization effects delivered by the injected Gaussian noise, undermining generalizability across multiple domains and substantially impeding data efficiency when paired supervision is scarce.

The paper provides an analytical and empirical argument that Gaussian noise, when injected at suitable magnitudes, functions as a "domain harmonizer"—minimizing the feature distribution gap between heterogeneous domains. This phenomenon is formalized by proving that the KL divergence between distributions decreases following the addition of iid Gaussian noise, thus facilitating more amenable learning landscapes for multi-task or unified I2I translation (Figure 1).

Figure 1: Left: Injecting Gaussian noise reduces feature representation gaps across source and target domains, as demonstrated by t-SNE visualizations; Right: DRDD decouples residual removal and denoising, preserving domain harmonization throughout the transformation.

DRDD: Methodological Design

Decoupled Forward and Reverse Process

DRDD reformulates the forward process as a two-stage sequence:

• Noise Diffusion Stage: Gaussian perturbations are stochastically injected into the target image, producing a noise-carrying target that facilitates domain harmonization.
• Residual Diffusion Stage: Within the (fixed) noise-carrying domain, deterministic transformation occurs via residual injection, modeling the semantic or structural mapping needed for I2I translation.

The reverse process is symmetrically decoupled:

• Residual Removal Stage: Starting from the noise-carrying input, the model first eliminates only the residual (i.e., source-to-target discrepancy) within the fixed noise domain.
• Denoising Stage: Final denoising brings the sample back to a clean target.

Crucially, this split ensures that domain harmonization—afforded by the noise—remains active throughout the semantic mapping, rather than being stripped away early by premature denoising as in conventional approaches (Figure 2).

Figure 2: The DRDD pipeline clearly decouples noise diffusion (gray arrows) and residual diffusion (blue arrows), disentangling harmonization and semantic mapping.

Training Protocol and Data Efficiency

A significant design decision in DRDD is to train the denoising network exclusively on unpaired, clean target images; it does not require paired supervision from corresponding source images. The residual removal network is responsible for deterministic semantic mapping, trained via paired data where available. This enables scalable leveraging of vast unpaired datasets, enhancing data efficiency and avoiding the intractability of collecting exhaustive paired samples.

Empirically, ablation and pruning studies consistently show that DRDD's performance diminishes much less than competing methods as paired data is removed (Figure 3).

Figure 3: DRDD maintains high SSIM and low LPIPS under drastic data pruning, highlighting strong data efficiency.

Noise Level Optimization

The paper provides a rigorous analysis of the effect of noise magnitude, showing that as the injected noise increases, inter-domain distances (MMD-based) decrease, but excessive noise introduces information loss and reconstruction difficulty. The authors formalize a trade-off function and empirically confirm the existence of an optimal regime (typically $\sigma \in [0.8, 1.3]$), beyond which model output quality deteriorates (Figure 4).

Figure 4: Performance is optimal at moderate noise injection strengths; excessive noise leads to utility decline.

Results

Unified Multi-Task and Multi-Domain I2I Translation

On challenging multi-degradation benchmarks (All-in-One-5, CDD-11), DRDD establishes new SOTA metrics in SSIM, LPIPS, and FID across a variety of restoration tasks, both outperforming recent diffusion and transformer-based methods and exceeding non-diffusion approaches in key metrics (Figure 5, Figure 6).

Figure 5: DRDD achieves SOTA SSIM across 11 degradation types in unified benchmarks.

Figure 6: Qualitative restoration: DRDD generates images with improved fidelity, less artifacting, and more faithful structure reconstruction compared to competing methods.

On cross-domain single-task benchmarks (MNMD), DRDD consistently surpasses specialized and previous unified methods, indicating robust domain harmonization.

Data Efficiency and Broad Compatibility

DRDD's decoupling strategy enables effective transfer to alternative diffusion architectures (e.g., SDE-based, DDIM). When tested with limited paired supervision, DRDD shows state-of-the-art robustness, affirming that its architectural decoupling is not only theoretically optimal but also practically advantageous. Extension experiments and analysis, including on inpainting, super-resolution, and style transfer, confirm these gains.

Theoretical Insights and Implications

Theoretical analysis centers on the insight that the denoising network does not generate semantic (target) content per se, but only removes stochastic noise—the semantic mapping is supplied by the residual removal mechanism, which is most tractable and generalizable in a harmonized feature space. The KL minimization proof formalizes this harmonization effect, and the selection of an optimal noise magnitude is both empirically tractable and theoretically justified.

The work repositions noise injection: not as a necessary evil for manifold lifting, but as a tool for domain alignment. This has implications for all generative models handling cross-domain tasks, especially as scalable paired data becomes less accessible or when the label space grows combinatorially.

Future Developments

DRDD's architecture suggests several future directions:

• Extending Decoupling: Applying DRDD's principles to more general conditional generation tasks or other modalities (text-to-image, video, etc.) where inter-domain discrepancies are critical.
• Scalable Unpaired Training: Leveraging large-scale, fully unpaired datasets for denoising, potentially with active or self-supervised residual component learning.
• Adaptive Noise Scheduling: Investigating dynamic, content-adaptive noise magnitudes for real-time domain harmonization in non-stationary data environments.
• Unified Generative Perception: Incorporation into vision-language or multimodal models for tasks requiring structured, controlled translation across representations.

Conclusion

This work presents a marked conceptual and practical advance for unified, data-efficient I2I translation, demonstrating that the core semantic transformation and denoising in diffusion models benefit strongly from architectural decoupling. The DRDD framework delivers SOTA quantitative and qualitative performance across a wide range of restoration and translation scenarios, with especially pronounced gains in low-resource (paired data scarce) environments. The broad compatibility and robust theoretical foundation underscore DRDD's value as a new default paradigm for unified generative modeling pipelines (2606.01048).