Plug-and-Play Diffusion Fusion

• Plug-and-play Diffusion Fusion is a modular framework that integrates diffusion-based generative priors with measurement constraints for inverse problems and multi-modal synthesis.
• It employs alternating steps between data-consistent projections and diffusion model denoising to rapidly adapt to diverse imaging and scientific inference tasks.
• The approach enables scalable, plug-and-play integration across applications—from medical imaging to 3D sensor fusion—backed by robust theoretical guarantees and strong empirical benchmarks.

Plug-and-play Diffusion Fusion refers to a class of algorithmic and architectural frameworks that enable the flexible integration ("plug-and-play") of learned diffusion priors with diverse data-fidelity, measurement, and modality-fusion components for inverse problems, generative modeling, and multi-modal synthesis. These techniques exploit the separation of the generative (diffusion-based) prior from task- or physics-specific constraints, allowing modular composition and rapid adaptation of state-of-the-art deep models in imaging, scientific inference, multi-modal perception, and generation. The plug-and-play paradigm encompasses both algorithmic strategies for alternating between generative priors and measurement constraints, and architectural strategies for feature/condition fusion in complex generative pipelines.

1. Theoretical Foundations and Inverse Problem Formulation

Plug-and-play diffusion fusion frameworks formalize inference as posterior sampling in Bayesian inverse problems. The canonical formulation involves reconstructing a latent variable $x$ (e.g., a high-resolution image) from observed, generally degraded measurements $y$ under a known or unknown (possibly nonlinear) forward model $\mathcal{A}$: $y = \mathcal{A}(x) + n,$ where $n$ denotes measurement noise. The posterior density is then

$p(x|y) \propto \exp[-f(x; y)] p(x),$

with $f(x; y)$ the negative log-likelihood ("data-fidelity") term encapsulating physical or statistical measurement constraints (e.g., quadratic for Gaussian noise), while $p(x)$ is an image, signal, or structure prior instantiated as the sampling process of a pretrained diffusion model (Wang et al., 2 Feb 2026, Banerjee et al., 28 Jul 2025, Xu et al., 2024).

Plug-and-play fusion frameworks further extend this to blind or multimodal inverse problems, i.e., simultaneous estimation of latent objects and measurement operators or latent features of multiple modalities. The resulting posteriors then have the form

$$p(x, k | y) \propto p(y | x, k) \, p_\theta(x) \, p_\phi(k),$$

where $k$ encodes unknown measurement or system parameters (e.g., blur kernels), each governed by potentially separate diffusion priors (Li et al., 28 May 2025).

2. Alternating Data-Consistency and Diffusion Prior Steps

A defining characteristic of plug-and-play diffusion fusion is the decoupled, alternating update scheme for posterior inference. At each iteration, the estimate is projected toward measurement consistency and then refined toward the manifold learned by the diffusion model. Major algorithmic realizations include:

• Gibbs/Split Sampling: The state alternates between a data-consistency step (conditional on measurements and previous estimate) and a prior-projection step (diffusion model denoising or reverse SDE/ODE integration). For instance, in OCT super-resolution, each iteration consists of:
  • Drawing from $$p(z \mid x^{(k-1)}, y) \propto \exp[-f(z; y) - \tfrac{1}{2}\lambda_k\|z-x^{(k-1)}\|^2]$$, yielding a Gaussian update for $z$;
  • Executing a sequence of discretized Langevin or predictor-corrector updates for $x$ under the diffusion prior, leading to the next sample $x^{(k)}$ (Wang et al., 2 Feb 2026).
• Hybrid Data-Consistency Modules: For problems such as compressive sensing, hybrid projections—e.g., a convex combination of GAP (hard projection) and HQS (soft constraint)—can be applied to the denoised estimate before proceeding to the next diffusion step (Wang et al., 11 Sep 2025).
• Blind Decomposition: When both the latent object and system parameters are unknown, two diffusion models are alternately used within a split-Gibbs scheme—each alternately updates object and operator estimates in a block-wise manner (Li et al., 28 May 2025).

This modular approach allows leveraging pretrained diffusion models as black-box priors and plugging them into any probabilistically specified measurement or constraint operator, without retraining or architecture modification.

3. Multi-Modal, Multi-Expert, and Cross-Condition Fusion

Plug-and-play diffusion fusion is also central in multi-modal and multi-condition generation. The fusion of feature representations or gradient signals from disparate sources is performed in a modular, training-free (or lightweight) fashion, often at inference time:

• MaxFusion: In text-to-image diffusion, MaxFusion combines independently trained condition branches (e.g., ControlNet/T2I-Adapter for segmentation, depth, pose) by fusing their intermediate feature maps via variance- and correlation-weighted rules. Coincident spatial features from different control branches are either averaged if similar, or fused by selecting the most "expressive" (i.e., highest variance) at each spatial location. This compositional fusion requires no retraining of the backbone diffusion model (Nair et al., 2024).
• Multi-Expert Plug-and-play Guidance: Practical Plug-And-Play (PPAP) employs multiple lightweight "experts" (parameter-efficient variants of an external guidance network), each specialized for a subset of the diffusion time steps, to guide the diffusion model via their gradients. This circumvents vanishing/exploding gradients in noisy regimes and avoids retraining or labeled data requirements by relying on data-free knowledge transfer (Go et al., 2022).
• Adaptive Multimodal Guidance: In protein structure inference, plug-and-play diffusion fusion fuses gradients from heterogeneous experimental modalities at each denoising step. Modality weights are adapted online by estimating noise scales, ensuring that high-uncertainty modalities are down-weighted, and guidance is dynamically balanced (Banerjee et al., 28 Jul 2025).

This compositionality enables modular scaling to new modalities, tasks, or sensor configurations with minimal retraining.

4. Architectural Realizations in Multi-Task and Unified 3D Frameworks

Beyond algorithmic schemes, plug-and-play fusion is realized directly at the architectural level:

• Multi-Sensor Latent Diffusion: DifFUSER fuses BEV features from multiple sensors (LiDAR, camera) by chaining multi-resolution cMini-BiFPN blocks, each equipped with Gated Self-conditioned Modulation (GSM). Robustness to sensor failure is achieved through progressive sensor dropout training, and fusion is achieved in latent space, allowing the model to reconstruct plausible features even under severe modality loss (Le et al., 2024).
• Unified 3D Understanding and Generation: PnP-U3D implements plug-and-play diffusion fusion by bridging an autoregressive (AR) next-token 3D understanding branch and a diffusion-based 3D generation branch via a minimal transformer bridge. The bridge is the only trainable component, mapping frozen LLM features to the conditional space of the 3D diffusion model, achieving state-of-the-art results in both 3D synthesis and semantic understanding (Chen et al., 3 Feb 2026).

These architectures showcase plug-and-play fusion as a scalable, modular paradigm for cross-modal and cross-task integration.

5. Convergence, Robustness, and Theoretical Guarantees

Plug-and-play diffusion fusion has been accompanied by advances in convergence guarantees and robustness analysis:

• Provable Consistency: Under alternating diffusion–consistency sampling, distributional convergence to the posterior $p(x|y)$ (as noise scales anneal to zero) has been established for broad classes of plug-and-play samplers (both stochastic and deterministic, e.g., DDPM and DDIM), with explicit bounds under regularity assumptions (Xu et al., 2024).
• Non-Vanishing Bias Correction: Classical memoryless PnP schemes can fail to strictly satisfy measurement constraints under corruptions. Dual-Coupled PnP Diffusion frameworks resolve this by integrating dual variables (as in ADMM), introducing integral feedback that eliminates bias and ensures asymptotic agreement with the data manifold (Du et al., 26 Feb 2026). However, this introduces colored artifacts in the dual variable; spectral homogenization is then required to map these artifacts to pseudo-AWGN, aligning the input distribution to the diffusion prior’s assumptions.

6. Quantitative Performance and Empirical Benchmarks

Plug-and-play diffusion fusion achieves state-of-the-art results across diverse application domains, including:

Application Domain	Task/Benchmark	Notable Metrics	Plug-and-Play Fusion Method (Best Result)
Medical Imaging (OCT)	4× Super-Resolution	PSNR=32.50 dB, SSIM=0.722, LPIPS=0.120	PnP-DM (VE/EDM) (Wang et al., 2 Feb 2026)
3D Sensor Fusion	BEV Segmentation (NuScenes)	mIoU=69.1% (+6.4% over baseline)	DifFUSER (Le et al., 2024)
Protein Structure	Multi-modal Constrained Inference	RMSD = 0.65 Å (partial coordinates + distances)	Adam-PnP (Banerjee et al., 28 Jul 2025)
Blind Deblurring	ImageNet faces	PSNR=27.42 dB, SSIM=0.795, LPIPS=0.176	Blind-PnPDM (Li et al., 28 May 2025)
Text-to-Image Fusion	COCO Contradictory/Complementary	FID=46.72, MSE-Seg=0.1080 (pose+seg)	MaxFusion (Nair et al., 2024)
Medical (CT, MRI)	Limited-angle/undersampled	PSNR=39.81 dB on SVCT-20, SSIM=0.960	DC-PnPDP (Du et al., 26 Feb 2026)

Experiments universally demonstrate superior reconstruction fidelity, perceptual quality, and robustness to missing or noisy modalities when compared to classical or supervised baselines.

7. Generalization, Modularity, and Extensions

The modularity of plug-and-play diffusion fusion allows rapid adaptation to diverse forward models, sensor setups, and task constraints. By decoupling the prior from the physics or task-specific constraints, pretrained generative models can be reused across domains such as CT, MRI, microscopy, compressive sensing, and even non-imaging scientific inference. New fidelity modules, constraints, or sensor types can be incorporated by defining appropriate projection, proximal, or gradient-correction steps interleaved with diffusion denoising—without retraining the core generative model (Wang et al., 11 Sep 2025, Xu et al., 2024). The paradigm is thus positioned as a central methodology for sample-efficient, scalable, and theoretically-grounded integration of learned generative models with physical, semantic, or multimodal information.