Plug-and-Play Conditional Diffusion

• The paper introduces a Bayesian framework that guides pretrained diffusion models via external constraints and auxiliary models without retraining.
• It combines data-consistency projections with score-based denoising to effectively address inverse problems in image restoration, video generation, and protein structure recovery.
• Adaptive scheduling and plug-in mechanisms enhance model robustness and convergence, ensuring high performance across diverse modalities and noise regimes.

Plug-and-Play conditional diffusion refers to a class of frameworks and algorithms enabling pretrained, often unconditional or weakly conditional, diffusion models to be guided at inference time by arbitrary constraints, conditions, or auxiliary models—without modifying the backbone or requiring retraining. This strategy enables conditional sampling, inverse problem solving, multi-modal fusion, and downstream editing in a flexible, modular fashion. Central to these approaches is the integration of learned generative priors (via diffusion models) with likelihood or constraint terms capturing the measurement process, desired conditioning signal, or external knowledge, often via a principled Bayesian or energy-based inference formulation. Contemporary work encompasses domains as diverse as image restoration, multimodal synthesis, protein structure recovery, video generation, and 3D object synthesis.

1. Core Bayesian and Algorithmic Principles

Plug-and-play (PnP) conditional diffusion methods recast conditional generative modeling as a Bayesian inference problem. Given a generative prior $p(x)$ (typically only accessible up to the score, $\nabla_x \log p(x)$, via a pretrained diffusion model) and a likelihood or constraint $p(y|x)$, the posterior is given by

$p(x|y) \propto p(y|x) p(x).$

For linear inverse problems, such as super-resolution and denoising, the measurement process is formulated as $y = A x + n$, $n \sim \mathcal N(0, \sigma^2 I)$, with $A$ denoting the degradation operator. The likelihood is then Gaussian, and the prior is implicitly represented by the diffusion model's score network evaluated at clean or noisy inputs (Wang et al., 2 Feb 2026, Wang et al., 20 May 2025). In more general conditional settings, auxiliary models—such as classifiers, segmentation networks, or arbitrary differentiable losses—define $c(x,y)$ or $p(y|x)$.

A common algorithmic pattern is a Split Gibbs or alternating minimization scheme:

• Data-consistency/likelihood step: Project the sample towards agreement with the measurement or guidance signal, typically via a proximal mapping.
• Prior/denoising step: Denoise, sample, or perform a score update using the diffusion prior. Adaptive coupling schedules or learnable guidance strengths are often used to balance these contributions dynamically.

2. Practical Plug-and-Play Conditional Diffusion Algorithms

Several algorithmic realizations embody plug-and-play diffusion:

a. MCMC and Langevin PnP

In inverse imaging problems, such as corneal OCT super-resolution, the method alternates Gaussian likelihood sampling with diffusion-driven denoising (score-based Langevin, reverse SDE, or EDM steps). Each PnP iteration samples an auxiliary latent variable enforcing data-consistency, then integrates the prior via a small number of score-based steps, with the relative strength scheduled by an exponential decay (Wang et al., 2 Feb 2026, Wang et al., 20 May 2025, Wang et al., 11 Sep 2025). This approach enables robust MAP and posterior sampling in high-dimensional ill-posed problems.

b. Product-of-Experts and Model Fusion

For multimodal or multi-constraint synthesis, closed-form score fusion via a product-of-experts (PoE) principle unifies scores from separately trained, off-the-shelf conditional diffusion models. The score at each denoising step is a weighted sum of the individual model scores, minus redundancy terms for overlapping priors. Reliability weights allow fine control over the influence of each modality (Nair et al., 2022).

c. Gradient-based Guidance

Gradient modification of the reverse diffusion step by the gradient of a task-specific loss or pretrained inverse model allows zero-shot control for arbitrary semantic or image-to-image conditions. This approach, termed "steered diffusion," perturbs each denoising step by $\nabla_x L_{\text{cond}}(x; c)$ to enforce constraints such as text, class, or geometric priors (Nair et al., 2023).

d. Dynamic Expert and Adapter Models

In scenarios where a single guidance model is ineffective across all noise regimes, practical frameworks (e.g., PPAP) assign a parameter-efficient, specialized expert to each noise interval. These experts, trained with adapter layers and knowledge distillation, provide robust, label-free guidance—even to off-the-shelf diffusion models—enabling control by plug-and-play external classifiers or estimators (Go et al., 2022).

e. Variational and SMC-based Density Modulation

Density ratio estimators and SMC schemes (e.g., RNE) can modulate the diffusion process at inference by rescaling path-wise likelihoods, controlling model composition or reward tilting via explicit Radon–Nikodym density modification (He et al., 6 Jun 2025).

3. Adaptive Guidance, Scheduling, and Plug-in Mechanisms

Static weighting of prior and likelihood limits plug-and-play generalization and performance. Recent advancements introduce:

• Adaptive Scale Tuning: Methods like SAIP compute a closed-form, data- and time-dependent balancing coefficient $s_t$ for the prior and likelihood terms at each step, thus mitigating the need for manual tuning and enhancing robustness across noise levels, tasks, and solvers (Wang et al., 29 Sep 2025).
• Memory and Conditional Token Injection: In video and 3D frameworks (e.g., DiT-Mem, PnP-U3D), reference modalities or external knowledge are encoded as memory tokens or connectors, concatenated into the model’s attention blocks at inference. These mechanisms support plug-and-play modularity without retraining the diffusion core (Song et al., 24 Nov 2025, Chen et al., 3 Feb 2026).

In all cases, the backbone diffusion model remains fixed; adaptation occurs through external components, lightweight adapters, or schedule-driven mechanisms orchestrated at inference time.

4. Methodological Flexibility and Application Domains

Plug-and-play conditional diffusion unlocks new domains and use cases:

• Biomedical and scientific imaging: State-of-the-art super-resolution and denoising in corneal OCT, compressive single-pixel imaging, and multimodal protein structure determination leverage the PnP paradigm for robust solution of ill-posed inverse problems, dynamic weighting of modalities, and adaptive noise estimation (Wang et al., 2 Feb 2026, Wang et al., 11 Sep 2025, Banerjee et al., 28 Jul 2025).
• Zero-shot and cross-modal synthesis: By freezing large zero-shot monocular depth estimators as global priors, a lightweight diffusion refiner can enhance any new predictor’s output, achieving superior depth accuracy without further retraining (Zhang et al., 2024).
• 3D and video generation: Modular plug-and-play connectors and memory encoders facilitate unified 3D understanding/generation and incorporation of world knowledge into diffusion video models, supporting both unconditional synthesis and highly conditioned editing or in-context adaptation (Chen et al., 3 Feb 2026, Song et al., 24 Nov 2025).
• Efficient compute and early exit: Properties of diffusion trajectories—such as accumulated score differences—enable plug-and-play, rejection-based filtering for computational savings and sample quality gains, without any changes to sampling schedule or model weights (Wang et al., 29 May 2025).

5. Quantitative Performance and Robustness

Plug-and-play conditional diffusion achieves competitive or superior results across diverse benchmarks:

Task / Method	Quantitative Results (select cases)
OCT super-res. (PnP-DM, EDM prior)	PSNR ≈ 32.14, SSIM ≈ 0.7150, LPIPS ≈ 0.1201 (bicubic: PSNR 28.13, SSIM 0.4607, LPIPS 0.4024)
ImageNet DDPM (PPAP, N=10 experts)	FID ≈ 27.86, IS ≈ 46.74 (DDIM25); FID ≈ 21.00, IS ≈ 57.38 (DDPM250)
Multimodal face synth. (PoE PnP)	CelebA FID: 26.1 (vs GAN FID ≈ 70), mIoU: 0.91, F1: 0.95
BetterDepth (MDE refiner, plug-in)	NYUv2 AbsRel/δ: 4.2/98.0, KITTI AbsRel/δ: 7.5/95.2, beating prior art; transfer to other MDEs matches or exceeds default performance
Denoising, deblurring, inpainting (SAIP)	Consistent PSNR boosts (+0.1–3 dB), improved SSIM/LPIPS across DPS, DMPS, πGDM on FFHQ and LSUN-bedroom. No degradation observed in any case (Wang et al., 29 Sep 2025).
Adaptive protein reconstruction (Adam-PnP)	p+D+E: avg RMSD 0.67 Å, outperforming single modality (P only: 0.74 Å); adaptive noise/weight estimation improves robustness to measurement uncertainty (Banerjee et al., 28 Jul 2025).

These findings demonstrate that plug-and-play provides robust, state-of-the-art conditioning without retraining, across modalities, domains, and tasks.

6. Convergence, Theoretical Guarantees, and Analysis

Rigorous mathematical analysis supports the stability and convergence of plug-and-play conditional diffusion methods even under nonconvexity and approximate updates:

• Split Gibbs MCMC with alternating Gaussian and diffusion prior steps achieves robust convergence in practical (≈100 iteration) settings (Wang et al., 2 Feb 2026, Wang et al., 20 May 2025).
• ADMM-based decoupling (ADMMDiff) establishes equivalence between reverse-diffusion steps and proximal operators, and proves sublinear $O(1/T)$ convergence to stationary points under mild conditions (Zhang et al., 2024).
• Adaptive scheduling (SAIP) is grounded in closed-form optimization of score-matching error, ensuring principled guidance adjustment and enhanced safety with negligible computational cost (Wang et al., 29 Sep 2025).
• Particle-based density estimators (RNE) rely on discrete Girsanov-based path-level weighting, with analytically tractable normalization and resampling for variational consistency (He et al., 6 Jun 2025).

Plug-and-play mechanisms remain robust to model class (DDPM, DDIM, EDM, VE-SDE, etc.), and inheritance of stability from the pretrained backbone is preserved regardless of the conditioning module’s provenance.

7. Outlook, Best Practices, and Limitations

Plug-and-play conditional diffusion establishes a general, extensible paradigm for modular, conditional generative modeling and inverse problem solving. Best practices include:

• Parameter-efficient adaptation (adapters, memory tokens, expert segmentation) for scalable, data-efficient control (Go et al., 2022, Song et al., 24 Nov 2025).
• Adaptive guidance schedules and dynamic weighting to accommodate non-stationary data and measurement uncertainty (Wang et al., 29 Sep 2025, Banerjee et al., 28 Jul 2025).
• Empirical validation across modalities, data regimes, and plug-in modules prior to deployment.

Current limitations include the need for careful design of auxiliary constraints or guidance modules, computation cost in certain iterative schemes (though mitigated by distillation (Hsiao et al., 2024)), and, in rare cases, suboptimal adaptation to extreme noise regimes if schedule parameters are not chosen judiciously.

Plug-and-play conditional diffusion is emerging as an essential approach for modular, extensible, and robust generative modeling, underpinned by principled Bayesian, variational, and energy-based inference (Wang et al., 2 Feb 2026, Nair et al., 2022, Chen et al., 3 Feb 2026, Zhang et al., 2024, Zhang et al., 2024, He et al., 6 Jun 2025, Nair et al., 2023, Wang et al., 29 Sep 2025, Go et al., 2022, Wang et al., 11 Sep 2025, Wang et al., 20 May 2025, Hsiao et al., 2024, Wang et al., 29 May 2025, Banerjee et al., 28 Jul 2025, Song et al., 24 Nov 2025).