---
title: Residual-Based Conditional Diffusion
url: https://www.emergentmind.com/topics/residual-based-conditional-diffusion
type: topic
---

# Residual-Based Conditional Diffusion

Residual-based conditional diffusion denotes a family of diffusion frameworks in which generation, restoration, or probabilistic prediction is organized around a reference signal—such as a coarse predictor, a degraded observation, a prior model, or a previously reconstructed state—and the diffusion component models residual structure relative to that reference rather than the full target from scratch. Across recent work, the residual may be the primary diffusion variable, a drift term in the forward process, a conditioning signal recomputed during deterministic sampling, or a physics residual injected through the objective. This design appears in medical image segmentation, event-driven video reconstruction, document enhancement, probabilistic time-series forecasting, sparse-view CT, image dehazing, PDE surrogates, semiconductor TCAD surrogates, image restoration, PET/MR denoising, and robust conditional generation, with a conceptual precursor in residual-bridge methods for conditioned diffusions [2509.01330], [2407.10636], [2305.03892], [2509.02341], [2603.02691], [2508.11134], [2507.06133], [2606.29272], [2305.20049], [2410.00184], [2408.02710], [1602.04439].

## 1. Conceptual scope and historical lineage

Residual-based conditional diffusion is not a single algorithmic template. The literature instead shows a recurring modeling principle: a strong but imperfect estimate handles large-scale or low-frequency structure, while diffusion models the remaining uncertainty, detail, or correction. In "Prior-Guided Residual Diffusion: Calibrated and Efficient Medical Image Segmentation" [2509.01330], the prior is a deterministic nnU-Net-like segmentor; in "DocDiff: Document Enhancement via Residual Diffusion Models" [2305.03892], it is a coarse predictor \(C_\theta\); in "Bridging Sequential Deep Operator Network and Video Diffusion: Residual Refinement of Spatio-Temporal PDE Solutions" [2507.06133], it is an S-DeepONet video prior; and in "Residual Prior Diffusion: A Probabilistic Framework Integrating Coarse Latent Priors with Diffusion Models" [2512.21593], it is a latent-variable Gaussian prior with decoder mean \(\hat{\mu}(z)\).

A second lineage uses residuals to align the diffusion dynamics with the measurement process rather than with a pretrained predictor. "Temporal Residual Guided Diffusion Framework for Event-Driven Video Reconstruction" [2407.10636] defines a temporal-domain residual \(x_0^t = I^t - \tilde I^{t-1}\), where \(\tilde I^{t-1}\) is a low-frequency estimate from the previous step. "Resfusion: Denoising Diffusion Probabilistic Models for Image Restoration Based on Prior Residual Noise" [2311.14900] inserts the residual \(R=\hat{x}_0-x_0\) directly into the forward process. "Volumetric Conditional Score-based Residual Diffusion Model for PET/MR Denoising" [2410.00184] models the volumetric residual \(r=x_{\text{Low}}-x_{\text{Nor}}\) rather than PET intensities themselves.

A third lineage uses residuals as step-wise correction signals inside deterministic or physics-guided sampling. "ReCo-Diff: Residual-Conditioned Deterministic Sampling for Cold Diffusion in Sparse-View CT" [2603.02691] computes an observation residual \(err_t=\mathcal{N}(x_t-D(\hat{x}_{0,t}^{\phi},v_t))\) at every step and conditions a second network pass on it. "PCGD: Physics-Guided Conditional Graph Diffusion for TCAD Device Simulation" [2606.29272] does not diffuse a residual target, but it enforces quasi-Fermi gradient matching and noise-aware PDE residuals on \(\hat{Y}_0\), so residuals enter as physics constraints rather than as the state variable.

The historical precursor is "Residual-Bridge Constructs for Conditioned Diffusions" [1602.04439], which applies modified diffusion bridge approximations to the difference between a true diffusion and an approximate diffusion driven by the same Brownian motion. This older bridge literature is not a DDPM literature, but it anticipates a central idea of later neural models: bridge or generate the discrepancy after extracting an approximate trajectory.

## 2. Residual variable and probabilistic parameterization

The literature instantiates the residual in several mathematically distinct locations.

| Formulation | Residual definition | Representative papers |
|---|---|---|
| Target-space residual | \(y-\text{prior}\), \(x_{\rm gt}-x^C\), \(I^t-\tilde I^{t-1}\), \(x_{\text{Low}}-x_{\text{Nor}}\), \(\mathbf{x}^{\text{GT}}-\mathbf{x}^{\text{prior}}\) | [2509.01330], [2305.03892], [2407.10636], [2410.00184], [2507.06133] |
| Prior-centered forward diffusion | Diffusion around \(\pi_\phi(x)\) or \(\hat{\mu}(z)\) instead of around zero-mean Gaussian only | [2509.01330], [2512.21593] |
| Residual embedded in forward drift | \(R=\hat{x}_0-x_0\) enters transition means; dual chains shift residuals between domains | [2311.14900], [2508.11134] |
| Residual used during sampling | Observation residual computed from current estimate and measurements | [2603.02691] |
| Residual used in objectives | PDE residuals and gradient proxies act as physics regularizers | [2606.29272] |

In PGRD, the state is centered on a prior prediction \(\pi_\phi(\mathbf X)\), with residual \(\mathbf r_t=\mathbf s_t-\pi_\phi(\mathbf X)\) and clean residual \(\mathbf r_0=\mathbf y_\star-\pi_\phi(\mathbf X)\). The forward process is
\[
q(\mathbf s_t\mid \mathbf s_{t-1},\mathbf X)=\mathcal N\!\big(\rho_t\,\mathbf s_{t-1}+(1-\rho_t)\,\pi_\phi(\mathbf X),\sigma_t^2\mathbf I\big),
\]
which induces a standard Gaussian diffusion on \(\mathbf r_t\). The denoiser predicts a residual-adapted velocity \(\widehat{\mathbf v}_t=f_\theta(\mathbf s_t,\mathbf X,\pi_\phi(\mathbf X),t)\), and reconstruction is always \(\widehat{\mathbf y}_0=\pi_\phi(\mathbf X)+\widehat{\mathbf r}_0\) [2509.01330].

DocDiff uses the simpler residual
\[
x_{\rm res}=x_{\rm gt}-x^C,
\]
then applies a standard DDPM forward process to \(x_0=x_{\rm res}\). Its reverse process uses \(x_0\)-prediction rather than \(\epsilon\)-prediction, and the final image is \(\hat x=x^C+\hat x_{\rm res}\) [2305.03892]. RDIT similarly separates a point estimator \(\hat{\mathbf y}=\mathscr M_{pt,\phi}(\mathbf x)\) from a residual diffusion model over
\[
\mathbf r=\mathbf y-\hat{\mathbf y},\qquad \mathbf r^0=\mathbf r/\boldsymbol{\sigma}_{trn},
\]
so that diffusion acts in a normalized residual space with approximately zero mean and unit variance [2509.02341].

Several works alter the forward law itself. Resfusion defines
\[
R=\hat{x}_0-x_0
\]
and modifies the Markov transition to
\[
q(x_t\mid x_{t-1},R)=\mathcal N\!\left(x_t;\sqrt{\alpha_t}\,x_{t-1}+\bigl(1-\sqrt{\alpha_t}\bigr)R,\,(1-\alpha_t)I\right),
\]
then predicts a weighted residual noise \(res\epsilon\) rather than pure \(\epsilon\) [2311.14900]. RBDM for dehazing defines \(e_0=y_0-x_0\) and \(e_1=x_0-y_0\), then uses dual Markov chains that interpolate between haze-free and hazy domains through these residuals, allowing bidirectional translation with only \(15\) sampling steps [2508.11134]. RPD moves still closer to a prior-conditioned probabilistic formalism: in prior-centered coordinates
\[
y_t=\frac{x_t-\hat{\mu}(z)}{\hat{\sigma}(z)},
\]
the forward process becomes standard DDPM diffusion on the residual deviation from the coarse latent prior [2512.21593].

This suggests that “residual-based” can refer either to the target variable, the coordinate system, or the actual stochastic dynamics.

## 3. Conditioning mechanisms

Conditioning is as important as the residual definition. The surveyed methods show four recurring conditioning patterns.

The first is **direct prior concatenation**. DocDiff conditions the HRR denoiser \(f_\theta(x_t,t,x^C)\) by channel-wise concatenation of the noisy residual and the coarse prediction \(x^C\) [2305.03892]. The unified image restoration framework of "A Unified Conditional Framework for Diffusion-based Image Restoration" [2305.20049] uses a lightweight UNet \(u_\phi(\mathbf x)\) as spatial guidance and injects it into every diffusion block through a Conditional Integration Module and an Adaptive Kernel Guidance Module, so the diffusion model learns the residual \(\mathbf y-u_\phi(\mathbf x)\) under spatially adaptive conditioning. CSRD conditions residual denoising on low-dose PET, MR, and 3D patch coordinates, thereby learning \(p(r\mid x_{\text{Low}},x_{\text{MR}})\) rather than an unconditional residual distribution [2410.00184].

The second is **time-dependent multi-path conditioning**. The event-driven video framework uses three conditioning paths: a pre-trained low-frequency intensity estimator \(\tilde I^{t-1}\), a temporal recurrent encoder on event voxels \(V^t\), and an attention-based high-frequency prior enhancement module that cross-attends between residual-event features, event features, and low-frequency features [2407.10636]. PGRD similarly conditions on both the image \(\mathbf X\) and the prior segmentation \(\pi_\phi(\mathbf X)\) at every diffusion step, and explicitly distinguishes this from classifier guidance because the guidance is built into the forward and reverse chains rather than injected via gradients of an external classifier [2509.01330].

The third is **global conditioning through structured tokens or operator priors**. In PDE surrogates, video diffusion is conditioned on the S-DeepONet prior video by channel concatenation and on the loading history \(\boldsymbol a\) via FiLM [2507.06133]. PCGD injects boundary conditions and device structure into a MeshGraphNet denoiser via boundary tokens, structure tokens, and global cross-attention from mesh nodes to condition tokens [2606.29272]. RDIT concatenates time-series history \(\mathbf x\), point forecasts \(\hat{\mathbf y}\), residual states \(\mathbf r^k\), and diffusion-step embeddings, then processes them with a bidirectional Mamba network [2509.02341].

The fourth is **conditioning through corrective residuals computed during inference**. ReCo-Diff first predicts a null reconstruction \(\hat x_{0,t}^{\phi}\), re-degrades it to the current view level, forms the residual \(err_t\), and then performs a second conditioned network call \(R_\theta([x_t,err_t],t)\) [2603.02691]. RCDM uses a different residual notion: the robust guidance term is
\[
\tilde{\varepsilon}_\theta=\varepsilon_\theta(x_t,y,t)+w\bigl(\varepsilon_\theta(x_t,y,t)-\varepsilon_\theta(x_t,t)\bigr),
\]
so the residual between conditional and unconditional denoisers becomes the adjustable guidance signal [2408.02710].

## 4. Objectives, parameterizations, and samplers

Residual-based conditional diffusion does not imply a single training objective. The surveyed methods use \(x_0\)-prediction, \(\epsilon\)-prediction, \(v\)-prediction, direct denoising, auxiliary supervision, and physics regularization.

PGRD adopts a residual-adapted velocity parameterization with
\[
\mathbf v_t=\sqrt{\bar\rho_t}\,\boldsymbol\varepsilon-\sqrt{1-\bar\rho_t}\,\mathbf r_0,
\]
and trains with an MSE loss on \(\mathbf v_t\), augmented by Deep Diffusion Supervision:
\[
\mathcal L_{\text{total}}=\mathcal L_{\mathrm{vel}}+\lambda\,\mathcal L_{\mathrm{DDS}},\qquad \lambda=0.1.
\]
DDS attaches decoder heads at selected timesteps and applies voxel-wise cross-entropy to intermediate predictions [2509.01330].

DocDiff instead uses \(x_0\)-prediction for the residual and a deterministic DDIM-style sampler with variance set to zero. It augments the diffusion loss with frequency separation: a low-frequency loss for the coarse predictor and a high-frequency loss for the residual diffusion model, with \(\beta_0=2\) and \(\beta_1=0.5\) [2305.03892]. Event-driven reconstruction uses standard DDPM \(\epsilon\)-prediction with an \(L_1\) loss on the residual image, and recurrent temporal state updates via ConvLSTM [2407.10636].

RPD proves that optimization reduces to familiar noise-prediction or velocity-prediction objectives in prior-centered coordinates, and introduces auxiliary variables such as
\[
\omega_t^\epsilon=\frac{x_t-\hat{\mu}(z)}{\sqrt{1-\bar{\alpha}_t}\,\hat{\sigma}(z)}
\]
to reduce prediction difficulty when the prior reconstruction is accurate [2512.21593]. Resfusion also changes the prediction target, replacing \(\epsilon\) with
\[
res\epsilon=\epsilon+\frac{(1-\sqrt{\alpha_t})\sqrt{1-\bar{\alpha}_t}}{\beta_t}\,R,
\]
while preserving the DDPM reverse form [2311.14900].

Physics-guided variants use hybrid objectives. PCGD adds exponent-free quasi-Fermi gradient matching
\[
\mathcal L_{\mathrm G}=\frac{1}{|E|}\sum_{(i,j)\in E}\sum_{r\in\{\psi,n,p\}}\bigl(\Phi_r(\hat Y_0,e_{ij})-\Phi_r(Y_0,e_{ij})\bigr)^2
\]
and a noise-aware PDE residual loss
\[
\mathcal L_{\mathrm P}(\hat Y_0,c)=\lambda_\psi\|R_\psi\|_2^2+\lambda_n\|R_n\|_2^2+\lambda_p\|R_p\|_2^2,
\]
combined as
\[
\mathcal L_{\mathrm{total}}=\lambda_{\mathrm D}\mathcal L_{\mathrm D}+\lambda_{\mathrm G}\mathcal L_{\mathrm G}+\alpha_{\mathrm{bal}}\,w_{\mathrm P}(t)\,\mathcal L_{\mathrm P}
\]
with \(\lambda_{\mathrm D}=1.0\), \(\lambda_{\mathrm G}=2\times10^{-2}\), \(r_{\mathrm P}=0.03\), and \(\lambda_p=10^4\) [2606.29272].

Sampling strategies are equally diverse. PGRD uses DDIM-style sampling with \(S\ll T\) and reports near-peak Dice at about \(300\) steps, whereas vanilla DDPM required more than \(800\) steps [2509.01330]. DocDiff emphasizes deterministic few-step sampling and reports strong perceptual quality even with \(5\) steps [2305.03892]. ReCo-Diff keeps a deterministic cold-diffusion update but adds residual-conditioned self-guidance at each step [2603.02691]. The image restoration framework introduces inter-step patch-splitting for arbitrary-resolution inference without grid artifacts [2305.20049].

## 5. Reported domains and empirical behavior

The empirical record is heterogeneous but consistent in one respect: most papers report that residual-based conditionalization improves either fidelity, calibration, efficiency, or robustness relative to direct diffusion baselines.

In medical segmentation, PGRD reports higher Dice and lower NLL/ECE than Bayesian, ensemble, Probabilistic U-Net, and vanilla diffusion baselines, while requiring fewer sampling steps. On BraTS2024, DDPM achieved DSC \(78.3\pm24.3\), NLL \(4.8\pm1.8\), ECE \(0.23\pm0.13\), whereas PGRD achieved DSC \(81.7\pm26.8\), NLL \(4.2\pm1.3\), ECE \(0.20\pm0.12\). On INSTANCE2022, DDPM achieved DSC \(59.8\pm19.3\), NLL \(1.12\pm0.32\), ECE \(0.68\pm0.26\), whereas PGRD achieved DSC \(62.2\pm21.8\), NLL \(1.00\pm0.23\), ECE \(0.65\pm0.24\) [2509.01330].

In document enhancement, DocDiff reports that DocDiff (Non-native)-5, with \(8.2\)M parameters, achieved MANIQA \(0.6873\), LPIPS \(0.0582\), PSNR \(22.17\), and SSIM \(0.9223\) on deblurring, compared with MPRNet’s MANIQA \(0.6675\), LPIPS \(0.0900\), PSNR \(21.27\), and SSIM \(0.8803\). It also reports OCR gains, with character error rate decreasing from about \(13.7\%\) after DE-GAN to about \(7.6\%\) after DE-GAN+HRR and about \(4.4\%\) after DocDiff [2305.03892].

In probabilistic time-series forecasting, RDIT is evaluated on Traffic, Weather, Electricity, Exchange, Solar, ETTh1, ETTm1, and ETTm2, and the paper states that it achieves the best overall CRPS on almost all datasets, the smallest or near-smallest PICP distance on most datasets, and state-of-the-art or second-best MAE and MSE among PTSF models when distributions are collapsed to means [2509.02341]. This suggests that residual diffusion can improve probabilistic calibration without sacrificing point accuracy when a strong point forecaster is available.

In sparse-view CT, ReCo-Diff reports quantitative gains over cold-diffusion baselines. For \(72\) views, CvG-Diff achieved RMSE \(15.77\), PSNR \(45.63\) dB, SSIM \(98.54\%\), whereas ReCo-Diff achieved RMSE \(13.50\), PSNR \(46.98\) dB, SSIM \(98.89\%\). For \(36\) views, CvG-Diff achieved RMSE \(24.67\), PSNR \(41.77\) dB, SSIM \(97.05\%\), whereas ReCo-Diff achieved RMSE \(22.45\), PSNR \(42.57\) dB, SSIM \(97.45\%\) [2603.02691].

In dehazing, RBDM reports PSNR \(32.68\) and SSIM \(0.976\) on RESIDE-6K, and on NTIRE 2020 it reports \(23.19/0.6665\), versus SCANet’s \(19.52/0.6488\). It further reports that \(15\) steps outperform both \(5\) and \(35\) steps on NTIRE2020, with \(23.19/0.6665\) at \(15\) steps compared with \(22.34/0.6411\) at \(5\) and \(22.68/0.6527\) at \(35\) [2508.11134].

In PDE surrogates, the residual video-diffusion model conditioned on S-DeepONet reduces the mean relative \(L_2\) error from \(4.58\%\) to \(0.829\%\) on the cavity-flow benchmark and from \(4.43\%\) to \(2.94\%\) on the dogbone plasticity benchmark [2507.06133]. In TCAD surrogates, PCGD reports that deterministic direct regression reaches mean relative field error \(1.207\%\), local diffusion reaches \(1.585\%\), pure condition-aware diffusion reaches \(0.720\%\), and full PCGD reaches \(0.835\%\) while reducing the aggregate residual metric from \(16.44\) to \(13.56\) [2606.29272].

## 6. Misconceptions, limitations, and open directions

A common misconception is that residual-based conditional diffusion is equivalent to classifier guidance or to ordinary conditional concatenation. The surveyed work shows otherwise. PGRD explicitly distinguishes segmentor-as-prior guidance from classifier or score guidance, because the prior enters the forward and reverse chains directly rather than through gradients of a separate classifier [2509.01330]. ReCo-Diff likewise contrasts residual-conditioned self-guided sampling with classifier guidance and classifier-free guidance: the guidance signal is the observation residual in image space, not a score-space interpolation [2603.02691].

Another misconception is that residualization is universally beneficial. The AR diffusion study on conditional dependence shows that structured factorization helps only when the partition and ordering align with the true dependency structure; when patches break the natural dependency, AR diffusion can underperform vanilla DDPM [2504.21314]. This suggests that residual targets and conditional factorizations must match the problem geometry or physics.

Several limitations recur. Residual methods often depend on a strong coarse prior. RPD states that if the prior \(\hat{\mu}(z)\) is poor, few-step benefits may disappear and auxiliary variables lose effectiveness [2512.21593]. The PDE surrogate paper notes that diffusion remains relatively slow versus S-DeepONet alone, even though it is far faster than CFD or FEA [2507.06133]. ReCo-Diff and the event-driven video framework both retain long iterative samplers in their strongest settings, with \(18\) or more steps for sparse-view CT and \(2000\) diffusion steps for event-driven reconstruction during analysis [2603.02691], [2407.10636]. PCGD reports poor zero-shot transfer to completely new SOI topologies before LoRA adaptation, indicating that residual-based physics-aware diffusion is not automatically topology invariant [2606.29272]. CSRD depends on 3D patch-wise training and MR conditioning to maintain volumetric coherence; without MR, residual noise remains more visible in background regions [2410.00184].

Open directions arise directly from these limits. RPD proposes multi-level hierarchical priors and latent-space variants [2512.21593]. The PDE surrogate work points to diffusion distillation, mixture-of-experts operator priors, and stronger physics-informed diffusion objectives [2507.06133]. PCGD identifies PDE-guided sampling, broader pretraining, and 3D scaling as natural next steps [2606.29272]. More broadly, the surveyed literature suggests that residual-based conditional diffusion is best understood as a modular pattern: select an informative reference, define a residual whose statistics are simpler than those of the full target, choose a conditioning mechanism that preserves the relevant dependencies, and align the sampler or loss with the structure of the residual itself.

Source: https://www.emergentmind.com/topics/residual-based-conditional-diffusion