Residual Prior Diffusion

• Residual Prior Diffusion is a generative framework that decomposes modeling by using a coarse prior to capture global structure with a diffusion process focused on fine-scale residuals.
• The method employs a two-stage approach where a pretrained prior provides an initial guess and a subsequent diffusion model refines the residual, leading to improved convergence and sample quality.
• Empirical results show RPD enhances performance in image restoration, medical segmentation, and physics-based surrogate modeling by efficiently leveraging multi-scale information.

Residual Prior Diffusion (RPD) refers to a broad class of generative models that systematically integrate a coarse prior—typically capturing large-scale, low-frequency, or physically important nuisance structure—with a diffusion process learning the fine-scale residual required to match the target data distribution or solution. The paradigm applies across domains, including probabilistic modeling of images, spatio-temporal PDE surrogates, inverse problems, and uncertainty-calibrated predictions. RPD architectures leverage the theoretical and practical advantages of splitting the learning burden: the prior provides global structure or an initial guess, and the diffusion model operates in the locally concentrated residual space, resulting in faster convergence, improved sample quality, and robustness with fewer inference steps (Kutsuna, 25 Dec 2025, Shi et al., 2023, Park et al., 8 Jul 2025, Mao et al., 1 Sep 2025).

1. Mathematical Formulation and Probabilistic Structure

Residual Prior Diffusion is built on a two-stage probabilistic model. Let $x_0$ represent the target data, $z$ auxiliary latent variables, and $\hat{\mu}(z)$ a coarse prior mapping (e.g., the output of a VAE, U-Net, or operator network). Instead of learning $p_\theta(x_0)$ from a basic prior (e.g., $\mathcal{N}(0,I)$), RPD defines a prior-driven generative model:

• Stage 1: Fit or pretrain a prior $\hat{p}(x_0|z)$ (or $\hat{p}(x_0)$ for deterministic priors). This prior captures the global/low-frequency structure:

$\hat{p}(x_0, z) = \hat{p}(z)\hat{p}(x_0|z)$

• Stage 2: Define the residual $R = x_0 - \hat{\mu}(z)$, and train a diffusion model on $R$, with the forward process given by:

$$x_t = \sqrt{\bar{\alpha}_t} x_0 + (1 - \sqrt{\bar{\alpha}_t}) \hat{\mu}(z) + \sqrt{1 - \bar{\alpha}_t}\,\hat{\sigma}(z)\,\epsilon$$

where $\epsilon \sim \mathcal{N}(0, I)$, and $(\bar{\alpha}_t)$ is the usual accumulated variance schedule.

• Reverse process: Learn $p_\theta(x_{t-1}|x_t, z)$ as a conditional Gaussian parameterized by $z$ and the current $x_t$; the output is typically parameterized by either noise or velocity prediction (Kutsuna, 25 Dec 2025).
• Inference: Given new data or conditioning, recover $x_0$ by applying the reverse diffusion from a noisy version of the coarse prior prediction.

The overall log-likelihood lower bound (ELBO), derived in (Kutsuna, 25 Dec 2025), decomposes into contributions from (i) prior model fit, and (ii) residual refinement matching the data distribution.

2. Diffusion Process with Residual Guidance

The distinctive feature of RPD is the explicit drift or centering of the forward and reverse chains on a prior output, modifying both the forward noising and denoising (reverse) steps. The residual process for discrete diffusion models is given by (Shi et al., 2023, Kutsuna, 25 Dec 2025):

$$q(x_t|x_{t-1}, R) = \mathcal{N}(x_t; \sqrt{\alpha_t}x_{t-1} + (1-\sqrt{\alpha_t})R,\, (1-\alpha_t)I)$$

with closed-form marginal:

$$x_t = \sqrt{\bar{\alpha}_t}x_0 + (1-\sqrt{\bar{\alpha}_t}) R + \sqrt{1 - \bar{\alpha}_t}\,\epsilon$$

For restoration or conditional generation, $R$ encodes the deviation between degraded input and target; for unconditional generation, $R=-x_0$ recovers the standard DDPM (Shi et al., 2023).

In multi-stage or physics-constrained surrogates, the prior $\hat{u}(x)$ may be obtained from, for example, an operator network or a physical simulation. The diffusion chain is trained only on the normalized residual field, with inference reconstructing the final solution as the sum of the prior and the denoised residual (Park et al., 8 Jul 2025).

3. Auxiliary Variables and Theoretical Analysis

Auxiliary variables $\omega_t$ capture the normalization offset between the noisy sample and the prior, providing improved targets for denoising prediction and tightening the theoretical bound on mean squared regression errors, as established in (Kutsuna, 25 Dec 2025). Specifically, in noise-prediction mode:

$$\omega_t^\epsilon := \frac{x_t - \hat{\mu}(z)}{\sqrt{1 - \bar{\alpha}_t}\,\hat{\sigma}(z)}$$

Proposition 5.1 quantifies $E[\|\epsilon_0 - \omega_t^\epsilon\|^2]$, which shrinks as $\hat{\mu}(z) \to x_0$. Inclusion of $\omega_t$ as an extra network input accelerates convergence and sharpens fine-scale details.

For velocity-prediction, the analogous auxiliary variable is:

$$\omega_t^v := \sqrt{\bar{\alpha}_t}(x_t - \hat{\mu}(z))/[\sqrt{1-\bar{\alpha}_t}\,\hat{\sigma}(z)]$$

with similar theoretical control (Kutsuna, 25 Dec 2025).

4. Conditioning Mechanisms and Architectural Implementations

RPD can be implemented with diverse prior generators—latent-variable models (VAE, $\beta$-VAE, MoG, VQ-VAE), deterministic surrogates (U-Net, DeepONet), or problem-specific operator networks. The architecture splits as follows:

• Prior Network: Trained independently (e.g., MLP, CNN, S-DeepONet with GRU-MLP for spatio-temporal PDEs (Park et al., 8 Jul 2025)), outputting coarse predictions.
• Residual Diffusion: Trained to denoise the residual with the prior prediction as a conditional input in every diffusion step (concatenation, FiLM layers, or channel-wise addition).
• Loss Functions: Standard $\ell_2$ losses in noise or velocity space; additional focal loss or deep diffusion supervision layers may be deployed to reweight hard examples and improve convergence (Mao et al., 1 Sep 2025).

Practical hyperparameter selection includes tuning the number of diffusion steps (RPD is robust to strong reduction; competitive with $T=3$–$10$), model capacity (the residual denoising network can be narrower), and using the auxiliary variables in the network input space (Kutsuna, 25 Dec 2025).

5. Algorithmic Workflows and Pseudocode

The training and inference pipelines for RPD are consistent across domains:

Stage	Step Description
Prior	Train $\hat{p}(x_0|z)$ or $\pi_\phi(x)$ via ELBO or supervised regression.
Training	Sample $x_0$, $z$; construct $x_t$ via residual-guided noising; compute auxiliary variable $\omega_t$; optimize $\ell_2$ loss between network output and $\epsilon_0$ or $\hat{v}_t$.
Inference	Sample $z$ from prior; generate $x_T$ from $\mathcal{N}(\hat{\mu}(z),\hat{\sigma}^2(z)I)$; reverse diffuse to $x_0$ using the denoiser conditioned on prior.

Residual learning enables acceleration by truncating the forward schedule (e.g., $T' \approx T/2$), and, in image restoration settings, initializing sampling from a noisy degraded input rather than pure Gaussian noise (Shi et al., 2023).

6. Empirical Performance and Benchmarks

RPD provides substantial quantitative and qualitative gains over traditional diffusion and prior-only baselines:

• Physics surrogates: On vortex-dominated lid-driven flow, mean relative $L^2$ error falls from $4.58\%$ (S-DeepONet only) to $0.83\%$ (video diffusion prior-corrected RPD, VD-PC-R); for elasto-plastic dogbone deformation, from $4.43\%$ to $2.94\%$ (Park et al., 8 Jul 2025).
• Medical segmentation: Dice increases compared to Bayesian, ensemble, and vanilla diffusion approaches; PGRD achieves $81.7\%$ Dice on BraTS2024 (vs $78.3\text{–}80.0\%$), lower NLL and ECE, with $300$ sampling steps instead of $800$–$1000$ (Mao et al., 1 Sep 2025).
• Image restoration (Resfusion): PSNR/SSIM improvements across ISTD (shadow removal), LOL (low-light), and Raindrop datasets with only $5$ sampling steps (Shi et al., 2023).
• Generative modeling: Synthetic hetero-scale benchmarks and real-world images (Butterflies, FMNIST) show that RPD matches or surpasses DDPMs and variants, especially when the number of inference steps is strongly reduced (as few as $3$), without loss of diversity or fine-scale coherence (Kutsuna, 25 Dec 2025).

7. Applications, Limitations, and Practical Considerations

RPD has been effectively deployed for:

• Surrogate modeling of nonlinear time-dependent PDEs, recovering global physics and sharp local features (Park et al., 8 Jul 2025).
• Probabilistic segmentation with uncertainty calibration in medical images (Mao et al., 1 Sep 2025).
• Accelerated image restoration and conditional/unconditional image generation (Shi et al., 2023).
• General-purpose generative modeling for distributions with multi-scale structure (Kutsuna, 25 Dec 2025).

Limitations include reliance on the quality of the prior (failure of the prior to encode coarse structure limits RPD's gains). Computational expense remains higher than single-shot deterministic models, but is considerably lower than unconditioned diffusion processes for equivalent quality (Mao et al., 1 Sep 2025). RPD frameworks assume the prior can be encoded as a continuous field; for discrete outputs, suitable embeddings (e.g., continuous one-hots) are typically used.

In summary, Residual Prior Diffusion unifies a family of models that exploit a coarse prior and a diffusion-based corrector trained solely on the task-relevant residual. This separation of concerns enables efficient and scalable learning in domains where target structure exhibits strong multi-scale or hierarchical properties, with demonstrated improvements in convergence, accuracy, calibration, and computational performance (Kutsuna, 25 Dec 2025, Park et al., 8 Jul 2025, Mao et al., 1 Sep 2025, Shi et al., 2023).