Score-Based Diffusion Models

• Score-based diffusion models are deep generative frameworks that synthesize data by reversing a forward stochastic process and approximating the reverse dynamics.
• They use neural networks to estimate the time-dependent score via denoising score matching, ensuring rigorous convergence and identifiability guarantees.
• These models support versatile sampling techniques—including SDE, ODE, and conditional schemes—with applications ranging from image synthesis to scientific data assimilation.

A score-based diffusion model is a deep generative modeling framework that synthesizes data by (i) progressively perturbing samples with a known stochastic process (the forward SDE or Markov chain) to noise, and (ii) learning a time-dependent vector field—the "score function," i.e., the gradient of the log-probability of the perturbed data distribution at every noise level. The generative process samples from noise and applies an approximate time-reverse SDE (or an equivalent ODE), replacing the unknown score with a neural estimator trained by denoising score matching. Score-based diffusion models admit rigorous connections to SDEs, continuous normalizing flows, and possess well-understood convergence and identifiability guarantees in both continuous and discrete state spaces. The framework has yielded state-of-the-art results in image synthesis, spatiotemporal forecasting, Bayesian inverse problems, function-space modeling, and scientific data assimilation.

1. Mathematical Foundations: Forward/Reverse SDEs and the Score

A core principle is the definition of a forward stochastic process—most commonly, an Itô SDE of the form

$$dx_t = f(x_t, t)\,dt + g(t)\,dw_t, \qquad x_0 \sim p_{\text{data}}$$

where $f$ is the drift, $g(t)$ controls the noise schedule, and $w_t$ is Brownian motion (Tang et al., 12 Feb 2024).

The forward process increases entropy such that $p_t$ (the marginal law at time $t$) becomes a simple distribution (e.g., a high-variance Gaussian). Under suitable smoothness, the time reversal can also be written as an SDE (Anderson 1982; Haussmann–Pardoux 1986): $$dy_t = [-f(y_t,T-t) + g^2(T-t) \nabla_y \log p(T-t, y_t)]\,dt + g(T-t)\,dw_t$$ where the key unknown term is the score $\nabla_x \log p(t, x)$ (Tang et al., 12 Feb 2024).

This formalism also generalizes to finite-step Markov chains (DDPM) and to discrete state space by Markov jump processes (CTMCs) (Zhang et al., 3 Oct 2024, Sun et al., 2022). For function space and infinite-dimensional settings, the SDEs act in Hilbert space with trace-class noise (Lim et al., 2023, Baldassari et al., 2023).

2. Score Estimation and Denoising Score Matching

Since the true score is intractable except for simple distributions, it is approximated by a neural network $s_\theta(x, t)$. Training is performed by minimizing a score-matching or denoising score-matching (DSM) objective. For instance, in the Gaussian SDE case: $$J_{\text{DSM}}(\theta) = \mathbb{E}_{x_0, t, x_t|x_0}\; \lambda(t)\,\|s_\theta(x_t, t) - \nabla_{x_t} \log p(x_t|x_0)\|^2$$ where $x_t|x_0$ is an explicit Gaussian, so the target score is computable analytically (Tang et al., 12 Feb 2024, Song et al., 2021, Chung et al., 2021).

For discrete state space, the score is replaced with singleton conditional ratios and trained via cross-entropy objectives matching the conditional marginals, yielding an unbiased estimator for the backward jump rates (Sun et al., 2022, Zhang et al., 3 Oct 2024).

Training procedures may be further enhanced by regularization terms enforcing PDE constraints derived from the underlying Fokker–Planck equation, such as the score-FPE (Lai et al., 2022). Efficiency improvements are enabled by pre-computing score fields or embedding the score into the data representation (Na et al., 10 Apr 2024).

3. Sampling Algorithms: SDE, ODE, PC Samplers, and Consistency Flows

Reverse SDE sampling: At generation time, one solves the reverse SDE, typically discretized by Euler–Maruyama steps. The Predictor–Corrector (PC) sampler alternates predictor (Euler–Maruyama) and corrector (Langevin) steps; the latter refines samples by approximate Langevin Monte Carlo using the neural score (Tang et al., 12 Feb 2024, Chung et al., 2021, Yi et al., 2023).

Reverse ODE sampling: By replacing the SDE noise term with zero, one obtains a "probability-flow" ODE: $$\frac{dx_t}{dt} = f(x_t, t) - \frac{1}{2}g^2(t) s_\theta(x_t, t)$$ which can be integrated by standard deterministic solvers (e.g., RK45) (Tang et al., 12 Feb 2024, Song et al., 2021).

Discrete/CTMC samplers: For categorical data or structured discrete problems, sampling is performed by running the reverse CTMC with rates parameterized by neural estimators of conditional ratios. Both Euler-type (numerical) and closed-form (analytical) backward kernels are available (Sun et al., 2022).

Latent and conditional sampling: For high-dimensional or structured outputs, diffusion can be applied in a learned latent space, or conditioned on side information using multi-channel or conditional architectures (Chase et al., 15 May 2025).

Consistency models: Recent work shows entire generative flows can be amortized into a one-step map by distilling trajectories of the ODE (Tang et al., 12 Feb 2024).

4. Theoretical Guarantees: Convergence, Adaptivity, and Likelihood

Explicit error bounds in total-variation (TV) and Wasserstein distances characterize the gap between the true data law $p_0$ and the sample law produced by a learned score model (Tang et al., 12 Feb 2024, Zhang et al., 3 Oct 2024). For continuous SDEs, TV error is at most $O(\epsilon \sqrt{T})$ in the $L^2$ score error $\epsilon$ and the time horizon $T$ (with an exponentially decreasing initialization bias).

For discrete diffusion, KL and TV error bounds scale nearly linearly with dimension $d$, matching the best continuous analogues and controlled by step size and average score entropy (Zhang et al., 3 Oct 2024).

Low-dimensional adaptivity: When data are concentrated near a $k$-dimensional manifold in ambient $\mathbb{R}^d$, special step-size schedules can yield total-variation convergence rates $O(k^2/\sqrt{T})$, so discretization error depends only on the intrinsic dimension $k$ (Li et al., 23 May 2024).

Maximum likelihood: By weighting the DSM loss with $g^2(t)$, the resulting (approximate) loss upper bounds the negative log-likelihood of the induced generative model, enabling practical ML training (Song et al., 2021).

5. Architectures, Conditional Models, and Physics-Informed Extensions

Network architectures: U-Net-style convolutional blocks with residual or linear skip structure dominate, often combined with noise/time embeddings (e.g., FiLM, sinusoidal), self-attention, and latent-space diffusion for efficiency (Chase et al., 15 May 2025, Tang et al., 12 Feb 2024).

Conditional models: Conditional diffusion is implemented by concatenating conditioning frames or auxiliary data along the channel dimension, adding pre-trained initial guesses (for residual-corrective approaches), or mapping observations into the diffusion's embedding space (Chase et al., 15 May 2025, Chung et al., 2021).

Physics-informed and inverse problems: Inverse tasks (e.g., MRI, CT, geophysical imaging, nowcasting) integrate the learned score function as a Bayesian prior by augmenting the sampling procedure with data-consistency or likelihood steps—alternately projecting onto the data constraint or fusing Fourier measurements (Chung et al., 2021, Han et al., 23 May 2024, Baldassari et al., 2023, Feng et al., 2023, Chase et al., 15 May 2025). Physics-informed methods embed the physical model into the score itself or perform ensemble-based filtering without neural training (Huynh et al., 9 Aug 2025).

Discretization-invariance and operator networks: Score-based frameworks in function space utilize neural operators (e.g., Fourier Neural Operator) allowing mesh-independent generalization and effective training on infinite-dimensional problems (Lim et al., 2023, Baldassari et al., 2023).

Training and efficiency: Score embedding, sliced losses, and analytic score computation can dramatically accelerate convergence, reduce data requirements, and lower the computational cost relative to conventional DSM training (Na et al., 10 Apr 2024).

6. Applications, Impact, and Extensions

Score-based diffusion models represent the state of the art for generation and imputation in high-dimensional data domains (images, medical signals, spatiotemporal fields), as well as for scientific filtering, uncertainty quantification, and adaptive solution of PDEs and inverse problems (Chase et al., 15 May 2025, Huynh et al., 9 Aug 2025, Han et al., 23 May 2024).

They enable rigorous posterior sampling for Bayesian inference—yielding calibrated uncertainties unavailable to classical point estimators or deterministic regularizers (McCann et al., 2023, Feng et al., 2023). Probabilistically principled formulations enable variational or MCMC approaches with explicit, learned priors, and task-adaptive strength without hand-tuned hyperparameters.

Recent developments include infinite-dimensional diffusion for scientific computing, discrete-state (CTMC) diffusion for language and music modeling, and flexible SDE parameterizations that subsume prior models, showing the breadth of the framework’s generality and adaptability (Sun et al., 2022, Du et al., 2022, Li et al., 23 May 2024).

Ongoing theoretical work addresses ELBO tightness, gap/variance tradeoffs with diffusion time, higher-order discretization, and convergence in manifold-structured data. Algorithmic advances seek to further unify model classes, reduce score estimation variance, and scale posterior inference to massive domains.

---

References:

• (Tang et al., 12 Feb 2024) "Score-based Diffusion Models via Stochastic Differential Equations -- a Technical Tutorial"
• (Song et al., 2021) "Maximum Likelihood Training of Score-Based Diffusion Models"
• (Li et al., 23 May 2024) "Adapting to Unknown Low-Dimensional Structures in Score-Based Diffusion Models"
• (Lai et al., 2022) "FP-Diffusion: Improving Score-based Diffusion Models by Enforcing the Underlying Score Fokker-Planck Equation"
• (Zhang et al., 3 Oct 2024) "Convergence of Score-Based Discrete Diffusion Models: A Discrete-Time Analysis"
• (Sun et al., 2022) "Score-based Continuous-time Discrete Diffusion Models"
• (Chase et al., 15 May 2025) "Score-based diffusion nowcasting of GOES imagery"
• (Na et al., 10 Apr 2024) "Efficient Denoising using Score Embedding in Score-based Diffusion Models"
• (Lim et al., 2023) "Score-based Diffusion Models in Function Space"
• (Baldassari et al., 2023) "Conditional score-based diffusion models for Bayesian inference in infinite dimensions"
• (McCann et al., 2023) "Score-based Diffusion Models for Bayesian Image Reconstruction"
• (Feng et al., 2023) "Score-Based Diffusion Models as Principled Priors for Inverse Imaging"
• (Huynh et al., 9 Aug 2025) "A Score-based Diffusion Model Approach for Adaptive Learning of Stochastic Partial Differential Equation Solutions"
• (Du et al., 2022) "A Flexible Diffusion Model"
• (Chung et al., 2021) "Score-based diffusion models for accelerated MRI"