Neural Diffusion Processes

• Neural Diffusion Processes are denoising diffusion models that generate distributions over functions by learning finite-dimensional marginals at arbitrary input locations.
• They apply a diffusion process to function values and utilize bi-dimensional attention to ensure exchangeability and input-dimension invariance.
• Empirical studies show that NDPs effectively emulate Bayesian posterior distributions in tasks such as synthetic regression, image regression, and Bayesian optimization.

Searching arXiv for core and closely related papers on Neural Diffusion Processes. arXiv search query: "Neural Diffusion Processes" Neural Diffusion Processes (NDPs) are denoising diffusion generative models for distributions over functions. Rather than modeling images or audio on a fixed grid, an NDP models a stochastic process by learning its finite-dimensional marginals over function values at arbitrary input locations: functions are represented by samples $(\mathbf{x},\mathbf{y})$ at finitely many locations, diffusion is applied to the function values $\mathbf{y}$, and a neural network learns to reverse the diffusion (Dutordoir et al., 2022). The framework was introduced as a novel approach that learns to sample from a rich distribution over functions through its finite marginals, with architecture-level treatment of stochastic-process symmetries such as exchangeability and input-dimension invariance (Dutordoir et al., 2022).

1. Conceptual position within stochastic process modeling

In the NDP formulation, the central object is a stochastic process $f:\mathbb{R}^D \to \mathbb{R}$, observed through finite sets of evaluations. This places NDPs in the same broad problem class as Gaussian Processes (GPs) and Neural Processes (NPs), but with a substantially different generative mechanism. GPs define a stochastic process such that every finite set of evaluations is jointly Gaussian; this gives exact posterior prediction but restricts all finite marginals to be multivariate Gaussian and makes prior design through kernels difficult in complex or high-dimensional settings. NPs use encoder-decoder architectures to model $p(\mathbf{y}^* \mid \mathbf{x}^*, \mathcal{D})$, but the NDP literature emphasizes that Conditional NPs factorize targets independently, Latent NPs use approximate inference, Attentive NPs often produce jittery samples, and NP predictive distributions are not Bayes-consistent in the GP sense (Dutordoir et al., 2022).

NDPs differ by generating samples through diffusion over the joint finite marginal rather than direct decoding. In the original formulation, this allows them to produce correlated samples, better approximate posterior structure, and maintain exchangeability within the sampled set. The reported empirical summary is that NDPs can capture functional distributions close to the true Bayesian posterior, can successfully emulate the behaviour of Gaussian processes, and surpass the performance of neural processes (Dutordoir et al., 2022).

A common shorthand in subsequent work is to view NDPs as diffusion probabilistic models over functions. In the wind-power extension, this is stated explicitly: NDPs adapt the idea of diffusion probabilistic models from data generation to function regression, learning arbitrary function distributions via diffusion rather than assuming Gaussian finite-dimensional marginals (Rawson et al., 3 Oct 2025). A related later critique describes standard NDPs as essentially conditional DDPMs for function regression: the inputs are used in the reverse model, but the default diffusion itself is not anchored to the input (Xu et al., 10 Aug 2025).

2. Finite-marginal diffusion formulation

The original NDP construction follows standard denoising diffusion models but applies them to finite samples from a function. For a generic variable $\mathbf{s}_0$, the forward process is

$$q(\mathbf{s}_{0:T}) = q(\mathbf{s}_0)\prod_{t=1}^T q(\mathbf{s}_t \mid \mathbf{s}_{t-1}),$$

with Gaussian corruption

$$q(\mathbf{s}_t \mid \mathbf{s}_{t-1}) = \mathcal{N}\!\left(\mathbf{s}_t;\sqrt{1-\beta_t}\,\mathbf{s}_{t-1},\beta_t\mathbf{I}\right).$$

Writing $\alpha_t = 1-\beta_t$ and $\bar{\alpha}_t=\prod_{j=1}^t \alpha_j$, the reparameterized form is

$$\mathbf{s}_t = \sqrt{\bar{\alpha}_t}\,\mathbf{s}_0 + \sqrt{1-\bar{\alpha}_t}\,\boldsymbol{\varepsilon}, \qquad \boldsymbol{\varepsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I}).$$

The denoising model learns $\mathbf{y}$0 by minimizing

$\mathbf{y}$1

For NDPs, the state is a function sample $\mathbf{y}$2, and the default construction keeps $\mathbf{y}$3 fixed while diffusing only $\mathbf{y}$4 (Dutordoir et al., 2022).

The forward kernel specialized to function values is

$\mathbf{y}$5

with $\mathbf{y}$6. The reverse kernel is parameterized as

$\mathbf{y}$7

with

$\mathbf{y}$8

The corresponding NDP objective is

$\mathbf{y}$9

This is the function-space analogue of DDPM training (Dutordoir et al., 2022).

Sampling proceeds by fixing a set of input locations $f:\mathbb{R}^D \to \mathbb{R}$0, drawing $f:\mathbb{R}^D \to \mathbb{R}$1, and iterating the learned reverse process until $f:\mathbb{R}^D \to \mathbb{R}$2 is obtained. The result is a sample from a learned distribution over functions evaluated at the specified inputs (Dutordoir et al., 2022).

3. Architecture, symmetry, and conditional inference

A central architectural contribution of the original NDP paper is the bi-dimensional attention block. Its motivation is explicitly stochastic-process-theoretic: a process model should be exchangeable over datapoint order and invariant to feature order. The model reshapes inputs $f:\mathbb{R}^D \to \mathbb{R}$3 into a tensor

$f:\mathbb{R}^D \to \mathbb{R}$4

by replicating outputs across input dimensions and concatenating features. The block

$f:\mathbb{R}^D \to \mathbb{R}$5

applies two multi-head self-attention layers in parallel, one across the feature dimension $f:\mathbb{R}^D \to \mathbb{R}$6 and one across the datapoint dimension $f:\mathbb{R}^D \to \mathbb{R}$7, with residual connections across $f:\mathbb{R}^D \to \mathbb{R}$8 blocks. The update is written as

$f:\mathbb{R}^D \to \mathbb{R}$9

For permutations $p(\mathbf{y}^* \mid \mathbf{x}^*, \mathcal{D})$0 and $p(\mathbf{y}^* \mid \mathbf{x}^*, \mathcal{D})$1, the block satisfies

$p(\mathbf{y}^* \mid \mathbf{x}^*, \mathcal{D})$2

and the resulting noise predictor inherits the corresponding equivariance property (Dutordoir et al., 2022).

Conditional inference is performed with a RePaint-style procedure. Given context $p(\mathbf{y}^* \mid \mathbf{x}^*, \mathcal{D})$3 and target inputs $p(\mathbf{y}^* \mid \mathbf{x}^*, \mathcal{D})$4, the model samples $p(\mathbf{y}^* \mid \mathbf{x}^*, \mathcal{D})$5, diffuses the context forward at each time step,

$p(\mathbf{y}^* \mid \mathbf{x}^*, \mathcal{D})$6

combines target and context into a joint finite set, and denoises. The original report notes that repeating the forward/backward context injection about 5 times improves sample quality (Dutordoir et al., 2022).

The same paper also introduces a joint model that diffuses both $p(\mathbf{y}^* \mid \mathbf{x}^*, \mathcal{D})$7 and $p(\mathbf{y}^* \mid \mathbf{x}^*, \mathcal{D})$8, replacing the output-only objective with

$p(\mathbf{y}^* \mid \mathbf{x}^*, \mathcal{D})$9

with

$\mathbf{s}_0$0

This enables sampling from $\mathbf{s}_0$1 and, in the reported experiments, a global-optimization strategy based on $\mathbf{s}_0$2 (Dutordoir et al., 2022).

The consistency discussion in the original paper is notably cautious. NDPs do not guarantee full Kolmogorov consistency for arbitrary sets in the exact mathematical sense, because the neural approximation $\mathbf{s}_0$3 breaks exact consistency. However, if the noise predictor were perfect, the reverse process would define consistent finite marginals, and in practice the paper reports that variation across different context/target partitions is small (Dutordoir et al., 2022).

4. Empirical profile and original use cases

The initial empirical program for NDPs covers synthetic GP emulation, implicit hyperparameter marginalization, non-Gaussian posterior prediction, image regression, and Bayesian optimization. On synthetic regression tasks, the paper trains on GP-sampled data using squared exponential and Matérn-$\mathbf{s}_0$4 kernels in dimensions $\mathbf{s}_0$5, with $\mathbf{s}_0$6 training paths, context size from $\mathbf{s}_0$7 to $\mathbf{s}_0$8, target size fixed to 50, and evaluation over 128 test samples. The reported outcome is that NDPs are competitive with Gaussian Neural Processes in $\mathbf{s}_0$9, are the most competitive neural baseline in $$q(\mathbf{s}_{0:T}) = q(\mathbf{s}_0)\prod_{t=1}^T q(\mathbf{s}_t \mid \mathbf{s}_{t-1}),$$0, and generally outperform standard NP variants in approximating GP posteriors (Dutordoir et al., 2022).

Implicit hyperparameter marginalization is demonstrated on GP samples with a Matérn-$$q(\mathbf{s}_{0:T}) = q(\mathbf{s}_0)\prod_{t=1}^T q(\mathbf{s}_t \mid \mathbf{s}_{t-1}),$$1 kernel and a log-normal prior over lengthscale,

$$q(\mathbf{s}_{0:T}) = q(\mathbf{s}_0)\prod_{t=1}^T q(\mathbf{s}_t \mid \mathbf{s}_{t-1}),$$2

With few context points, samples reflect the prior over lengthscales; as more data arrive, the implied lengthscale distribution narrows around the true generating value. Non-Gaussian posterior prediction is illustrated with a step-function family having random jump location $$q(\mathbf{s}_{0:T}) = q(\mathbf{s}_0)\prod_{t=1}^T q(\mathbf{s}_t \mid \mathbf{s}_{t-1}),$$3,

$$q(\mathbf{s}_{0:T}) = q(\mathbf{s}_0)\prod_{t=1}^T q(\mathbf{s}_t \mid \mathbf{s}_{t-1}),$$4

The reported result is that NDPs capture a bimodal posterior at $$q(\mathbf{s}_{0:T}) = q(\mathbf{s}_0)\prod_{t=1}^T q(\mathbf{s}_t \mid \mathbf{s}_{t-1}),$$5, whereas GPs cannot represent this because every finite marginal is Gaussian (Dutordoir et al., 2022).

Image regression treats MNIST and CelebA $$q(\mathbf{s}_{0:T}) = q(\mathbf{s}_0)\prod_{t=1}^T q(\mathbf{s}_t \mid \mathbf{s}_{t-1}),$$6 as coordinate-to-pixel regression tasks. In that setting, the architecture is modified by removing the MHSA block over the feature dimension $$q(\mathbf{s}_{0:T}) = q(\mathbf{s}_0)\prod_{t=1}^T q(\mathbf{s}_t \mid \mathbf{s}_{t-1}),$$7, since coordinate order matters in the image setting. The reported qualitative pattern is that NDPs learn coherent image structure from sparse context points and that MSE decreases sharply as context size increases (Dutordoir et al., 2022).

For Bayesian optimization, NDPs are used as surrogate models on Hartmann 3D, Hartmann 6D, Rastrigin 4D, and Ackley 5D. The paper reports that NDP nearly matches Gaussian Process Regression, beats NP, ANP, and random search, and—unlike GPR—does not need retraining inside the BO loop. This use case is closely tied to the joint input-output diffusion model, which makes it possible to sample promising query points by conditioning on a low target value $$q(\mathbf{s}_{0:T}) = q(\mathbf{s}_0)\prod_{t=1}^T q(\mathbf{s}_t \mid \mathbf{s}_{t-1}),$$8 (Dutordoir et al., 2022).

The implementation profile reported in the appendix is also distinctive: 4 or 5 bi-dimensional attention blocks, hidden size $$q(\mathbf{s}_{0:T}) = q(\mathbf{s}_0)\prod_{t=1}^T q(\mathbf{s}_t \mid \mathbf{s}_{t-1}),$$9, 8 heads, typically 500 or 1000 diffusion steps, a cosine schedule for $$q(\mathbf{s}_t \mid \mathbf{s}_{t-1}) = \mathcal{N}\!\left(\mathbf{s}_t;\sqrt{1-\beta_t}\,\mathbf{s}_{t-1},\beta_t\mathbf{I}\right).$$0, Adam, cosine decay with 20-epoch warm-up and max learning rate typically $$q(\mathbf{s}_t \mid \mathbf{s}_{t-1}) = \mathcal{N}\!\left(\mathbf{s}_t;\sqrt{1-\beta_t}\,\mathbf{s}_{t-1},\beta_t\mathbf{I}\right).$$1, 250 epochs for most tasks, 500 epochs for hyperparameter marginalization, and 100 epochs for image regression. The same report notes that the model is input-size agnostic: weights do not depend on $$q(\mathbf{s}_t \mid \mathbf{s}_{t-1}) = \mathcal{N}\!\left(\mathbf{s}_t;\sqrt{1-\beta_t}\,\mathbf{s}_{t-1},\beta_t\mathbf{I}\right).$$2 or $$q(\mathbf{s}_t \mid \mathbf{s}_{t-1}) = \mathcal{N}\!\left(\mathbf{s}_t;\sqrt{1-\beta_t}\,\mathbf{s}_{t-1},\beta_t\mathbf{I}\right).$$3, so the same trained model can handle different dataset sizes and dimensions (Dutordoir et al., 2022).

5. Major extensions and reformulations

Several later works retain the core NDP viewpoint—diffusion models over stochastic processes or finite marginals of functions—while altering the noising process, symmetry assumptions, or task structure.

Framework	Modification relative to NDPs	Reported emphasis
NDP	Diffusion over finite marginals of function values	Exchangeability and input-dimension invariance
GeomNDP	Kernelized noising toward a geometric Gaussian process and equivariant score network	Scalar fields, vector fields, spherical codomains
MT-NDP	Task encoder $$q(\mathbf{s}_t \mid \mathbf{s}_{t-1}) = \mathcal{N}\!\left(\mathbf{s}_t;\sqrt{1-\beta_t}\,\mathbf{s}_{t-1},\beta_t\mathbf{I}\right).$$4 with $$q(\mathbf{s}_t \mid \mathbf{s}_{t-1}) = \mathcal{N}\!\left(\mathbf{s}_t;\sqrt{1-\beta_t}\,\mathbf{s}_{t-1},\beta_t\mathbf{I}\right).$$5	Cross-turbine correlations and few-shot adaptation
NBP	Input-dependent bridge kernel	Stronger input coupling and endpoint coherence
DANP	Diffusion-style noising over target fidelities	Conceptual relative, not the same model class

Geometric Neural Diffusion Processes extend NDPs to settings with symmetry and non-Euclidean codomains. Their forward finite-marginal noising process is a kernelized multivariate Ornstein–Uhlenbeck SDE,

$$q(\mathbf{s}_t \mid \mathbf{s}_{t-1}) = \mathcal{N}\!\left(\mathbf{s}_t;\sqrt{1-\beta_t}\,\mathbf{s}_{t-1},\beta_t\mathbf{I}\right).$$6

with limiting process $$q(\mathbf{s}_t \mid \mathbf{s}_{t-1}) = \mathcal{N}\!\left(\mathbf{s}_t;\sqrt{1-\beta_t}\,\mathbf{s}_{t-1},\beta_t\mathbf{I}\right).$$7. Equivariance is enforced both in the geometric Gaussian process prior,

$$q(\mathbf{s}_t \mid \mathbf{s}_{t-1}) = \mathcal{N}\!\left(\mathbf{s}_t;\sqrt{1-\beta_t}\,\mathbf{s}_{t-1},\beta_t\mathbf{I}\right).$$8

and in the score network,

$$q(\mathbf{s}_t \mid \mathbf{s}_{t-1}) = \mathcal{N}\!\left(\mathbf{s}_t;\sqrt{1-\beta_t}\,\mathbf{s}_{t-1},\beta_t\mathbf{I}\right).$$9

The reported applications include scalar fields, vector fields, and cyclone trajectories on $\alpha_t = 1-\beta_t$0, together with a Langevin-based conditional sampler (Mathieu et al., 2023).

A different line of work argues that standard NDPs suffer from weak input coupling because the forward process is unconditional and the conditioning signal enters only through the denoising network. Neural Bridge Processes replace the standard DDPM kernel

$\alpha_t = 1-\beta_t$1

with the input-dependent bridge kernel

$\alpha_t = 1-\beta_t$2

and correspondingly

$\alpha_t = 1-\beta_t$3

The paper states that this gives stronger gradient signals, guarantees endpoint coherence, and improves performance on synthetic regression, EEG signal regression, and image regression tasks. It also summarizes the comparison as follows: for NDP, endpoint match is implicit and path consistency is weak; for NBP, endpoint match and path consistency are both marked as present (Xu et al., 10 Aug 2025).

The multi-task extension for wind-power forecasting keeps the NDP backbone but introduces a task encoder that maps context information to a task embedding $\alpha_t = 1-\beta_t$4, with $\alpha_t = 1-\beta_t$5 in the reported experiments. The context embeddings are averaged into $\alpha_t = 1-\beta_t$6, replicated across target inputs, and concatenated to the original covariates. On SCADA data from six Senvion MM92 turbines at Kelmarsh wind farm, spanning January 2016 to July 2021 at 10-minute resolution with over 1.7 million records and 110 variables, the paper reports that NDP-based models provide better calibration than a GP baseline and that MT-NDP improves both point accuracy and calibration for a more distinctive target turbine. On turbine 1, GP coverage error is reported as $\alpha_t = 1-\beta_t$7 versus $\alpha_t = 1-\beta_t$8 for a 1D single-task NDP; on turbine 6 with 50 context points, 5D MT-NDP gives MAE $\alpha_t = 1-\beta_t$9, RMSE $\bar{\alpha}_t=\prod_{j=1}^t \alpha_j$0, and CE $\bar{\alpha}_t=\prod_{j=1}^t \alpha_j$1, compared with 5D ST-NDP at MAE $\bar{\alpha}_t=\prod_{j=1}^t \alpha_j$2, RMSE $\bar{\alpha}_t=\prod_{j=1}^t \alpha_j$3, and CE $\bar{\alpha}_t=\prod_{j=1}^t \alpha_j$4 (Rawson et al., 3 Oct 2025).

Diffusion-Augmented Neural Processes occupy a nearby but explicitly non-identical position. That work describes DANPs as not the same as NDPs: NDPs are generative diffusion models whose denoising process is parameterized by a Neural Process, whereas DANPs are Neural Process models for direct conditional prediction using a diffusion-style auxiliary noising scheme over target fidelities. The relationship is therefore conceptual rather than architectural (Bonito et al., 2023).

6. Terminological boundaries, misconceptions, and adjacent literatures

The phrase “neural diffusion” is used across several research strands that are not Neural Diffusion Processes in the function-distribution sense. This terminological overlap is a recurrent source of confusion.

“Neural Network Diffusion” generates high-performing neural network parameters by training an autoencoder on subsets of trained weights and then learning a latent diffusion model over the resulting codes. The reported pipeline is explicitly parameter generation rather than stochastic-process regression, and the method usually synthesizes only selected normalization parameters rather than all weights (Wang et al., 2024). “HyperDiffusion” likewise operates in parameter space, training a diffusion model directly on flattened MLP weights that encode per-instance implicit neural fields; its target is unconditional generative modeling of 3D shapes and 4D mesh animations through synthesized neural-field parameters (Erkoç et al., 2023). These methods are diffusion models over neural representations, but not NDPs as introduced in (Dutordoir et al., 2022).

Other papers use “diffusion” in still different senses. “Dynamic Tensor Decomposition via Neural Diffusion-Reaction Processes” proposes a continuous-time latent embedding model whose core dynamics are

$\bar{\alpha}_t=\prod_{j=1}^t \alpha_j$5

combining graph diffusion with neural reaction terms for dynamic tensor decomposition (Wang et al., 2023). “Infinitely deep neural networks as diffusion processes” studies a depth limit of residual networks and derives convergence in law to an SDE under $\bar{\alpha}_t=\prod_{j=1}^t \alpha_j$6 drift and $\bar{\alpha}_t=\prod_{j=1}^t \alpha_j$7 random increments, establishing a link between infinitely deep residual networks and diffusion processes rather than introducing a diffusion model over function samples (Peluchetti et al., 2019). “Neural McKean-Vlasov Processes” are described as a closely related alternative framework and partial precursor, with explicit law dependence $\bar{\alpha}_t=\prod_{j=1}^t \alpha_j$8 in the drift of an MV-SDE; they are adjacent to NDPs but specialized to interacting-particle and mean-field settings (Yang et al., 2024). An earlier physics-informed line represents the evolving pdf or characteristic function of diffusion processes with neural networks constrained by Fokker–Planck or characteristic-function equations, and is explicitly described as not an NDP in the modern generative-model sense (Uy et al., 2020).

The original NDP literature also records substantive limitations. More diffusion steps improve sample quality but slow inference; performance degrades when test inputs are outside the training range; full Kolmogorov consistency is not guaranteed in the general mathematical sense; and, in later critiques, the default unconditional forward process is said to produce weak input coupling and semantic mismatch at the diffusion endpoint (Dutordoir et al., 2022). The subsequent appearance of bridge-conditioned, geometric, and multi-task variants suggests that the principal active directions in the area concern stronger conditioning, stricter inductive structure, and better calibrated posterior sampling rather than abandonment of the finite-marginal diffusion paradigm itself (Xu et al., 10 Aug 2025).