---
title: Diffusion Contrastive Divergence in EBMs
url: https://www.emergentmind.com/topics/diffusion-contrastive-divergence-dcd
type: topic
---

# Diffusion Contrastive Divergence in EBMs

Diffusion Contrastive Divergence (DCD) is a family of training objectives for energy-based models (EBMs) in which the model-dependent Langevin dynamics of classical Contrastive Divergence (CD) are replaced by a predefined, EBM-parameter-free diffusion process. In its canonical formulation, DCD measures the amount by which a shared diffusion contracts the Kullback–Leibler divergence between the data distribution and the model distribution, thereby reinterpreting CD as one member of a broader diffusion-induced divergence family rather than as a procedure tied specifically to short-run MCMC [2307.01668]. The term has since coexisted with several adjacent but technically distinct lines of work—contrastive objectives for denoisers, joint EBM–diffusion training, and diffusion-based representation learning—so the literature surrounding “DCD” requires careful disambiguation [2312.03397].

## 1. Classical contrastive divergence and the motivation for DCD

Classical CD trains an EBM by replacing the exact negative phase of maximum-likelihood learning with samples produced by a short Markov chain initialized at the data. For an EBM written as
$$
p_\theta(x) = \frac{\exp(f_\theta(x))}{Z_\theta},
$$
the maximum-likelihood gradient has the familiar positive-phase minus negative-phase form,
$$
\frac{\partial}{\partial \theta}\mathbb{E}_{p_d}\log \frac{\exp(f_\theta(x))}{Z_\theta}
=
\mathbb{E}_{p_d}\frac{\partial}{\partial \theta} f_\theta(x)
-
\mathbb{E}_{p_\theta}\frac{\partial}{\partial \theta}f_\theta(x).
$$
In standard CD, the model expectation is approximated by a short MCMC chain, typically Langevin dynamics,
$$
d x_t = \frac{1}{2}\nabla_{x_t}\log p_\theta(x_t) d t + d \bm{w}_t,
$$
and the objective is written as
$$
\mathcal{D}_{CD}(p_d,p_\theta)
=
\mathcal{D}_{KL}(p_d, p_\theta)
-
\mathcal{D}_{KL}(p_{d,\theta}^{(T)}, p_\theta).
$$
The central difficulty is that the short-run distribution $p_{d,\theta}^{(T)}$ depends on $\theta$, so differentiating CD introduces an additional term,
$$
\frac{\partial}{\partial \theta}\mathcal{D}_{CD}(p_d,p_\theta)
=
\mathbb{E}_{ p_{d,\theta}^{(T)}}\bigg[\frac{\partial}{\partial \theta}f_\theta(x) \bigg]
-\mathbb{E}_{ p_d}\bigg[\frac{\partial}{\partial \theta} f_\theta(x)\bigg]
-\mathbb{E}_{p_{d,\theta}^{(T)}}\bigg[\log p_\theta(x) \frac{\partial}{\partial \theta}\log p_{d,\theta}^{(T)}(x) \bigg],
$$
and that final term is non-negligible in general [2307.01668].

A complementary line of analysis showed that CD is not best understood as approximate descent on a fixed KL-difference objective. Instead, when the transition kernel is reversible and satisfies detailed balance with respect to $p_\theta$, the usual CD update is exactly proportional to the gradient of a binary cross-entropy objective that classifies whether a Markov trajectory is presented in forward order or time-reversed. That derivation also yields a correction weight for inexact chains such as finite-step Langevin dynamics [2012.03295]. This perspective does not define DCD, but it clarifies why a reformulation of CD in diffusion-theoretic terms became attractive: the classical short-run MCMC negative phase is both computationally burdensome and theoretically awkward.

## 2. Formal definition of Diffusion Contrastive Divergence

DCD starts from a general diffusion process
$$
d x_t = \bm{F}(x_t,t)d t + \bm{G}(t)d \bm{w}_t,
$$
whose marginal density evolves by the Fokker–Planck equation
$$
\frac{d}{d t}p(x,t) = - \langle \nabla_{x}, p(x,t)\bm{F}(x,t)\rangle + \frac{1}{2}\bm{G}^2(t)\Delta_{x} p(x,t),
\qquad p(x,0)=p_0(x).
$$
The decisive restriction is that the diffusion be EBM-parameter-free: $\bm{F}$ and $\bm{G}$ do not depend on $\theta$. Under that condition, the diffused data distribution and the diffused model distribution are transported by the same known process, and the problematic model-dependent MCMC derivative of classical CD is avoided [2307.01668].

The core DCD divergence is
$$
\mathcal{D}_{DCD}^{(\bm{F},\bm{G},T)}(p_d, p_\theta)
:=
\mathcal{D}_{KL}(p_d, p_\theta)
-
\mathcal{D}_{KL}(p_d^{(T)}, p_\theta^{(T)}),
$$
where $p_d^{(T)}$ and $p_\theta^{(T)}$ are the time-$T$ marginals obtained by diffusing the data distribution and model distribution under the same process. The main theorem gives the integral representation
$$
\mathcal{D}_{DCD}^{(\bm{F},\bm{G},T)}(p,q)
=
\frac{1}{2}\int_{0}^T
\mathbb{E}_{x_t \sim p^{(\bm{F},\bm{G},t)}(x)}
\bm{G}^2(t)\|\nabla_{x_t} \log p^{(\bm{F},\bm{G},t)}(x_t) - \nabla_{x_t} \log q^{(\bm{F},\bm{G},t)}(x_t)\|^2_2 d t.
$$
This immediately implies nonnegativity,
$$
\mathcal{D}_{DCD}^{(\bm{F},\bm{G},T)}(p,q)\ge 0.
$$
Under the assumptions stated in the paper, DCD is therefore a proper probability divergence, and it can be interpreted as the accumulated discrepancy between diffused scores along the diffusion path [2307.01668].

Within this framework, classical CD appears as a special case. If the diffusion is chosen to be the EBM-induced Langevin dynamics,
$$
\bm{F}(x,t)= \nabla_x f_\theta(x)/2,\qquad \bm{G}(t)=\mathbf{I},
$$
then $p_\theta$ is stationary, so $p_\theta^{(t)}=p_\theta$, and DCD reduces to CD. This is the sense in which DCD generalizes CD rather than merely replacing one sampler with another [2307.01668].

## 3. Practical instantiation: variance-exploding diffusion and the one-step objective

The practical version developed in the original DCD paper is variance-exploding diffusion,
$$
d x_t = g(t)d \bm{w}_t.
$$
This process has no drift and simply adds Gaussian noise. The paper writes the transition kernel as
$$
p(x_t|x_0) = \mathcal{N}(x_0, \sigma(t)\mathbf{I}),
\qquad
\sigma(t)=\int_0^t g(s)ds.
$$
Because the transition is explicit and parameter-free, drawing $x_t$ from data is inexpensive: sample $x_0\sim p_d$ and perturb it with Gaussian noise [2307.01668].

For parameter-free diffusions, DCD minimization is equivalent to minimizing
$$
\mathcal{L}_{DCD}(\theta)
=
\mathbb{E}_{x_0\sim p_d, x_t\sim p(x_t|x_0)}\big[f_\theta^{(\bm{F},\bm{G},T)}(x_t)\big]
-
\mathbb{E}_{x_0\sim p_d}\big[f_\theta(x_0) \big].
$$
The difficulty shifts from MCMC sampling to the evaluation of the evolved model energy $f_\theta^{(\bm{F},\bm{G},T)}$. In general that evolution is governed by a PDE. For VE diffusion, the paper derives
$$
d f_\theta^{(t)}(x)/d t
=
\frac{1}{2}\bm{G}^2(t)\big[ \|\nabla_{x} f_\theta^{(t)}(x)\|^2 + \Delta_{x} f_\theta^{(t)}(x) \big].
$$
Rather than solving this long-time evolution exactly, the practical algorithm uses a small-$t$, one-step approximation, yielding
$$
\mathcal{L}_{DCD}^{(VE)}(\theta)
=
\mathbb{E}_{p_t}\frac{1}{2}\bm{G}^2(0)\bigg[ \|\nabla_{x} f_\theta(x_t)\|^2 + \Delta f_\theta(x_t) \bigg]
+\frac{1}{t} \bigg[ \mathbb{E}_{p_t}[f_\theta(x_t)] - \mathbb{E}_{p_d}[f_\theta(x_0)] \bigg].
$$
The first bracketed term has the same structure as score matching, while the second term is a contrastive energy difference between clean and noised data. The paper explicitly connects DCD to Diffusion Recovery Likelihood, identifying DRL as a special case of the same KL-contraction form under Gaussian perturbation [2307.01668].

A time-dependent variant is also described, in which one trains a time-dependent energy model $f_\theta(x,t)$ over diffused data distributions by sampling $t \sim \text{Unif}[0,T]$, diffusing data to $x_t$, computing the relevant DCD objective, and updating $\theta$ by gradient descent [2307.01668].

## 4. Computational characteristics, implementation, and algorithmic trade-offs

DCD changes the computational bottleneck of EBM training. Classical CD requires sequential, model-dependent MCMC, and its quality depends on chain length and mixing. DCD-VE eliminates EBM-driven MCMC and replaces it with direct Gaussian perturbation, but it requires higher-order differentiation through the energy network. In particular, the practical loss depends on $\nabla_x f_\theta(x)$ and the Laplacian $\Delta f_\theta(x)$ [2307.01668].

In low dimensions, the Laplacian can be computed directly. In high dimensions, the paper uses Hutchinson trace estimation,
$$
\Delta_x f_\theta(x)=\operatorname{tr}(\nabla_x s_\theta(x))
\approx
\epsilon^\top \nabla_x s_\theta(x)\epsilon,
\quad
\epsilon\sim p_\epsilon,\ \mathbb E[\epsilon\epsilon^\top]=I,
$$
where $s_\theta(x)=\nabla_x f_\theta(x)$. The resulting trade-off is explicit in the paper: DCD is often faster in wall-clock time than CD, but it is more memory-intensive because of second-order derivative computation [2307.01668].

The original experiments instantiate this trade-off in three regimes. In 2D synthetic density modeling, DCD-VE uses a 4-layer MLP with 300 hidden units per layer, GELU activations, $t=0.0005$, $G(0)^2=1$, Adam with learning rate $10^{-3}$, batch size $1000$, and $5000$ iterations. The CD baseline uses Langevin step size $0.001$ and $10$ Langevin steps; the PCD baseline uses replay buffer size $10000$, step size $0.001$, $20$ MCMC steps, and replay update frequency $5\%$ [2307.01668].

For image denoising, the paper uses Wide ResNet architectures with SiLU and no normalization. MNIST and FashionMNIST use depth $16$ and widen factor $8$; CIFAR10 and SVHN use depth $28$ and widen factor $10$. Inputs are scaled to $[-1,1]$, training data are additionally corrupted with Gaussian noise $\sigma=0.3$, and DCD-VE uses $t=0.018$, $G(0)^2=1$, Adam, learning rate $2\times 10^{-4}$, and $\beta=(0.9,0.99)$ [2307.01668].

For image generation on CelebA $32\times 32$, the paper trains a time-dependent EBM with a residual/UNet-style architecture from an EDM VP backbone plus an extra SiLU before the last pooling layer. The forward process uses VE diffusion with $g(t)=t$, training samples time as $t\sim \operatorname{LogNormal}(-1.2,1.2)$, and reverse sampling uses a Heun solver with 18 discretized noise levels from $0.01$ to $80.0$ [2307.01668].

## 5. Empirical results, performance profile, and limitations

The empirical results reported for DCD divide naturally into synthetic density learning, denoising, and image generation. On seven 2D datasets—Swissroll, Circles, Rings, Moons, 8 Gaussians, 2 Spirals, and Checkerboard—the evaluation metric is the score matching loss
$$
\operatorname{L}(\theta)
\coloneqq
\mathbb{E}_{x \sim p_d} \bigg[ \frac{1}{2}\|\nabla_{x} f_\theta(x)\|_2^2 + \Delta_{x} f_\theta(x) \bigg].
$$
DCD-VE outperforms CD and PCD on all seven datasets. The reported values include $-2398.81$ on Swissroll for DCD-VE versus $+\infty$ for both CD and PCD, $-758.33$ on Rings versus $+\infty$ for both baselines, and $-178.43$ on Checkerboard versus $-67.22$ for CD and $-124.27$ for PCD [2307.01668].

In image denoising, the metric is average RMSE after adding Gaussian noise at levels $0.3$, $0.6$, and $0.9$. DCD consistently outperforms CD across MNIST, FashionMNIST, CIFAR10, and SVHN. The gap becomes especially large at high noise: on CIFAR10, DCD reports $0.244$ at high noise versus $8.572$ for CD; on SVHN, the corresponding values are $0.294$ versus $6.938$ [2307.01668]. This suggests that the learned energy landscape remains useful farther from the clean data manifold, although that inference concerns interpretation rather than a theorem.

In image generation on CelebA $32\times 32$, EBM-DCD reports FID $13.85$. The paper places that result as better than ABP at $51.50$, ABP-SRI at $36.84$, VAE at $38.76$, and Glow at $23.32$, comparable to DCGAN at $12.50$ and EBM-FCE at $12.21$, and worse than GEBM at $5.21$ and CoopFlow(T=30) at $6.44$ [2307.01668]. The paper therefore presents DCD generation as viable rather than leading.

Two limitations are explicit. First, DCD requires higher-order derivatives, so the energy model must be at least twice differentiable. Second, exact long-time energy evolution is difficult, and the practical method relies on a one-step approximation rather than a full consistency result for long diffusion times [2307.01668]. A plausible implication is that DCD is especially attractive when MCMC instability dominates and second-order autodiff is feasible.

## 6. Related formulations, neighboring methods, and common confusions

The phrase “Diffusion Contrastive Divergence” now sits beside several technically different frameworks. The following distinctions are essential.

| Method | Core mechanism | Relation to DCD |
|---|---|---|
| **DCD** [2307.01668] | KL contraction under parameter-free diffusion for EBM training | Canonical use of the term |
| **GCD** [2312.03397] | Joint minimax training of an EBM and a trainable sampler, including diffusion samplers | Generalizes CD by replacing MCMC with a learned sampler |
| **CDL** [2407.08946] | Logistic classification between clean and more heavily noised marginals | Adjacent contrastive diffusion training, not classical CD |
| **DCR** [2603.04803] | InfoNCE-style loss on predicted noises in diffusion reconstruction | Contrastive regularization, not contrastive divergence |
| **DCG** [2503.09185] | DDPM-style latent diffusion plus contrastive alignment for incomplete multi-view clustering | Diffusion with contrastive losses, not CD for EBMs |
| **DCDM** [2605.15676] | Dynamic semantic chunking for discrete diffusion language modeling | Acronym overlap only |

Generalized Contrastive Divergence (GCD) is the nearest extension in spirit. It replaces the MCMC negative phase with an arbitrary trainable sampler $\pi$, and when $\pi$ is instantiated as a diffusion model the joint training becomes
$$
\min_{\theta} \max_{\phi} \mathcal{L}(\theta,\phi),
\qquad
\mathcal{L}
=
\mathbb{E}_{p}[E_\theta(x)]
-
\mathbb{E}_{\pi_\phi}[E_\theta(x)]
+
\tau \mathcal{H}(\pi_\phi).
$$
The paper formulates this joint learning as a minimax problem and interprets it through inverse reinforcement learning, where the energy is a negative reward and the diffusion model is a policy [2312.03397]. Relative to DCD, the difference is structural: DCD uses a predefined diffusion and optimizes a divergence on the EBM, whereas GCD learns the sampler itself.

A second adjacent line reinterprets diffusion denoisers as implicit classifiers. “Your Diffusion Model is Secretly a Noise Classifier and Benefits from Contrastive Training” introduces Contrastive Diffusion Loss (CDL), a logistic objective
$$
\mathcal L_{\mathrm{CDL}}
=
\mathbb E_{q(x,y)}
\left[
\operatorname{softplus}\!\left( y\bigl(\log p_\zeta(x)-\log p(x)\bigr) \right)
\right],
$$
which discriminates between the clean data distribution $p(x)$ and a noisier marginal $p_\zeta(x)$. The paper frames CDL as a self-supervised binary classification or density-ratio objective between two points on the diffusion path, and reports improvements in OOD denoising and especially in parallel sampling, but it does not define an energy-based diffusion model or a contrastive-divergence objective in the classical sense [2407.08946].

Other recent works use “contrastive” in representation-learning rather than EBM-training senses. DCR places an InfoNCE-style objective directly in diffusion predicted-noise space to enhance CLIP representations, explicitly arguing that a naive weighted sum of contrastive and reconstruction losses suffers from gradient conflict, with $86.3\%$ of training steps having negative gradient cosine similarity [2603.04803]. DCG combines DDPM-style latent diffusion, generated-view contrastive loss, mutual-information alignment, and KL-based clustering objectives for incomplete multi-view clustering, but it contains no energy function, no partition function, and no CD-style positive-versus-negative phase [2503.09185].

A recurrent source of confusion is acronymic rather than conceptual. “Dynamic Chunking for Diffusion Language Models” introduces DCDM, the Dynamic Chunking Diffusion Model, for discrete diffusion language modeling. That work explicitly states that it is not about “Diffusion Contrastive Divergence (DCD)” in the usual sense; its contribution is dynamic semantic chunking and chunk-causal attention in block diffusion language models, not a contrastive-divergence training objective [2605.15676].

Taken together, the literature supports a narrow and a broad usage. In the narrow sense, Diffusion Contrastive Divergence denotes the KL-contraction family for EBM training introduced in [2307.01668]. In the broader sense, it has become a reference point for several diffusion methods that use contrastive, classifier-based, or adversarial signals. Maintaining that distinction is necessary for precise reading of current arXiv work.

Source: https://www.emergentmind.com/topics/diffusion-contrastive-divergence-dcd