Energy-Based Diffusion Models Overview

• Energy-Based Diffusion Models (EBDMs) are generative models that integrate energy-based modeling with diffusion processes, enabling structured density estimation and enhanced controllability.
• They employ training strategies such as diffusion recovery likelihood, contrastive divergence, and cooperative training to improve sampling efficiency and maintain theoretical guarantees.
• EBDMs have shown high performance in image synthesis, text modeling, robotics, and molecular simulation, offering practical benefits like robust out-of-distribution detection.

Energy-Based Diffusion Models (EBDMs) are a class of generative models that combine the probabilistic structure and flexibility of energy-based models (EBMs) with the multi-scale, tractable learning and sampling properties of diffusion models. EBDMs leverage parameterizations wherein the score function in diffusion is realized as the gradient of an explicit energy function, often time-dependent, enabling a conservative vector field and enabling new forms of compositionality, controllability, and likelihood estimation. The energy-based perspective unifies advances in score-based generative modeling, contrastive divergence, and inverse reinforcement learning, facilitating a spectrum of algorithms for density modeling, sampling, and downstream applications.

1. Mathematical Foundations and Core Parameterizations

Energy-Based Diffusion Models instantiate a continuum of energy-based models indexed by diffusion (noise) time, with each step of the diffusion (or reverse denoising) process linked to a time-indexed energy $E_\theta(x,t)$ such that the model density at time $t$ is

$$p_\theta(x,t) \propto \exp\left(-E_\theta(x,t)\right)$$

with $\theta$ denoting model parameters and $t$ the diffusion index (Aarts et al., 1 Oct 2025, Thornton et al., 18 Feb 2025, Diamond et al., 21 Nov 2025). The reverse-time drift in the corresponding SDE is given by $-\nabla_x E_\theta(x,t)$, i.e., the negative gradient of energy, ensuring a conservative flow (Diamond et al., 21 Nov 2025). This setup generalizes both classical MCMC-based EBMs and score-based diffusion models, with $E_\theta(x,t)$ either directly parameterized (for explicit energy models) or obtained via distillation from pre-trained score networks (Thornton et al., 18 Feb 2025).

Training objectives are typically variants of denoising score matching (DSM), regularized likelihood, or contrastive divergence, often adapted for the diffusion setting by introducing time-dependent energy functions and leveraging tractable conditional distributions in the diffusion ladder (Aarts et al., 1 Oct 2025, Gao et al., 2020, Plainer et al., 20 Jun 2025, Luo et al., 2023). In particular, the diffusion recovery likelihood objective leverages conditional EBMs at each diffusion level, enabling efficient short-run MCMC sampling from narrower, locally unimodal densities (Gao et al., 2020, Zhu et al., 2023).

2. Learning Algorithms: Recovery Likelihood, Contrastive Divergence, and Beyond

EBDMs encompass a range of training paradigms including recovery likelihood maximization, contrastive divergence (CD) and its generalizations, cooperative training with amortized initializers, and adversarial and inverse RL-inspired minimax formulations.

• Diffusion Recovery Likelihood (DRL): Each conditional EBM $p_\theta(x_{t-1}|x_t)$ is trained to maximize the likelihood of recovering a less-noised sample from a more-noised one. The gradient is computed as a two-term expectation—one over the data joint $(x_{t-1}, x_t)$, and one over model samples from $p_\theta(x_{t-1}|x_t)$—with the latter approximated by short-run Langevin dynamics (Gao et al., 2020, Zhu et al., 2023).
• Diffusion Contrastive Divergence (DCD): DCD generalizes CD by evolving both data and model distributions under a parameter-free diffusion process, yielding an unbiased, computationally efficient objective and enabling MCMC-free EBM learning in high dimension (Luo et al., 2023).
• Ranking NCE and Joint Sampler Learning: Multi-class ranking NCE enables consistent EBM training by pairing a learnable negative sampler with the energy network, jointly optimizing the sampler and the EBM for minimal divergence between sampled and true data distributions (Singh et al., 2023).
• Amortized Cooperative Training: Learnable proposal networks ("initializers") are trained to mimic refined EBM samples, reducing the burden of expensive MCMC per training batch (Zhu et al., 2023). This cooperative scheme is particularly effective in the diffusion setting, allowing a sharp reduction in MCMC steps while preserving generation and likelihood performance.
• Adversarial Minimax/Inverse RL Objectives: Recent works formulate joint EBM-diffusion training as a minimax problem, where the EBM plays the role of a discriminator or reward function and the diffusion model acts as a trainable sampler or policy (Yoon et al., 2023, Yoon et al., 2024, Geng et al., 2024). The joint optimization aligns the sampler's marginal with the data and the learned energy to the negative log-density, stabilized via entropy regularization.

Algorithmic Paradigm	Key Feature	MCMC-Free?	Example EBDM Papers
Recovery Likelihood	Timewise conditional EBM	No	(Gao et al., 2020, Zhu et al., 2023)
Diffusion CD / DCD	Diffusion-based divergence	Yes	(Luo et al., 2023)
Amortized Cooperative	Learnable initializer + EBM	Effective	(Zhu et al., 2023)
Minimax/IRL	Joint EBM & sampler	Yes	(Yoon et al., 2023, Yoon et al., 2024)

3. Hierarchical, Latent, and Structured EBDMs

EBDMs are not limited to data space, but are effectively deployed as priors and generators in latent-variable and hierarchical models (Cui et al., 2024, Yu et al., 2022, Yu et al., 2023, Zhang et al., 2024). Hierarchical EBM priors in deep latent variable models bridge the "prior hole" between generic Gaussian priors and expressive posteriors. Diffusion in a transformed, uni-scale latent space enables learning highly multi-modal latent priors, with tractable sampling per diffusion level reducing the MCMC burden. Latent EBDMs have demonstrated strong results in image generation (CIFAR-10, CelebA-HQ) and interpretable text modeling, outperforming vanilla latent-EBMs and variational baselines, especially when coupled with geometric clustering and information bottleneck regularization (Cui et al., 2024, Yu et al., 2022).

EnergyMoGen (Zhang et al., 2024) demonstrates advanced compositionality in latent EBDMs for text-to-motion synthesis by combining latent-aware and semantic-aware energy terms and enabling logical operations (e.g., conjunction, negation) via gradient composition and energy fusion.

4. Advanced Sampling, Control, Composition, and Physical Modeling

Energy parameterization of diffusion models unlocks principled sampling and controllability advances beyond conventional score-based pipelines:

• Sequential Monte Carlo & Feynman–Kac Control: Distilled EBDMs enable SMC via Feynman–Kac measures using the explicit energy as a time-indexed potential, supporting composition (AND-type logical combination), temperature control, and guidance with precise control over distributional support (Thornton et al., 18 Feb 2025).
• Metropolis-Corrected and Analytical Samplers: MCMC-corrected samplers utilizing the energy at each diffusion step, as well as analytical proposals based on the Boltzmann form, restore stationarity and sample efficiency while supporting nontrivial operators (composition, rare event targeting) (Diamond et al., 21 Nov 2025, Thornton et al., 18 Feb 2025).
• Physical Simulation and Molecular Dynamics: Energy-based diffusion samplers can be recast as discretized Langevin integrators for overdamped (and even underdamped) dynamics, with the learned energy defining the force field and the step size controlling the temperature (Diamond et al., 21 Nov 2025, Plainer et al., 20 Jun 2025). Fokker–Planck regularization aligns the energy gradient and marginal density evolution, stabilizing simulation and generation even under rapid sampling regimes.
• Object-Conditioned and Attention-Driven EBDMs: In text-to-image and multi-modal settings, EBDMs can be injected into cross-attention map dynamics for attribute binding, object preservation, and improved prompt adherence via max-MLE and regularization terms defined in the attention space (Zhang et al., 2024).

5. Theoretical Guarantees, Consistency, and Model Comparison

The energy-based diffusion framework inherits and extends several theoretical guarantees:

• Consistency and Asymptotic Efficiency: Properly designed R-NCE, minimax, and DCD-based training schemes yield estimators that are consistent for the true density in the infinite data limit, with provable variance close to the Fisher bound when the negative sampler approaches optimality (Singh et al., 2023, Luo et al., 2023, Yoon et al., 2023).
• Error Control: Explicit separation of path-space error for generated trajectories into discretization and score/model error is possible, with arbitrarily high accuracy in the limit of many diffusion steps and increased model capacity (Diamond et al., 21 Nov 2025, Plainer et al., 20 Jun 2025).
• Superior Out-of-Distribution Detection: EBMs trained via diffusion (joint or recovery likelihood) maintain highly faithful density estimation and achieve state-of-the-art AUROC on OOD benchmarks, overcoming the “OOD reversal” problem seen in flow-based and vanilla diffusion models (Zhang et al., 2023, Zhu et al., 2023, Yoon et al., 2024).
• Comparative Sample Quality: Cooperative and adversarial EBDMs attain FID and IS competitive with, or surpassing, score-based and flow-based models on CIFAR-10, CelebA, and ImageNet, particularly when leveraging amortized proposals and SMC-based sampling (Zhu et al., 2023, Cui et al., 2024, Geng et al., 2024, Thornton et al., 18 Feb 2025).

6. Applications and Extensions Across Modalities

EBDMs have demonstrated their flexibility and impact across diverse domains:

• High-Dimensional Image and Video Synthesis: State-of-the-art image generation, inpainting, and compositional synthesis via hierarchical, cooperative, and compositional EBM architectures (Zhu et al., 2023, Cui et al., 2024, Zhang et al., 2024).
• Interpretable and Structured Text Models: Latent EBDMs with interpretable clustering and bottleneck regularizations for text representation and disentangled generation (Yu et al., 2022).
• Robotics and Policy Learning: EBDM policy classes optimized via ranking-NCE compete with and outperform diffusion-based policies in complex robotic control and path planning tasks (Singh et al., 2023).
• Molecular Simulation and Physical Modeling: Data-driven MD simulation and consistent Boltzmann/distributional emulation via EBDMs, with applications in biomolecular conformational sampling and coarse-grained trajectory generation (Diamond et al., 21 Nov 2025, Plainer et al., 20 Jun 2025).
• Speech Synthesis: Non-autoregressive speech synthesis models realized as energy-based diffusions provide explicit density evaluation and competitive MOS and objective performance (Sun et al., 2023).

7. Challenges, Limitations, and Future Directions

Despite their strengths, EBDMs face notable challenges:

• Computational Cost: Energy parametrization typically incurs higher computational overhead than direct score modeling, especially when higher-order derivatives (e.g., Laplacians in DCD) are required (Luo et al., 2023, Aarts et al., 1 Oct 2025).
• Stability and Mode Coverage: Careful design of training objectives, entropy regularization, and cooperative learning is necessary to avoid mode collapse and instability, especially in minimax and MCMC-free regimes (Geng et al., 2024, Yoon et al., 2023, Yoon et al., 2024).
• Sampling Efficiency: Amortized and variational methods (e.g., initializers, dynamic programming-based policies) are critical for practical scalability, and further improvements may be needed for very high-dimensional domains (Yu et al., 2023, Yoon et al., 2024).
• Extension to Complex and Non-Euclidean Domains: Early evidence supports EBDM learning in physical simulation and complex Langevin scenarios, yet efficient extension to non-Euclidean, graph-based, or symbolic data spaces remains an open topic (Aarts et al., 1 Oct 2025, Diamond et al., 21 Nov 2025).
• Integration with Large Pretrained Models: Combining EBDMs with foundation models (e.g., LLMs, large vision models, pretrained generative networks) and leveraging cross-modal energies is an active direction with substantial promise (Zhang et al., 2024, Cui et al., 2024).

Broadly, EBDMs are a rapidly maturing methodological nexus unifying generative modeling, structured density estimation, contrastive learning, and probabilistic simulation. Their explicit, learnable energy parameterization affords principled control over generation, robust density-based anomaly detection, and stable simulation in both data and latent spaces. Continued innovations are expected along algorithmic efficiency, plug-and-play compositionality, scalable inference, and transfer to new scientific and multi-agent systems domains.