Energy-Based Model Structure

• Energy-Based Model (EBM) Structure is defined by an energy function that assigns values to configurations, lowering probabilities for higher energies.
• The structure leverages parametrizations such as feature maps and deep neural networks to flexibly model complex data distributions.
• Advanced sampling and learning methods, including Langevin dynamics and MCMC, enable effective inference and diverse applications across domains.

An energy-based model (EBM) specifies a probability distribution over a domain by assigning to each configuration an energy value via an energy functional. The unnormalized probability of a configuration decreases monotonically with its assigned energy, typically through an exponential or general monotonic map. The resulting probabilistic model is "unnormalized"—the normalization constant (partition function) is defined as an integral or sum over the entire domain. EBM structure is mathematically and algorithmically rich, encompassing graphical models, shallow and deep neural architectures, and structure-preserving frameworks for physical systems. EBMs are fundamental in machine learning, computer vision, inverse imaging, speech and language processing, and computational physics.

1. Mathematical and Structural Definition

An EBM over domain $\mathcal{X}$ and parameters $\theta$ defines a probability density as: $$p_\theta(x) = \frac{\exp\left[-E_\theta(x)\right]}{Z(\theta)}, \qquad \text{where} \quad Z(\theta) = \int_\mathcal{X} \exp\left[-E_\theta(x)\right] dx$$ where $E_\theta(x)$ is the energy function parameterized by $\theta$ and $Z(\theta)$ is the partition function ensuring normalization (Ou, 16 Mar 2024, Habring et al., 16 Jul 2025). The negative log density is interpreted as energy, driving learning and inference via gradient methods and MCMC.

Classic EBMs use the exponential link, but semiparametric generalizations admit any strictly decreasing $g(E)$: $$p(x; \alpha, g) = \frac{g(E(x;\alpha))}{Z(\alpha, g)},\quad g(E) > 0,\; g'(E)<0$$ enabling flexible tail behavior and latent-variable mixtures (Humplik et al., 2016).

In undirected graphical models, e.g., Markov random fields, the energy decomposes over cliques: $$p(x_V) = \frac{1}{Z} \exp\left\{ -\sum_{C\in\mathcal{C}} E_C(x_C) \right\}$$ Each $E_C(x_C)$ is a potential on clique $C$ (Ou, 16 Mar 2024).

2. Parametrization: Feature Maps, Architectures, and Decomposition

EBMs are instantiated with various architectures:

• Linear/Feature Map Structure: The energy is a function of feature activations:

$$h_\theta(x) = \sum_{i=1}^D w_i f_i(x), \qquad E_\theta(x) = \ell(h_\theta(x), y)$$

Decorrelation regularization on $\{f_i(x)\}_{i=1}^{D}$ improves generalization by promoting diversity, as formalized by $\vartheta$-diversity and Rademacher complexity bounds (Laakom et al., 2023).

• Neural Networks:
  • Shallow: $$E_\theta(x) = \sum_{j=1}^m a_j \sigma(w_j^T x + b_j)$$ for single-layer networks (Domingo-Enrich et al., 2021).
  • Deep ConvNets: $E_\theta(x) =g(F_\theta(x))$, where $F_\theta$ is a CNN or stack of nonlinear layers (Ou, 16 Mar 2024).
  • Joint Architectures: For data $x$ and latent $z$, $E_\alpha(x,z)$ is parameterized by concatenating $h_x = \text{Enc}_\alpha(x)$ and $h_z = \text{MLP}_\alpha(z)$ into deep layers (Han et al., 2020).
• Energy Decomposition For image domains, decompositions into "semantic" and "texture" components have demonstrated improved mixing and learning:

$$E(x) = E_{\text{semantic}}(z) + E_{\text{texture}}(x)$$

where $E_{\text{semantic}}$ operates in feature/latent space and $E_{\text{texture}}$ in pixel space, both learned via deep autoencoders and generators (Zeng, 2023).

• Latent Variable and Hierarchical Models:

Multi-layer generators $$z^{(L)} \rightarrow \ldots \rightarrow z^{(1)} \rightarrow x$$ with layer-wise energy terms:

$$E(z^{(1)},\ldots,z^{(L)}) = -\sum_{i=1}^L f_{\alpha_i}(z^{(i)}) - \sum_{i=1}^{L-1}\log p_{\beta_i}(z^{(i)}|z^{(i+1)}) - \log p(z^{(L)})$$

These models capture intra- and inter-layer dependencies beyond conditional Gaussian chains (Cui et al., 2023, Cui et al., 22 May 2024).

3. Learning Objectives and Algorithms

Learning EBMs centers on maximizing the likelihood or minimizing divergences. A canonical maximum likelihood gradient is: $$\nabla_\theta\log p_\theta(x) = \mathbb{E}_{p_{\text{data}}}[\nabla_\theta E_\theta(x)] - \mathbb{E}_{p_\theta}[\nabla_\theta E_\theta(x)]$$ The empirical ("positive phase") energy is subtracted from the model ("negative phase") energy; the latter usually needs MCMC estimation. In joint, latent-variable, or amortized frameworks:

• Latent-EBM Joint Objectives:

Divergence-triangle loss unifies VAEs and EBMs by combining three KL terms, with flow between generator, inference, and EBM "critic" (Han et al., 2020).

• Feature Diversity Regularization:

Feature decorrelation penalties directly impact generalization, as proven by PAC analysis (Laakom et al., 2023).

• Amortized/MCMC Sampling:

Sampling from both prior and posterior of latent variables uses Langevin dynamics, preconditioned by efficient bottom-up encoders for the positive phase (Cui et al., 2023, Pang et al., 2020).

4. Sampling and Inference Strategies

Sampling from $p_\theta(x)$ is essential for both learning and generation:

Sampler	Principle	Structural Implications
Metropolis-Hastings	Accept/reject moves	Relies only on evaluating $E(x)$
(Stochastic) Langevin	Gradient-based updates	Requires differentiability; convexity yields mixing
Hamiltonian MC	Hamiltonian flow	Volume-preservation favors high dimensionality
Gibbs Sampling	Conditional sampling	Exploits graph factors for block-wise updates
Two-Stage MCMC	Latent then data space	Accelerates mixing by first sampling semantic z

In high-dimensional or multimodal cases, sampling in latent or "transported" latent space, as with flow models, dramatically improves mixing and sample fidelity (Nijkamp et al., 2020, Zeng, 2023). For hierarchical settings, diffusion over a whitened latent space allows local, conditional EBMs at each reverse step (Cui et al., 22 May 2024).

5. Applications Across Domains

• Vision and Imaging:

Classic EBMs, fields-of-experts, convolutional energy functions, and denoising autoencoder-based decompositions have enabled state-of-the-art performance in unconditional image generation, inverse problems, and OOD detection (Zeng, 2023, Habring et al., 16 Jul 2025).

• Speech and Language:

EBMs structured as undirected random fields handle marginal, conditional (CRF), and joint distributions for modeling sequential data, NLP, and speech recognition (Ou, 16 Mar 2024).

• Physical System Modeling:

Structure-preserving discretizations, such as port-Hamiltonian or Dirac formulations, maintain energy dissipation and interconnection invariants, enabling robust simulation of mechanical, electrical, and multiphysics systems (Rashid, 9 Dec 2025, Altmann et al., 18 Jun 2024).

• Protein Design and Scientific ML:

Recasting structure-prediction metrics as energies—e.g., pTMEnergy derived from predicted alignment errors—provides likelihood-based losses for generative hallucination and virtual screening in molecular design (Nori et al., 27 May 2025).

6. Advanced Extensions and Theoretical Properties

• Semiparametric EBMs:

Replacing the exponential link with a learned map $g(E)$ yields tail-flexible distributions and connects to implicit latent-variable mixtures (Humplik et al., 2016).

• Overparametrized Regimes:

In shallow-net EBMs, the "active" regime—training both features and weights in wide networks—enables adaptivity to low-dimensional structure; in contrast, kernel (lazy) training lacks this adaptivity (Domingo-Enrich et al., 2021).

• Generalization Bounds:

Feature diversity directly shrinks Rademacher complexity and bounds the empirical-to-true energy expectation gap, establishing decorrelation as essential for tight generalization (Laakom et al., 2023).

• Score-based and Diffusion Learning:

Noise-conditional score matching and diffusion reversals make EBM priors tractable even in deep hierarchical generators (Cui et al., 22 May 2024).

7. Structure-Preserving Principles in Dynamical and Constrained Systems

For physical and engineering applications, port-Hamiltonian and energy-balanced forms ensure invariance of energy dissipation:

• States partitioned as $z=[z_1, z_2, z_3]$ (energy, co-energy, constraints)
• Dynamics:

$$[\partial_{z_1}H,\,\dot z_2,\,0]^T = (J - R)[\dot z_1,\,\partial_{z_2}H,\,z_3]^T + B u$$

where $J$ is skew-symmetric, $R \geq 0$ symmetric, and $H$ the Hamiltonian (Rashid, 9 Dec 2025, Altmann et al., 18 Jun 2024)

• Structure-preserving discretizations (midpoint, discrete gradient) guarantee monotonic energy dissipation at the time-discrete level.

These methodologies extend EBMs to constrained optimization, multi-physics simulations, and dissipative system identification, preserving qualitative and quantitative invariants from continuum models to numerical solvers.

---

In summary, EBM structure encompasses a broad spectrum of frameworks unified by the formalism of energy-based statistical modeling, spanning classic graphical models, neural architectures, generative priors, and structure-preserving physical models. The architectural design—feature structure, form of energy decomposition, and diversity regularization—directly influences both statistical and computational properties. The integration of advanced sampling, amortization, and diffusion-based learning has rendered EBMs practically viable for high-dimensional, multi-modal, and dynamic settings (Ou, 16 Mar 2024, Zeng, 2023, Laakom et al., 2023, Cui et al., 2023, Rashid, 9 Dec 2025, Cui et al., 22 May 2024, Habring et al., 16 Jul 2025, Nori et al., 27 May 2025).