Energy-Based Models (EBM)

• Energy-Based Models (EBMs) are probabilistic models that define data density via an energy functional, playing a key role in inverse imaging tasks like denoising, tomography, and inpainting.
• They ensure well-posed and stable Bayesian inference by satisfying integrability and continuity conditions, which are essential for reliable image reconstruction.
• EBMs combine advanced training strategies—such as score matching and bilevel optimization—with efficient sampling algorithms like Langevin MCMC to achieve robust estimations and uncertainty quantification.

An energy-based model (EBM) is a probabilistic model that defines the density of data via a Gibbs or Boltzmann distribution, $p(x) \propto \exp\{-E(x)\}$, where $E(x)$ is an energy functional. In the context of inverse imaging problems, EBMs are used within a rigorous Bayesian framework to model the unknown image $x$ given forward measurements $y$ and a typically ill-posed measurement process. The use of EBMs as priors enables highly expressive modeling of natural image statistics, and—if properly constructed—yields well-posed, stable Bayesian inference for a broad range of inverse problems, including denoising, tomographic inversion, and inpainting. This modeling paradigm unites theory, learning strategies, and practical computational algorithms under a common formalism and supports both point estimation and principled uncertainty quantification.

1. Bayesian Formulation and Theoretical Well-Posedness

In the Bayesian framework for inverse imaging, the measurement process is modeled as $y = P(F(x))$, where $F$ is the forward operator and $P$ represents noise or corruption. Both $x$ (the unknown image) and $y$ (the observed data) are random variables, and the posterior is defined as

$p(x \mid y) \propto p(y \mid x)\,p(x)$

where $p(x)$ is the prior and $p(y \mid x)$ is the data likelihood.

For EBMs, the prior is assigned as a Gibbs distribution: $p(x) \propto \exp(-E(x))$ where $E(x)$ is an energy functional, often with learnable parameters. Validity and stability of the posterior in both finite- and infinite-dimensional spaces (e.g., when $x$ is a function or image) require sufficient coercivity and integrability conditions on $E(x)$, i.e., that $\int \exp(-E(x))\,dx < \infty$. Under these conditions, one can prove existence, uniqueness, and stability of the posterior. Continuity and stability are often measured using metrics such as the Hellinger distance

$$H(p,q) = \left( \frac{1}{2} \int \left( \sqrt{p(x)} - \sqrt{q(x)} \right)^2 dx \right)^{1/2}$$

with the property that if $y \mapsto L(y\,|\,x)$ is continuous and dominated, $y \mapsto p(x\,|\,y)$ is stable in the Hellinger and total variation sense (Habring et al., 16 Jul 2025).

In infinite-dimensional settings (common for function-space or PDE-constrained imaging), Gaussian reference measures are employed, and additional care is taken to guarantee absolute continuity and stability of the posterior.

2. Learning Energy-Based Priors

The energy functional $E(x)$ must be expressive enough to capture the high-order structure of natural image statistics. Several training strategies are discussed:

• Divergence Minimization/Maximum Likelihood: For sample-based (empirical) distributions $\hat{P}$, maximum likelihood minimizes

$$\mathrm{KL}(\hat{P} \,\|\, p_\theta) = E_{x \sim \hat{P}} [-\log p_\theta(x)] + \mathrm{const}$$

but since the normalization constant (partition function) is intractable, direct estimation is often circumvented.

• Score Matching and Denoising Score Matching: Instead of matching densities, one matches gradients—i.e., the "score" $\nabla_x \log p_\theta(x)$. The denoising score-matching objective

$$L(\theta) = E_{x, \varepsilon}[ \| \sigma \nabla E_\theta(x + \sigma \varepsilon) - \varepsilon \|^2 ]$$

directly relates to the optimal denoiser via Tweedie's formula, allowing stochastic gradient-based training without the need to compute $Z$.

• Bilevel Optimization for MAP Quality: Instead of only matching densities, a bilevel approach optimizes for end-task performance by embedding $E_\theta(x)$ in the lower-level variational reconstruction problem

$$x^*(\theta) = \arg\min_x \{ D(y, F(x)) + \lambda E_\theta(x) \}$$

and using the upper-level loss (e.g., squared error to ground truth) to refine $\theta$, requiring (implicit) differentiation through the optimization process.

• Architectural Choices: The classical Fields-of-Experts (FoE) model is employed, with

$$E_\theta(x) = \sum_{i=1}^n \sum_{j=1}^o \phi_j((K_j x)_i)$$

where convolutions $K_j$ and nonlinear potentials $\phi_j$ (e.g., negative-log Gaussian mixtures) are learnable. This guarantees both interpretability and control over the model's mathematical properties.

3. Sampling Algorithms for Posterior Inference

EBMs define distributions only up to an intractable normalization constant; consequently, inference algorithms rely on Markov chain Monte Carlo (MCMC) approaches:

• Metropolis–Hastings (MH): Proposes samples $y \sim Q(x,\cdot)$, and accepts transitions with probability

$$\rho(x, y) = \min \left\{ \frac{p(y) q(y, x)}{p(x) q(x, y)},\, 1 \right\}$$

For EBM priors, the normalization constant cancels and samples are generated according to the posterior.

• Gibbs Sampling: Alternating sampling for model blocks or augmented latent variables when conditionals are tractable.
• Langevin Monte Carlo and Hamiltonian Monte Carlo: For high-dimensional problems, overdamped Langevin diffusion is utilized:

$$X_{k+1} = X_k - \tau \nabla E(X_k) + \sqrt{2\tau} Z_k$$

More advanced samplers use underdamped (kinetic) Langevin or HMC, introducing momentum variables $(v)$ and symplectic integration (e.g., leapfrog), with accept/reject steps to ensure detailed balance. Under suitable smoothness and coercivity assumptions, convergence rates and ergodicity can be established, ensuring that empirical and theoretical distributions coincide asymptotically.

These sampling strategies enable MMSE or MAP estimates, and—through sample-based statistics—uncertainty quantification.

4. Numerical Experiments in Inverse Imaging

Empirical results are provided on standard inverse imaging tasks:

• Denoising, where $F$ is the identity and $P$ is additive Gaussian noise.
• Reconstruction from Fourier Samples, simulating MRI or compressed sensing, with $F$ as a partial Fourier transform.
• Reconstruction from Radon Samples, for computed tomography (CT), using the Radon transform as $F$.

Both MAP and MMSE estimates are computed, with the latter requiring efficient sampling. Learned EBM priors (trained via score-matching or bilevel approaches) consistently outperform classical regularizers (such as total variation), as shown by PSNR improvements and qualitative visual results that display fewer artifacts and better naturalistic details. Uncertainty maps (pixelwise standard deviations) provide additional information not available with deterministic approaches.

5. Verification and Theoretical Properties

Explicit attention is devoted to verifying that learned energy functions $E(x)$ yield proper, stable Bayesian posteriors:

• Integrability and Coercivity: $E(x)$ is constructed so that $p(x) = \exp(-E(x)) / Z$ is a proper density (i.e., normalization constant $Z < \infty$), with coercivity or added nullspace penalization where necessary.
• Continuity and Stability: Under regularity and boundedness conditions, the mapping $y \mapsto p(x\,|\,y)$ is continuous (e.g., in Hellinger distance), ensuring that reconstructions are stable to data perturbations.
• Ergodicity of Samplers: Markov properties and detailed balance are established for each class of MCMC methods, guaranteeing that empirical averages converge to posterior expectations.
• Structural Properties: The architecture (e.g., Fields-of-Experts with full-rank convolutional filters) is chosen to avoid degeneracies (e.g., singularities in the induced precision matrix). Visualizations of learned filters and associated potentials further validate proper regularization behavior.

6. Significance and Context

Energy-based models provide a unifying regularization framework in Bayesian inverse imaging, enabling the integration of learned, data-adaptive priors with explicit measure-theoretic guarantees. Their flexibility supports the modeling of highly complex image distributions, while properly constructed training and sampling algorithms provide principled uncertainty quantification—an essential requirement in high-stakes imaging tasks such as medical reconstruction, astronomical imaging, and geophysical inversion.

Advances in learning (e.g., denoising score matching, bilevel optimization), sampling (e.g., ULA/HMC), and theory (well-posedness in infinite-dimensions) ensure that EBMs can be robustly applied to inverse imaging. Verification steps are critical to ensure all necessary assumptions for statistical validity are satisfied. Consistent empirical improvements over classical priors and compatibility with uncertainty-driven inference make energy-based modeling for inverse problems a central paradigm in contemporary computational imaging (Habring et al., 16 Jul 2025).