Hat-EBMs in Energy-Based Generative Models

• Hat-EBMs are a probabilistic framework that integrates a learned energy correction with generator outputs to refine image synthesis without explicit latent code inference.
• They use Langevin sampling to update residuals, thereby bypassing costly computations like Jacobian determinants and simplifying training.
• Empirical results demonstrate competitive FIDs and robust out-of-distribution detection, underlining their versatility in generative modeling tasks.

Hat Energy-Based Models (Hat-EBMs) are a probabilistic modeling framework that augments an existing generator network with a learned energy correction, enabling explicit energy-based modeling directly in the generator’s output space. In this approach, observed images are assumed to decompose into a latent variable passed through a generator and a residual that bridges the gap to the data manifold. Hat-EBMs allow training and sampling without the need for explicit inference of latent codes corresponding to observed data or the computation of generator Jacobians, and can incorporate both probabilistic and deterministic generators as model backbones (Hill et al., 2022).

1. Model Formulation

Let $x \in \mathbb{R}^d$ denote an observed image. The fundamental generative mechanism in Hat-EBMs is

$x = G(z) + r$

where $z \in \mathbb{R}^m$ is an unobserved latent code sampled from a simple prior $p_0(z) = \mathcal{N}(0, I_m)$, and $G : \mathbb{R}^m \to \mathbb{R}^d$ is a fixed or learned generator network. The residual $r \in \mathbb{R}^d$ (also denoted $y$) accounts for the gap between the generator manifold and the true data distribution. A typical prior for $r$ is also Gaussian: $$p_0(r) = (2\pi \sigma^2)^{-d/2} \exp\left(-\frac{1}{2\sigma^2}\|r\|^2\right)$$

The joint density over $(x, z, r)$ is given by

$x = G(z) + r$0

and the marginal density of $x = G(z) + r$1 becomes

$x = G(z) + r$2

In the unconditional variant, the residual prior can be omitted, with both $x = G(z) + r$3 and $x = G(z) + r$4 modeled via an energy function.

2. Energy Parameterization and Model Variants

Hat-EBMs define an unnormalized density over $x = G(z) + r$5 pairs via a parameterized “hat” energy network $x = G(z) + r$6. The energy function is

$x = G(z) + r$7

giving the unnormalized joint distribution

$x = G(z) + r$8

There are two principal variants:

• Joint Hat EBM: No explicit priors on $x = G(z) + r$9; the energy function governs the entire joint space.
• Conditional Hat EBM: With $z \in \mathbb{R}^m$0 fixed, the model defines

$z \in \mathbb{R}^m$1

3. Training Objectives and Algorithms

Training is performed by minimizing the Kullback–Leibler divergence between the empirical joint distribution and the model: $z \in \mathbb{R}^m$2 The gradient of the log-likelihood with respect to network parameters $z \in \mathbb{R}^m$3 follows the standard energy-based model “positive–negative phase” form: $z \in \mathbb{R}^m$4

In practice, data expectations are taken over observed images, while model expectations are generated by running Markov chain Monte Carlo (MCMC) to sample negative pairs $z \in \mathbb{R}^m$5. Key computational advantages include:

• No need for Jacobian determinant computation ($z \in \mathbb{R}^m$6)
• No requirement for latent code inference ($z \in \mathbb{R}^m$7) per datum—data $z \in \mathbb{R}^m$8 is directly input to the energy network

4. Langevin Sampling and Algorithmic Structure

Negative samples are generated via alternating Langevin updates (a Metropolis-within-Gibbs scheme) in $z \in \mathbb{R}^m$9 and $p_0(z) = \mathcal{N}(0, I_m)$0: $p_0(z) = \mathcal{N}(0, I_m)$1 with $p_0(z) = \mathcal{N}(0, I_m)$2. For the conditional Hat EBM, only residuals $p_0(z) = \mathcal{N}(0, I_m)$3 are updated (latent $p_0(z) = \mathcal{N}(0, I_m)$4 is fixed), simplifying sampling for unconditional generation.

A summary of the conditional Hat EBM training algorithm:

Step	Operation	Details
1	Update EBM parameters	Negative samples via Langevin in $p_0(z) = \mathcal{N}(0, I_m)$5 per $p_0(z) = \mathcal{N}(0, I_m)$6
2	Update generator from replay bank	Regression to align $p_0(z) = \mathcal{N}(0, I_m)$7 to EBM sample $p_0(z) = \mathcal{N}(0, I_m)$8

This algorithmic design allows the incorporation of both deterministic and probabilistic generator networks.

5. Empirical Performance and Use Cases

Hat-EBMs have been evaluated in multiple generative modeling tasks:

A. Refinement of Pretrained GANs (Joint Hat EBM)

• Datasets: CIFAR-10 ($p_0(z) = \mathcal{N}(0, I_m)$9), CelebA ($G : \mathbb{R}^m \to \mathbb{R}^d$0)
• SN-GAN FID: 18.58 (CIFAR-10), 6.13 (CelebA)
• DDLS refinement FID: 14.59, 6.06
• Hat EBM refinement FID: 14.04, 5.98
• Visual assessment indicates the residual $G : \mathbb{R}^m \to \mathbb{R}^d$1 is nearly imperceptible; refinements occur mainly in latent space.

B. Retrofit of Deterministic Autoencoders

• Pretrained autoencoder with latent codes on a $G : \mathbb{R}^m \to \mathbb{R}^d$2-sphere
• Application of joint Hat EBM sampling achieves FID ≈ 26.01 on CIFAR-10 via residual-corrected generations.

C. Unconditional Synthesis (Conditional Hat EBM)

• Short-run Langevin ($G : \mathbb{R}^m \to \mathbb{R}^d$3); competitive FIDs achieved:
  • CIFAR-10 ($G : \mathbb{R}^m \to \mathbb{R}^d$4): 19.30 vs. 18.6 (SN-GAN)
  • CelebA ($G : \mathbb{R}^m \to \mathbb{R}^d$5): 11.57 vs. 6.1 (GAN)
  • ImageNet ($G : \mathbb{R}^m \to \mathbb{R}^d$6): 29.37 (large model), approaching SS-GAN benchmark (23.4)
• Qualitatively, samples exhibit sharp detail and diversity; residual correction primarily enhances fine texture.

D. Out-of-Distribution Detection

• Using the energy $G : \mathbb{R}^m \to \mathbb{R}^d$7 as an anomaly score, Hat-EBMs show state-of-the-art unsupervised AUROC:
  • SVHN: 0.92
  • CIFAR-100: 0.87
  • CelebA: 0.94

6. Distinctive Features and Advancements

Hat-EBMs provide the following key features:

• Modularity: Any generator network, deterministic or probabilistic, can be leveraged as an EBM component.
• No Jacobian or Latent Inference: Explicit computation of generator Jacobians and inference of latents for observed data are unnecessary; only Langevin updates are required during training and sampling.
• Scalability: The approach demonstrates competitive Fréchet Inception Distance (FID) at high image resolutions with moderate computational demands.
• Versatility: Effective out-of-distribution detection and generator refinement, and compatibility with both pretrained and jointly trained generators.

A plausible implication is that Hat-EBMs effectively unify energy-based and generator-based modeling paradigms by enabling energy correction at the generator output level, potentially broadening the applicability of EBMs to settings previously dominated by adversarial or likelihood-based generator models (Hill et al., 2022).