Energy Discrepancies: A Score-Independent Loss for Energy-Based Models (2307.06431v2)

Abstract: Energy-based models are a simple yet powerful class of probabilistic models, but their widespread adoption has been limited by the computational burden of training them. We propose a novel loss function called Energy Discrepancy (ED) which does not rely on the computation of scores or expensive Markov chain Monte Carlo. We show that ED approaches the explicit score matching and negative log-likelihood loss under different limits, effectively interpolating between both. Consequently, minimum ED estimation overcomes the problem of nearsightedness encountered in score-based estimation methods, while also enjoying theoretical guarantees. Through numerical experiments, we demonstrate that ED learns low-dimensional data distributions faster and more accurately than explicit score matching or contrastive divergence. For high-dimensional image data, we describe how the manifold hypothesis puts limitations on our approach and demonstrate the effectiveness of energy discrepancy by training the energy-based model as a prior of a variational decoder model.

• The paper introduces energy discrepancy, a dual energy contribution loss that bridges score matching and maximum likelihood estimation without relying on MCMC simulations.
• The paper presents a novel variance reduction technique, w-stabilisation, to overcome numerical instabilities in Monte Carlo estimations.
• Empirical analyses on synthetic and image datasets demonstrate that ED accurately captures multi-modal distributions and stabilizes EBM training in high-dimensional settings.

Energy Discrepancies: A Score-Independent Loss for Energy-Based Models

The paper "Energy Discrepancies: A Score-Independent Loss for Energy-Based Models" introduces a novel approach to the training of energy-based models (EBMs) by proposing a method called Energy Discrepancy (ED). Energy-based models are a flexible class of probabilistic models that have been constrained in their applicability due to the computational cost associated with training. The traditional methods, such as Contrastive Divergence (CD) and Score Matching (SM), have limitations including theoretical inconsistency and nearsightedness, respectively. This paper addresses these issues by introducing ED as an alternative loss function that efficiently interpolates between score matching and maximum likelihood estimation, while not requiring Markov Chain Monte Carlo (MCMC) simulations or the computation of score functions.

Energy discrepancy is derived from the basic problem of training EBMs, where the main challenge lies in dealing with the intractability of the normalisation constant, often referred to as the partition function. Traditional methods like CD approximate the gradient of the log-likelihood but suffer from limitations such as convergence to unstable solutions due to short MCMC runs. On the other hand, score-based methods avoid the partition function altogether but fail to capture global features of the data distribution, particularly in scenarios involving well-separated mixture distributions.

In contrast, energy discrepancy introduces a dual energy contribution approach, wherein it utilises a loss function based on contrasting energies from data distributions and the model, circumventing the need for explicit score information. The authors demonstrate theoretically that ED effectively interpolates across different loss functions, capturing both local and global features. By incorporating a convolution with Gaussian noise, ED is shown to establish a connection between the biases of score matching and the generality of maximum likelihood estimation when the integration limit is sufficiently large.

The authors conduct thorough experimental analyses, initially on synthetic two-dimensional datasets, which highlight ED's ability to capture multi-modal distributions more accurately than either score matching or contrastive divergence. Additionally, when leveraged in high-dimensional problems like image data modelling, ED shows robust performance when used as a prior for variational decoders. Here, the manifold hypothesis presents challenges, as such datasets often do not conform to positive probability densities. In these scenarios, the authors employ a lower-dimensional latent representation to maintain computational tractability and effectiveness.

Furthermore, the paper introduces a novel variance reduction trick called $w$-stabilisation. This mechanism addresses the numerical instability and potential biases in the contrastive potential computation by adjusting the Monte Carlo estimation in a way that robustly approximates the energy discrepancy, even with fewer samples.

Practically, energy discrepancy offers significant computational advantages. It provides a stable training methodology that does not over-rely on tuning, reducing computational expenses linked to score computation or MCMC simulations typical of other EBM training methods. The empirical demonstrations underscore that energy discrepancy converges more rapidly to accurate solutions, stabilising the training of EBMs across diverse scenarios and improving the application scope of EBMs in generative modelling, especially for high-dimensional data.

For future developments, several directions are promising. Exploring the generalisation of ED for non-Gaussian perturbations and further optimising the w-stabilisation could yield improvements. Moreover, deeper investigations into training EBMs directly on manifolds or in other non-Euclidean spaces can be explored, given the theoretical framework the authors establish for non-parametric estimation on general measure spaces. Such pursuits can significantly broaden the applicability of EBMs in contexts where data do not adhere to traditional probabilistic model assumptions.