A unifying view of contrastive learning, importance sampling, and bridge sampling for energy-based models

Abstract: In the last decades, energy-based models (EBMs) have become an important class of probabilistic models in which a component of the likelihood is intractable and therefore cannot be evaluated explicitly. Consequently, parameter estimation in EBMs is challenging for conventional inference methods. In this work, we provide a unified framework that connects noise contrastive estimation (NCE), reverse logistic regression (RLR), multiple importance sampling (MIS), and bridge sampling within the context of EBMs. We further show that these methods are equivalent under specific conditions. This unified perspective clarifies relationships among existing methods and enables the development of new estimators, with the potential to improve statistical and computational efficiency. Furthermore, this study helps elucidate the success of NCE in terms of its flexibility and robustness, while also identifying scenarios in which its performance can be further improved. Hence, rather than being a purely descriptive review, this work offers a unifying perspective and additional methodological contributions. The MATLAB code used in the numerical experiments is also made freely available to support the reproducibility of the results.

• The paper presents a unified framework that connects contrastive learning, importance sampling, and bridge sampling for parameter estimation in energy-based models.
• It demonstrates the theoretical equivalence of NCE, RLR, and optimal bridge sampling under specific assumptions through classification and Monte Carlo integration.
• Empirical results highlight the impact of proposal distributions and scoring rules on estimator variance, offering practical guidelines for efficient EBM inference.

A Unified Framework for Contrastive Learning, Importance Sampling, and Bridge Sampling in Energy-Based Models

Introduction and Motivation

Energy-based models (EBMs) provide a powerful class of probabilistic models where the likelihood is often only defined up to an intractable partition function $Z(\theta)$. This intractability makes likelihood-based parameter inference difficult and precludes direct use of common inference methods, a challenge intrinsic to models such as Markov random fields and Boltzmann machines. Traditional statistical approaches address this via methods like MCMC or score matching, but EBMs remain a central challenge for methods both frequentist and Bayesian, especially in high dimensions or when $Z(\theta)$ depends strongly on parameters.

Recent literature has proposed a variety of solutions, including noise-contrastive estimation (NCE), reverse logistic regression (RLR), multiple importance sampling (MIS), and bridge sampling. Each emerged from different theoretical traditions—classification, generalized regression, and Monte Carlo methodologies—but all strive to approximate or bypass the intractable normalizing constant. This paper presents a rigorous, unified mathematical framework that makes explicit the structural connections between these methods, demonstrates their equivalence under specific assumptions, and highlights opportunities for generalized and improved estimators.

Core Contributions and Theoretical Insights

Unified Perspective

The paper develops a shared mathematical infrastructure connecting NCE, RLR, MIS, and bridge sampling when parameter estimation in EBMs is the objective. The unification is accomplished by expressing these methods in terms of (i) discriminative estimators formulated as classification tasks between data and auxiliary samples, and (ii) importance weighting schemes defined via Monte Carlo integration.

Specifically, NCE recasts inference in EBMs as a binary classification task distinguishing between true samples from the model and samples from a tractable reference distribution $q(\mathbf{y})$. The NCE cost function depends on the parameters $\theta$ and the normalizing constant $Z(\theta)$, and its minimization leads to estimators for both. The RLR framework, inspired by mixture model classification, provides an equivalent likelihood-based viewpoint focused on ratios of normalizing constants. Both methods can be embedded in and derived from the optimal bridge sampling estimator when reduced to the two-distribution ($K=2$) case.

The paper establishes that, with $K=2$ and specific choices of scoring rule and mixture weights, NCE, RLR, and optimal bridge sampling yield identical fixed-point equations for estimating $Z(\theta)$. The derivation of the bridge sampling estimator from NCE, via derivatives of the classification loss and the construction of a suitable “bridge function”, provides both theoretical clarity and generalization.

Generalization to Broader Classes of Estimators

The authors extend this unified framework to cover MIS and variants, showing how empirical mixture distributions over both observed and auxiliary samples can be employed to design estimators with improved variance or convergence properties. The work further extends to optimal umbrella estimators (requiring additional sampling challenges) and considers the performance of estimators derived from different proper scoring rules, showing that the choice of scoring rule (i.e., the cost in the classification/Fenchel duality framework) interacts significantly with the choice of proposal/reference distribution and can yield significantly different statistical behavior.

Crucially, the paper explores the use of adaptive mixtures, multiple proposal densities, and incorporates insights from recent advances in importance sampling [MIS2019] toward adaptive or optimized proposal design, showing how classification-based estimators naturally extend to these more general settings.

Numerical and Empirical Analysis

Extensive numerical simulations are performed using a univariate Gaussian target and auxiliary Gaussian proposals to provide empirical evaluation of each estimator’s mean squared error in $Z(\theta)$ and $\theta$. These experiments systematically vary the number of model and proposal samples, the reference distribution’s variance, and scoring rule in the cost function, permitting a comprehensive comparison in both idealized and practical settings. Key findings include:

• Statistical efficiency is highly sensitive to the proposal distribution $Z(\theta)$0 and the chosen scoring rule.
• The so-called optimal bridge estimator is outperformed by MIS estimators when proposals are well-chosen and $Z(\theta)$1 can be evaluated on the right-hand side of recursion, but can be inferior in realistic scenarios due to recursive dependence and initialization.
• NCE and related classification-based approaches are robustly effective, and their performance in estimating $Z(\theta)$2 reflects their theoretical equivalence with optimal bridge sampling under suitable conditions.
• Alternative scoring rules, even non-proper rules, can be used to derive new estimators; the geometric mean of standard IS and reverse IS arises as a limit, but is biased unless properly balanced.

The authors support all results with reproducible code and systematic statistical analysis, emphasizing practical scenarios where sample sizes are finite and proposals may not perfectly overlap the model.

Implications, Practical Guidance, and Future Directions

This unified framework clarifies why NCE has consistently shown empirical success in EBM inference: it is in fact an instance of an optimal bridge estimator in the normalizing constant space, and can be interpreted as an importance sampling scheme optimized for mixture-classification. The equivalence with RLR further demystifies the connection between discriminative and generative estimation paradigms.

Practically, the work provides a comprehensive recipe for constructing and analyzing estimators for $Z(\theta)$3 and $Z(\theta)$4 in EBMs, including:

• How to choose and tune proposal/reference distributions for efficient estimation.
• When recursion is required, and what impact initialization and proposal support have on estimator performance.
• The duality between choosing scoring rules (losses for the classification problem) and bridge functions (in importance-weighted estimators), and how this duality can guide both theoretical and empirical optimization.

The exploration of generalized bridge estimators using multiple proposals, proper scoring rules, and hybrid schemes (such as optimal umbrella sampling) suggests many directions for future research. Of particular interest is the analytical design of proposals and losses/scoring rules that jointly optimize estimator variance for finite samples, the use of adaptive or tempered sequences of EBMs as proposals (i.e., adaptive importance schemes), and the rigorous characterization of estimator bias and finite-sample efficiency.

Conclusion

This paper delivers a technically rigorous, comprehensive framework unifying NCE, RLR, importance sampling, and bridge sampling for normalizing constant and parameter estimation in EBMs ["A unifying view of contrastive learning, importance sampling, and bridge sampling for energy-based models" (2604.08116)]. By elucidating the mathematical equivalence and relationships among these methods, the work offers both theoretical understanding and practical guidance for inference in unnormalized models. The insights on interplay between scoring rule and proposal selection pave the way for new estimator design, variance reduction, and adaptive modeling strategies—avenues of continuing significance as EBMs find broader application in machine learning and statistical inference.