Bayesian Empirical Likelihood

Updated 27 October 2025

Bayesian empirical likelihood is a method that replaces parametric likelihoods with nonparametric, constraint-based likelihoods, enabling robust Bayesian inference in challenging models.
It leverages advanced computational techniques like importance sampling and Hamiltonian Monte Carlo to navigate complex, nonconvex posterior supports arising from moment-based constraints.
Recent developments extend BEL to applications such as quantile regression, variable selection, and robust inference, addressing both computational challenges and optimal constraint specification.

Bayesian empirical likelihood (BEL) is a methodology within Bayesian statistics that replaces the parametric likelihood with an empirical likelihood—a nonparametric, data-driven likelihood constructed from moment or estimating equation constraints. BEL facilitates Bayesian inference in models where the true likelihood is computationally intractable or unavailable, while maintaining the desirable properties of both Bayesian and nonparametric inference. This strategy has seen rapid development, with critical advances in theoretical foundations, computational algorithms, and application domains including quantile regression, dynamic models, variable selection, and survey analysis.

1. Formulation and Theoretical Foundations

BEL centers on integrating empirical likelihood (EL) into the Bayesian framework by treating the EL as a (pseudo-)likelihood for posterior updating. Given independent data $y_1, \ldots, y_n$ , the empirical likelihood for parameter $\theta$ under a set of constraints (e.g., moment or score equations) is constructed as

$L_{EL}(\theta|y) = \max_{p \in P_\theta} \prod_{i=1}^n p_i,$

where $P_\theta = \left\{ p \in [0,1]^n : \sum_{i=1}^n p_i = 1, \sum_{i=1}^n p_i h(y_i, \theta) = 0 \right\}$ for a suitably chosen constraint function $h(y, \theta)$ (such as $y - \theta$ for the mean, or more generally, a vector of estimating equations).

In Bayesian computation via empirical likelihood (BCel), the posterior is then

$\pi(\theta|y) \propto \pi(\theta) \cdot L_{EL}(\theta|y).$

This construction yields posteriors with convergence properties analogous to those of fully parametric Bayesian posteriors when the constraint functions are correctly specified and as $n \to \infty$ (Mengersen et al., 2012, Drovandi et al., 2018).

A major theoretical result is that under regularity, the BEL posterior is asymptotically normal with mean centered at the maximum empirical likelihood estimator (MELE) and variance matching that of MELE (e.g., for quantile regression (Yang et al., 2012)). Wilks-type and Bernstein–von Mises theorems hold under suitable conditions, ensuring valid asymptotic uncertainty quantification (Kim et al., 2023, Liu et al., 2017, Tang et al., 2021).

2. Algorithmic and Computational Aspects

The BCel algorithm typically employs importance sampling:

Step	Description
1. Prior Sampling	Draw $\theta_i \sim \pi(\theta)$
2. Weighting	Compute $\omega_i = L_{EL}(\theta_i\|y)$
3. Posterior Sample	Use $(\theta_i, \omega_i)$ as a weighted posterior sample

For higher efficiency, adaptive multiple importance sampling (AMIS) can refine the proposal distribution iteratively (e.g., mixtures of Student’s $t$ ) (Mengersen et al., 2012).

Key computational considerations include:

Constrained optimization: Each EL evaluation requires solving a convex optimization problem (solved in practice via Newton–Lagrange methods, e.g., R package "emplik").
Support irregularity: The convex hull constraint restricts BEL support to a subset of the parameter space where constraints are feasible, leading to nonconvex and possibly disconnected posterior support (Kien et al., 2022).
Advanced MCMC: Standard random walk proposals may perform poorly due to irregular support, motivating the use of Hamiltonian Monte Carlo (HMC), which leverages the diverging gradients of the log empirical likelihood at the support boundary to "reflect" proposals back inside. This improves mixing and effective exploration (Kien et al., 2022, Moon et al., 2020).
Variational Inference and Expectation Propagation: Recent advances employ expectation-propagation (EP) to achieve fast, scalable Gaussian approximations of the BEL posterior, rigorously shown to be asymptotically equivalent to the exact BEL under increasing sample size (Ng et al., 24 Oct 2025).

3. Choice and Role of Constraints

The performance of BEL is critically dependent on the specification of constraint functions $h(y, \theta)$ :

Classical settings: For mean estimation, $h(y, \theta) = y - \theta$ suffices.
Complex models: Score functions or composite scores (e.g., pairwise composite likelihoods in population genetics) are often used where the likelihood is analytically intractable (Mengersen et al., 2012).
ABC integration: In ABC-style inference, summary statistics (moments, quantiles, etc.) serve as constraints, and the empirical likelihood is constructed to match the observed and simulated summaries (Chaudhuri et al., 2018, Chaudhuri et al., 2020, Chaudhuri et al., 8 Mar 2024).
Quantile regression: In the absence of a parametric likelihood, sample quantile estimating equations define the empirical likelihood constraints (Yang et al., 2012).

Over- or under-specification of constraints can negatively impact both efficiency and identifiability. Testing constraint sets against simulated data or prior predictive checks helps calibrate their informativeness (Mengersen et al., 2012).

4. Posterior Inference, Extensions, and Robustification

BEL has been extended and adapted for multiple inferential tasks:

Quantile Regression: BEL yields asymptotically normal posteriors for quantile parameters, with variance matching regression quantile estimators. Informative (shrinkage) priors can be incorporated across quantile levels to borrow strength and reduce uncertainty in sparse data regions (Yang et al., 2012).
Hierarchical and Spatial Modeling: Empirical likelihood can be specified as the data level in hierarchical models, with parametric or reduced-rank process models inducing dependencies (e.g., spatial autocorrelation), as in semiparametric hierarchical empirical likelihood (SHEL) models for areal or point-referenced spatial data (Porter et al., 2014).
Robust BEL: Exponentially tilted empirical likelihood (BETEL) and its robust variant (RBETEL) accommodate outlier contamination by introducing latent indicators for "good" vs. "bad" data, up-weighting uncontaminated observations in the likelihood through modified estimating equations and marginalization (Liu et al., 2017).
Regularization: In high-dimensional or ill-posed scenarios, penalization (e.g., penalized empirical likelihood) targets both Lagrange multipliers and parameters to enhance identification and stability, yielding consistent and asymptotically normal estimators (Chang et al., 23 Dec 2024, Moon et al., 2020).
Convex Hull Relaxation: Regularized exponentially tilted empirical likelihood (RETEL) introduces pseudo-data from continuous exponential family distributions, removing the support-restricting convex hull constraint and ensuring that posterior inference is well defined even for small samples or extreme parameter values (Kim et al., 2023).

5. Applications and Performance

BEL has demonstrated strong empirical performance in a broad array of settings:

Domain	Summary of BEL Implementation and Outcomes
Population genetics	Composite moment constraints in models with intractable likelihoods, large reduction in simulation and improved calibration compared to ABC (Mengersen et al., 2012).
Time series (e.g., ARCH, GARCH)	Moment-based constraint functions on reconstructed latent innovations, robust and effective estimation (Mengersen et al., 2012).
Quantile-based models	Robust uncertainty quantification when analytic likelihood is unavailable; improved efficiency via shrinkage across quantiles (Yang et al., 2012).
Bayesian variable selection	Nonparametric BEL with sparsity-inducing priors (such as LASSO, SCAD) demonstrates consistency for large parameters and correct zeroing for noise variables; effective MCMC via Laplace-approximation-based proposals (Cheng et al., 2022).
Survey data	Bayesian jackknife empirical likelihood enables calibrated inference with complex design weights or calibration constraints in complex surveys (Shang et al., 2023).
Model selection	Extension to reversible jump MCMC (RJMCMC) with hierarchical proposals based on empirical likelihood maximization achieves feasible, higher-acceptance sampling in models of varying dimension (Chaudhuri et al., 2022).

Furthermore, simulation experiments widely confirm the asymptotic theory: credible intervals from BEL achieve correct coverage, and, in numerous real-data cases, BEL point estimators exhibit lower mean squared error and credible intervals with improved calibration compared to parametric, pseudo-likelihood, or frequentist approaches.

6. Methodological Developments and Limitations

BEL methodologies manifest several strengths:

No requirement for full parametric likelihood: Enables posterior inference under moment information alone.
Self-diagnostics: Effective sample size, sensitivity to constraint choice, and comparison across constraint sets are naturally embedded in importance sampling and weight diagnostics.
Flexible prior structures: Priors (including shrinkage for variable selection and hierarchical models) can be integrated directly, allowing incorporation of domain knowledge.

Critical limitations remain:

Constraint specification: Faithful inference relies on valid, identifying constraint functions. Mis-specification leads to poorly calibrated posteriors. There is an ongoing need for principled constraint selection or data-driven diagnostic tools.
Computational burden: Each EL evaluation is a constrained optimization, posing substantial computational challenges in high dimensions or complex models. While HMC and expectation-propagation mitigate some issues, scaling to very high dimensions remains difficult (Ng et al., 24 Oct 2025).
Irregular support: The presence of nonconvex support complicates both practical implementation and theoretical analysis. Regularization via pseudo-data (e.g., RETEL) and robustification via indicator augmentation (RBETEL) offer partial solutions.

7. Extensions, Variants, and Research Directions

Recent research has expanded BEL in the following directions:

Approximate Bayesian Computation (ABC): Empirical likelihood is used to compute approximate posteriors in ABC without requiring user-specified distances or tolerances, instead employing constraints on summary statistics. Such methods are theoretically justified via information projection and shown to be consistent (Chaudhuri et al., 2018, Chaudhuri et al., 2020, Chaudhuri et al., 8 Mar 2024).
Variational Inference and EP: Expectation-propagation and variational methods for BEL deliver efficient, large-scale computation while attaining asymptotic equivalence to the exact BEL posterior (Ng et al., 24 Oct 2025).
Regularized and penalized BEL: Approaches such as penalized empirical likelihood and RETEL address nonconvexity, parameter identifiability, and support shrinkage, providing improved finite-sample performance and broader support for Bayesian inference (Kim et al., 2023, Chang et al., 23 Dec 2024).
Software: R packages such as "emplik" and "elhmc" implement core optimization routines and HMC sampling for BEL, facilitating practical application across diverse models (Kien et al., 2022).

Open problems include: automatic constraint selection, theoretical guarantees for high-dimensional or misspecified constraint scenarios, improved robustness features, and more scalable algorithms for high-throughput applications.

Bayesian empirical likelihood represents a robust, flexible alternative to parametric likelihood-based Bayesian inference, particularly suitable for models with intractable likelihoods or only partial moment information. Its ongoing development spans foundational theory, computational methods, and broad applications ranging from quantile regression to population genetics and survey analysis, with significant recent progress in computational tractability and robustness.