---
title: Penalized Empirical Likelihood (PEL)
url: https://www.emergentmind.com/topics/penalized-empirical-likelihood-pel
type: topic
---

# Penalized Empirical Likelihood (PEL)

Penalized empirical likelihood (PEL) denotes a family of methods that modifies empirical likelihood by adding an explicit penalty or, in one influential high-dimensional construction, by replacing exact moment feasibility with a quadratic penalty on weighted moment discrepancies. Across this literature, the common empirical-likelihood core is optimization over probability weights subject to estimating-equation structure, while the penalty is used for bias reduction, sparsity, moment selection, computational stabilization, or problem-specific regularization. The term does not refer to a single canonical estimator: different papers penalize the model parameter, the Lagrange multipliers, the discrepancy in empirical moments, or a target parameter such as abundance in capture–recapture models [1704.00566, 1808.06222, 1302.3071, 2208.05635].

## 1. Empirical-likelihood foundation and the scope of penalization

A standard empirical-likelihood formulation for estimating-equation models is
\[
L(\theta)=\sup\left\{\prod_{i=1}^n \pi_i:\pi_i>0,\ \sum_{i=1}^n \pi_i=1,\ \sum_{i=1}^n \pi_i g(X_i;\theta)=0\right\},
\]
with a dual representation involving Lagrange multipliers \(\lambda\) through terms of the form \(\sum_{i=1}^n \log\{1+\lambda^\top g(X_i;\theta)\}\) [1704.00566, 2412.17354]. Penalized empirical likelihood retains this structure but augments it with a regularizing term. In some papers the penalty is added directly to the empirical-likelihood objective; in others it appears in the dual problem through the multiplier vector; in still others it regularizes a particular scientific target such as the population size \(N\) [1808.06222, 2412.17354, 2208.05635].

This breadth matters because the same label covers substantively different methodological aims. In high-dimensional estimating-equation problems, PEL is primarily a sparse estimation and moment-selection device [1704.00566, 2108.03382]. In bias-reduction work, it is the empirical-likelihood analogue of Jeffreys-prior penalization [1808.06222]. In capture–recapture, it is a one-sided stabilizer against implausibly large abundance estimates [2208.05635]. In recent causal work, PEL is used on the treatment-weighting side to impose covariate-balancing propensity score restrictions with an \(\ell_1\) penalty [2509.15846].

A further terminological complication is that the acronym “PEL” is not stable across subfields. One causal-inference paper explicitly uses “PEL” to mean *pseudo-empirical likelihood*, not *penalized empirical likelihood*, and contains no regularization penalty in the high-dimensional or shrinkage sense [2401.06919]. By contrast, the empirical-likelihood-based consistent information criterion ELCIC is adjacent to PEL because it evaluates candidate penalized fits, but it is not a penalized empirical-likelihood estimator [2006.13281].

## 2. Main penalization architectures

The literature contains several recurring penalty placements. Their differences are structural rather than cosmetic.

| Penalization target | Representative criterion | Main role |
|---|---|---|
| Moment discrepancy | \(L_n(\mu)=\sup_{\pi\in\Pi_n}\left\{\left(\prod_i\pi_i\right)\exp\!\left(-\lambda\sum_j\delta_j\left[\sum_i\pi_i(X_{ij}-\mu_j)\right]^2\right)\right\}\) | High-dimensional mean inference |
| Model parameter \(\theta\) | \(\tilde\theta=\arg\max_\theta\{l(\theta)-\tfrac12\log \sigma^2(\theta)\}\) | Bias reduction |
| Parameters and multipliers | \(\arg\min_\theta\max_\lambda\left[\sum_i\log\{1+\lambda^\top g_i(\theta)\}-n\sum_jP_{2,\nu}(|\lambda_j|)+n\sum_kP_{1,\pi}(|\theta_k|)\right]\) | Variable and moment selection |
| Multipliers only | \(\arg\min_\theta\max_\lambda\left[\sum_i\log\{1+\lambda^\top g_i(\theta)\}-n\sum_jP_\nu(|\lambda_j|)\right]\) | Many-moment dimension reduction |
| Target parameter \(N\) | \(\ell_p(N,\beta,\alpha)=\ell_e(N,\beta,\alpha)+C f(N)\) | Stabilization in capture–recapture |
| Propensity parameter \(\beta\) | \(\max_{\mathbf p\in\mathcal P_n}\sum_i\log p_i-\lambda\|\beta\|_1\) subject to EL constraints | CBPS-based causal weighting |

The 2013 high-dimensional mean paper is distinctive because it does not impose the exact empirical-likelihood moment constraint at all. Instead it replaces the hard feasibility condition by a variance-normalized quadratic penalty on the weighted discrepancy of the sample mean from \(\mu\). This makes the criterion well-defined even when \(p>n/2\), precisely the regime in which the convex-hull condition for ordinary empirical likelihood can fail with positive probability [1302.3071].

By contrast, the double-penalty estimating-equation literature keeps the usual EL dual form and adds penalties to \(\theta\), to \(\lambda\), or to both. The representative formulation is
\[
\hat\theta_n = \arg\min_{\theta\in\Theta} \max_{\lambda\in\widehat\Lambda_n(\theta)} \left[ \sum_{i=1}^n \log\{1+\lambda^\top g(X_i;\theta)\} - n\sum_{j=1}^r P_{2,\nu}(|\lambda_j|) + n\sum_{k=1}^p P_{1,\pi}(|\theta_k|) \right],
\]
with \(P_{1,\pi}\) inducing sparsity in \(\theta\) and \(P_{2,\nu}\) inducing sparsity in the active estimating equations through \(\lambda\) [1704.00566].

The bias-reduction paper uses a different logic. It derives an empirical-likelihood prior
\[
\pi(\theta)\propto \{\sigma^2(\theta)\}^{-1/2}, \qquad 
\sigma^2(\theta)=\frac{E\{G(X,\theta)\}^2}{[E\{G'(X,\theta)\}]^2},
\]
and then defines the penalized estimator
\[
\tilde\theta=\arg\max_\theta \exp\{l(\theta)\}\pi(\theta)
=\arg\max_\theta\left\{l(\theta)-\frac12\log \sigma^2(\theta)\right\}.
\]
Here the penalty is not a sparsity device; it is a Jeffreys-type information correction aimed at reducing the first-order asymptotic bias of the maximum empirical-likelihood estimator [1808.06222].

## 3. High-dimensional sparse estimation and moment selection

The most developed PEL strand is the high-dimensional estimating-equation literature. Its central claim is that the main obstacle in empirical likelihood is often not only the parameter dimension \(p\), but also the moment dimension \(r\), because the dual variable \(\lambda\) is \(r\)-dimensional [1704.00566]. This perspective led to the “new scope” in which a sparse \(\theta_0\) is estimated using a sparse subset of informative moments, rather than treating all \(r\) moments as equally active.

A key insight is that sparsity in \(\lambda\) is equivalent to sparsity in the set of active estimating equations. In the 2017 double-penalty framework, for fixed \(\theta\),
\[
f(\lambda;\theta)=\frac1n\sum_{i=1}^n \log\{1+\lambda^\top g(X_i;\theta)\}
-\sum_{j=1}^r P_{2,\nu}(|\lambda_j|),
\]
and the inner optimizer \(\hat\lambda(\theta)\) selects only a thresholded subset of moments. The support of \(\hat\lambda(\theta)\) is shown, with high probability, to be contained in a set of coordinates with sufficiently large empirical discrepancies \(|\bar g_j(\theta)|\) [1704.00566]. This provides a direct moment-selection interpretation of the multiplier penalty.

The theoretical payoff is substantial. Under suitable conditions, including \(\log r=o(n^{1/3})\), the estimator is sparse and consistent, with
\[
P(\hat\theta_{n,\mathcal S^c}=0)\to 1,
\]
and after a bias correction \(\hat\psi_{\mathcal R_n}\), the nonzero coordinates satisfy
\[
n^{1/2}\alpha^\top \hat J_{\mathcal R_n}^{1/2}
\left(\hat\theta_{n,\mathcal S}-\theta_{0,\mathcal S}-\hat\psi_{\mathcal R_n}\right)
\overset{d}{\longrightarrow} N(0,1)
\]
for any unit vector \(\alpha\in\mathbb R^s\) [1704.00566]. The same paper emphasizes that both \(p\) and \(r\) may grow exponentially with \(n\) under the stated rates.

The 2021 paper “Culling the herd of moments with penalized empirical likelihood” recasts possible moment invalidity as a sparse nuisance-parameter problem. It introduces an auxiliary vector \(\xi\) with components
\[
\xi_{0,k}=E\{g_{i,k}^{(\mathcal D)}(\theta_0)\},
\]
so that valid moments correspond to \(\xi_{0,k}=0\) and invalid moments to \(\xi_{0,k}\neq 0\). The proposed PEL criterion penalizes both \(\xi\) and the dual variables \(\lambda_j\) for candidate moments,
\[
(\hat\theta_{\rm PEL}^\top,\hat\xi_{\rm PEL}^\top)^\top
=\arg\min_{\psi\in\Psi}\max_{\lambda\in\hat\Lambda_n^{\mathcal T}(\psi)}
\left[
\frac1n\sum_{i=1}^n\log\{1+\lambda^\top g_i^{(\mathcal T)}(\psi)\}
-\sum_{j\in\mathcal D}P_{2,\nu}(|\lambda_j|)
+\sum_{k\in\mathcal D}P_{1,\pi}(|\xi_k|)
\right].
\]
This yields consistent detection of invalid moments through
\[
\hat{\mathcal A}=\{k\in\mathcal D:\hat\xi_k=0\},
\qquad
\mathbb P(\hat{\mathcal A}=\mathcal A)\to 1,
\]
and an oracle-type asymptotic theory after bias correction [2108.03382].

The 2013 mean-inference paper occupies a different place within high-dimensional PEL. Its target is the \(p\)-dimensional population mean, and the emphasis is not sparse selection but asymptotic behavior under component-wise dependence structures. The paper derives different limit laws for the PEL ratio statistic under non-Ergodic, long-range dependence, and short-range dependence, showing that the limit is typically not the classical fixed-dimensional \(\chi^2\) law. It also establishes a unified subsampling calibration valid across those dependence regimes [1302.3071].

## 4. Bias reduction, robustness, and Bayesian computation

Bias reduction enters PEL through the empirical-likelihood prior of Vexler, Zou, and Hutson. Starting from an integrated Kullback–Leibler or Shannon mutual information criterion,
\[
I_n(\pi)=\int \pi(\theta)\, E\!\left[\log\!\left\{\frac{\pi_e(\theta\mid X)}{\pi(\theta)}\right\}\right]\,d\theta,
\]
the paper shows that the asymptotically optimal prior is
\[
\pi(\theta)\propto \{\sigma^2(\theta)\}^{-1/2}.
\]
The resulting penalized empirical likelihood removes the \(E\{GG'\}\)-component of the \(O(n^{-1})\) bias of the ordinary maximum empirical-likelihood estimator, and when \(E\{G''(X_1,\theta_0)\}=0\), the first-order bias vanishes entirely [1808.06222]. This PEL variant is therefore closest in spirit to Firth-type penalization, not to sparse regularization.

Robustness is the defining feature of the high-dimensional longitudinal PEL developed for marginal models with repeated measurements. That paper replaces the ordinary QIF-style estimating equations by bounded robust estimating functions
\[
g(X_i;\beta)=
\begin{pmatrix}
D_i^T A_i^{-1/2} M_1 h_i(\mu_i(\beta))\\
\vdots\\
D_i^T A_i^{-1/2} M_\ell h_i(\mu_i(\beta))
\end{pmatrix},
\qquad
h_i(\mu_i)=W_i[\psi(Y_i-\mu_i(\beta))-C_i(\mu_i(\beta))],
\]
and combines them with penalties on both \(\lambda\) and \(\beta\):
\[
S_n^\star(\beta)=
\sum_{i=1}^n \log_\star\{1+\lambda^T g(X_i;\beta)\}
- n\sum_{j=1}^r P_{1,\nu}(|\lambda_j|)
+ n\sum_{k=1}^p P_{2,\omega}(|\beta_k|).
\]
The concrete variants ERPEL, HRPEL, and TRPEL correspond to exponential, Huber, and Tukey robust score functions. The paper argues that robustness in the estimating equations improves estimating-equation selection and therefore improves variable selection under heavy tails and contamination. Its formal robustness measure is a bounded influence function [2103.10613].

Bayesian penalized empirical likelihood, or BPEL, shifts the focus from point estimation to posterior computation. The defining estimator penalizes the dual variables only,
\[
\hat\theta_n = \arg\min_{\theta\in\Theta} \max_{\lambda\in\hat\Lambda_n(\theta)}
\left[
\sum_{i=1}^n\log\{1+\lambda^\top g(X_i;\theta)\}
- n\sum_{j=1}^r P_\nu(|\lambda_j|)
\right],
\]
and the corresponding posterior is
\[
\pi^\dagger(\theta\mid\mathcal X_n)\propto \pi_0(\theta)\,{\rm PEL}_\nu(\theta)\,I(\theta\in\Theta).
\]
The penalty creates sparse multiplier vectors and hence automatic moment selection, while the Bayesian layer replaces difficult global optimization over \(\theta\) by sampling. The paper develops both Metropolis–Hastings and a modified adaptive multiple importance sampler (MAMIS), proves convergence of both schemes, and establishes a Bernstein–von Mises theorem:
\[
\mathcal D_{\rm TV}\!\left( \Pi_n^\dagger,\,
\mathcal N\big(\hat\theta_n,\ n^{-1}\widehat V_{\mathcal R_n}^{-1}\big) \right)\to 0
\quad\text{in probability.}
\]
The posterior is asymptotically Gaussian around the penalized estimator, not directly around \(\theta_0\), reflecting the bias induced by the penalty [2412.17354].

## 5. Domain-specific formulations and applications

One of the most problem-specific PEL constructions arises in closed-population capture–recapture. In that setting the instability concerns the abundance parameter \(N\), especially when the capture probability is low or the empirical-likelihood profile in \(N\) has a flat right tail. The proposed penalized log empirical likelihood is
\[
\ell_p(N,\beta,\alpha)=\ell_e(N,\beta,\alpha)+C f(N),
\]
with the recommended one-sided quadratic penalty
\[
f(N)=-(N-\widetilde N_c)^2 I(N>\widetilde N_c),
\qquad
\widetilde N_c=n+\frac{m_1^2}{2m_2}.
\]
This shrinks overly large EL abundance estimates toward Chao’s lower bound without penalizing smaller values. The paper proves asymptotic normality of the maximum PEL estimator and Wilks-type limits for the PEL ratio statistic, and it develops standard EM algorithms for PEL, EL, and CL. In a difficult scenario B with \(K=2\) and \(N_0=200\), the reported RMSEs were
\[
\text{RMSE}(\widehat N_v)=70302,\quad
\text{RMSE}(\widehat N_c)=10271,\quad
\text{RMSE}(\widehat N_e)=726,\quad
\text{RMSE}(\widehat N_p)=44,
\]
which the paper presents as direct evidence of stabilization [2208.05635].

A recent causal paper integrates generalized propensity score, covariate balancing propensity score, and outcome regression into a PEL formulation for continuous outcomes under multiple treatment levels. Its defining treatment-side optimization is
\[
\underset{\mathbf p\in\mathcal P_n}{\text{maximize}}
\quad \sum_{i=1}^n \log p_i-\lambda\|\beta\|_1,
\]
subject to
\[
\sum_{i=1}^n p_i\,g(Y_i,A_i,X_i;\beta)=0,\qquad
\sum_{i=1}^n p_i=1,\qquad
p_i>0.
\]
The paper states that \(g(\cdot)\) encodes CBPS and outcome-regression moment conditions, and it gives the explicit CBPS balance restriction
\[
\frac{1}{n}\sum_{i=1}^n
\left(
\frac{A_i-\pi_{\beta}(X_i)}{\pi_{\beta}(X_i)\{1-\pi_{\beta}(X_i)\}}X_i
\right)=0.
\]
Its stated goals are covariate balance, high-dimensional regularization, and support for a doubly robust estimator [2509.15846].

That causal paper also illustrates a recurring limitation in applied PEL work: the conceptual framework can be broader than the displayed formulas. It claims accommodation of multi-valued treatments, yet the explicit propensity-score and CBPS equations are written in binary-treatment form. It also claims double robustness, finite-sample validity, and outlier resistance, but the supplied manuscript excerpt does not provide theorem statements for consistency of the PEL treatment-model estimator, oracle properties, or asymptotic normality [2509.15846].

## 6. Adjacent literature, misconceptions, and unresolved issues

A persistent misconception is that any empirical-likelihood procedure attached to penalized or sparse fits is automatically PEL. The 2020 empirical-likelihood-based consistent information criterion is an instructive counterexample. ELCIC is
\[
\mathrm{ELCIC}=-2\log R^F(\hat\gamma_{EE},\hat\lambda_{EE})+p\log n,
\]
and is designed for model selection among candidate models, including models generated by penalized procedures such as PGEE with SCAD. Its relevance to PEL is indirect: it uses empirical likelihood to score candidate penalized fits, but it does not formulate or analyze a penalized empirical-likelihood estimator [2006.13281].

A second misconception concerns acronym reuse. The 2024 causal-inference paper on average treatment effects is explicitly about *pseudo*-empirical likelihood. Its PEL objective uses estimated propensity-score weights and calibration constraints, and the resulting maximum PEL estimator is equivalent to a Hájek-type IPW estimator. There is no regularization penalty on parameters, multipliers, or moments, so it belongs to a different methodological lineage despite the shared acronym [2401.06919].

Several open issues recur across the genuine PEL literature. First, the field lacks a single canonical penalty geometry. Penalties have been placed on \(\theta\), on \(\lambda\), on moment discrepancies, on nuisance misspecification vectors such as \(\xi\), and on scientific targets such as \(N\). This suggests that “PEL” is better understood as a design principle than as a single estimator class [1704.00566, 2108.03382, 2208.05635]. Second, multiplier penalization typically introduces asymptotic bias, which then requires explicit correction, projected inference, or posterior centering around the penalized estimator rather than the truth [1704.00566, 2108.03382, 2412.17354]. Third, tuning remains problem-dependent: the literature uses BIC-type criteria, theoretical rate conditions, or ad hoc simulation-scale choices, but no universal rule emerges from the cited papers [1704.00566, 2108.03382, 2412.17354].

The resulting picture is technically coherent but heterogeneous. Penalized empirical likelihood has evolved into a broad toolkit for semiparametric and estimating-equation problems: sparse estimation in high dimensions, selection among many moments, invalid-moment screening, bias reduction, robust longitudinal analysis, Bayesian sampling, abundance stabilization, and treatment-model regularization. Its unifying idea is not a particular penalty form, but the use of regularization to make empirical-likelihood methods workable outside the classical low-dimensional, well-specified regime [1302.3071, 1704.00566, 1808.06222, 2412.17354].

Source: https://www.emergentmind.com/topics/penalized-empirical-likelihood-pel