Parameter-Expanded EM (PX-EM)

• Parameter-Expanded EM (PX-EM) is an extension of the EM algorithm that augments the parameter space with auxiliary parameters to achieve faster and more stable convergence.
• It uses a systematic expansion and reduction mechanism to preserve the observed-data likelihood while enabling implicit bias correction and acceleration of convergence.
• PX-EM has wide applications in latent variable models, mixture models, and penalized logistic regression, offering robust optimization for complex, incomplete data scenarios.

Parameter-Expanded EM (PX-EM) is an extension of the standard Expectation-Maximization (EM) algorithm for maximum likelihood estimation in the presence of incomplete data. PX-EM augments the original parameter space with auxiliary expansion parameters, enabling faster and more stable convergence while preserving the observed-data likelihood and the monotonicity property of EM. The core idea is to embed the complete-data model into a larger family parameterized by both the original and expanded parameters, coupled with a reduction mapping that projects solutions back to the original parameter space.

1. Mathematical Framework and Definition

In standard EM, the observed-data $X_{\rm obs}$ and missing-data $X_{\rm mis}$ jointly specify a complete-data model $g(X_{\rm obs}, X_{\rm mis}; \theta)$ for $\theta \in \Theta$. The EM algorithm iteratively computes the expected complete-data log-likelihood (Q-function) and maximizes it to update $\theta$. PX-EM generalizes this setup by:

• Constructing an expanded model $g_*(X_{\rm obs}, X_{\rm mis}; \theta_*, \alpha)$, where $(\theta_*, \alpha) \in \Theta \times A$,
• Ensuring the observed-data likelihood is preserved: $$f_*(X_{\rm obs}; \theta_*, \alpha) = \int g_*(X_{\rm obs}, X_{\rm mis}; \theta_*, \alpha)\, dX_{\rm mis} \equiv f(X_{\rm obs}; \theta)$$,
• Defining a reduction mapping $R: \Theta \times A \rightarrow \Theta$ by $\theta = R(\theta_*, \alpha)$, with $R(\theta, \alpha_0) = \theta$ at a fixed null value $\alpha_0$.

The PX-EM iteration consists of:

• PX-E-step: Compute $Q(\theta_*, \alpha \mid \theta^{(t)}, \alpha_0)$ by expectation under the imputation model $(\theta^{(t)}, \alpha_0)$,
• PX-M-step: Jointly maximize $Q(\theta_*, \alpha \mid \theta^{(t)}, \alpha_0)$ over $(\theta_*, \alpha)$,
• Reduction: Map $(\theta_*^{(t+1)}, \alpha^{(t+1)})$ back via $$\theta^{(t+1)} = R(\theta_*^{(t+1)}, \alpha^{(t+1)})$$, maintaining the essential monotonicity and simplicity of EM (Lewandowski et al., 2011).

2. Statistical Interpretation and Theoretical Properties

PX-EM’s theoretical appeal centers on monotonicity, convergence acceleration, and bias reduction:

• Monotonicity: Under standard regularity, $\ell(\theta^{(t+1)}) \geq \ell(\theta^{(t)})$ for the observed-data log-likelihood.
• Convergence rate: Let $\rho_{\rm EM}$ and $\rho_{\rm PX}$ denote the local linear rates for EM and PX-EM. PX-EM satisfies $\rho_{\rm PX} \leq \rho_{\rm EM}$ and, in many cases, achieves much faster convergence (Lewandowski et al., 2011).
• Bias correction: The expansion parameters allow the PX-M-step to adjust for bias from the imputation model used in EM, yielding an implicit covariance adjustment. The Fisher information matrix in the expanded space has reduced missing-data information, making the update closer to Newton-Raphson. This perspective relates directly to efficient inference principles.

3. Illustrative Examples of PX-EM

PX-EM’s mechanics are best demonstrated in canonical examples:

Poisson–Binomial Toy Model

• Original model: $Z \sim \text{Poisson}(\lambda)$; $X|Z \sim \text{Binomial}(Z, \pi)$; $X \sim \text{Poisson}(\lambda \pi)$.
• PX-EM expansion: Replace $\pi$ with $\alpha$; $X|Z \sim \text{Binomial}(Z, \alpha)$; $Z \sim \text{Poisson}(\lambda_*)$; $X \sim \text{Poisson}(\lambda_* \alpha)$.
• Reduction: $\lambda = (\alpha / \pi) \lambda_*$; null value $\alpha_0 = \pi$.
• Performance: PX-EM may achieve one-step convergence, compared to much slower EM at rate $1-\pi$.

Robit Regression

• Original latent-variable EM: $z_i|\tau_i \sim N(x_i'\beta, 1/\tau_i)$; $\tau_i \sim \Gamma(\nu/2, \nu/2)$; $y_i = \mathbf{1}\{z_i > 0\}$.
• Expanded model: $(\tau_i/\alpha) \sim \Gamma(\nu/2, \nu/2)$; $z_i|\tau_i \sim N(x_i'\beta_*, \sigma^2/\tau_i)$.
• Reduction: $\beta = (\sqrt{\alpha}/\sigma)\beta_*$; null $(\alpha_0,\sigma_0) = (1, 1)$.
• Empirical behavior: PX-EM converges in 10–20 iterations versus 200–300 for EM in vaso-constriction data.

These examples manifest the dramatic gains in convergence from parameter expansion (Lewandowski et al., 2011).

4. PX-EM and Over-Parameterization in Mixture Models

Parameter expansion can be interpreted as a form of over-parameterization, applicable even when expansion parameters are statistically redundant. In Gaussian mixture models:

• Original EM: Estimates means $\theta$ with known mixture weights $w^*$.
• Over-parameterized/PX-EM: Treats $w$ as free parameters, expanding the domain to $(\theta, w)$.
• PX-EM steps: E-step computes responsibilities using $(\theta^{(t)}, w^{(t)})$; M-step updates both; reduction discards the auxiliary $w$.

In the symmetric two-component case, PX-EM provably avoids spurious local optima and converges globally from nearly any initialization. For general mixtures (including higher dimensions and sample sizes), empirical results show much higher success rate in finding the global maximum versus standard EM (Xu et al., 2018).

The redundancy introduced by $w$ “smooths” the likelihood, converting spurious maxima to saddle directions and imparting permutation invariance. This suggests that parameter expansion in EM has broad utility for solving non-convex optimization problems in latent variable models.

5. Parameter Expansion in Logistic and Penalized Logistic Regression

The parameter expansion principle extends to monotone optimization methods for logistic regression, including EM, MM, and variational Bayes. For Polya-Gamma-augmented logistic regression:

• Parameter expansion: Expand the regression parameter $\beta \in \mathbb{R}^p$ to $(\theta, \alpha)$, $\beta = \alpha \theta$.
• Complete-data model: $$y_i | W_i, (\theta, \alpha) \sim \text{Binomial}(m_i, \mathrm{expit}(\alpha x_i^\top \theta))$$; $$W_i | (\theta, \alpha) \sim \text{PG}(m_i, \alpha x_i^\top \theta)$$.
• Expanded Q-function: Includes arbitrary penalty functions $P_\eta(\cdot)$.
• Algorithmic structure: E-step estimates expectations; M-step updates $\theta$ via weighted least squares; C-step or line-search updates $\alpha$; reduction combines into $\beta = \alpha \theta$.
• Guaranteed monotonicity: If each substep increases its target, the overall iteration maintains monotonicity in penalized log-likelihood.
• Rate improvement: Spectral radius of PX-ECME Jacobian is no larger than EM, and practical convergence is often one or two orders of magnitude faster.

Further, generalized PX-ECME algorithms can nest EM, MM, proximal-gradient, and their PX counterparts based on surrogate Q-functions and curvature adjustments (Henderson et al., 2023).

6. Connections, Generalizations, and Relationships

Parameter expansion is not restricted to EM. It links efficiently with broader iterative optimization frameworks:

• Minorization-Maximization (MM): Expanded surrogates or curvature adjustments connect MM schemes to PX-EM.
• Variational approaches: Polya-Gamma EM with fixed $\alpha$ coincides with Jaakkola-Jordan variational Bayes reweighted least squares.
• Generalized PX-ECME: By selecting appropriate surrogate Q-functions, PX-EM algorithms can be tailored for high-dimensional and penalized settings, inheriting both monotonicity and acceleration while retaining per-iteration simplicity.
• Newton-Raphson: PX-EM steps resemble Newton-Raphson near the optimum due to the covariance adjustment in the augmented Fisher information.

This suggests a broad foundational role for parameter expansion across monotone iterative algorithms in statistical inference.

7. Applications and Impact

PX-EM and its generalizations have practical impact in:

• Accelerating inference for latent variable models (e.g., Gaussian mixtures, regression with random effects)
• Reliable optimization in non-convex likelihood scenarios
• Enhancing monotonicity and robustness in penalized and weighted estimation
• Practical computational improvements, with PX-EM and PX-ECME often running ten times faster than EM while preserving stability (Henderson et al., 2023)

Parameter expansion thus directly influences methodology in large-scale statistical inference, machine learning, and empirical Bayes, connecting efficient computation with bias reduction and theoretical guarantees of convergence (Lewandowski et al., 2011, Xu et al., 2018, Henderson et al., 2023).