Maximum Likelihood Estimation (MLE)

• Maximum Likelihood Estimation (MLE) is a method for estimating model parameters by maximizing the likelihood function based on observed data.
• MLE’s theoretical efficiency, invariance, and asymptotic normality provide robust guidelines for applications in Gaussian, high-dimensional, and nonparametric models.
• Practical MLE implementations range from closed-form solutions and EM algorithms to algebraic and convex optimization methods in latent variable and mixture models.

Maximum Likelihood Estimate (MLE) is a foundational methodology in statistical inference for estimating parameters of probabilistic models. Given a parametric statistical model and observed data, the MLE is the value of the parameter that maximizes the likelihood function, that is, the probability (or probability density) of the observed data as a function of the parameters. MLE underpins a vast range of modern data analysis, from classical Gaussian inference, through high-dimensional models, to nonparametric and algebraic statistics. Its theoretical tractability, efficiency, and universal applicability make it central in both probability theory and mathematical statistics.

1. Definition and Mathematical Framework

Given data $Y = (Y_1,\dots,Y_n)$ generated independently from a distribution parametrized by $\theta \in \Theta$, with density or mass function $p_\theta(y)$, the likelihood function is $L(\theta;Y) = \prod_{i=1}^n p_\theta(Y_i)$. The MLE is defined as

$$\hat{\theta}_{\mathrm{MLE}} = \arg\max_{\theta \in \Theta} L(\theta;Y)$$

or, equivalently, maximizing the log-likelihood $\ell(\theta;Y) = \sum_{i=1}^n \log p_\theta(Y_i)$ (Vella, 2018). The score function is the gradient $$s(\theta;Y) = \partial \ell(\theta;Y)/\partial \theta$$, and the Fisher information matrix is

$$I(\theta) = \mathbb{E}_{Y \sim p_\theta}\left[ s(\theta;Y) s(\theta;Y)^T \right]$$

MLE is invariant under one-to-one reparameterizations; if $\phi = g(\theta)$, maximizing $L(\theta)$ over $\theta$ yields the same estimate in $\phi$-space (Vella, 2018, Ramos et al., 2021). For regular models, the asymptotic distribution of the MLE is multivariate normal, centered at the true parameter, with covariance $I(\theta)^{-1}/n$.

2. Existence, Uniqueness, and Phase Transitions

The existence and uniqueness of an MLE are not universally guaranteed and depend on both the model and observed data. In high-dimensional logistic regression with Gaussian covariates, there is a sharp phase transition for the existence of the MLE: for $p/n \to \kappa$, there exists a model-specific boundary $h_{\mathrm{MLE}}$ such that if $\kappa > h_{\mathrm{MLE}}$, the MLE does not exist with high probability, and if $\kappa < h_{\mathrm{MLE}}$, it exists almost surely (Candes et al., 2018). The boundary is characterized via a minimization over expected squared positive parts involving auxiliary random variables, demarcating regimes where separation precludes or guarantees the existence of a maximizing parameter.

For matrix normal models, boundedness, existence, and uniqueness of the MLE can be decided explicitly in terms of the sample size $m$, dimensions $p, q$, and their greatest common divisor, using results from quiver representation theory (Derksen et al., 2020). Specifically, for the density

$$\mathrm{vec}(Y) \sim \mathcal{N}_{pq}(\mathrm{vec}(M), \Sigma_2 \otimes \Sigma_1),$$

there exist sharp thresholds for $m$ that ensure, almost surely: (i) bounded likelihood, (ii) existence of an MLE, (iii) uniqueness of the MLE.

In nonparametric settings, e.g., population estimation of Bernoulli probabilities from pooled binomial data, the MLE is always well-defined as a solution to a convex program over the space of mixing distributions supported on a finite set; Carathéodory's theorem yields discrete support with at most $t+1$ atoms (Vinayak et al., 2019).

3. Optimality and Distributional Properties

MLE exhibits optimality properties both in the classical and a distributional sense:

• Among all unbiased estimators within exponential families, the MLE uniquely minimizes the Kullback-Leibler (KL) variance, i.e., it is the uniformly minimum distribution variance unbiased (UMV$^{\dagger}$U) estimator (Vos et al., 2015). Here, estimators are viewed as random probability measures, and risk is measured via expected KL divergence.
• Distributional analogs of classical results (Rao-Blackwell theorem) hold: conditional estimators (conditioned on sufficient statistics) possess smaller or equal distributional variance, and the MLE arises as the projected, Rao-Blackwellized estimator.
• Robustness persists: even when the true data-generating distribution is not in the model family, the MLE remains distribution-unbiased for the KL-projection of the true law onto the model, provided the mean of the sufficient statistic is preserved (Vos et al., 2015).

4. Computational and Algorithmic Aspects

MLE computation varies substantially across models and data structures.

Latent Variable Models and Monotonicity

For models with latent variables, a new inequality establishes a necessary and sufficient criterion for strict improvement of the observed-data likelihood by comparing two candidate parameter values via integrals of truncated likelihood ratios over the posterior distributions of the latent variables (Olsen, 2019): $$L(\theta_{1}) < L(\theta_{2}) \iff \int_{\mathcal W} \min\!\left(1,\frac{L(\theta_{2},w)}{L(\theta_{1},w)}\right) dP_{\theta_1}(w|y) > \int_{\mathcal W} \min\!\left(1,\frac{L(\theta_{1},w)}{L(\theta_{2},w)}\right) dP_{\theta_2}(w|y)$$ This result provides a likelihood-increasing acceptance test, generalizing the EM-algorithm's monotonicity property and permitting arbitrary parameter proposals in global optimization schemes.

Nonparametric and Finite Mixture Models

MLE for finite mixtures, nonparametric populations, or models with discrete structure often takes the form of an infinite-dimensional convex program. For population estimation from binomial data, the log-likelihood is convex as a function on the space of distributions, and the optimizer can be sought over a finite-dimensional simplex using support-reduction algorithms (Vinayak et al., 2019).

Algebraic Geometry Approaches

MLE for discrete models with polynomial constraints benefits from dual algebraic reformulations. By working with dual varieties and the conormal variety, the MLE problem translates into solving the "dual likelihood equations," enabling solution without explicit factorization or elimination of the original model constraints (Rodriguez, 2014). This dual approach reduces symbolic algebraic complexity and facilitates computation of ML degrees for complex statistical models.

Closed-form and Real-time Estimation

Generalized closed-form MLEs can be constructed by embedding the likelihood into a larger family with auxiliary parameters, under suitable conditions. Such estimators are computable by solving algebraic equations based on sample statistics and preserve strong consistency, asymptotic normality, and invariance (Ramos et al., 2021). This approach allows efficient MLE computation in models (Gamma, Beta, Nakagami, bivariate Gamma) relevant for streaming and real-time hardware, trading negligible bias for dramatic computational gains.

5. Applications Across Statistical Models

MLE constitutes the backbone of statistical inference in numerous specialized models:

• Optical and High-dimensional Data: In photon counting and spatial intensity estimation, MLE underpins the design of experiments and the quantification of uncertainty via the Fisher information matrix and Cramér–Rao bounds (Vella, 2018).
• Stochastic Processes: For Wishart processes, MLE yields the optimal convergence rate for drift parameter estimation in both ergodic and nonergodic regimes, with explicit limit laws and attainability of minimax lower bounds (Alfonsi et al., 2015).
• High-dimensional Regimes: In high-dimensional settings (e.g., when $p \gtrsim n$), phase transitions dictate regimes of MLE nonexistence, signaling fundamental limits on statistical inference unless regularization is employed (Candes et al., 2018).
• Mixture and Latent Models: MLE for Gaussian mixtures, latent variable models, and population estimation leverages posterior-based criteria and monotonic algorithms to achieve robust and computationally feasible estimates (Olsen, 2019, Vinayak et al., 2019).

6. Limitations, Extensions, and Theoretical Implications

MLE, despite its optimality under standard conditions, possesses limits:

• Nonexistence and Boundary Behavior: In finite samples, separation or parameter boundaries may prevent the MLE from existing as a finite maximizer, as elucidated in logistic regression and boundary cases in exponential families (Candes et al., 2018, Vos et al., 2015).
• Computational Complexity: For models with nonconvex likelihoods (e.g., coupled matrix normal covariances), the likelihood surface may have multiple local maxima, and global optimization is nontrivial (Derksen et al., 2020).
• Model Misspecification: Under model mismatch, the MLE estimates the "closest" distribution in the model class as measured by KL divergence (Vos et al., 2015). This phenomenon generalizes the classical notion of consistency to misspecified models.
• Algebraic and Coding Constraints: Algebraic manipulations in discrete models may become intractable as degrees increase; dual likelihood solutions bypass but do not eliminate all computational burdens (Rodriguez, 2014).

Advances in MLE theory—such as new monotonicity inequalities, duality principles, and explicit phase transition results—extend its applicability and deepen its foundational role in mathematical statistics, optimization, and data science.