Maximum Likelihood Estimates (MLEs)

• Maximum Likelihood Estimates (MLEs) are defined as the parameter values that maximize the likelihood function for observed data, ensuring statistical consistency and invariance.
• MLEs are computed via iterative optimization methods like Newton–Raphson and EM, which handle high-dimensional and latent variable models effectively.
• The framework extends to nonparametric and complex models, supporting techniques such as bootstrap uncertainty quantification and algebraic methods for robust inference.

Maximum Likelihood Estimates (MLEs) are central constructs in statistical inference, representing parameter values that maximize the likelihood function for observed data under a given probabilistic model. The MLE formalism is essential across parametric, latent variable, and high-dimensional statistical models, with fundamental implications for estimation theory, model selection, uncertainty quantification, and computational statistics.

1. Mathematical Definition and General Properties

Given observed data $x$ and a parametric model $\{P_\theta : \theta\in\Theta\}$, the likelihood function is $L(\theta|x) = P_\theta(x)$ (discrete case) or $f_\theta(x)$ (continuous case) (Vella, 2018). The MLE $\hat\theta$ is the maximizer

$$\hat\theta = \operatorname{argmax}_{\theta\in\Theta} L(\theta|x).$$

Typically, log-likelihood $\ell(\theta|x) = \log L(\theta|x)$ is used for computational tractability, as it converts products to sums and enhances numerical stability. MLEs inherit key invariance properties: if $\psi:\Theta\to\Lambda$ is a bijection, then the MLE for $\lambda=\psi(\theta)$ is $\psi(\hat\theta)$. Under standard regularity conditions, $\{P_\theta : \theta\in\Theta\}$0 is strongly consistent and asymptotically normal (Ramos et al., 2021).

2. Existence, Uniqueness, and Geometric Perspectives

Existence and uniqueness of the MLE are nontrivial, particularly in high-dimensional or structured models.

Exponential families and log-linear models: The MLE exists and is unique if and only if the sufficient statistic lies in the relative interior of the marginal cone generated by the design matrix (Fienberg et al., 2011). Structural zeros in observed data may expose faces of the marginal polytope, obstructing MLE existence; in such cases, the extended MLE is defined over the exposed face to maximize the likelihood.

Matrix normal models: The existence and uniqueness criteria are characterized by explicit algebraic thresholds. For the real or complex matrix normal model with $\{P_\theta : \theta\in\Theta\}$1 observations and $\{P_\theta : \theta\in\Theta\}$2 samples, the log-likelihood is bounded and the MLE exists uniquely almost surely if and only if $\{P_\theta : \theta\in\Theta\}$3; otherwise, no MLE exists (Derksen et al., 2020). These thresholds are derived using results from quiver representation theory (Kac, King, Schofield), connecting statistical estimation to invariant theory and geometric invariant theory stability conditions.

Algebraic statistics and ML degree: For algebraic models $\{P_\theta : \theta\in\Theta\}$4 (e.g., toric or log-linear models), the number of complex critical points for generic data—the ML degree—quantifies the algebraic complexity of the likelihood equations. Dual varieties and conormal varieties allow dual formulations of MLE, often simplifying solution of the likelihood equations and enabling numerical computation by homotopy continuation or Gröbner basis methods (Rodriguez, 2014, Améndola et al., 2020).

3. MLEs in Models with Latent Variables and the EM Paradigm

Latent-variable models pose distinct computational and theoretical challenges due to unobserved structure.

Olsen's inequality: For models with complete-data likelihood $\{P_\theta : \theta\in\Theta\}$5 and posterior $\{P_\theta : \theta\in\Theta\}$6 of the latent variable $\{P_\theta : \theta\in\Theta\}$7, Olsen (2024) introduces the criterion

$\{P_\theta : \theta\in\Theta\}$8

(Olsen, 2019). This condition, leveraging two different posteriors and truncated likelihood ratios, is both necessary and sufficient for likelihood improvement and generalizes the classic EM monotonicity principle. Unlike EM's one-sided, Jensen-based lower bound, Olsen's approach does not require differentiability, concavity, or the construction of global majorizers, and can be deployed in hybrid EM–Monte Carlo algorithms for likelihood maximization.

4. Computational Algorithms and Uncertainty Quantification

Iterative optimization: In most practical contexts, MLEs are computed by iterative numerical optimization (trust-region, quasi-Newton, Newton–Raphson, or EM). The solution generally requires evaluation of the gradient and Hessian of the log-likelihood. For models depending nonlinearly on parameters (e.g., radiometric calibration, mixture models), closed-form solutions are rare (Pintar et al., 2022).

Closed-form and generalized MLEs: The absence of a closed-form for the score equations can limit application in real-time and embedded settings. Generalized MLE methodology introduces an auxiliary family $\{P_\theta : \theta\in\Theta\}$9 and modifies the score equations so that, under regularity conditions, closed-form solutions emerge for distributions such as Gamma, Beta, and Nakagami-m. These estimators retain strong consistency, asymptotic normality, and invariance under smooth reparametrization (Ramos et al., 2021).

Bootstrap and Fisher information: The uncertainty of MLEs can be assessed via the Fisher information matrix—whose inverse yields the Cramér–Rao lower bound for covariance—or by nonparametric bootstrap methods. For instrument calibration problems, the non-parametric pairs bootstrap produces empirical estimates of variance and confidence intervals that demonstrate nominal coverage properties (90–97%) and negligible bias for typical signal strengths (Pintar et al., 2022).

5. Nonparametric and Population-Level MLEs

MLE methodology extends beyond finite-dimensional parametric inference.

Population mixture models: When learning the distribution $L(\theta|x) = P_\theta(x)$0 of individual-level parameters (e.g., $L(\theta|x) = P_\theta(x)$1 in binomial/Bernoulli models with limited per-individual trials), the MLE is formulated as a nonparametric convex optimization over mixture distributions. The population MLE achieves minimax-optimal rates in earth mover distance: $L(\theta|x) = P_\theta(x)$2 for $L(\theta|x) = P_\theta(x)$3 and $L(\theta|x) = P_\theta(x)$4 for $L(\theta|x) = P_\theta(x)$5 beyond that, significantly outperforming plug-in or moment-matching estimators in the sparse-data regime (Vinayak et al., 2019).

6. Applications and Extensions

MLEs are pivotal across application domains:

• Quantum and optical parameter estimation: MLEs provide optimal estimators for unknown parameters in photon-counting, spatially-resolved experiments. The Fisher information guides experiment design, as in off-null ellipsometry and quantum weak measurement scenarios (Vella, 2018).
• Gaussian graphical models: In high-dimensional models with structured covariance (e.g., Markov networks), MLE computation is converted to the vanishing of polynomial "score" equations, and the number of real maxima is bounded by the ML-degree, which is calculated exactly in algebraic packages such as GraphicalModelsMLE for Macaulay2 (Améndola et al., 2020).
• Log-linear and exponential family models: MLE theory underpins model selection, goodness-of-fit assessment, and the development of extended MLEs when the standard MLE does not exist due to sampling zeros or other structural constraints (Fienberg et al., 2011).
• Radiometric and sensor calibration: In instrument calibration, MLEs obtained from physically-motivated polynomial models with soft constraints support rigorous uncertainty quantification and demonstrate robust empirical coverage properties (Pintar et al., 2022).

7. Theoretical and Computational Challenges

Contemporary research in MLE theory confronts several open questions:

• The precise conditions for MLE existence in non-regular models and the geometry of extended parametric spaces.
• Efficient and robust algorithms for likelihood maximization in large-scale and algebraically complex models (e.g., via dual varieties, convex relaxations, or homotopy methods).
• The integration of Monte Carlo methods with MLE optimization for intractable likelihoods in latent-variable or generative modeling frameworks.
• The use of MLEs in population estimation and high-dimensional inference, especially in the context of privacy, regularization, and model misspecification.

Ongoing advances in algebraic, computational, and nonparametric statistics continue to expand the scope and applicability of MLEs across statistical science.