Maximum-Likelihood Fitting Methods

• Maximum-likelihood fitting methods are statistical techniques that estimate model parameters by maximizing the likelihood function, ensuring asymptotic unbiasedness and efficiency.
• They are widely used across disciplines such as astronomy, econometrics, and machine learning, effectively handling various data types and noise models.
• Practical implementations involve gradient-based optimizers, Monte Carlo simulations, and specialized routines to address challenges like non-Gaussian noise and high-dimensionality.

Maximum-likelihood (ML) fitting methods constitute a set of statistical techniques for estimating parameters of parametric models by maximizing the likelihood function—that is, the probability density (or mass) of the observed data as a function of the model parameters. These methods underpin a vast array of procedures across experimental sciences, astronomy, econometrics, machine learning, and signal processing. ML fitting guarantees key properties under regularity conditions: asymptotic unbiasedness, efficiency (variance attains the Cramér–Rao lower bound), and invariance under reparameterization. The methodology accommodates a range of data-generating scenarios, statistical noise models, and computational challenges, including latent variables, hierarchical models, non-Gaussian noise, and high dimensionality.

1. Mathematical Framework and Canonical Forms

At the core is the construction of the likelihood $\mathcal{L}(\theta)$ as a function of parameters $\theta$ given data $\mathcal{D}$. For independently observed data points $x_i$, with probability density or mass function $f(x_i|\theta)$, the likelihood is:

$$\mathcal{L}(\theta) = \prod_{i} f(x_i|\theta), \quad \ell(\theta) = \log \mathcal{L}(\theta).$$

Parameter estimation is performed by maximizing $\ell(\theta)$ or, equivalently, minimizing $-2\ell(\theta)$ ("deviance" or "fit statistic" in some domains). In non-Gaussian problems (e.g., Poisson or exponential noise, censored or binned data, or latent-variable models), the explicit form of $f(x|\theta)$ is dictated by the process and structure of the observed data (Fowler, 2013, Barret et al., 2011, Yu et al., 2023).

The score function and observed Fisher information,

$$\mathcal{S}_j(\theta) = \frac{\partial \ell(\theta)}{\partial \theta_j}, \qquad \mathcal{I}_{jk}(\theta) = - \mathrm{E}\left[ \frac{\partial^2 \ell(\theta)}{\partial \theta_j \partial \theta_k} \right],$$

serve as the basis for iterative maximization, uncertainty quantification, and model assessment.

2. Maximum-Likelihood Fitting in Practice

The practical implementation revolves around constructing the likelihood appropriate for data structure and noise statistics:

• Unbinned and binned data: For individually observed events, unbinned likelihoods are preferable, maximizing statistical power and eliminating information loss due to binning. For histograms or discrete event counts subject to Poisson fluctuations, the exact Poisson likelihood is recommended over $\chi^2$-type approximations, especially when event counts per bin are small (Fowler, 2013, Parada et al., 2023).
• Weighted and efficiency-corrected fits: When experimental observations are subject to non-uniform efficiencies, the event-by-event likelihood must be weighted accordingly,

$$\ell(\theta) = \sum_i \ln \epsilon(x_i) + \ln f(x_i|\theta) - N \ln \int \epsilon(x)f(x|\theta)\,dx,$$

yielding unbiased MLEs when the efficiency map is known (Yu et al., 2023).

• Multivariate and latent-structure models: For data with latent variables or hierarchical structure, the likelihood involves integration or marginalization over unobserved quantities. For tractable cases, likelihood maximization can proceed directly; otherwise, expectation-maximization (EM), h-likelihood joint maximization, or stochastic approximations (e.g., Monte Carlo, particle methods) are used (Han et al., 2022, Lim et al., 2023).
• Mixed models and random effects: Simulated maximum-likelihood, using importance or quasi-random draws from the distribution of individual-level coefficients, becomes crucial for mixed logit and mixed regret minimization models (Zhu et al., 2023). The likelihood is then estimated by Monte Carlo numerical quadrature at each iteration of parameter optimization.

3. Algorithms, Optimization, and Computational Considerations

Standard optimization for ML fitting leverages the form of the likelihood and the structure of the parameter space:

• Gradient-based optimizers (Newton–Raphson, BFGS, conjugate gradient): Used when analytic or numerically stable first and (sometimes) second derivatives are available. For example, the SFit method computes a Gauss–Newton approximation to the Hessian for $\chi^2$ minimization, delivering robust, positive-definite covariance estimation (Yee et al., 6 Feb 2025).
• Gauss–Newton and Levenberg–Marquardt: Extensively used to adapt least-squares procedures to likelihood contexts—either by changing the objective function to the Poisson or exponential likelihood, or by replacing residuals and weights in standard routines (Fowler, 2013, Yee et al., 6 Feb 2025).
• Monte Carlo and stochastic optimization: For likelihoods involving high-dimensional integrals (e.g., random coefficients or latent variables), simulated maximum likelihood employs draws from the coefficient or latent-variable distribution, using quasi-Monte Carlo sequences for variance reduction (Zhu et al., 2023, Lim et al., 2023).
• Markov-Chain Monte Carlo (MCMC): When parameter posteriors are high-dimensional or non-convex, MCMC (e.g., Metropolis–Hastings with small-world proposals and simulated annealing) can be used to map out credible intervals and locate global maxima (Johnson et al., 2011).
• Specialized ML routines: For graphical models and covariance selection, convex optimization (often with sparsity constraints, as in the graphical lasso) is employed, subject to existence thresholds linked to the sample size and the graphical structure (Bernstein et al., 2023, Turčičová et al., 2017).
• Hierarchical and multilevel ML: By maximizing over nested subspaces (parameter hierarchies), one can systematically reduce asymptotic variance and sampling noise, which is effective in high-dimensional covariance estimation under model constraints (Turčičová et al., 2017).

4. Goodness-of-Fit, Uncertainty Quantification, and Model Assessment

ML fitting inherently provides uncertainty quantification via:

• Fisher Information and Profile Likelihoods: Variance of estimators is given (asymptotically) by the inverse Fisher information. Profile-likelihood-based intervals (the set where the fit statistic increases by a specified amount, e.g., $\Delta S=1$), provide robust confidence regions (Barret et al., 2011).
• Bootstrap and Resampling: Empirical quantification of uncertainties, especially when analytic error propagation is problematic, is implemented by repeated resampling and refitting (as in bootstrap error bars for luminosity function fits) (Parada et al., 2023).
• Nonparametric metrics: Traditional $R^2$ is inapplicable for nonlinear/flexible models—maximum-likelihood frameworks support rigorous, bounded, and model-agnostic measures such as the Empirical Survival Jensen–Shannon divergence (ESJS), which quantifies distributional discrepancy in survival space, providing interpretable metrics and confidence intervals for fitted models (Levene et al., 2018).
• Bias and Consistency Checks: Simulation studies and asymptotic theory underpin the assessment of systematic biases. As in extreme-value analysis, block-maxima MLEs for the Fréchet distribution have explicit formulas for bias and variance, guiding block-size selection and efficiency considerations (Bücher et al., 2015).

5. Specialized Domain Applications

Maximum-likelihood fitting adapts flexibly to a range of methodological challenges:

• Power Density Spectrum Fitting: ML fitting in the non-Gaussian regime is critical for unbiased extraction of quasi-periodic oscillation parameters, with the exponential likelihood correctly modeling periodogram statistics and outperforming $\chi^2$ approximations except in the infinite-averaging limit (Barret et al., 2011).
• Astrometry and Signal Processing: ML-based expansions (in orthonormal bases optimized for underlying PSF statistics) minimize astrometric bias, crucial for missions with microarcsecond-scale requirements, such as Gaia. This approach enables fit calibration across thousands of parameters while maintaining rigorous bias/error control (Gai et al., 2013).
• Template and Binned Data Fitting: For template-based fits, the exact multinomial or Poisson likelihoods (Barlow–Beeston) are computationally challenging, prompting the use of highly accurate closed-form approximations that maintain correct uncertainty propagation even for weighted data (2206.12346).
• Non-uniform Efficiency Correction: Likelihood weighting by the inverse efficiency addresses parameter recovery in experiments with non-uniform selection functions, provided the efficiency is reliably mapped and properly included in both the numerator (per-event likelihood) and normalization (Yu et al., 2023).
• Latent Variable and Hierarchical Models: Joint maximization via h-likelihood dispenses with intractable EM integration, allowing one-shot ML imputation for missing data and direct prediction of random effects by mode-imputation, as opposed to conditional mean (Han et al., 2022).
• Particle-based and Dynamical System Approaches: Recent advances employ free-energy minimization in likelihood spaces equipped with Wasserstein geometry, blending momentum (Nesterov/underdamped Langevin) with particle discretization for accelerated fitting in latent variable models (Lim et al., 2023).

6. Theoretical Properties and Limitations

Key properties essential to ML fitting include:

• Consistency and Asymptotic Normality: ML estimators are consistent and asymptotically normal under standard regularity, with explicit variance formulas depending on Fisher information and, in dependent/complex settings, explicit bias corrections (Bücher et al., 2015).
• Existence and Uniqueness: Well-posedness depends on data structure and model constraints. For instance, in high-dimensional Gaussian graphical models, the sample size must exceed the maximum-likelihood threshold $\tau(G)$ to guarantee the existence and uniqueness of the MLE, which is tightly linked to combinatorial properties of the underlying graph (Bernstein et al., 2023).
• Robustness and Practical Recommendations: In real-world data, small sample sizes or noise-model misspecification can induce bias or non-existence of the MLE. Strategies include the use of robust statistics (e.g., Cash's $C$ statistic for low-count Poisson data), regularization (graphical lasso or subspace constraints), and data-driven tuning of block sizes or expansions to control the bias–variance trade-off.

Limitations persist in settings with very sparse data, model misspecification, or poorly characterized efficiency maps. The reliability of uncertainty estimates degrades if the weight distribution is highly variable or if the ML equations are not globally concave. In some cases, computational complexity scales prohibitively (e.g., $O(n^3)$ for direct GP ML), but algorithmic innovations (e.g., skeletonization, low-rank or randomized factorization, and peeling) allow $O(n^{3/2})$ scaling, extending applicability to large datasets (Minden et al., 2016).

7. Summary Table: Key ML Fitting Recommendations

Scenario	Likelihood/statistic	Optimization/notes
Low-count Poisson/histogram data	Poisson likelihood / Cash $C$	LM with modified weights (Fowler, 2013)
Non-uniform efficiency	Weighted likelihood, $1/\epsilon(x)$	Requires efficiency map, analytic normalization preferred (Yu et al., 2023)
Binned fits with simulation-based templates	(Approx.) Barlow–Beeston likelihood	Fast closed-form approximations for large bins or weighted templates (2206.12346)
Random coefficients/mixed models	Simulated maximum likelihood	Halton/Sobol draws, BFGS (Zhu et al., 2023)
Dependent block maxima/extreme value	Explicit Fréchet MLE equations with bias correction	ROOT finding for $\alpha$, Fisher info for errors (Bücher et al., 2015)
High-dim Gaussian graphical model	$\operatorname{tr}(SK) - \log\det K$	Existence threshold $\tau(G)$, convex optimization (Bernstein et al., 2023)
Latent variable models	Free energy/EM/h-likelihood	Optimization on augmented parameter space (Lim et al., 2023, Han et al., 2022)

Maximum-likelihood fitting methods provide the theoretical and computational backbone for modern parameter inference, with methodological developments tailored to data statistics, efficiency corrections, model dependence, and computational scale. Rigorous application—and awareness of model-specific challenges—ensures unbiased parameter estimation, reliable uncertainty quantification, and interpretable model comparison across scientific disciplines.