Minimum-Distance Estimators

• Minimum-distance estimators are statistical methods that estimate parameters by minimizing a divergence between empirical and model distributions.
• They utilize metrics like L2, Hellinger, and Wasserstein to offer a unified, robust, and asymptotically normal approach to parameter estimation.
• These methods are applied in econometrics, machine learning, and survey inference, with practical strategies for high-dimensional and complex data.

Minimum-Distance Estimators

Minimum-distance estimators (MDEs) constitute a broad and powerful class of statistical inference methods defined by minimizing a well-chosen distance or divergence between the empirical (or nonparametrically estimated) data distribution and a parametric model family. They provide a unified approach to robust, efficient, and often easy-to-compute parameter estimation, applicable across a wide range of statistical, econometric, and machine learning models.

1. Mathematical Formulation and General Properties

Let $\mathcal{P} = \{P_\theta: \theta \in \Theta\}$ be a family of probability distributions/densities on a sample space $\mathcal{X}$, and let $P_n$ be an empirical or nonparametric estimate (KDE, empirical cdf, etc.) of the true data-generating distribution $P_0$. The archetypal MDE is defined as

$$\hat\theta_n = \arg\min_{\theta \in \Theta} D(P_n, P_\theta)$$

where $D(\cdot,\cdot)$ is a suitable distance metric or divergence quantifying discrepancy between two probability laws.

Prominent choices include:

• Integrated $L^2$-distance (Cramér–von Mises, quadratic loss): $D(f,g) = \int (f(x) - g(x))^2 w(x)\,dx$
• Hellinger distance: $$H^2(f, g) = \frac{1}{2} \int (\sqrt{f(x)} - \sqrt{g(x)})^2 dx$$
• Wasserstein ($p$-transport) distance: $\mathcal{X}$0
• $\mathcal{X}$1-distance or generalized empirical processes: $\mathcal{X}$2
• Disparity-based criteria: $\mathcal{X}$3

Well-posedness and consistency typically require identifiability and regularity:

• The mapping $\mathcal{X}$4 must be injective (identifiable).
• The population criterion $\mathcal{X}$5 must have a unique minimizer.

MDEs enjoy favorable theoretical properties: under standard conditions, they are consistent and, with suitable choices of $\mathcal{X}$6 and weighting, asymptotically normal and efficient (Keepplinger et al., 15 Oct 2025, Kolesár, 2015, Wei et al., 2023, Hooker, 2013, Kim, 2017).

2. Major Instantiations: Distance Choices and Specialized Designs

Distance/Objective	Typical Setting	Example Reference
Hellinger	Parametric models, complex surveys	(Keepplinger et al., 15 Oct 2025)
Wasserstein	Mixture modeling, finite mixtures	(Zhang et al., 2021, Wei et al., 2023)
$\mathcal{X}$7 (CvM)	Univariate parametrics, Lomax	(Nombebe et al., 2022, Kim, 2022)
$\mathcal{X}$8-distance	Mixture deconvolution, general MMs	(Wei et al., 2023)
Maximum Mean Discrepancy (MMD)	Time series, dependent data	(Alquier et al., 16 Jan 2026)
Conditional disparity	Regression, semiparametrics	(Hooker, 2013)

Specific implementations frequently adapt MDE to:

• Survey inference under complex designs: Hellinger distance combined with survey-weighted kernel density estimators yields robust and efficient MHDEs, especially when high-leverage weights and outliers threaten design-based MLEs (Keepplinger et al., 15 Oct 2025).
• Finite mixture models: Both norm-based ($\mathcal{X}$9, Wasserstein, $P_n$0-distance/MMD) and kernel methods naturally enforce convergence rates on estimated mixing measures under various metrics (Zhang et al., 2021, Wei et al., 2023).
• Panel and time series models: MDEs generalize to empirical characteristic functions, autocorrelation structures, and quantile models via minimum distance to sample moments or conditional fits (Melly et al., 25 Feb 2025, Lana et al., 2018, Borodavka et al., 14 Jun 2025).
• Semiparametric and independence models: Minimum integrated distance/MMD or optimal transport embed independence and weak identifiability via kernel-based or primal-dual empirical objectives (Gao et al., 2014).

3. Asymptotic Theory: Consistency, Rates, and Efficiency

• Consistency: Under identifiability and suitable control of the stochastic error between empirical and population distances, $P_n$1 (where $P_n$2 minimizes the expected distance to $P_n$3 or $P_n$4 for mixing laws).
• Asymptotic Normality: With additional smoothness (differentiable model, suitably regular $P_n$5), the minimized sample contrast admits an expansion yielding

$P_n$6

where the specific form of $P_n$7 ("sandwich" variance) depends on the curvature and influence structure of $P_n$8 (Keepplinger et al., 15 Oct 2025, Kim, 2017, Kolesár, 2015, Wei et al., 2023, Boulin et al., 25 Nov 2025).

• Efficiency: For certain distances and weights, the limiting covariance matches or nearly matches the inverse Fisher information of the MLE (in the fully specified correct model), i.e., MDE can be asymptotically efficient at the model (Keepplinger et al., 15 Oct 2025, Kolesár, 2015, Gach et al., 2010, Wei et al., 2023).
• Robustness: Boundedness of influence functions characterizes robustness. Hellinger, $P_n$9, and Wasserstein-based MDEs provide bounded and redescending influence, conferring stability against outlier contamination and high-leverage sample points (Keepplinger et al., 15 Oct 2025, Kim, 2017, Hooker, 2013, Jana et al., 2022).

4. Algorithmic and Computational Strategies

Methodological choices are closely tied to efficient computation:

• Empirical cdf–based or $P_0$0–distance MDEs: Reduce to sum-of-squares or quadratic programs, often with straightforward numerical minimization, sometimes exploitable for closed-form in exponential families (Nombebe et al., 2022, Kim, 2022, Betsch et al., 2019).
• Hellinger/KDE-based MDEs: Grid-based quadrature over a fine grid, with a precomputed KDE and derivative-free iterative maximization (Keepplinger et al., 15 Oct 2025).
• Wasserstein/MMD/Optimal transport MDEs: Minimum Wasserstein distance over finite mixtures can be computed by minimizing quantile- or order-statistic–based expressions in 1-d, or via linear programming and Sinkhorn/entropic regularization in multivariate settings (Zhang et al., 2021, Wei et al., 2023, Gao et al., 2014, Alquier et al., 16 Jan 2026).
• Panel/data-adaptive MDEs: Two-stage minimum-distance GMM in quantile panel settings, exploiting closed-form IV-type solutions when moments are just-identified (Melly et al., 25 Feb 2025).
• Simulation-based/Indirect inference MDEs: Fit model by simulating nonparametric or NPML densities under candidate $P_0$1 and minimizing an $P_0$2 or Hellinger distance to nonparametric (NPML/sample) density estimates from observed data (Gach et al., 2010).
• High-dimensional/structured data: Sparse robust regression via capped loss/Lasso (MD-Lasso), iteratively reweighted ℓ2 minimization and composite-gradient methods (Lozano et al., 2013).

5. Empirical Evidence and Performance Trade-offs

Results from applied studies and simulations reveal nuanced efficiency-robustness tradeoffs:

• Robustness to contamination: MDEs (particularly Hellinger-based) maintain bounded bias and stable variance under moderate to severe contamination, where MLE-based methods often break down catastrophically (Keepplinger et al., 15 Oct 2025, Kim, 2017, Lozano et al., 2013, Jana et al., 2022).
• Efficiency: With clean data and correctly specified models, MDEs approach MLE in bias and RMSE; efficiency loss is generally modest, and often negligible for $P_0$3 and Hellinger distances (Keepplinger et al., 15 Oct 2025, Nombebe et al., 2022, Boulin et al., 25 Nov 2025, Betsch et al., 2019).
• Finite mixture estimation: Wasserstein and $P_0$4-MDEs are consistent and achieve minimax optimal rates for estimating mixing measures, but empirical evidence confirms no robustness gains over penalized MLE in typical mixture scenarios (Zhang et al., 2021, Wei et al., 2023).
• Time series and complex error structure: Koul’s MDEs and bias-corrected minimum-distance estimators outperform Whittle/MLE under nonconstant means and heavy-tailed innovations, especially in small samples (Lana et al., 2018, Kim, 2017, Kim, 2016, Alquier et al., 16 Jan 2026).
• Small sample regimes: In distributional fitting (e.g., Lomax), MDEs show superiority for $P_0$5 and heavy-tails, whereas MLE (possibly bias-corrected) remains preferable for large $P_0$6 (Nombebe et al., 2022).

6. Extensions and Contemporary Directions

Modern research extends MDEs through:

• Complex survey designs: Integration of design-based weights (Horvitz–Thompson adjustment), effective sample size adaptations, and finite-population corrections (Keepplinger et al., 15 Oct 2025).
• Nonparametric/classical limitations: Handling non-normalized and exponential-polynomial models via $P_0$7-distance MDEs and generalized Stein characterizations (Betsch et al., 2019).
• Empirical Bayes estimation and mixture deconvolution: Minimum-distance (e.g., Hellinger or L2) plug-in for prior estimation in compound discrete models with sharp non-asymptotic regret bounds (Jana et al., 2022).
• Empirical process and kernel-based methods: Use of reproducing kernel Hilbert spaces (MMD, minimum-$P_0$8 distance), supporting non-Euclidean, high-dimensional, and structured data (Wei et al., 2023, Alquier et al., 16 Jan 2026).
• Algorithmic advances: Fast coordinate-wise and net-gain based algorithms in high-dimensional and massive-data scenarios, as with Koul’s MDEs for image segmentation (Kim, 2022, Kim, 2016).

7. Practical Considerations and Recommendations

• Choice of distance: $P_0$9, Hellinger, and Wasserstein distances are effective across a wide range of applications; squared Hellinger often yields improved robustness with minimal efficiency loss (Keepplinger et al., 15 Oct 2025, Jana et al., 2022). Wasserstein and MMD are valuable for mixtures and non-Euclidean data (Zhang et al., 2021, Wei et al., 2023, Alquier et al., 16 Jan 2026).
• Design adaptation: Survey-weighted and effective-sample-size tuning is vital under complex sampling (Keepplinger et al., 15 Oct 2025).
• Computation: Derivative-free methods suffice in low dimensions; BFGS and stochastic gradients are applicable in high-dimensional/complex settings (Zhang et al., 2021, Lozano et al., 2013, Alquier et al., 16 Jan 2026).
• Asymptotics and inference: Sandwich variance form is prototypical; under regularity, plug-in and bootstrap intervals are valid (Keepplinger et al., 15 Oct 2025, Hooker, 2013, Lana et al., 2018).
• Robustness vs. efficiency: In "clean" iid contexts, MDE and MLE are nearly equivalent; with leverage, contamination, or model misspecification, MDEs provide substantial gains in stability and outlier resistance (Keepplinger et al., 15 Oct 2025, Kim, 2017, Lozano et al., 2013).

Minimum-distance estimation remains an essential pillar of robust and semiparametric inference, with ongoing developments in empirical process theory, kernel methods, and high-dimensional statistics expanding its reach and applicability.