Non-Parametric Maximum Likelihood Estimation

• NPMLE is a nonparametric method that estimates complex models by optimizing the likelihood over infinite-dimensional function spaces with regularity constraints such as smoothness.
• It employs sieves like Sobolev balls to balance flexibility and computational tractability, ensuring uniform convergence rates and robust theoretical guarantees.
• NPMLE underpins advanced techniques including Donsker-type theorems and simulation-based minimum-distance estimation, facilitating efficient and reliable statistical inference.

Non-Parametric Maximum Likelihood Estimation (NPMLE) is a fundamental approach in modern statistics for estimating complex, potentially infinite-dimensional models directly from data, imposing only mild regularity constraints such as smoothness or shape. Its most prominent applications are in density estimation, mixture models, and simulation-based (indirect inference) estimators. The methodology centers on maximizing the likelihood over a nonparametric (often function space) class, frequently defined via smoothness sieves such as Sobolev balls. Recent advances, as synthesized here, have established a comprehensive asymptotic theory for NPMLEs, especially their uniform convergence rates, their behavior as stochastic processes, and their use as auxiliary estimators in indirect inference frameworks.

1. Definition and Framework

The non-parametric maximum likelihood estimator is constructed by maximizing the likelihood function over a nonparametric class of densities or measures—often constrained by smoothness, boundedness, or structural features such as monotonicity or log-concavity. For density estimation, a typical NPMLE solves

$$\hat{p}_n = \mathop{\mathrm{arg\,max}}_{p \in \mathcal{P}} \sum_{i=1}^n \log p(X_i),$$

where $\mathcal{P}$ is a suitable class of densities, such as those with Sobolev regularity (order $t$) or supported on some compact domain. In simulation-based minimum distance or indirect inference applications, the NPMLE is used as an "auxiliary" estimator, facilitating the construction of test statistics or discrepancies between observed and simulated data.

A key characteristic is the use of "sieves"; rather than optimizing over the infinite-dimensional class directly, estimation is performed over a sequence of growing, regularized function spaces (e.g., Sobolev balls with increasing radius or smoothness), allowing for both theoretical control and practical computation.

2. Uniform Convergence Rates in Sobolev Norms

One of the central achievements of the NPMLE methodology is the establishment of sharp, uniform convergence rates in Sobolev (and related) norms over the entire parameter space. For the estimator $\hat{p}_k(0)$ based on a sample of size $k$, the paper proves: $$\sup_{\theta \in \Theta} \| \hat{p}_k(0) - p_\theta \|_{s,2} = O_p\left(k^{-\frac{t-s}{2t+1}}\right)$$ for Sobolev norms of order $s < t$ (Gach et al., 2010). This form of "uniform-in-parameters" rate is substantially stronger than typical pointwise or $L_2$ convergence; it guarantees that the NPMLE's rate of approximation is controlled uniformly over all $\theta$ in the parameter space, which is crucial for applications involving function-valued parameters or for guaranteeing robust inference in the presence of model misspecification.

This result is achieved via empirical process theory and entropy calculations tailored to the sieve classes, exploiting the concentration of measure and metric entropy properties of Sobolev spaces.

3. Donsker-Type (Uniform CLT) Theorems

Beyond rates, the paper establishes uniform central limit theorems (Donsker theorems) for the stochastic process formed by the NPMLE. For the process

$$(\theta, f) \mapsto \sqrt{n} \int \left( \hat{p}_n(\theta)(x) - p_\theta(x) \right) f(x)\, dx,$$

it is shown that, under suitable smoothness and regularity, this process converges weakly to a centered Gaussian process indexed by both $\theta$ and test functions $f$ in a bounded set of the Sobolev space (Gach et al., 2010). More precisely,

$$\sqrt{n} \left[ \int \left(\hat{p}_n(\theta) - p_\theta\right) f\,dx - \int (\mathbb{P}_n - P) f\,dx \right] = o_p(1)$$

uniformly over $\theta$ and $f$.

This Donsker-type theorem is essential as it permits linearization of the NPMLE—showing the estimator is, asymptotically, as "simple" as the empirical process itself—which dramatically simplifies further asymptotic development, notably for simulation-based minimum-distance estimators.

4. Asymptotic Normality, Efficiency, and the Fisher Information Matrix

A principal result is the asymptotic normality of simulation-based (minimum-distance or SMD) estimators when the NPMLE is used as an auxiliary estimator. The minimum-distance estimator $\hat{\theta}_n$ that minimizes

$$Q_n(\theta) = \int (\hat{p}_n(\theta) - p_\theta)^2 w\,dx,$$

admits a linear expansion: $$\sqrt{n} (\hat{\theta}_n - \theta_0) = -J(\theta_0)^{-1} \sqrt{n} \nabla Q_n(\theta_0) + o_p(1)$$ which leads to the Gaussian limit

$$\sqrt{n} (\hat{\theta}_n - \theta_0) \xrightarrow{d} N(0,\, J(\theta_0)^{-1} I(\theta_0) J(\theta_0)^{-1})$$

where

• $J(\theta_0)$ is half the Hessian of the limiting objective,
• $I(\theta_0)$ represents the information in the empirical process.

If the model is correctly specified (i.e., the true density is $p_{\theta_0}$), then $I(\theta_0) = J(\theta_0)$ and the variance reduces to $J(\theta_0)^{-1}$, the inverse Fisher information as in the parametric efficient MLE case (Gach et al., 2010). This establishes the efficiency of the NPMLE-based minimum distance estimator in the classical (parametrically correct) setting.

5. Implementation Considerations and Practical Workflow

Sieve Construction

• Smoothness-based sieves (e.g., Sobolev balls of order $t$) are constructed to approximate the nonparametric class. For practical computation, a finite basis (such as B-splines or finite Fourier series) represents functions in the sieve.

Optimization

• The NPMLE is typically found by maximizing the empirical log-likelihood over the sieve via convex optimization techniques. The convexity of both the likelihood and the function class ensures global optimality.
• In simulation-based inference, simulated data are generated for each parameter value; the NPMLE is applied to both real and simulated datasets, and their empirical discrepancies form the basis of indirect inference.

Uniform Convergence Checks

• Uniform convergence and Donsker property checks require, in practice, verifying sufficient smoothness (appropriate sieve order), boundedness, and entropy control for candidate sieves.

Variance Estimation

• For inference post-NPMLE, the sandwich form of the asymptotic variance $J^{-1} I J^{-1}$ is used generically, with simplification to $J^{-1}$ if the model is correctly specified.

Scaling

• The computational burden scales polynomially with sieve dimension. Choice of basis size must balance approximation error (dictated by smoothness) with estimation error and computational tractability.

6. Implications and Applications

• The uniformity of convergence rates and the applicability of Donsker-type theorems justify the use of NPMLE in high- or infinite-dimensional models, especially as auxiliary estimators in indirect inference, where robustness to parameterization and model regularization is critical.
• The strong theory removes longstanding obstacles to deploying simulation-based minimum-distance estimators in econometrics and applied statistics, giving rigorous backing to estimation/prediction even when the auxiliary model is infinite-dimensional.
• The asymptotic normality and variance results connect the practical behavior of SMD estimators directly to the Fisher information, ensuring that, under correct specification, practitioners recover parametric efficiency.

7. Summary Table: Key Results

Result Type	Formula or Rate	Context/Condition
Uniform Sobolev Rate	$$\sup_{\theta}\|\hat{p}_k(0) - p_\theta\|_{s,2} = O_p(k^{-(t-s)/(2t+1)})$$	Sieve of order $t$, $0 \leq s < t$; uniform over parameter space
Donsker Theorem	$$\sqrt{n} \left[ \int (\hat{p}_n(\theta) - p_\theta)f\,dx - \int (\mathbb{P}_n - P)f\,dx \right] = o_p(1)$$	Uniform over parameter $\theta$ and test function $f$
Asymptotic Normality	$$\sqrt{n} (\hat{\theta}_n-\theta_0) \xrightarrow{d} N(0,J^{-1}IJ^{-1})$$	SMD estimator with NPMLE auxiliary; $J=I$ under correct specification
Fisher Information Case	$J(\theta_0)^{-1}$	Under correct specification; efficiency of minimum-distance estimator

These results provide the technical backbone of the nonparametric maximum likelihood paradigm and validate its use in both theoretical and real-world applications involving simulation-based or indirect inference procedures (Gach et al., 2010).