Estimator in Statistical Models

• Estimator is a data-based rule that maps observed data to a target quantity, with its performance influenced by model specification and efficiency criteria.
• Optimization-defined estimators are derived from maximizing or minimizing empirical criteria, as seen in Kaplan–Meier and partition function methods.
• Robust estimators balance bias and variance to ensure stability and reliable inference in the presence of misspecification and outlying data.

An estimator is a data-based rule $\hat\theta(X^n)$. Under correct specification, several estimators may target the same statistical parameter and differ mainly in efficiency; under misspecification, especially in over-identified models, different estimators generally converge to different probability limits, so changing the estimator changes the estimand (Andrew et al., 18 Aug 2025). Across current research, estimators appear as sample-survey corrections, M-estimators and EM fixed points in survival analysis, solutions of mathematical programs, residual-process functionals, spectral corrections in network regression, and surrogate-gradient devices in binary neural networks (Gu et al., 2020, Hsieh et al., 2017, Sassi, 2023, Wu et al., 2023).

1. Conceptual status of the estimator

In the modern formalization, the estimator is distinct from the estimand, the identification problem, and the efficiency criterion. Identification is a property of the model, not of the estimator; efficiency compares estimators that target the same object; and, under misspecification, the natural target of an estimator is its probability limit $plim(\hat\theta,P)$ rather than an idealized parameter that may cease to be well defined outside the maintained model (Andrew et al., 18 Aug 2025). In over-identified GMM, this point is explicit: if $g_P(\psi)=E_P[g(X_i,\psi)]$ and $\Omega_P=plim(\hat\Omega,P)$, then

$$plim(\hat\psi_\Omega,P)=\arg\min_{\psi\in\Psi} g_P(\psi)'\Omega_P g_P(\psi),$$

so different weighting matrices imply different pseudo-true values under misspecification (Andrew et al., 18 Aug 2025).

This interpretation changes the meaning of familiar comparisons. The distinction between “more efficient” and “estimating something else” is model dependent. In the misspecified, over-identified setting reviewed in recent econometric work, identity weighting, diagonal weighting, or hand-selected moments are not merely computational or efficiency choices; they alter the population criterion and therefore the estimand itself (Andrew et al., 18 Aug 2025).

The same conceptual separation appears in estimator-selection problems. In the Gaussian sequence model $Y\sim\mathcal N(f,\sigma^2 I_n)$, one may start from an arbitrary and possibly infinite family $\mathbb F=\{\hat f_\lambda,\lambda\in\Lambda\}$ of estimators of $f$, with no assumptions on their structure and with unknown dependence on the same data used for selection (Baraud et al., 2010). There, the central object is not a single estimator but a selection rule over competing estimators, evaluated by oracle-type Euclidean risk bounds rather than by any single closed-form expression (Baraud et al., 2010).

2. Construction by optimization, equations, and fixed points

A large class of estimators is defined as the optimizer of an empirical criterion over a structured parameter space. In right-censored survival analysis, the Kaplan–Meier estimator can be reconstructed exactly as

$\hat S=\arg\max_{S\in\mathcal S}\widetilde M_n(S),$

where $\mathcal S$ is the class of nonincreasing survival functions and $plim(\hat\theta,P)$0 is a quadratic concordance-based objective that treats censoring through an EM completion step (Gu et al., 2020). The EM iterates are monotone in the observed-data criterion, their fixed point is the classical product-limit estimator, and the reconstruction places the Kaplan–Meier procedure inside standard M-estimation theory (Gu et al., 2020).

Optimization-defined estimators also arise when the parameter is itself the solution to a mathematical program with estimated coefficients. For linear programming, the optimizer can be characterized through primal feasibility, dual feasibility, and complementarity,

$plim(\hat\theta,P)$1

and for convex quadratic programming the stationarity condition adds the quadratic term $plim(\hat\theta,P)$2 (Hsieh et al., 2017). Inference then proceeds by testing whether a candidate $plim(\hat\theta,P)$3 can be embedded in the corresponding KKT system with estimated coefficients, rather than by analyzing a nonsmooth argmax map directly (Hsieh et al., 2017).

A related construction appears in statistical mechanics, where the Partition Function Estimator rewrites the configurational partition function through the exact identity

$plim(\hat\theta,P)$4

The cutoff $plim(\hat\theta,P)$5 is then chosen by minimizing the relative standard error of the truncated exponential observable, and the excluded high-energy region is compensated by an explicit sublevel-set volume term $plim(\hat\theta,P)$6 (Chiang et al., 2024). Here the estimator is neither a plug-in moment nor a direct optimizer of a smooth criterion; it is an exact identity turned into a practical estimator by truncation and correction (Chiang et al., 2024).

3. Statistical criteria: bias, risk, consistency, and limit laws

The classical criteria attached to estimators remain bias, variance, mean squared error, consistency, and limiting distribution, but current work emphasizes explicit trade-offs among them. In survey sampling, the improved Horvitz–Thompson estimator

$plim(\hat\theta,P)$7

replaces very small inclusion probabilities by a threshold, thereby introducing bias but reducing the instability generated by extreme inverse-probability weights (Zong et al., 2018). Its normalized bias is of order $plim(\hat\theta,P)$8, its normalized MSE remains $plim(\hat\theta,P)$9, and, under the stated conditions, its MSE is asymptotically no larger than that of the classical Horvitz–Thompson estimator; for Poisson sampling the comparison is stronger and non-asymptotic (Zong et al., 2018).

Consistency may itself be weak, strong, or accompanied by a refined limit law. For the $g_P(\psi)=E_P[g(X_i,\psi)]$0-index estimator

$g_P(\psi)=E_P[g(X_i,\psi)]$1

weak consistency holds under $g_P(\psi)=E_P[g(X_i,\psi)]$2, strong consistency holds when $g_P(\psi)=E_P[g(X_i,\psi)]$3 with $g_P(\psi)=E_P[g(X_i,\psi)]$4, and a Hall-type tail expansion yields asymptotic normality for the modified version involving the constant $g_P(\psi)=E_P[g(X_i,\psi)]$5 (Cadena, 2015). In nonparametric regression, the smoothed residual empirical cdf

$g_P(\psi)=E_P[g(X_i,\psi)]$6

based on an under-smoothed local quadratic estimator satisfies the uniform expansion

$g_P(\psi)=E_P[g(X_i,\psi)]$7

and its influence function $g_P(\psi)=E_P[g(X_i,\psi)]$8 is the efficient influence function in the model studied there (Müller et al., 2018).

The limiting law may also be available in exact finite-sample form. For the robust similarity estimator,

$g_P(\psi)=E_P[g(X_i,\psi)]$9

the local similarity variable has density

$\Omega_P=plim(\hat\Omega,P)$0

under bivariate elliptical distributions with homogeneous variances, which yields an exact finite-sample characteristic function for the standardized sample mean and therefore exact inference on the Fisher-transformed correlation parameter $\Omega_P=plim(\hat\Omega,P)$1 (Archakov, 18 Jan 2026). In network regression with latent interacted effects, the Ordinary Least Eigenvalues estimator improves the usual $\Omega_P=plim(\hat\Omega,P)$2 rate of dyadic OLS to rate $\Omega_P=plim(\hat\Omega,P)$3, and, when $\Omega_P=plim(\hat\Omega,P)$4, the one-step corrected estimator attains the same asymptotic distribution as the oracle model without latent effects (Sassi, 2023).

4. Robustness, stability, and misspecification

Robustness enters estimator design through several distinct channels. In the similarity estimator, robustness is geometric: the resemblance statistic

$\Omega_P=plim(\hat\Omega,P)$5

depends only on relative direction and relative magnitude, not on the radial scale of the observation vector, so heavy tails and outliers in elliptical models do not alter its sampling law except through the correlation parameter (Archakov, 18 Jan 2026). In survey sampling, robustness is obtained by hard-thresholding extreme design weights, which sacrifices unbiasedness to attenuate variance inflation from tiny inclusion probabilities (Zong et al., 2018).

Misspecification introduces a different notion of robustness: robustness of interpretation. In over-identified econometric models, the central warning is that different estimators should not automatically be read as different approximations to the same parameter. Under misspecification, the estimand becomes estimator specific, and even the Hansen $\Omega_P=plim(\hat\Omega,P)$6-statistic acquires a new role: asymptotically, it measures the range of estimates attainable at a given standard error by varying the weighting matrix (Andrew et al., 18 Aug 2025). A plausible implication is that transparent reporting of estimator choice, weighting choice, and $\Omega_P=plim(\hat\Omega,P)$7-statistics is part of estimator interpretation, not merely of diagnostic testing (Andrew et al., 18 Aug 2025).

In machine learning, robustness may concern optimization stability rather than statistical bias. In binary neural networks, the Straight Through Estimator is used because the sign function has zero derivative almost everywhere, but the estimator design problem is formulated as an equilibrium between reducing the mismatch to the sign function and maintaining stable gradients (Wu et al., 2023). The Rectified Straight Through Estimator,

$\Omega_P=plim(\hat\Omega,P)$8

interpolates from the original STE at $\Omega_P=plim(\hat\Omega,P)$9 to the sign function as $$plim(\hat\psi_\Omega,P)=\arg\min_{\psi\in\Psi} g_P(\psi)'\Omega_P g_P(\psi),$$0, and its derivative

$$plim(\hat\psi_\Omega,P)=\arg\min_{\psi\in\Psi} g_P(\psi)'\Omega_P g_P(\psi),$$1

makes the trade-off explicit: lower estimating error comes with less stable gradients, particularly near zero (Wu et al., 2023).

5. Representative estimator families

The contemporary literature uses the term “estimator” for a wide range of constructions that share the common role of mapping data to a target quantity.

Domain	Estimator	Defining feature
Survival analysis	Kaplan–Meier as quadratic M-estimator (Gu et al., 2020)	Exact maximizer of a concordance-based M-objective and EM fixed point
Survey sampling	Improved Horvitz–Thompson estimator (Zong et al., 2018)	Hard-thresholded inclusion probabilities to trade small bias for variance reduction
Nonparametric regression	Smoothed residual cdf estimator (Müller et al., 2018)	Under-smoothed local quadratic residuals plus kernel smoothing
Data streams	Optimal quantile estimator for Compressed Counting (0808.1766)	Stable-law scale estimation by a variance-optimal quantile of $$plim(\hat\psi_\Omega,P)=\arg\min_{\psi\in\Psi} g_P(\psi)'\Omega_P g_P(\psi),$$2
Statistical mechanics	Partition Function Estimator (Chiang et al., 2024)	Truncated inverse-Boltzmann average plus sublevel-set volume correction
Network regression	Ordinary Least Eigenvalues Estimator (Sassi, 2023)	OLS criterion corrected by removing the leading spectral component of latent interactions
Robust dependence	Similarity estimator (Archakov, 18 Jan 2026)	Average Fisher-transformed resemblance, with exact elliptical sampling distribution
Binary neural networks	Rectified Straight Through Estimator (Wu et al., 2023)	Power-function backward estimator balancing estimating error and gradient stability

These examples show that “estimator” is not confined to unbiased sample analogues or smooth plug-in formulas. The same term covers objective-function maximizers, KKT-characterized optimizers, spectral corrections, quantile rules, robust angular statistics, and gradient surrogates. What unifies them is not algebraic form but the role they play: each defines a reproducible rule for extracting a target quantity from observed data or from an empirical optimization state.

6. Estimator selection, aggregation, and tuning

Once a collection of candidate estimators is available, a second-order problem arises: how to select among estimators. In the Gaussian setting, the selection rule is built from approximation spaces $$plim(\hat\psi_\Omega,P)=\arg\min_{\psi\in\Psi} g_P(\psi)'\Omega_P g_P(\psi),$$3, weights $$plim(\hat\psi_\Omega,P)=\arg\min_{\psi\in\Psi} g_P(\psi)'\Omega_P g_P(\psi),$$4, and a penalized criterion

$$plim(\hat\psi_\Omega,P)=\arg\min_{\psi\in\Psi} g_P(\psi)'\Omega_P g_P(\psi),$$5

with $$plim(\hat\psi_\Omega,P)=\arg\min_{\psi\in\Psi} g_P(\psi)'\Omega_P g_P(\psi),$$6 (Baraud et al., 2010). The selected estimator satisfies a non-asymptotic oracle-type inequality, and the framework covers aggregation, model selection, choosing a window and a kernel for estimating a regression function, and tuning the parameter involved in a penalized criterion (Baraud et al., 2010).

For linear estimators $$plim(\hat\psi_\Omega,P)=\arg\min_{\psi\in\Psi} g_P(\psi)'\Omega_P g_P(\psi),$$7, the same framework yields oracle inequalities with only logarithmic dependence on the number of candidate procedures, while in variable selection it can compare Lasso-type paths, Dantzig-type rules, random-forest rankings, and exhaustive subset search by evaluating the fitted subspaces they generate (Baraud et al., 2010). In that sense, estimator selection becomes a meta-estimation problem: the object being estimated is no longer $$plim(\hat\psi_\Omega,P)=\arg\min_{\psi\in\Psi} g_P(\psi)'\Omega_P g_P(\psi),$$8 or $$plim(\hat\psi_\Omega,P)=\arg\min_{\psi\in\Psi} g_P(\psi)'\Omega_P g_P(\psi),$$9 directly, but the identity of the procedure whose risk-complexity trade-off is most favorable on the observed sample.

This selection problem must still be read through the lens of estimands. In correctly specified Gaussian risk problems, selecting among estimators can be understood as approximating the best Euclidean-risk procedure in a common target class. Under misspecification, especially in over-identified models, the same act of selection may also select a different population target (Andrew et al., 18 Aug 2025). A comprehensive view of the estimator therefore includes not only its formula, bias, and asymptotic law, but also the model under which those properties are meaningful, the computational mechanism by which the estimator is obtained, and the estimand it converges to when the model is only approximately true.