M-Estimators for Non-IID Data

• M-estimators for non-iid data are robust techniques that extend classical estimation by aggregating heterogeneous sample-specific criteria for flexible modeling.
• They employ adaptive weighting and one-step Newton-Raphson methods to achieve asymptotic normality and control error rates even under contamination.
• This framework supports diverse applications such as robust regression, extreme value analysis, and adaptive sequential inference in heterogeneous environments.

M-estimators for non-identically distributed (non-iid) data generalize the principle of robust and efficient parameter estimation from classical homogeneous sampling models to the diverse, heteroscedastic, contaminated, adaptively sampled, or weakly dependent regimes encountered in modern statistical applications. Formally, an M-estimator is the solution to an estimating equation or an optimization problem which aggregates observation-specific score functions or criterion values, enabling maximum flexibility in modeling complex data generating processes. This paradigm has spawned a rich literature spanning robust regression, distributional regression, non-parametric smoothing, adaptive inference, model contamination, and invariance to nuisance parameters or local dependence.

1. General Definition and Triangular Array Framework

The non-iid setting is fundamentally broader than classical iid asymptotics. Let $\{Z_{n,i}\}$ for $i=1,\ldots,n$ be a triangular array of independent, but not necessarily identically distributed, observations; each $Z_{n,i}$ follows its own law, possibly depending on $n$ or covariate design. Consider a compact parameter space $(H,d_H)$ and an upper semicontinuous criterion function $m:H\times Z\rightarrow[-\infty,\infty)$ such that $M_n(\eta) = n^{-1}\sum_{i=1}^n m_\eta(Z_{n,i})$, and $M(\eta)$ is its population average. An M-estimator $\hat\eta_n$ is any maximizer (or root, for score-based estimators) of the empirical criterion: $$M_n(\hat\eta_n) \geq M_n(\eta_0) - o_{p,\ a.s.}(1),$$ with $\eta_0$ characterized as the unique maximizer of $M(\eta)$ (Bücher et al., 14 Nov 2025). This formulation includes conditional and unconditional maximum likelihood, proper scoring rules, minimum pseudodistance, nonlinear and weighted regression, and robust or penalized estimators under contamination.

2. Robustness and Tractability: Contamination and Non-convexity

Robust estimation under the Huber gross-error model—where each observation is either an inlier or an adversarial outlier—illustrates the paradigm for highly non-iid data. Write $y_i = x_i^\top\beta^* + \varepsilon_i$, with

$\varepsilon_i \sim (1-\delta) f_0 + \delta g_i,$

where $f_0$ is light-tailed noise and $g_i$ arbitrary. The empirical risk for robust regression, e.g. with Welsch loss $\rho(t)=1-\exp(-t^2/\sigma^2)$, is

$$R_n(\beta)=\frac{1}{n}\sum_{i=1}^n \rho(y_i-x_i^\top\beta).$$

Under mild smoothness and sub-Gaussian design, both the population and sample risks possess a unique stationary point near $\beta^*$, with error rates

$$\|\hat\beta_n-\beta^*\|_2 = O\big(\delta + \sqrt{p\log n/n}\big),$$

even if $\rho$ is non-convex (Zhang et al., 2019). For high-dimensional sparse parameterizations ($p\gg n$, $s$ sparse), penalization with $\lambda\|\beta\|_1$ preserves robustness and tractability, with error $O(\delta + \sqrt{s\log p/n})$ provided $\delta$ is small and design is well-conditioned.

3. Asymptotics and Strong Consistency

Strong and weak consistency of M-estimators for non-iid designs follow from generalizations of the argmax theorem. Primitive conditions are upper semicontinuity (in parameter and data), identifiability ($M(\eta)<M(\eta_0)$ for $\eta\neq\eta_0$), and $L^2$ or uniform envelope dominance for criterion values. Under such conditions, the law of large numbers for triangular arrays yields

$$M_n(\hat\eta_n)\to M(\eta_0) \quad \implies \quad \hat\eta_n\to\eta_0,$$

almost surely or in probability depending on the integrability regime, without requiring uniform convergence (Bücher et al., 14 Nov 2025). This applies even when the criterion takes $-\infty$ due to parameter-dependent support (e.g., in extreme value likelihood problems).

4. One-step Weighted and Newton-Type Estimation

Explicit one-step Newton–Raphson corrections deliver asymptotically optimal M-estimators even in complex non-iid settings. Suppose independent observations $\{X_{i,n}\}$ with differentiable score functions $\psi_i(X_{i,n},\theta)$ and deterministic weights $w_{i,n}$. Starting from a preliminary root $\theta_n^{(0)}$, the weighted one-step estimator is

$$\theta^{(1)}_n = \theta^{(0)}_n - \Big[\sum_{i=1}^n w_{i,n}\ \partial_\theta\psi_i(X_{i,n},\theta^{(0)}_n)\Big]^{-1} \sum_{i=1}^n w_{i,n}\psi_i(X_{i,n},\theta^{(0)}_n),$$

achieving asymptotic normality with variance $A^{-1} B (A^{-1})^\top$ for $$A=\lim_n \sum w_{i,n} \mathbb E[\partial_\theta \psi_i]$$, $B=\lim_n \sum w_{i,n}^2 \operatorname{Var}[\psi_i]$ (Linke, 2015, Linke, 2015). Optimal variance is reached using Cauchy–Schwarz weights

$$w_{i,n}^\ast \propto \frac{\mathbb E[\psi_i'(X_{i,n},\theta_0)]}{\mathbb E[\psi_i^2(X_{i,n},\theta_0)]}.$$

These methods generalize nonlinear least squares, weighted regression, and moment-based M-estimation.

5. Rényi Pseudodistance and Robust Wald-Type Inference

Minimum Rényi pseudodistance estimators provide robust alternatives to classical MLE, especially in the presence of contamination or heterogeneity. For data $Y_i$ with model densities $f_i(\cdot;\theta)$ and order $\alpha>0$, the minimum-RP estimator $\hat\theta_\alpha$ solves

$$\hat\theta_\alpha = \arg\max_\theta \frac{1}{n}\sum_{i=1}^n V_i(Y_i;\theta),$$

where

$$V_i(Y_i;\theta) = f_i(Y_i;\theta)^\alpha \left[\int f_i(y;\theta)^{\alpha+1} dy\right]^{\alpha/(\alpha+1)}.$$

These estimators admit bounded influence functions for $\alpha>0$ and achieve $\sqrt{n}$-consistency and asymptotic normality under mild regularity (Castilla et al., 2021). Wald-type tests based on $\hat\theta_\alpha$ maintain nominal size and power under modest contamination in both simple and composite hypothesis regimes, and an optimal value $\alpha\approx 0.2$–$0.4$ yields good compromise between robustness and efficiency in regression.

6. Adaptive and Dependent Data: Weighted and Local M-Estimation

M-estimation retains asymptotic validity for adaptively collected (e.g., bandit-sampled) or weakly dependent data using stabilized weighting or kernel localization. In adaptive designs, square-root importance weights relative to a pre-specified distribution correct non-stationarity: $$W_t = \sqrt{\frac{\pi_t^{sta}(A_t,X_t)}{\pi_t(A_t,X_t,H_{t-1})}},$$ yielding weighted M-estimators with asymptotically valid confidence regions in the presence of adaptivity (Zhang et al., 2021). For dependent or locally identified criterion functions, maximal inequalities for $\beta$-mixing arrays and empirical processes enable cube-root asymptotics,

$\|\hat\theta_n-\theta_0\| = O_p((n h_n)^{-1/3}),$

with non-Gaussian limit laws via the argmax of a Gaussian process (Seo et al., 2016). Subsampling inference is justified, while naive bootstrapping typically fails.

7. Applications and Regimes

Non-iid M-estimation encompasses a wide array of applied contexts, including:

• Robust linear and logistic regression under heteroscedasticity or contamination,
• Extreme value and Pareto regression with parameter-dependent supports,
• Weighted nonlinear regression and design-based inference,
• Adaptively collected sequential data (bandit policies, reinforcement learning),
• Partially identified models and set estimation,
• Local maximum score estimation via kernel smoothing.

Theoretical results provide explicit conditions for consistency, minimax rates, robustness, and tractability, often under minimal moment or continuity assumptions; practical implementation relies on proper weighting, root-finding algorithms, and simulation diagnostics. The approach extends to multivariate, high-dimensional, semi-parametric, and non-convex optimization settings, ensuring broad relevance for robust inference in heterogeneous and modern data environments.