M-Estimation with Convex Loss

• M-Estimation with Convex Loss is a framework that uses convex objective functions to ensure unique, robust parameter estimation in regression, classification, and related tasks.
• The convex loss guarantees global minimization and supports strong asymptotic convergence, even under high-dimensional settings and complex constraints.
• Applications span robust regression, shape-constrained optimization, and machine learning models, with performance analyzed via risk bounds and Gaussian limit laws.

M-Estimation with Convex Loss

M-estimation with convex loss is a foundational paradigm in statistics and machine learning, encompassing broad classes of problems such as regression, classification, robust location/scatter estimation, and modern empirical risk minimization. The convexity of the loss function enables both powerful asymptotic theory and robust algorithmic techniques, even in high dimensions or under weak smoothness assumptions. The paper of convex M-estimation is characterized by geometric, probabilistic, and optimization-theoretic insights, particularly when extended to constrained, nonparametric, or manifold-based settings.

1. Formal Definition and Setting

Let $(X_i)_{i\ge1}$ be independent and identically distributed observations in a measurable space $(E,\mathcal E)$ with law $P$. For parameter inference, consider a loss function $\rho:E\times\Theta_0\to\mathbb R$ where $\Theta_0 \subset \mathbb R^d$ is an open convex set, and possibly a closed convex constraint set $\Theta\subset\Theta_0$. The fundamental object is the population risk

$$\Phi(\theta) = \mathbb{E}[\rho(X_1, \theta)], \quad \theta \in \Theta_0$$

and its empirical counterpart

$$\Phi_n(\theta) = \frac{1}{n} \sum_{i=1}^n \rho(X_i, \theta).$$

The M-estimator is a (measurable) minimizer,

$$\hat{\theta}_n \in \arg\min_{\theta\in\Theta} \Phi_n(\theta),$$

where $\rho(x,\theta)$ is convex in $\theta$ for almost every $x$. The framework allows for nondifferentiable losses (e.g., absolute deviation, quantile loss) and admits constraints (e.g., parameter nonnegativity, affine restrictions).

The convexity of $\rho(x, \cdot)$ ensures the convexity of both the empirical and population risk, enabling strong minimization guarantees even in infinite-dimensional or functional settings (Brunel, 6 Nov 2025).

2. Theoretical Foundations: Existence, Uniqueness, and Consistency

Main assumptions for classical asymptotic analysis include:

• (A1) Convexity: $\rho(x, \cdot)$ is convex, and $\Theta$ is closed convex.
• (A2) Local integrability: $\rho(\cdot, \theta)\in L^2(P)$ for all $\theta$ near the minimizer.
• (A3) Population minimizer uniqueness: $\Theta^* = \arg\min_{\theta\in\Theta}\Phi(\theta)$ is a singleton $\{\theta^*\}$.
• (A4) Second-order local structure: $\Phi$ is twice differentiable at $\theta^*$, with Hessian $S \succ 0$.

Under these, the estimator is consistent: $\hat\theta_n \rightarrow \theta^*$ almost surely (Brunel, 2023, Brunel, 6 Nov 2025), and one obtains parameter convergence rates and risk bounds. Convexity alone, without differentiability, enables uniform convergence by Rockafellar’s argument and localization via empirical process theory (Chinot et al., 2018, Chinot, 2019).

Convexity is the key ingredient—no small-ball, explicit identifiability, or stochastic equicontinuity arguments are required for classical risk bounds or in the geodesic (manifold) setting (Brunel, 2023).

3. Asymptotic Distribution and the Impact of Geometry

In the finite-dimensional, differentiable case, the local behavior around $\theta^*$ is captured by a quadratic approximation: $$\sqrt{n}\,(\hat\theta_n-\theta^*) \xrightarrow{d} U,$$ with

$$Z \sim \mathcal{N}(0, S^{-1} B S^{-1}), \quad B = \mathrm{Var}[g(X_1,\theta^*)]$$

where $g(\cdot, \theta^*)$ is a measurable selection of subgradients.

Influence of Constraints and Boundary: The asymptotic distribution is determined by the interplay between $\rho$ and the constraint set’s boundary structure. If $\theta^*$ is in the interior of $\Theta$, the limiting distribution is Gaussian: $$\sqrt{n}(\hat{\theta}_n - \theta^*) \xrightarrow{d} Z.$$ At the boundary, the tangent cone $T_\Theta(\theta^*)$ modifies the distribution: fluctuations are “clipped” to a (potentially polyhedral) cone via the directional derivative of the projection mapping: $U = \mathrm d^+\pi^S_{\,\Theta-\theta^*}(u_0; Z),$ where $u_0 = -S^{-1} \nabla \Phi(\theta^*)$ and the minimization is over $u\in T_\Theta(\theta^*)$.

For general convex constraints, the asymptotic law is that of a Gaussian vector projected onto the tangent cone, i.e., the law of

$$U = \arg\min_{u\in T_\Theta(\theta^*)} \left\{ Z^\top u + \frac{1}{2} u^\top S u \right\}.$$

The structure of $T_\Theta(\theta^*)$ (interior, facet, corner) determines the degree of constraint-induced “shrinkage” in the limit (Brunel, 6 Nov 2025).

4. Examples and Illustrative Special Cases

Example	Loss Function	Limiting Law and Structure
Constrained mean	$\rho(x,\theta) = \frac{1}{2}\|x - \theta\|^2$	Projection of Gaussian onto $\Theta$
Geometric median	$\rho(x,\theta) = \|x-\theta\|$	Classical $n^{-1/2}$ limits (possibly with polyhedral projection)
Oja depth median	$U$-statistic: determinant-based	Cone-projected Gaussian limit
Pairwise scatter (Gini)	$\rho(x_1,x_2,\theta)=\ell(\|x_1-x_2\|^p - \theta)$	Bahadur expansion controlled by cone geometry

In each case, the limit law combines convexity, an L2 process expansion, and a conic projection. This structure applies in both mean and robust/median estimation, and for U-estimators arising in “deepest point” location estimation (Brunel, 6 Nov 2025).

5. Extensions: U-Estimators, High Dimensions, and Metric Spaces

U-Estimators and Depth-Functionals: For $U$-statistics of order $k$,

$$\Phi_n(\theta) = \frac{1}{\binom{n}{k}} \sum \rho(X_{i_1},\ldots, X_{i_k},\theta),$$

the asymptotic distribution is

$$\sqrt{n}(\hat{\theta}_n - \theta^*) \xrightarrow{d} k\, \mathrm d^+\pi^S_{\,\Theta-\theta^*}(u_0; Z),$$

with $B$ replaced by a conditional variance respecting the Hoeffding decomposition.

High-dimensional and Non-Euclidean Settings: Convex M-estimation extends to geodesic metric spaces and Riemannian manifolds (Brunel, 2023). If the cost is geodesically convex and the population risk is twice differentiable at the minimizer, consistency and asymptotic normality follow, with the limiting covariance determined by the Hessian of the population risk and the covariance of the gradient field, regardless of differentiability of the loss.

Risk Bounds and Rates: Non-asymptotic deviation inequalities for convex M-estimators are available under weak boundedness or moment conditions. These yield exponential or polynomial tail bounds for deviation rates and enable statements about almost-sure, r-complete, and quick convergence that are not accessible for nonconvex estimators (Ferger, 2023, Chinot et al., 2018, Chinot, 2019).

6. Role of Convexity, Regularity, and Efficiency Considerations

Convexity of $\rho(x, \theta)$ is essential because it:

• Ensures the existence, uniqueness (when strict), and computability of the M-estimator.
• Enables the minimizer to be characterized as a solution to variational inequalities, even without differentiability.
• Provides amenability to projection and geometric arguments needed for explicit limit laws, especially under constraints.

When the loss is strictly convex, the minimizer is unique and the asymptotic expansion reduces to conventional central limit behavior. For losses that are only convex (not strictly), the set of minimizers may be enlarged, but under the above regularity assumptions, local uniqueness is typically restored by the behavior of the population risk (Brunel, 6 Nov 2025, Dimitriadis et al., 2022).

Efficiency and Optimality: Within the class of convex M-estimators, explicit efficiency bounds exist: the minimal achievable asymptotic variance is determined by the infimum over all consistent decreasing scores (i.e., $-\ell'$ decreasing), as shown via score matching and convex order arguments (Feng et al., 25 Mar 2024). For heavy-tailed noise, the Huber-type loss arises as the minimax variance convex loss.

In semiparametric models, the structure of convex losses eliciting functional parameters is fully characterized in terms of consistent loss functions and Bregman divergences, enabling tailored efficiency-robustness trade-offs (Dimitriadis et al., 2022).

7. Applications and Broader Context

Convex M-estimation is central in:

• Robust statistics: Geometric median and scatter functionals.
• Machine learning: Empirical risk minimization with hinge, logistic, or pinball losses.
• Shape-constrained and constrained optimization: Nonnegativity, sparsity, and boundary-constrained inference.
• High-dimensional statistical learning: Regularized M-estimation, Lasso, and structured penalties, where convexity enables precise error characterizations even in $n \asymp p$ asymptotics (Thrampoulidis et al., 2016, Advani et al., 2016).
• Functional data and nonparametrics: Sieve and partition-based convex M-estimators, with uniform inference enabled by Bahadur representation and strong approximation theory (Cattaneo et al., 9 Sep 2024).

Algorithmically, convexity ensures polynomial-time solvers (gradient, projected subgradient, interior-point methods), global optimality, and (in some cases) distributed or online implementability.

Summary Table: Key Theoretical Elements in Convex M-Estimation

Aspect	Description/Condition	Source(s)
Population Risk	$\Phi(\theta) = \mathbb{E}[\rho(X, \theta)]$	(Brunel, 6 Nov 2025)
Existence	Convexity $\to$ minimizer exists	(Brunel, 6 Nov 2025)
Uniqueness	Strict convexity or local strong convexity	(Brunel, 6 Nov 2025)
Asymptotic Normality	$\sqrt{n}(\hat\theta_n-\theta^*)$ limit law via cone	(Brunel, 6 Nov 2025)
Constraints	Projected limit law; tangent cone modifies fluctuation	(Brunel, 6 Nov 2025)
Efficiency Bound	Minimal variance for convex M-estimator	(Feng et al., 25 Mar 2024)
Extensions	U-estimators, geodesic metric spaces	(Brunel, 6 Nov 2025Brunel, 2023)

• Asymptotics of constrained $M$-estimation under convexity (Brunel, 6 Nov 2025)
• Geodesically convex $M$-estimation in metric spaces (Brunel, 2023)
• Supremal inequalities for convex M-estimators (Ferger, 2023)
• Characterizing M-estimators (Dimitriadis et al., 2022)
• Optimal convex $M$-estimation via score matching (Feng et al., 25 Mar 2024)
• Precise Error Analysis of Regularized M-estimators in High-dimensions (Thrampoulidis et al., 2016)
• An equivalence between high dimensional Bayes optimal inference and M-estimation (Advani et al., 2016)
• Uniform Estimation and Inference for Nonparametric Partitioning-Based M-Estimators (Cattaneo et al., 9 Sep 2024)

This body of work establishes convex M-estimation as a mathematically transparent, computationally tractable, and broadly adaptable tool for modern statistical inference and learning, even in the face of non-smoothness, high dimensionality, and model constraints.