Median-Stabilized LASSO Methods

• LASSO-based Median Estimators are penalized methods that combine LASSO regularization with median-based stabilization to mitigate issues from heavy-tailed noise and outliers.
• These methods include diverse constructions like MOM-LASSO, robust square-root LASSO, Bayesian group selection, distributed MESSAGE, and adaptive-LASSO IV, each applying a unique median operation.
• The approach achieves robust sparse recovery with theoretical guarantees, exemplified by bounds such as ||hat t − t*||2^2 ≲ s log(ed/s)/N, ensuring performance under contamination.

A LASSO-based median estimator is a penalized estimator that retains the $\ell_1$ or adaptive-Lasso sparsity mechanism of LASSO while replacing a fragile averaging, aggregation, or coefficient-extraction step by a median rule. In the arXiv literature, the term covers several non-equivalent constructions: median-of-means (MOM) LASSO and related Le Cam and tournament procedures for heavy-tailed or contaminated regression, a MOM analogue of square-root LASSO for unknown noise variance, posterior median thresholding under group-lasso-type spike-and-slab priors, the distributed MESSAGE procedure based on a median model, and adaptive-Lasso instrumental-variable methods initialized by a median-of-medians estimator (Guillaume et al., 2017, Lugosi et al., 2017, Finocchio et al., 2021, Xu et al., 2015, Wang et al., 2014, Liang et al., 2022).

1. Scope of the term

In these literatures, “median” does not refer to a single operation. It may denote a median of blockwise empirical means, a majority vote across blocks, a posterior median under a spike-and-slab posterior, a coordinatewise median of subset inclusion indicators, or a recursive median-of-medians across just-identified instrumental-variable estimators.

Family	Median operation	LASSO component
MOM-LASSO / tournament LASSO	Median of block means or majority of blocks	$\|t\|_1$ penalty
Robust square-root LASSO	MOM convex-concave criterion	$\|\beta\|_1$ penalty
Bayesian group selection	Posterior median thresholding	Group lasso or sparse group lasso slab
MESSAGE	Coordinatewise median of inclusion vectors	Local Lasso on each subset
Multiple-exposure IV	Median-of-medians initial estimator	Adaptive Lasso weights

Accordingly, the common structure is not a unique objective function but the combination of LASSO-type regularization with a median-based stabilization device. In robust regression papers, the median is introduced to control heavy tails and arbitrary outliers. In Bayesian group selection, it is the estimator that converts posterior spike mass into exact thresholding. In distributed inference and IV selection, it aggregates local or just-identified estimators without reverting to simple averaging.

2. Le Cam aggregation and the MOM-LASSO construction

The most direct robust interpretation arises in the MOM-LASSO of Lecué and Lerasle. In sparse linear regression, the target is an oracle

$$t^*\in\arg\min_{t\in\mathbb R^d}\, \mathbb E\big(Y-\langle X,t\rangle\big)^2,$$

estimated from possibly corrupted data. The procedure defines pairwise MOM tests

$$T_{K,\lambda}(g,f)=\operatorname{MOM}_K\!\big(\ell_f^\lambda-\ell_g^\lambda\big),\qquad \ell_f(x,y)=(y-f(x))^2,\qquad \ell_f^\lambda=\ell_f+\lambda\|f\|,$$

so that

$$T_{K,\lambda}(g,f)=\operatorname{MOM}_K(\ell_f-\ell_g)+\lambda(\|f\|-\|g\|).$$

The sample is split into $K$ blocks, and $\operatorname{MOM}_K$ is the median of the blockwise empirical means. Each candidate $f$ induces a beat-set

$$\mathcal B_{K,\lambda}(f)=\{g\in F: T_{K,\lambda}(g,f)\ge 0\},$$

and Le Cam’s construction selects an estimator by minimizing the radius of this set. A second estimator augments this rule with a MOM-estimated $\|t\|_1$0-type criterion

$\|t\|_1$1

and chooses a minimizer of the corresponding combined radius functional. The regularization norm is arbitrary in the general theory; the LASSO specialization sets it to $\|t\|_1$2 (Guillaume et al., 2017).

The $\|t\|_1$3 case is governed by the sparsity equation

$\|t\|_1$4

The paper uses the subdifferential geometry of $\|t\|_1$5 to show that when the oracle is close to an $\|t\|_1$6-sparse vector, the penalty contributes a linear lower bound of order $\|t\|_1$7 in the relevant directions. This yields sparse localization. In the theorem stated for MOM-LASSO, with exponentially high probability,

$\|t\|_1$8

which implies

$\|t\|_1$9

The robustness claim is explicit. The general risk behavior is

$\|\beta\|_1$0

and the sparse theorem allows

$\|\beta\|_1$1

without degrading the rate. The assumptions are weak: the noise only needs $\|\beta\|_1$2 for some $\|\beta\|_1$3; design coordinates satisfy, for every coordinate $\|\beta\|_1$4 and every $\|\beta\|_1$5,

$\|\beta\|_1$6

and independence between noise and design is not required, since the theorem assumes only

$\|\beta\|_1$7

3. Tournament and square-root extensions

A second robust line replaces empirical-risk minimization by regularized MOM tournaments. In tournament LASSO, the data are split into three independent parts, and the estimator is built in phases: a distance oracle based on medians of block averages of $\|\beta\|_1$8; an elimination tournament in which $\|\beta\|_1$9 beats $$t^*\in\arg\min_{t\in\mathbb R^d}\, \mathbb E\big(Y-\langle X,t\rangle\big)^2,$$0 if a majority of blocks support

$$t^*\in\arg\min_{t\in\mathbb R^d}\, \mathbb E\big(Y-\langle X,t\rangle\big)^2,$$1

a “champions league” based on a linearized excess-risk test; and a final selection of the smallest class in a hierarchy

$$t^*\in\arg\min_{t\in\mathbb R^d}\, \mathbb E\big(Y-\langle X,t\rangle\big)^2,$$2

that still contains a winner. In the sparse linear case, $$t^*\in\arg\min_{t\in\mathbb R^d}\, \mathbb E\big(Y-\langle X,t\rangle\big)^2,$$3 is indexed by sparsity level, $$t^*\in\arg\min_{t\in\mathbb R^d}\, \mathbb E\big(Y-\langle X,t\rangle\big)^2,$$4, and the estimator automatically adapts to the correct complexity scale. If $$t^*\in\arg\min_{t\in\mathbb R^d}\, \mathbb E\big(Y-\langle X,t\rangle\big)^2,$$5 and $$t^*\in\arg\min_{t\in\mathbb R^d}\, \mathbb E\big(Y-\langle X,t\rangle\big)^2,$$6 is sufficiently well approximated by an $$t^*\in\arg\min_{t\in\mathbb R^d}\, \mathbb E\big(Y-\langle X,t\rangle\big)^2,$$7-sparse vector, then with high probability

$$t^*\in\arg\min_{t\in\mathbb R^d}\, \mathbb E\big(Y-\langle X,t\rangle\big)^2,$$8

and

$$t^*\in\arg\min_{t\in\mathbb R^d}\, \mathbb E\big(Y-\langle X,t\rangle\big)^2,$$9

The confidence is exponential, which is the main contrast with standard LASSO under heavy tails (Lugosi et al., 2017).

A further extension handles unknown noise variance through a robust MOM analogue of square-root LASSO. The estimator is built from the stabilized functional

$$T_{K,\lambda}(g,f)=\operatorname{MOM}_K\!\big(\ell_f^\lambda-\ell_g^\lambda\big),\qquad \ell_f(x,y)=(y-f(x))^2,\qquad \ell_f^\lambda=\ell_f+\lambda\|f\|,$$0

aggregated by MOM over blocks and regularized by $$T_{K,\lambda}(g,f)=\operatorname{MOM}_K\!\big(\ell_f^\lambda-\ell_g^\lambda\big),\qquad \ell_f(x,y)=(y-f(x))^2,\qquad \ell_f^\lambda=\ell_f+\lambda\|f\|,$$1. The estimator is the saddle-point solution

$$T_{K,\lambda}(g,f)=\operatorname{MOM}_K\!\big(\ell_f^\lambda-\ell_g^\lambda\big),\qquad \ell_f(x,y)=(y-f(x))^2,\qquad \ell_f^\lambda=\ell_f+\lambda\|f\|,$$2

In the sparse linear case, this yields a robust square-root LASSO. Under weak moment conditions, with $$T_{K,\lambda}(g,f)=\operatorname{MOM}_K\!\big(\ell_f^\lambda-\ell_g^\lambda\big),\qquad \ell_f(x,y)=(y-f(x))^2,\qquad \ell_f^\lambda=\ell_f+\lambda\|f\|,$$3 and $$T_{K,\lambda}(g,f)=\operatorname{MOM}_K\!\big(\ell_f^\lambda-\ell_g^\lambda\big),\qquad \ell_f(x,y)=(y-f(x))^2,\qquad \ell_f^\lambda=\ell_f+\lambda\|f\|,$$4,

$$T_{K,\lambda}(g,f)=\operatorname{MOM}_K\!\big(\ell_f^\lambda-\ell_g^\lambda\big),\qquad \ell_f(x,y)=(y-f(x))^2,\qquad \ell_f^\lambda=\ell_f+\lambda\|f\|,$$5

and

$$T_{K,\lambda}(g,f)=\operatorname{MOM}_K\!\big(\ell_f^\lambda-\ell_g^\lambda\big),\qquad \ell_f(x,y)=(y-f(x))^2,\qquad \ell_f^\lambda=\ell_f+\lambda\|f\|,$$6

The paper also gives an adaptive Lepski-type version producing $$T_{K,\lambda}(g,f)=\operatorname{MOM}_K\!\big(\ell_f^\lambda-\ell_g^\lambda\big),\qquad \ell_f(x,y)=(y-f(x))^2,\qquad \ell_f^\lambda=\ell_f+\lambda\|f\|,$$7 when sparsity is unknown (Finocchio et al., 2021).

4. Posterior median under group-lasso-type priors

In Bayesian variable selection, the median operation appears in a different role. The grouped linear model is

$$T_{K,\lambda}(g,f)=\operatorname{MOM}_K\!\big(\ell_f^\lambda-\ell_g^\lambda\big),\qquad \ell_f(x,y)=(y-f(x))^2,\qquad \ell_f^\lambda=\ell_f+\lambda\|f\|,$$8

and the standard Bayesian group lasso assigns each group a multivariate Laplace prior

$$T_{K,\lambda}(g,f)=\operatorname{MOM}_K\!\big(\ell_f^\lambda-\ell_g^\lambda\big),\qquad \ell_f(x,y)=(y-f(x))^2,\qquad \ell_f^\lambda=\ell_f+\lambda\|f\|,$$9

To obtain exact group selection, the spike-and-slab group lasso adds a point mass at zero: $$T_{K,\lambda}(g,f)=\operatorname{MOM}_K(\ell_f-\ell_g)+\lambda(\|f\|-\|g\|).$$0 The posterior mode then corresponds to a criterion combining an $$T_{K,\lambda}(g,f)=\operatorname{MOM}_K(\ell_f-\ell_g)+\lambda(\|f\|-\|g\|).$$1 group penalty with an $$T_{K,\lambda}(g,f)=\operatorname{MOM}_K(\ell_f-\ell_g)+\lambda(\|f\|-\|g\|).$$2-like penalty on the number of active groups, while the posterior median becomes the thresholding estimator of interest (Xu et al., 2015).

The reason is structural. For spike-and-slab posteriors, the marginal posterior has a discrete spike at $$T_{K,\lambda}(g,f)=\operatorname{MOM}_K(\ell_f-\ell_g)+\lambda(\|f\|-\|g\|).$$3, so the posterior median can be exactly zero whenever the posterior mass at $$T_{K,\lambda}(g,f)=\operatorname{MOM}_K(\ell_f-\ell_g)+\lambda(\|f\|-\|g\|).$$4 is large enough. The paper generalizes the Johnstone–Silverman thresholding phenomenon to the multivariate setting: if

$$T_{K,\lambda}(g,f)=\operatorname{MOM}_K(\ell_f-\ell_g)+\lambda(\|f\|-\|g\|).$$5

then under suitable conditions there exists a threshold $$T_{K,\lambda}(g,f)=\operatorname{MOM}_K(\ell_f-\ell_g)+\lambda(\|f\|-\|g\|).$$6 such that

$$T_{K,\lambda}(g,f)=\operatorname{MOM}_K(\ell_f-\ell_g)+\lambda(\|f\|-\|g\|).$$7

Under block orthogonality, the least-squares estimator for each group is explicit, the posterior conditional is spike-and-slab, and the posterior median has an explicit soft-thresholding form. The threshold is random and adaptive through the posterior spike probability

$$T_{K,\lambda}(g,f)=\operatorname{MOM}_K(\ell_f-\ell_g)+\lambda(\|f\|-\|g\|).$$8

The asymptotic message differs from frequentist group lasso. Under orthogonal design, the posterior median has the oracle property if

$$T_{K,\lambda}(g,f)=\operatorname{MOM}_K(\ell_f-\ell_g)+\lambda(\|f\|-\|g\|).$$9

Then selection consistency and asymptotic normality on the active set both hold. By contrast, if

$K$0

the frequentist group lasso is not selection consistent, whereas forcing selection consistency requires

$K$1

which yields convergence rate $K$2 rather than the usual $K$3 rate. The same Bayesian logic is extended to sparse-group settings through bi-level spike-and-slab priors.

5. Distributed median aggregation and adaptive-Lasso IV selection

In distributed inference, the median is used to aggregate subset selections rather than blockwise losses. The MESSAGE procedure starts from the linear model

$K$4

randomly partitions the data into $K$5 disjoint subsets, performs variable selection independently on each subset using Lasso or GIC, collects the subset inclusion vectors

$K$6

and defines the final model by the coordinatewise median rule

$K$7

After the median model is selected, each subset refits by OLS on the selected variables and the final coefficient estimate is the arithmetic mean

$K$8

The theory gives model selection consistency for the median model and coefficient estimation efficiency for the averaged refit. The key probability amplification is that each subset selector needs to be correct with probability greater than $K$9, after which the median aggregation yields an exponential rate for recovering the true model via a Chernoff bound (Wang et al., 2014).

In multiple-exposure instrumental-variable models, the median is again used differently. The structural equation is

$\operatorname{MOM}_K$0

and the adaptive Lasso is applied to $\operatorname{MOM}_K$1 in the transformed regression

$\operatorname{MOM}_K$2

The required initial estimator is provided by a median-of-medians construction. For $\operatorname{MOM}_K$3, each instrument $\operatorname{MOM}_K$4 defines a column median

$\operatorname{MOM}_K$5

and the final estimator is

$\operatorname{MOM}_K$6

For general $\operatorname{MOM}_K$7, the procedure becomes a recursive median-of-medians-of-medians estimator, consistent under the majority condition

$\operatorname{MOM}_K$8

This yields a root-$\operatorname{MOM}_K$9 stable initial estimator and, under the adaptive-Lasso tuning conditions, consistent selection of the invalid set together with oracle asymptotics for the post-selection IV estimator (Liang et al., 2022).

6. Robustness notions, computation, and recurring distinctions

The general MOM viewpoint rewrites the oracle

$f$0

as the minimax problem

$f$1

then replaces the unknown expectation by

$f$2

over a partition into $f$3 blocks. The regularized estimator becomes

$f$4

with

$f$5

When $f$6, MOM reduces to the empirical mean, so the construction reduces to ERM or regularized ERM. The block-median mechanism is robust because outliers corrupt only a minority of blocks as long as $f$7 is at least about twice the number of outliers (Lecué et al., 2017).

This literature also formalizes robustness in non-asymptotic statistical terms. The learning-theoretic breakdown number is defined as the minimal number of added outliers needed to destroy a target rate and confidence level, and is of order

$f$8

For sparse LASSO-type estimation, this becomes

$f$9

whereas beyond that contamination level the error becomes of order

$$\mathcal B_{K,\lambda}(f)=\{g\in F: T_{K,\lambda}(g,f)\ge 0\},$$0

These statements align with the robust sparse bounds derived in MOM-LASSO and related procedures.

Computation is a notable feature of the MOM line. Standard optimization routines can be turned into MOM versions by selecting, at each iteration, the block whose empirical loss difference equals the current median over blocks and then performing an update on that block only. The paper gives MOM analogues of subgradient descent, proximal gradient descent / ISTA / FISTA, ADMM / Douglas–Rachford, and cyclic coordinate descent; it also proposes re-randomizing the block partition at each iteration and MOM $$\mathcal B_{K,\lambda}(f)=\{g\in F: T_{K,\lambda}(g,f)\ge 0\},$$1-fold cross-validation for tuning $$\mathcal B_{K,\lambda}(f)=\{g\in F: T_{K,\lambda}(g,f)\ge 0\},$$2 and $$\mathcal B_{K,\lambda}(f)=\{g\in F: T_{K,\lambda}(g,f)\ge 0\},$$3. This suggests that the unifying role of the median is operational rather than purely notational: in each family of estimators, it replaces a statistically unstable step by a rule that is less sensitive to heavy tails, outliers, or subset-level instability (Lecué et al., 2017).

A common misconception is that all LASSO-based median estimators are robust in exactly the same sense. The literature does not support that equivalence. In MOM-LASSO, tournament LASSO, and robust square-root LASSO, the median acts on blockwise criteria and directly controls contamination and weak moments. In Bayesian group selection, the posterior median is a thresholding estimator induced by a spike at zero. In MESSAGE, the median aggregates model inclusion across machines. In multiple-exposure IV, the median-of-medians provides an initial estimator for adaptive-Lasso weights. The phrase therefore names a family of median-regularized constructions rather than a single canonical estimator.