Median-of-Means U-Statistics

• Median-of-Means U-Statistics is a robust estimator that partitions data into blocks, computes U-statistics for each, and takes their median to mitigate heavy-tailed effects and contamination.
• It achieves minimax-optimal deviation bounds under finite variance and mild moment conditions, even in the presence of outliers or adversarial contamination.
• The framework extends to high-dimensional settings and kernel methods, enabling robust risk minimization in a variety of modern statistical and machine learning tasks.

The median-of-means (MoM) U-statistic framework extends robust mean estimation to the estimation of general statistical functionals defined as U-statistics, allowing for minimax-optimal deviation bounds under only finite variance or mild moment conditions, even in the presence of heavy tails or adversarial contamination. This estimator operates by partitioning data into blocks, computing U-statistics within each block, and taking the median of these blockwise estimates, thereby safeguarding against the influence of outliers or heavy-tailed observations. The approach generalizes unions of robust statistics, high-probability bounds, and algorithmic learning guarantees to the context of U-processes, offering a toolkit for robust risk minimization in a variety of modern statistical and machine learning tasks.

1. Formal Definitions and Construction

Let $X_1, \dots, X_n$ be i.i.d. observations from distribution $P$ on a measurable space, and let $h: \mathcal{X}^m \to \mathbb{R}$ be a symmetric kernel with finite variance. The classical $m$-sample U-statistic is

$$U_n(h) = \frac{1}{\binom{n}{m}}\sum_{1 \leq i_1 < \cdots < i_m \leq n} h(X_{i_1}, \dots, X_{i_m}),$$

which is the unbiased minimum-variance estimator of the parameter $\theta(h) = \mathbb{E} h(X_1, ..., X_m)$. For heavy-tailed data or contaminated samples, the empirical U-statistic loses its sub-Gaussian deviation properties.

The median-of-means U-statistic (MoM-U) partitions the sample $\{1, ..., n\}$ into $K$ (possibly random or deterministic) blocks of size $B \approx n/K$. For each block $\mathcal{B}_k$, compute the block U-statistic $P$0, and define

$P$1

Randomized versions (where each block is drawn without replacement from the full sample) are denoted MoRU; both deterministic and randomized schemes achieve the same asymptotic rates (Laforgue et al., 2022).

A further permutation-invariant variant, viewing the estimator as the median over all possible subsample means from micro-blocks, can achieve optimal constants up to $P$2 factors under mild moment growth assumptions, via a U-statistic kernel of arbitrary order (Minsker, 2022, Fu et al., 2024).

2. Main Deviation Guarantees and Concentration Results

Under minimal moment assumptions, MoM-U estimators achieve high-probability deviation bounds essentially matching classical U-statistics for bounded or sub-Gaussian kernels, but remain robust for heavy-tailed or contaminated data.

For symmetric, square-integrable kernel $P$3, and confidence parameter $P$4, deviate bounds take the form: $P$5 with explicit dependence of the constants on $P$6, $P$7, and the variance/projection structure of $P$8 (Joly et al., 2015, Laforgue et al., 2022).

In the presence of arbitrary contamination, with $P$9 the outlier fraction and $h: \mathcal{X}^m \to \mathbb{R}$0, the estimator satisfies, with probability $h: \mathcal{X}^m \to \mathbb{R}$1,

$h: \mathcal{X}^m \to \mathbb{R}$2

where the blow-up constant $h: \mathcal{X}^m \to \mathbb{R}$3 diverges only as $h: \mathcal{X}^m \to \mathbb{R}$4 (Laforgue et al., 2020). For canonical kernels of order $h: \mathcal{X}^m \to \mathbb{R}$5, fully degenerate, the error rate improves to $h: \mathcal{X}^m \to \mathbb{R}$6.

If $h: \mathcal{X}^m \to \mathbb{R}$7 is only $h: \mathcal{X}^m \to \mathbb{R}$8-integrable for $h: \mathcal{X}^m \to \mathbb{R}$9, the deviation rate is $m$0 (Joly et al., 2015). In the permutation-invariant U-statistic MoM variant, provided $m$1 and mild polynomial moment conditions, the sub-Gaussian deviation bound is nearly sharp: $m$2 (Minsker, 2022).

3. Robustness to Outliers and Heavy Tails

The MoM-U approach tolerates contamination fractions $m$3, preserving the $m$4 rates (up to a constant) as long as the majority of blocks remain uncontaminated. This property extends to both univariate means and general U-statistics, including applications to pairwise losses and multivariate kernels (Laforgue et al., 2020). The blocking and median structure ensures that outliers or heavy-tailed blocks are discarded, as the median selects a value from the majority of "sane" blocks.

This estimator also attains optimal rates and sub-Gaussian tails for kernel mean embeddings and maximum mean discrepancy (MMD) in reproducing kernel Hilbert spaces—up to $m$5 arbitrary sample corruptions can be handled while maintaining consistency (Lerasle et al., 2018).

4. Comparison with Classical (Unrobust) U-Statistics

Classical U-statistics require boundedness or sub-Gaussian tails for concentration, failing for heavy-tailed or adversarial data (e.g., when the kernel is $m$6-stable). In such regimes, deviation inequalities no longer hold (Joly et al., 2015). MoM-U estimators, however, only assume finite variance (or finite $m$7-th moment) and maintain high-confidence, nonasymptotic bounds.

A comparison of rates:

Method	Assumptions	Deviation Rate (degenerate $m$8)	Constant (order)
Classical U-stat ($m$9 bounded) [Arcones-Ginè]	Boundedness	$$U_n(h) = \frac{1}{\binom{n}{m}}\sum_{1 \leq i_1 < \cdots < i_m \leq n} h(X_{i_1}, \dots, X_{i_m}),$$0	$$U_n(h) = \frac{1}{\binom{n}{m}}\sum_{1 \leq i_1 < \cdots < i_m \leq n} h(X_{i_1}, \dots, X_{i_m}),$$1
MoM-U (finite variance) (Joly et al., 2015, Laforgue et al., 2022)	$$U_n(h) = \frac{1}{\binom{n}{m}}\sum_{1 \leq i_1 < \cdots < i_m \leq n} h(X_{i_1}, \dots, X_{i_m}),$$2	$$U_n(h) = \frac{1}{\binom{n}{m}}\sum_{1 \leq i_1 < \cdots < i_m \leq n} h(X_{i_1}, \dots, X_{i_m}),$$3	$$U_n(h) = \frac{1}{\binom{n}{m}}\sum_{1 \leq i_1 < \cdots < i_m \leq n} h(X_{i_1}, \dots, X_{i_m}),$$4
MoM-U (contamination, bounded $$U_n(h) = \frac{1}{\binom{n}{m}}\sum_{1 \leq i_1 < \cdots < i_m \leq n} h(X_{i_1}, \dots, X_{i_m}),$$5) (Laforgue et al., 2020)	Outliers $$U_n(h) = \frac{1}{\binom{n}{m}}\sum_{1 \leq i_1 < \cdots < i_m \leq n} h(X_{i_1}, \dots, X_{i_m}),$$6	$$U_n(h) = \frac{1}{\binom{n}{m}}\sum_{1 \leq i_1 < \cdots < i_m \leq n} h(X_{i_1}, \dots, X_{i_m}),$$7	$$U_n(h) = \frac{1}{\binom{n}{m}}\sum_{1 \leq i_1 < \cdots < i_m \leq n} h(X_{i_1}, \dots, X_{i_m}),$$8

For permutation-invariant MoM-U (growing-order U-statistic), the deviation constants are nearly optimal, matching the Gaussian minimax constant up to $$U_n(h) = \frac{1}{\binom{n}{m}}\sum_{1 \leq i_1 < \cdots < i_m \leq n} h(X_{i_1}, \dots, X_{i_m}),$$9 (Minsker, 2022, Fu et al., 2024).

5. Extensions, Algorithmic Schemes, and Applications

Median-of-means U-statistics have been applied throughout robust learning, especially for pairwise or higher-order losses. Key algorithmic mechanisms include block-wise risk minimization and median-of-blocks (MoM-U) gradient descent for parameter learning; these approaches ensure robust excess risk bounds even in contaminated settings: $\theta(h) = \mathbb{E} h(X_1, ..., X_m)$0 for a VC-class hypothesis set $\theta(h) = \mathbb{E} h(X_1, ..., X_m)$1 and bounded loss (Laforgue et al., 2020).

The method generalizes to randomized block construction (random resampling of blocks) and incomplete U-statistic computation for computational efficiency (Laforgue et al., 2022, Fu et al., 2024). Applications include:

• Robust kernel mean embedding and MMD, via blockwise geometric medians in Hilbert space (the "MONK" algorithm), relevant for distributional hypothesis testing and two-sample testing (Lerasle et al., 2018)
• Robust clustering, by plugging MoM-U estimators into within-cluster dissimilarity objectives, yielding oracle inequalities and minimax rates under only finite variance and low-noise conditions (Joly et al., 2015)
• Classical shadows in quantum tomography, where U-statistic MoM estimators optimize the measurement complexity and improve deviation constants in high-dimensional expectation estimation (Fu et al., 2024)

6. Proof Techniques and Structural Insights

MoM-U deviation bounds rest on a two-level combination of classical U-statistic and median-of-means arguments, relying crucially on the Hoeffding (or Hájek) projection decomposition:

• For each blockwise U-statistic, block independence (or near-independence) enables Chebyshev or Hoeffding-type concentration for the block estimators.
• The median-of-means principle discards up to $\theta(h) = \mathbb{E} h(X_1, ..., X_m)$2 corrupted blocks without biasing the central estimator.
• In higher-order or randomized settings, concentration for the median is derived from Bernoulli tail bounds (binomial deviations) for the proportion of "good" blocks (Laforgue et al., 2022, Laforgue et al., 2020).
• For growing-order U-statistics, symmetrization and control of higher-order degenerate terms imply that the estimator inherits the tight deviation properties of first-order (mean-like) projections, with all higher-order terms exponentially negligible (Minsker, 2022).

7. Practical Considerations, Limitations, and Current Frontiers

The choice of block size and number is critical, with the standard recommendation $\theta(h) = \mathbb{E} h(X_1, ..., X_m)$3 for confidence $\theta(h) = \mathbb{E} h(X_1, ..., X_m)$4. Computational trade-offs appear in large-scale or high-order U-statistic scenarios, leading to incomplete U-statistics or randomized block designs (random or cyclic sampling of micro-blocks) for tractable implementation at modest efficiency loss (Fu et al., 2024).

A distinctive feature is the retention of $\theta(h) = \mathbb{E} h(X_1, ..., X_m)$5 rates and tolerance of up to $\theta(h) = \mathbb{E} h(X_1, ..., X_m)$6 adversarial contamination—a sharp breakdown point. However, as $\theta(h) = \mathbb{E} h(X_1, ..., X_m)$7, constants diverge; performance deteriorates for high contamination fractions. The methods also necessitate careful handling of block dependencies for higher-order U-statistics, often requiring conditional concentration or diagonal block restrictions (Laforgue et al., 2020).

Recent work emphasizes sharper deviation constants via permutation-invariant U-statistics and extends the robust MoM-U methodology to risk minimization, kernel methods, clustering, and large-scale quantum information settings (Minsker, 2022, Lerasle et al., 2018, Fu et al., 2024). Open questions remain on optimal adaptivity to unknown moment conditions and computationally efficient, high-order, high-dimensional variants.

---

Key References:

• "Robust estimation of U-statistics" (Joly et al., 2015)
• "Generalization Bounds in the Presence of Outliers: a Median-of-Means Study" (Laforgue et al., 2020)
• "On Medians of (Randomized) Pairwise Means" (Laforgue et al., 2022)
• "MONK -- Outlier-Robust Mean Embedding Estimation by Median-of-Means" (Lerasle et al., 2018)
• "U-statistics of growing order and sub-Gaussian mean estimators with sharp constants" (Minsker, 2022)
• "Classical Shadows with Improved Median-of-Means Estimation" (Fu et al., 2024)