Median-of-Means U-Statistics

• Median-of-Means U-Statistics is a robust estimator that partitions data into blocks, computes U-statistics for each, and takes their median to mitigate heavy-tailed effects and contamination.
• It achieves minimax-optimal deviation bounds under finite variance and mild moment conditions, even in the presence of outliers or adversarial contamination.
• The framework extends to high-dimensional settings and kernel methods, enabling robust risk minimization in a variety of modern statistical and machine learning tasks.

The median-of-means (MoM) U-statistic framework extends robust mean estimation to the estimation of general statistical functionals defined as U-statistics, allowing for minimax-optimal deviation bounds under only finite variance or mild moment conditions, even in the presence of heavy tails or adversarial contamination. This estimator operates by partitioning data into blocks, computing U-statistics within each block, and taking the median of these blockwise estimates, thereby safeguarding against the influence of outliers or heavy-tailed observations. The approach generalizes unions of robust statistics, high-probability bounds, and algorithmic learning guarantees to the context of U-processes, offering a toolkit for robust risk minimization in a variety of modern statistical and machine learning tasks.

1. Formal Definitions and Construction

Let $X_1, \dots, X_n$ be i.i.d. observations from distribution $P$ on a measurable space, and let $h: \mathcal{X}^m \to \mathbb{R}$ be a symmetric kernel with finite variance. The classical $m$-sample U-statistic is

$$U_n(h) = \frac{1}{\binom{n}{m}}\sum_{1 \leq i_1 < \cdots < i_m \leq n} h(X_{i_1}, \dots, X_{i_m}),$$

which is the unbiased minimum-variance estimator of the parameter $\theta(h) = \mathbb{E} h(X_1, ..., X_m)$. For heavy-tailed data or contaminated samples, the empirical U-statistic loses its sub-Gaussian deviation properties.

The median-of-means U-statistic (MoM-U) partitions the sample $\{1, ..., n\}$ into $K$ (possibly random or deterministic) blocks of size $B \approx n/K$. For each block $\mathcal{B}_k$, compute the block U-statistic $\hat{U}_k(h)$, and define

$$\hat{\theta}_{\text{MoU}}(h) = \mathrm{median}(\hat{U}_1(h), ..., \hat{U}_K(h)).$$

Randomized versions (where each block is drawn without replacement from the full sample) are denoted MoRU; both deterministic and randomized schemes achieve the same asymptotic rates (Laforgue et al., 2022).

A further permutation-invariant variant, viewing the estimator as the median over all possible subsample means from micro-blocks, can achieve optimal constants up to $1 + o(1)$ factors under mild moment growth assumptions, via a U-statistic kernel of arbitrary order (Minsker, 2022, Fu et al., 4 Dec 2024).

2. Main Deviation Guarantees and Concentration Results

Under minimal moment assumptions, MoM-U estimators achieve high-probability deviation bounds essentially matching classical U-statistics for bounded or sub-Gaussian kernels, but remain robust for heavy-tailed or contaminated data.

For symmetric, square-integrable kernel $h$, and confidence parameter $\delta \in (0,1)$, deviate bounds take the form: $$\Pr\left( |\hat{\theta}_{\text{MoU}}(h) - \theta(h)| \geq O\left( \sqrt{\frac{\log(1/\delta)}{n}} \right) \right) \leq \delta,$$ with explicit dependence of the constants on $m$, $\|h\|_\infty$, and the variance/projection structure of $h$ (Joly et al., 2015, Laforgue et al., 2022).

In the presence of arbitrary contamination, with $\epsilon = n_0 / n$ the outlier fraction and $K > 2 n_0$, the estimator satisfies, with probability $1 - \delta$,

$$|\hat{\theta}_{\text{MoU}}(h) - \theta(h)| \leq C\, \Gamma(\epsilon) \sqrt{\frac{\log(1/\delta)}{n}},$$

where the blow-up constant $\Gamma(\epsilon)$ diverges only as $\epsilon \rightarrow 1/2$ (Laforgue et al., 2020). For canonical kernels of order $m=2$, fully degenerate, the error rate improves to $O((1 + \log(1/\delta)) / n)$.

If $h$ is only $p$-integrable for $1 < p \leq 2$, the deviation rate is $O\left( ( \log(1/\delta) / n )^{m(p-1)/p} \right)$ (Joly et al., 2015). In the permutation-invariant U-statistic MoM variant, provided $m = o(N)$ and mild polynomial moment conditions, the sub-Gaussian deviation bound is nearly sharp: $$\Pr\left( |\hat{\mu}_N - \mu| \geq t \right) \leq 2 \exp\left( - (1-o(1)) \frac{N t^2}{2 \sigma^2} \right)$$ (Minsker, 2022).

3. Robustness to Outliers and Heavy Tails

The MoM-U approach tolerates contamination fractions $\epsilon < 1/2$, preserving the $O(n^{-1/2})$ rates (up to a constant) as long as the majority of blocks remain uncontaminated. This property extends to both univariate means and general U-statistics, including applications to pairwise losses and multivariate kernels (Laforgue et al., 2020). The blocking and median structure ensures that outliers or heavy-tailed blocks are discarded, as the median selects a value from the majority of "sane" blocks.

This estimator also attains optimal rates and sub-Gaussian tails for kernel mean embeddings and maximum mean discrepancy (MMD) in reproducing kernel Hilbert spaces—up to $n/4$ arbitrary sample corruptions can be handled while maintaining consistency (Lerasle et al., 2018).

4. Comparison with Classical (Unrobust) U-Statistics

Classical U-statistics require boundedness or sub-Gaussian tails for concentration, failing for heavy-tailed or adversarial data (e.g., when the kernel is $\alpha$-stable). In such regimes, deviation inequalities no longer hold (Joly et al., 2015). MoM-U estimators, however, only assume finite variance (or finite $p$-th moment) and maintain high-confidence, nonasymptotic bounds.

A comparison of rates:

Method	Assumptions	Deviation Rate (degenerate $m=2$)	Constant (order)
Classical U-stat ($\|h\|_\infty$ bounded) [Arcones-Ginè]	Boundedness	$O( \log(1/\delta)/n )$	$c_1 \|h\|_\infty$
MoM-U (finite variance) (Joly et al., 2015, Laforgue et al., 2022)	$\mathbb{E}[h^2]<\infty$	$O( \sqrt{ \log(1/\delta) / n })$	$C_m \sigma$
MoM-U (contamination, bounded $h$) (Laforgue et al., 2020)	Outliers $\epsilon<1/2$	$O( \Gamma(\epsilon) \sqrt{ \log(1/\delta) / n })$	$4\sqrt{d} M \Gamma(\epsilon)$

For permutation-invariant MoM-U (growing-order U-statistic), the deviation constants are nearly optimal, matching the Gaussian minimax constant up to $1 + o(1)$ (Minsker, 2022, Fu et al., 4 Dec 2024).

5. Extensions, Algorithmic Schemes, and Applications

Median-of-means U-statistics have been applied throughout robust learning, especially for pairwise or higher-order losses. Key algorithmic mechanisms include block-wise risk minimization and median-of-blocks (MoM-U) gradient descent for parameter learning; these approaches ensure robust excess risk bounds even in contaminated settings: $$R(\hat{g}) - R(g^\star) \leq C\, \Gamma(\epsilon) \sqrt{ (d_{VC} (1+\log n) + \log(1/\delta)) / n }$$ for a VC-class hypothesis set $G$ and bounded loss (Laforgue et al., 2020).

The method generalizes to randomized block construction (random resampling of blocks) and incomplete U-statistic computation for computational efficiency (Laforgue et al., 2022, Fu et al., 4 Dec 2024). Applications include:

• Robust kernel mean embedding and MMD, via blockwise geometric medians in Hilbert space (the "MONK" algorithm), relevant for distributional hypothesis testing and two-sample testing (Lerasle et al., 2018)
• Robust clustering, by plugging MoM-U estimators into within-cluster dissimilarity objectives, yielding oracle inequalities and minimax rates under only finite variance and low-noise conditions (Joly et al., 2015)
• Classical shadows in quantum tomography, where U-statistic MoM estimators optimize the measurement complexity and improve deviation constants in high-dimensional expectation estimation (Fu et al., 4 Dec 2024)

6. Proof Techniques and Structural Insights

MoM-U deviation bounds rest on a two-level combination of classical U-statistic and median-of-means arguments, relying crucially on the Hoeffding (or Hájek) projection decomposition:

• For each blockwise U-statistic, block independence (or near-independence) enables Chebyshev or Hoeffding-type concentration for the block estimators.
• The median-of-means principle discards up to $K/2$ corrupted blocks without biasing the central estimator.
• In higher-order or randomized settings, concentration for the median is derived from Bernoulli tail bounds (binomial deviations) for the proportion of "good" blocks (Laforgue et al., 2022, Laforgue et al., 2020).
• For growing-order U-statistics, symmetrization and control of higher-order degenerate terms imply that the estimator inherits the tight deviation properties of first-order (mean-like) projections, with all higher-order terms exponentially negligible (Minsker, 2022).

7. Practical Considerations, Limitations, and Current Frontiers

The choice of block size and number is critical, with the standard recommendation $K \simeq c \log(1/\delta)$ for confidence $1 - \delta$. Computational trade-offs appear in large-scale or high-order U-statistic scenarios, leading to incomplete U-statistics or randomized block designs (random or cyclic sampling of micro-blocks) for tractable implementation at modest efficiency loss (Fu et al., 4 Dec 2024).

A distinctive feature is the retention of $O(n^{-1/2})$ rates and tolerance of up to $\epsilon < 1/2$ adversarial contamination—a sharp breakdown point. However, as $\epsilon \to 1/2$, constants diverge; performance deteriorates for high contamination fractions. The methods also necessitate careful handling of block dependencies for higher-order U-statistics, often requiring conditional concentration or diagonal block restrictions (Laforgue et al., 2020).

Recent work emphasizes sharper deviation constants via permutation-invariant U-statistics and extends the robust MoM-U methodology to risk minimization, kernel methods, clustering, and large-scale quantum information settings (Minsker, 2022, Lerasle et al., 2018, Fu et al., 4 Dec 2024). Open questions remain on optimal adaptivity to unknown moment conditions and computationally efficient, high-order, high-dimensional variants.

---

Key References:

• "Robust estimation of U-statistics" (Joly et al., 2015)
• "Generalization Bounds in the Presence of Outliers: a Median-of-Means Study" (Laforgue et al., 2020)
• "On Medians of (Randomized) Pairwise Means" (Laforgue et al., 2022)
• "MONK -- Outlier-Robust Mean Embedding Estimation by Median-of-Means" (Lerasle et al., 2018)
• "U-statistics of growing order and sub-Gaussian mean estimators with sharp constants" (Minsker, 2022)
• "Classical Shadows with Improved Median-of-Means Estimation" (Fu et al., 4 Dec 2024)