Stability beyond Bounded Differences: Sharp Generalization Bounds under Finite $L_p$ Moments

Abstract: While algorithmic stability is a central tool for understanding generalization of learning algorithms, existing high-probability guarantees typically rely on uniform boundedness or sub-Gaussian/sub-Weibull tail assumptions, which can be overly restrictive for modern settings with heavy-tailed or unbounded losses. We develop a stability-based framework that requires only a finite $L_p$ moment condition. Our first contribution is sharp concentration inequalities for functions of independent random variables under $L_p$ constraints, extending McDiarmid's bounded-differences techniques beyond the classical regime. Leveraging these results, we derive sharp high-probability generalization bounds across a range of learning paradigms, including empirical risk minimization, transductive regression, and meta-learning. These guarantees show that $L_p$ stability suffices for robust generalization even when boundedness fails, substantially weakening the standard assumptions in the stability literature.

• The paper introduces a novel framework for algorithmic stability using finite L_p moments, enabling sharp concentration bounds for heavy-tailed losses.
• It extends McDiarmid’s inequality to unbounded settings, providing robust high-probability generalization guarantees for ERM, transductive regression, and meta-learning.
• The results reveal a dual tail regime with sub-Gaussian behavior for moderate deviations and polynomial decay for extreme events, enhancing risk assessment.

Stability under Finite $L_p$ Moments: A Technical Analysis of Modern Generalization Bounds

Overview and Motivation

Algorithmic stability is central to theoretical generalization guarantees in statistical learning. Classical approaches—most notably founded on uniform stability and bounded loss functions—enable concentration results via tools such as McDiarmid’s inequality, yielding tight high-probability deviation bounds. However, modern machine learning applications, especially those encountering unbounded or heavy-tailed losses (e.g., deep networks, robust/contaminated regression), frequently violate these restrictive assumptions. While recent work has relaxed tails (e.g., via sub-Gaussian or sub-Weibull surrogates), these still require existence of exponential moments and thus fail to accommodate the power-law phenomena observed empirically in contemporary models.

The paper "Stability beyond Bounded Differences: Sharp Generalization Bounds under Finite $L_p$ Moments" (2606.06855) introduces a framework that leverages only a finite $L_p$ moment condition for algorithmic stability. The core contribution is a suite of sharp concentration inequalities that extend McDiarmid-type results to the heavy-tailed regime, and high-probability generalization guarantees for canonical learning setups—empirical risk minimization (ERM), transductive regression, and meta-learning—under $(L_p, \beta)$-Lipschitz stability. The bounds feature a two-regime tail: sub-Gaussian for moderate deviations and polynomial for extreme events, reflecting precisely the distributional properties permitted by finite-$L_p$ increments.

Theoretical Contributions: Concentration without Exponential Moments

$(L_p, \beta)$-Lipschitz Stability

The technical linchpin is a generalized notion of stability: For independent samples $x_1,\ldots,x_n$ and a measurable statistic $f(x_1,\ldots,x_n)$,

• The sensitivity of $f$ to coordinate $i$ is controlled not by a deterministic bound, but by a random variable $L_p$0, where $L_p$1 is an independent copy.
• The only assumption is $L_p$2 for some $L_p$3.

This setup is strictly weaker than both bounded differences and Orlicz/sub-Weibull diameter conditions. The approach naturally models scenarios where one-sample perturbations can be unbounded but their distributions possess sufficiently light polynomial tails.

Main Concentration Inequality

A fundamental result (Theorem 1) is as follows: If $L_p$4 for each $L_p$5 and the $L_p$6 satisfy $L_p$7, then, for any $L_p$8,

$L_p$9

with $L_p$0 dependent on $L_p$1. Notably, this captures both moderate and large deviations. For $L_p$2, the tail is essentially polynomial, matching the heavy-tailed regime.

Figure 1: Plot of $L_p$3 versus $L_p$4 for a heavy-tailed linear regression; stabilization near $L_p$5 confirms the polynomial tail in high deviations.

Sharpness in the tail-order sense is guaranteed by reference to the classical Fuk–Nagaev inequalities: the coexistence of sub-Gaussian and polynomial regimes is unavoidable under mere $L_p$6 assumptions.

Applications: Robust Generalization in Modern Learning

Empirical Risk Minimization (ERM)

For ERM in the i.i.d. setting, the paper proves high-probability oracle inequalities for the gap between the population risk and empirical risk (or leave-one-out estimate), under $L_p$7-Lipschitz stability: $L_p$8 where $L_p$9 controls the fluctuation of the loss across test samples. Unlike uniform stability, vanishing generalization now requires not only $(L_p, \beta)$0 but also that stability coefficients $(L_p, \beta)$1 decay faster than $(L_p, \beta)$2.

Transductive Regression

For transductive setups (where train and test points are splits of a fixed finite population), the paper develops concentration results for sampling without replacement, extending bounded differences to the finite-moment scenario. The generalization bound for the difference between the test and training error again features both Gaussian and heavy-tailed terms, now scaled by subsample sizes $(L_p, \beta)$3 and $(L_p, \beta)$4 and their moments.

Figure 3: Plot of $(L_p, \beta)$5 versus $(L_p, \beta)$6 showing robustness of polynomial tail ratio across increasing train sizes in transductive regression.

Meta-Learning

The meta-learning framework considers a learn-to-learn setting: a meta-learner is trained across tasks (task distributions drawn i.i.d. from a meta-distribution), and performance is measured on new unseen tasks. The analysis demands meta-stability both across training tasks and within-task samples. The result is a high-probability bound for the meta-generalization error, which, while retaining a polynomial tail, only presupposes finite $(L_p, \beta)$7-moment meta-stability.

Empirical Results: Verifying the Two-Regime Theory

Large-scale simulations in regression (linear and two-layer neural networks) establish the theoretical sharpness of the proposed bounds. The key diagnostic, $(L_p, \beta)$8, stabilizes at $(L_p, \beta)$9 for large $L_p$0, as predicted for polynomial tails, and displays sub-Gaussian scaling in the moderate-deviation regime.

Figure 4: Plot of $L_p$1 versus $L_p$2 on a neural network regression task confirming both sub-Gaussian and polynomial regimes for heavy-tailed data under ADAM optimization.

These results highlight in particular that classical bounds, by omitting such polynomial terms, can dramatically underestimate tail risks in presence of heavy-tailed stability increments.

Implications and Future Directions

The move to $L_p$3-based stability fundamentally broadens the applicability of concentration-based generalization theory. In practical terms, this guarantees that algorithms can be certified for robust out-of-sample performance—even when data or losses violate exponential tail conditions—provided that finite moments and demonstrable asymptotic stability are present.

Theoretical implications include:

• Non-exponential tails are fundamental: No uniform $L_p$4 (Orlicz) exponential-type bound can be valid in Pareto-like or power-law heavy-tailed settings.
• Finite-moment stability is both necessary and sufficient: In such regimes, the polynomial term is not a technicality but a lower-bound-matching feature.
• Sharper understanding of "rare" overfitting: While moderate errors remain tightly controlled, rare extreme deviations are quantifiably possible—a risk that traditional uniform stability obfuscates.

Practically, this framework invites further empirical investigation and supports algorithm design for robust generalization:

• Design and tuning of regularization/hyperparameters to enforce rapid decay of stability coefficients.
• Algorithmic policies (e.g., clipped gradients, robust means) that can ensure finite moments even in highly non-ideal data regimes.

Figure 2: Plot of $L_p$5 versus $L_p$6 for transductive ridge regression, highlighting the two-regime tail and alignment with theoretical predictions.

Open directions include studying stability in the presence of dependent data, fully unsupervised settings, and matching lower bounds for various learning paradigms under only $L_p$7-moment assumptions.

Conclusion

This work decisively advances the theory of algorithmic stability for generalization beyond bounded or sub-exponential settings, providing precise and sharp probabilistic bounds under minimal finite-moment conditions. The framework and concentration inequalities formalize, for the first time, the exact way in which heavy-tailed data and losses impact generalization in modern learning algorithms. This creates a robust bridge from abstract stability concepts to the realities of deep and heavy-tailed data regimes in modern AI.