Subsampled Ensemble Can Improve Generalization Tail Exponentially (2405.14741v4)

Abstract: Ensemble learning is a popular technique to improve the accuracy of machine learning models. It traditionally hinges on the rationale that aggregating multiple weak models can lead to better models with lower variance and hence higher stability, especially for discontinuous base learners. In this paper, we provide a new perspective on ensembling. By selecting the best model trained on subsamples via majority voting, we can attain exponentially decaying tails for the excess risk, even if the base learner suffers from slow (i.e., polynomial) decay rates. This tail enhancement power of ensembling is agnostic to the underlying base learner and is stronger than variance reduction in the sense of exhibiting rate improvement. We demonstrate how our ensemble methods can substantially improve out-of-sample performances in a range of numerical examples involving heavy-tailed data or intrinsically slow rates. Code for the proposed methods is available at https://github.com/mickeyhqian/VoteEnsemble.

• The paper demonstrates that bagging at the parameter level converts polynomial decay in generalization errors to exponential decay.
• It introduces novel algorithms using majority and ε-optimal votes applicable to both discrete and continuous model spaces.
• Empirical results across diverse applications confirm significant improvements in model reliability and stability for heavy-tailed data.

Exponential Generalization through Bagging of Model Parameters

The paper pivots on the longstanding ensemble technique of bagging (Bootstrap Aggregating) in machine learning to offer a novel perspective on its utility, specifically focusing on improving the generalization performance of models. Historically, bagging contributes to variance reduction by resampling data and averaging predictions from multiple models. However, the researchers propose a significant shift: instead of the traditional output-level aggregation, they emphasize aggregation at the parametrization level, leading to an exponential decay in generalization errors, even under conditions commonly resulting in slow (polynomial) convergence rates such as heavy-tailed data distributions.

Main Contributions and Results

1. Theoretical Foundation:
   • The authors formulate a generic stochastic optimization problem:

   $$\min_{x \in \mathcal{X}} Z(x) := \mathbb{E}[h(x, \xi)],$$

   where $x$ represents the decision variable, and $\xi$ embodies the inherent randomness. - They prove that under scenarios where generalization errors decay polynomially, bagging can reduce these errors to an exponential decay. This assertion extends across conventional empirical risk minimization (ERM), distributionally robust optimization (DRO), and various regularization techniques. - The exponential decay is quantified such that for any general stochastic optimization problem with polynomially decaying generalization errors, bagging achieves:

   $$P\Big(Z(\hat{x}) > \min_{x \in \mathcal{X}} Z(x) + \delta \Big) \leq C_2 \gamma^{n/k},$$

   where $C_2$ and $\gamma$ are constants, and $k$ is a properly chosen sub-sample size.

2. Intuition and Mechanism:

   • For discrete solution spaces, the bagging approach involves a majority-vote mechanism where models frequently appearing among resamples are selected. This transforms the error convergence dynamics from heavy-tailed to robust exponential bounds, driven by bounded analyses of random indicator functions and U-statistics.
   • For continuous spaces, the approach adapts by voting on models within an $\epsilon$-optimal set considering subsample performance, hence circumventing the degeneracy issue of simple majority votes in such contexts.
3. Algorithms and Empirical Validation:
   • The authors propose several algorithms including a basic procedure for discrete solutions and more sophisticated ones for continuous spaces. Algorithm 1 involves bagging with majority vote whereas Algorithm 2 introduces $\epsilon$-Optimality Vote ensuring robustness across model spaces.
   • Extensive numerical experiments validate theoretical claims across varied problems such as resource allocation, supply chain network design, portfolio optimization, model selection, maximum weight matching, and linear programming. These experiments demonstrate not only the practical improvement but also the stability of proposed methods over traditional bagging.

Implications and Future Directions

The implications of this research are multifaceted. Generalization performance, particularly for models facing heavy-tailed data distributions, is a pivotal concern in modern machine learning applications inclusive of LLMs, finance, and physics. The exponential decay in errors introduced here versus traditional polynomial decay potentially shifts the benchmark for model reliability and effectiveness in these areas.

Theoretically, this paradigm aligns with and expands upon findings in risk minimization under heavy-tailed scenarios, offering a robust statistical basis for practitioners. Additionally, practical applications illuminated by the paper suggest that model bias and stability could witness significant improvements through versatilized bagging strategies.

Future Research

Directions for future investigations include extending the proposed framework to more complex machine learning architectures, such as deep neural networks, and exploring its integration with other robust statistical methods like Median-of-Means. Another promising avenue could be the empirical and theoretical exploration of the interaction between sample size $n$, sub-sample size $k$, and the number of aggregated models $B$ to further elucidate optimal strategies under various data conditions.

In conclusion, this paper provides a substantial leap in understanding and applying the principle of bagging to enhance generalization performances exponentially, reaffirming the pliability and potential of ensemble techniques in advanced machine learning paradigms.