Tilted Empirical Risk Minimization (2007.01162v2)

Abstract: Empirical risk minimization (ERM) is typically designed to perform well on the average loss, which can result in estimators that are sensitive to outliers, generalize poorly, or treat subgroups unfairly. While many methods aim to address these problems individually, in this work, we explore them through a unified framework -- tilted empirical risk minimization (TERM). In particular, we show that it is possible to flexibly tune the impact of individual losses through a straightforward extension to ERM using a hyperparameter called the tilt. We provide several interpretations of the resulting framework: We show that TERM can increase or decrease the influence of outliers, respectively, to enable fairness or robustness; has variance-reduction properties that can benefit generalization; and can be viewed as a smooth approximation to a superquantile method. We develop batch and stochastic first-order optimization methods for solving TERM, and show that the problem can be efficiently solved relative to common alternatives. Finally, we demonstrate that TERM can be used for a multitude of applications, such as enforcing fairness between subgroups, mitigating the effect of outliers, and handling class imbalance. TERM is not only competitive with existing solutions tailored to these individual problems, but can also enable entirely new applications, such as simultaneously addressing outliers and promoting fairness.

• The paper introduces tilted empirical risk minimization (TERM), extending ERM by using a tilt parameter to modulate loss aggregation for improved robustness, fairness, and variance reduction.
• The authors develop batch and stochastic first-order optimization methods that preserve computational efficiency across a spectrum of tilt values, even under challenging nonsmooth conditions.
• Empirical results show that TERM enhances model generalization and subgroup fairness while effectively mitigating outlier impact and handling label noise.

Tilted Empirical Risk Minimization: A Unified Framework for Robustness, Fairness, and Variance Reduction

Empirical risk minimization (ERM) has long been foundational in statistical estimation, but its susceptibility to outliers and shortcomings in achieving fairness across subgroups have prompted the exploration of alternative methodologies. This paper introduces tilted empirical risk minimization (TERM), a versatile framework that extends ERM through a hyperparameter, referred to as the tilt, which adjusts the influence of individual data points. The framework aims to address issues such as sensitivity to outliers, fairness among different subgroups, and poor generalization by modulating how losses are aggregated.

Framework and Methodology

TERM is defined by the objective in Equation 2 of the paper, where the tilt parameter $t$ enables the control of loss aggregation. The framework unifies several approaches: for $t = 0$, TERM reduces to ERM; as $t \to +\infty$, it approximates min-max formulations; and as $t \to -\infty$, it approaches min-min problems. This flexibility allows for a continuum of solutions between min and max losses, providing adaptability beyond conventional ERM strategies.

The authors propose batch and stochastic first-order optimization methods tailored to solve TERM efficiently. These methods maintain the computational tractability of traditional ERM approaches, even for $t \to \pm \infty$, which typically suffer from slower convergence due to reduced smoothness in objective landscapes.

Computational Properties and Interpretation

TERM leverages exponential smoothing, a technique frequently employed in approximation methods, herein to approximate superquantile methods—useful in scenarios where loss quantiles are more informative than average loss. The authors provide several results detailing TERM's theoretical underpinnings, including:

• Gradient Re-weighting: Celebrated in Lemma 1, TERM facilitates altering sample influence dynamically based on their loss, therefore magnifying or suppressing outliers contingent on the tilt $t$.
• Bias-Variance Tradeoff: Theorem 6 illustrates TERM's capacity for variance reduction and generalization improvement via modulating $t$, thereby attaining better bias-variance balancing for enhanced out-of-sample performance.

Moreover, the solution continuity across varying tilts underscores TERM's potential to adapt to diverse datasets with smooth adjustments.

Empirical Validation

Extensive empirical evaluations substantiate TERM's effectiveness over a multitude of applications, including mitigating label noise in classification settings, ensuring fairer principal component analysis (PCA), and addressing class imbalance and suboptimal generalization:

• Robust Regression and Classification: TERM outperforms robust variants of regression in regimes of high label and input noise, offering a competitive edge in scenarios traditionally challenging for ERM.
• Fairness and Generalization: By exploiting positive tilt values, TERM improves fairness, yielding more balanced subgroup performance while promoting more uniform representations—a critical advancement in fairness-aware learning algorithms.

Additionally, hierarchical TERM implementations adeptly handle compound issues like simultaneous class imbalance and noise—a testament to the framework's extensibility.

Implications and Future Directions

Remarkably versatile, TERM exemplifies a minimalist yet potent extension to ERM—enabling novel insights into robustness, fairness, and loss variance control across both theoretical and practical domains. Importantly, it prompts ongoing discourse on the implications of loss re-weighting and smooth approximations in real-world contexts, inviting subsequent inquiry into generalization bounds and theoretical guarantees for the proposed stochastic optimization solvers.

In conclusion, TERM signifies a noteworthy advancement in machine learning methodologies, offering a streamlined path towards achieving robust, fair, and generalizable models. As research progresses, it remains imperative to interrogate the broader applications and limitations, ensuring the ethical deployment of such innovations in diverse learning environments.