Tilted Empirical Risk Minimization (TERM)

• TERM is a robust learning framework that replaces the arithmetic mean with an exponential aggregation, enabling a tunable trade-off between average, minimum, and maximum losses.
• It applies an exponential reweighting of per-sample losses to enhance robustness against outliers and heavy-tailed data, leading to improved generalization bounds.
• The framework supports fairness by emphasizing high-loss subpopulations, making it suitable for diverse tasks including regression, classification, and online anomaly detection.

Tilted Empirical Risk Minimization (TERM) is an extension of the classical empirical risk minimization paradigm in statistical learning. By integrating exponential tilting into the risk aggregation process, TERM introduces a tunable hyperparameter that enables flexible trade-offs among average-case performance, robustness to outliers, and fairness across subpopulations. The framework generalizes the standard sample-mean-based ERM by replacing the arithmetic mean of the sample losses with a nonlinear, exponentially weighted aggregation, thereby interpolating between min-loss, average-loss, and max-loss objectives. This operation, and its variants, are directly related to log-sum-exponential constructions and connect TERM to several pillars of modern risk-sensitive and robust learning.

1. Mathematical Formulation and Core Principle

Let $S = \{z_1, \ldots, z_n\}$ be a training dataset, $\ell(h, z)$ a loss function for hypothesis $h$, and $t$ (or $\gamma$) the scalar tilt parameter. The classical ERM objective

$$\text{ERM:}\quad \hat{h}_{\text{ERM}} = \arg\min_{h\in\mathcal{H}}~\frac{1}{n}\sum_{i=1}^n \ell(h, z_i)$$

is replaced in TERM by the tilted empirical risk:

$$R_t(h; S) = \frac{1}{t}\log\left(\frac{1}{n}\sum_{i=1}^n \exp\left( t \ell(h, z_i) \right) \right),$$

with $t \to 0$ recovering standard ERM. Positive values of $t$ emphasize higher losses (approximate max-loss as $t \to +\infty$), whereas negative values of $t$ suppress large losses (approximate min-loss as $t \to -\infty$) (Li et al., 2020, Li et al., 2021, Aminian et al., 28 Sep 2024).

The gradient of TERM with respect to model parameters $\theta$ is a weighted sum of per-sample gradients,

$$\nabla_\theta R_t(\theta) = \sum_{i=1}^n w_i(t; \theta) \nabla_\theta \ell(\theta, z_i),$$

where

$$w_i(t; \theta) = \frac{\exp \left( t\ell(\theta, z_i) \right)}{ \sum_j \exp \left( t\ell(\theta, z_j) \right)}.$$

This exponential reweighting mechanism is central: for $t > 0$, high-loss samples incur greater weight; for $t < 0$, large losses are downweighted. This log-sum-exp structure is a smooth approximation to the maximum function, yielding a continuum between mean and worst-case formulations.

2. Theoretical Properties and Generalization Bounds

Sub-Gaussian and Robust Estimation

In settings with heavy-tailed losses, naive sample means may exhibit high variance and lack concentration. Early robust TERM formulations replaced the empirical mean with robust mean estimators such as Catoni's estimator:

$$\mu_f \text{ defined by } r_f(\mu_f) = \frac{1}{n\alpha} \sum_{i=1}^n \phi\left(\alpha(f(X_i) - \mu_f)\right) = 0,$$

where $\phi(\cdot)$ is a truncation function and $\alpha$ is tuned based on data variance and confidence parameters. Such estimators provide sub-Gaussian deviation bounds under mere finite variance conditions, even for heavy-tailed or unbounded losses (Brownlees et al., 2014).

Performance Bounds

Generalization error for TERM is bounded using both uniform and information-theoretic analyses. For bounded losses ($0 \leq \ell \leq M$), uniform deviation bounds

$$\sup_{h \in \mathcal{H}} \left| R(h, \mu) - R_t(h; S) \right| \leq \mathcal{O}\left(\frac{e^{|t|M}-1}{|t|} \sqrt{\frac{\log|\mathcal{H}|}{n}}\right),$$

with corresponding information-theoretic expressions involving the mutual information $I(H; S)$ between hypothesis and data (Aminian et al., 28 Sep 2024). For unbounded losses with finite moments, similar error rates can be achieved under negative tilt, critical for robustness.

Complexity penalization via chaining functionals ($\gamma_1$, $\gamma_2$), local Rademacher complexity, or Gaussian width can further refine rates in high-dimensional settings and for structured function classes (Brownlees et al., 2014, Roy et al., 2021, Qiu et al., 29 Aug 2025).

Variance Reduction and Smoothness

Incrementing $t$ from zero reduces empirical variance of losses for certain classes of loss functions, leading to improved bias-variance trade-offs and generalization error (Li et al., 2020, Li et al., 2021).

3. Robustness, Fairness, and Distributional Shift

Robustness to Outliers

With negative tilt ($t < 0$), the exponential downweighting of high losses imparts robustness to outliers or corrupted data. Theoretical analysis quantifies the impact of distribution shift: the difference in tilted risks under clean and corrupted distributions is controlled by total variation distance and the tilt magnitude (Aminian et al., 28 Sep 2024).

TERM with robust loss functions (Huber, Catoni-type) or median-of-means block aggregation can yield sub-Gaussian guarantees without requiring high-order moments (Brownlees et al., 2014, Minsker et al., 2019).

Fairness and Max-Loss Behavior

With positive tilt ($t > 0$), TERM interpolates to max-loss minimization, focusing attention on high-loss or underrepresented subgroups—an approach effective in fair learning and class-imbalance problems (Li et al., 2020, Szabo et al., 2021).

Hierarchical extensions support structured scenarios, e.g., minimizing group-level tilted losses (by class or annotator). This extends TERM to compound objectives involving both fairness and robustness.

Outlier and Minority-Class Detection in Online Settings

The online variant of TERM removes the log from the classical objective, maintaining tilt sensitivity per sample so that negative $t$ confers robustness while positive $t$ improves recall in minority-class detection, even in streaming data regimes (Yildirim et al., 18 Sep 2025).

4. Connections with Related Risk Measures

TERM is tightly linked to risk-sensitive and robust optimization frameworks:

• Value-at-Risk (VaR), Conditional Value-at-Risk (CVaR): As $t$ increases, TERM approximates the upper quantile or CVaR objective, providing a smooth surrogate for tail-risk minimization.
• Rényi Cross Entropy and Entropic Risk: TERM can be recast as a minimization of the Rényi (or entropic) risk between uniform and exponential-weighted loss distributions (Li et al., 2021).
• Distributionally Robust Optimization (DRO): For $t\to +\infty$, TERM converges to a DRO or adversarial formulation, emphasizing worst-case instances.
• Bayesian Inference: Bayesian versions of TERM employ exponentially tilted empirical likelihoods, yielding uncertainty quantification calibrated via the Bernstein-von Mises theorem without requiring model correctness (Tang et al., 2021).

These connections make TERM a unified lens for classical, robust, and adversarial learning strategies.

5. Optimization Algorithms and Computational Considerations

Batch and stochastic first-order optimization schemes for TERM rely on the differentiability and smoothness of the log-sum-exp risk surrogate. The gradient's exponential reweighting maintains computational efficiency, and stochastic approximations are tractable for large datasets (Li et al., 2020, Li et al., 2021). In hierarchical/multi-objective formulations, efficient weight aggregation is critical.

In online learning contexts, the per-sample computation for the online TERM objective (i.e., $(1/t)e^{t\ell(\theta)}$ and its gradient) remains comparable with classic stochastic ERM updates (Yildirim et al., 18 Sep 2025).

Sample complexity and covering-number analyses extend to the quantum learning setting (QTERM), bridging classical and quantum risk minimization under tunable exponential loss aggregation (Qiu et al., 29 Aug 2025).

6. Applications Across Domains

TERM and its variants have demonstrated empirical and theoretical benefits across multiple domains:

• Robust Regression: Outperforms L1/L2/Hubble losses under contamination or heavy-tailed noise, with improved convergence and stability (Li et al., 2020, Minsker et al., 2019).
• Classification under Label Noise: Protects against overfitting to mislabeled data, competitive with advanced robust methods (e.g., MentorNet, GCE) on datasets like CIFAR-10 (Li et al., 2020).
• Semantic Segmentation: Used in Tilted Cross Entropy (TCE), focusing optimization on worst-performing classes and reducing fairness disparity in mIoU across classes (Szabo et al., 2021).
• Federated and Meta-Learning: Ensures fairness across clients/tasks via group-level tilting, applicable to imbalanced or non-IID data settings (Li et al., 2020).
• Streaming and Online Learning: Online TERM supports efficient, per-sample robust or fairness adjustments in real-time applications such as anomaly detection, resource allocation, and autonomous systems (Yildirim et al., 18 Sep 2025).
• Quantum Learning: QTERM generalizes TERM to quantum process learning with explicit sample complexity and generalization results (Qiu et al., 29 Aug 2025).

7. Comparative Analysis and Theoretical Implications

TERM generalizes ERM by providing a continuous trade-off between average-case (ERM), fairness (max-loss), and robustness (min-loss) via the tilt parameter. It subsumes special cases such as robust mean estimation (Catoni, median-of-means), superquantile/CVaR objectives, and variational DRO. Unlike handcrafted loss reweighting, TERM's single-parameter control is systematic and interpretable.

When compared with block-based robust approaches or convex M-estimator techniques, TERM offers smooth optimization landscapes and efficient solvers, though careful tuning of the tilt is necessary, with potential sensitivity of constants in error bounds.

A notable implication is that, for heavy-tailed or dependent data, the use of tilt (particularly negative for robustness) can maintain convergence rates similar to sub-Gaussian scenarios, provided that complexity measures and local curvature are carefully tracked (Roy et al., 2021, Brownlees et al., 2014).

Summary Table: TERM Variants and Key Features

Variant/Setting	Tilted Objective Form	Key Feature
Classical TERM	$(1/t)\log(\frac{1}{n}\sum e^{t\ell_i})$	Interpolates ERM/max/min loss
Online TERM	$(1/t) e^{t\ell(\theta)}$	Per-sample, tilt preserved
Catoni/MoM-TERM	Implicitly via truncation/MoM	Heavy tail robustness
Hierarchical/Group TERM	Nested log-sum-exp across groups	Multi-scale fairness/robust
Bayesian PETEL	Tilted empirical likelihood	Calibrated uncertainty
Quantum TERM (QTERM)	Log-moment over quantum losses	Quantum setting adaptation

TERM provides a principled, theoretically grounded toolset for robust and fairness-aware learning, efficiently bridging statistical risk minimization with robust optimization and risk-sensitive paradigms.