---
title: Empirical Risk Minimization Objective
url: https://www.emergentmind.com/topics/empirical-risk-minimization-objective
type: topic
---

# Empirical Risk Minimization Objective

Empirical risk minimization (ERM) is the central objective in modern statistical learning theory and machine learning, defining the principle by which algorithms select predictive models based on finite data samples. ERM formalizes the substitution of intractable population risk minimization—optimization over an unknown data-generating distribution—by tractable minimization over the sample mean loss computed from observed data. The ERM framework underlies not only standard supervised learning but also recurrent, dynamical, robust, relational, fairness-aware, and explainable machine learning paradigms.

## 1. Formal Definition and Basic Objective

Let $(x_i, y_i)_{i=1}^n$ be observed data, a loss function $\ell(f(x), y)$, and a hypothesis class $\mathcal{F}$. ERM seeks
\[
\hat f_{\rm ERM} = \arg\min_{f \in \mathcal{F}} \frac{1}{n} \sum_{i=1}^{n} \ell(f(x_i), y_i)
\]
where the sum is the empirical risk, approximating the population risk
\[
R(f) = \mathbb{E}_{(x, y) \sim P}\left[\ell(f(x), y)\right]
\]
In the case of parameterized models $f_\theta$, the optimizer becomes
\[
\hat\theta_n = \arg\min_{\theta} \frac{1}{n} \sum_{i=1}^n \ell(f_\theta(x_i), y_i)
\]
For time series prediction, with recursive forecasters $f_{\theta, t}$, the ERM selects
\[
\hat\theta = \arg\min_{\theta \in \Theta} \frac{1}{T} \sum_{t=1}^T L(Y_t, f_{\theta, t})
\]
where $L$ is typically a Bregman-type loss, e.g.,
\[
L(Y_t, \hat{Y}_t) = \psi(Y_t) - \psi(\hat{Y}_t) - \psi'(\hat{Y}_t)(Y_t-\hat{Y}_t)
\]
[2108.05184].

## 2. Fundamental Theoretical Properties

ERM provides consistency and optimality guarantees under suitable conditions:
- For i.i.d. data and bounded/sub-Gaussian loss functions, the excess risk decays at $O(1/\sqrt{n})$ [1406.2462, 1412.6606, 2106.01723].
- In time series settings, assuming stationarity and strong mixing, ERM achieves an oracle inequality:
\[
\mathbb{P}\Big(R(\hat{\theta}) \leq \inf_{\theta\in\Theta} R(\theta) + C \sqrt{\frac{p \log T}{T}} \Big) \geq 1 - O(1/\log T)
\]
indicating rate-optimal, non-asymptotic convergence of ERM to the best risk achievable in the model class [2108.05184].

For convex, smooth losses and strongly convex risk, the ERM estimator matches the statistical minimax rate, $O(\sigma^2 / n)$, with explicit finite-sample constants [1412.6606].

## 3. Generalizations and Robustifications

ERM is the basis for several generalizations:

- **Tilted ERM (TERM):** Introduces a tilt parameter $t$ to weight individual losses, defining the tilted risk:
  \[
  L_t(w) = \frac{1}{t} \log \Big(\frac{1}{n} \sum_{i=1}^n e^{t\ell(f_w(x_i), y_i)}\Big)
  \]
  For $t\to 0$, recovers ERM; $t>0$ emphasizes large losses (worst-case/fairness); $t<0$ suppresses outliers (robustness). TERM interpolates between mean, max, and min loss, offering a smooth mechanism for robustness and fairness control [2007.01162].

- **Robust ERM via Catoni's Estimator:** Standard ERM's arithmetic mean is replaced with robust M-estimation. Catoni's loss estimator $\widehat\mu_f$ satisfies:
  \[
  r_f(\widehat\mu_f) = \frac{1}{n\alpha} \sum_{i=1}^n \phi(\alpha(\ell(f(z_i), y_i) - \widehat\mu_f)) = 0
  \]
  leading to excess risk bounds under heavy-tailed losses, maintaining $O(1/\sqrt{n})$ rates even without boundedness [1406.2462].

- **Robust Newton Methods:** Second-order ERM optimization can use robust mean estimators for both gradient and Hessian, enhancing statistical and algorithmic robustness to contamination [2301.13192].

- **Functional Risk Minimization (FRM):** A strict generalization, where the loss is computed in function space, allowing per-sample functional perturbations and capturing richer noise models. ERM emerges as a special case when the functional variability is restricted to output noise [2412.21149].

- **Explainable ERM (EERM):** Incorporates an information-theoretic regularization term, such as the conditional entropy of the predictions given user feedback, to balance predictive performance with subjective, user-dependent explainability [2009.01492].

## 4. Empirical Risk Minimization in Structured and Non-i.i.d. Settings

- **Time Series and Dynamical Models:** ERM is extended to settings where data exhibit dependence (e.g., stationary time series, ergodic processes). Here, the empirical risk is computed over recursive predictors, and the model class can be highly structured (e.g., regime-switching autoregressive models) [2108.05184]. For dynamical systems, the empirical risk includes minimization over unknown initial conditions and addresses signal-noise separation under complexity (entropy) constraints on the model class [1611.06173].

- **Relational and Graph Data:** In non-i.i.d., relational data contexts (e.g., graphs), ERM is based on sampling subgraphs (via random walks, edge/vertex sampling), defining risk as the expected loss over sampled substructures [1806.10701]:
  \[
  \hat R_k(\theta;G) = \mathbb{E}_{G_k \sim \text{Sample}(G,k)} [L(G_k; \theta) \mid G]
  \]
  This enables mini-batch SGD procedures with unbiased gradient estimators over relational data.

- **Adaptively Collected Data:** In bandit or adaptive designs, empirical risk minimization must incorporate inverse-propensity or importance weights to correct for non-uniform sampling. The IS-weighted ERM objective is
  \[
  \hat R_T(f) = \frac{1}{T} \sum_{t=1}^T w_t \ell(f, O_t),\quad w_t = \frac{g^*(A_t\mid X_t)}{g_t(A_t\mid X_t)}
  \]
  with theory providing rates parameterized by the maximum and average importance weight inflation [2106.01723].

## 5. Optimization and Algorithmic Implementations

ERM objectives are predominantly optimized via first-order (gradient descent, stochastic gradient descent) or second-order (Newton-type) methods. Key algorithmic aspects:

- **Batch and Stochastic Procedures:** For convex, smooth losses, SGD and variance-reduced methods (e.g., Streaming SVRG) efficiently minimize ERM, attaining optimal sample complexity $O(1/\epsilon)$ or $O(\log(1/\epsilon))$ depending on algorithm and problem structure [1412.6606, 2011.02522].

- **Single-Pass and Memory-Efficient Solvers:** Streaming SVRG realizes the statistical rates of batch ERM using a single pass through the data and $O(d)$ memory by combining staged reference gradient computation with variance-reduced updates [1412.6606].

- **Handling Heavy-Tailed and Noisy Gradients:** Robustification of gradient and Hessian via estimators such as Huber-aggregation or median-of-means manages outliers and non-Gaussian sampling noise [2301.13192].

- **Nonparametric Gradient Learning:** If the loss admits smoothness in data, local polynomial regression can be exploited to approximate the gradient, yielding ERM solvers with improved oracle complexity when data dimension is low [2011.02522].

## 6. Limitations, Extensions, and Open Problems

- **Risk Monotonicity:** Counter to expectation, ERM does not guarantee monotonic improvement in expected risk as sample size increases; "risk curves" for ERM may be non-monotonic even for standard loss/hypothesis pairs (linear regression, classification, density estimation). Risk-monotonicity, defined as
  \[
  \mathbb{E}\left[R_D(A(S_{n+1})) - R_D(A(S_n))\right] \leq 0
  \]
  often fails for ERM [1907.05476].

- **Complexity-Driven Signal-Noise Separation:** In dynamical, non-i.i.d. settings, the ability of ERM to recover the underlying signal is contingent on the entropy of the model class; zero-entropy (or bounded mean-width) ensures model selection is driven by signal, while high complexity classes can overfit additive noise [1611.06173].

- **Consistency Under Misspecification and Structured Noise:** Under heavy-tailed losses, only robustified (e.g., Catoni-based) ERM attains favorable concentration, suggesting the necessity of robustification in practical, non-sub-Gaussian regimes [1406.2462].

- **Function-Space Generalization:** Modern over-parameterized models, such as deep neural nets, empirically benefit from extensions of ERM to function-space regularization (FRM), explaining generalization and robustness beyond classical statistical learning theory [2412.21149].

## 7. Summary Table: ERM Variants and Generalizations

| Formulation               | Objective Structure                                                 | Key Property/Context                             |
|---------------------------|---------------------------------------------------------------------|--------------------------------------------------|
| Classical ERM             | $\arg\min_f \frac{1}{n}\sum_i \ell(f(x_i), y_i)$                   | i.i.d. data, bounded/sub-Gaussian loss           |
| Term (Tilted ERM)         | $\frac{1}{t}\log \sum_i e^{t\ell_i}$                              | Tunable robustness/fairness via $t$              |
| Robust ERM (Catoni)       | $\arg\min_f \widehat{\mu}_f$, $\widehat{\mu}_f$ solves $r_f(\mu)=0$| Finite-variance, heavy-tailed loss               |
| Explainable ERM (EERM)    | $\arg\min_h R_S(h) + \lambda \hat{H}(h|u)$                         | Subjective explainability-accuracy trade-off     |
| Relational ERM            | $\mathbb{E}_{G_k}[L(G_k;\theta)]$ over sampled subgraphs           | Dependent/relational (graph) data structures     |
| Weighted ERM (ISWERM)     | $\sum_t w_t \ell(f, O_t)$ with $w_t$ correcting for adaptivity     | Off-policy, adaptively collected data            |
| Functional RM (FRM)       | Minimize average function-space divergence                         | Over-parameterized, structured noise, DNNs       |

Each of these extends the ERM principle to address specific statistical or application-driven desiderata, retaining or extending the original statistical optimality guarantees under new regimes and constraints.

Source: https://www.emergentmind.com/topics/empirical-risk-minimization-objective