Rolling Empirical Risk Minimization

• Rolling empirical risk minimization is the process of retraining models as new data points are added, with the expectation—but not guarantee—of improved performance.
• The analysis shows that common ERM procedures (e.g., linear regression, SVM, density estimation) can exhibit nonmonotonic learning curves where more data can increase expected risk.
• A two-point construction demonstrates that exact minimization of training loss does not ensure decreased true risk, highlighting risk monotonicity as an independent property from PAC guarantees.

Rolling empirical risk minimization denotes the repeated retraining of an empirical risk minimizer as the training sample grows from $n$ to $n+1$ observations. In this setting, a natural expectation is that adding one more training example should improve, or at least not worsen, expected performance under the same loss optimized during training. The analysis in "Minimizers of the Empirical Risk and Risk Monotonicity" shows that this expectation is false in general: empirical risk minimization (ERM) does not guarantee learning-curve monotonicity, even when the learner is evaluated by the very same loss it minimizes (Loog et al., 2019). The paper formalizes this issue through the notion of risk monotonicity and proves that several standard ERM learners can be nonmonotone at every sample size.

1. Formal setup for ERM under increasing sample size

The paper studies supervised learning with training sample

$$S_n = (z_1,\ldots,z_n), \qquad z_i \stackrel{i.i.d.}{\sim} D,$$

where $D$ is a distribution over a domain $\mathcal{Z}$, together with a hypothesis class $\mathcal{H}$ and loss function

$\ell: \mathcal{Z}\times \mathcal{H}\to \mathbb{R}.$

The true risk is defined as

$R_D(h) := \mathbb{E}_{z\sim D}\,\ell(z,h),$

and the empirical risk on $S_n$ as

$R_{S_n}(h) := \frac{1}{n}\sum_{i=1}^n \ell(z_i,h).$

The ERM learner is

$n+1$0

This setup covers many standard learners. The paper explicitly notes that Fisher’s linear discriminant, SVMs, linear regression, lasso, and maximum likelihood / density estimation fit the ERM template (Loog et al., 2019). In the context of rolling ERM, the object of interest is not only whether $n+1$1 is consistent or achieves low excess risk, but how the expected true risk changes when the sample is enlarged by one observation.

2. Risk monotonicity as a finite-sample property

The paper introduces risk monotonicity to capture whether expected risk deteriorates when the training set size increases. A learner $n+1$2 is locally monotonic at sample size $n+1$3 if

$n+1$4

or, more explicitly,

$n+1$5

Two stronger notions are then defined. Weak $n+1$6-monotonicity requires that there exists some $n+1$7 such that for all $n+1$8, the learner is locally monotonic for all distributions on the domain $n+1$9. Global $$S_n = (z_1,\ldots,z_n), \qquad z_i \stackrel{i.i.d.}{\sim} D,$$0-monotonicity requires local monotonicity for every $$S_n = (z_1,\ldots,z_n), \qquad z_i \stackrel{i.i.d.}{\sim} D,$$1 (Loog et al., 2019).

The significance of this definition is that it isolates a finite-sample comparison between risk at $$S_n = (z_1,\ldots,z_n), \qquad z_i \stackrel{i.i.d.}{\sim} D,$$2 and risk at $$S_n = (z_1,\ldots,z_n), \qquad z_i \stackrel{i.i.d.}{\sim} D,$$3. Standard learning-curve analysis often asks whether risk decreases asymptotically, or whether one can bound excess risk as a function of $$S_n = (z_1,\ldots,z_n), \qquad z_i \stackrel{i.i.d.}{\sim} D,$$4. By contrast, risk monotonicity asks whether expected risk improves when one more sample is added. It is an expectation over both training samples and test samples, and it concerns monotonicity at every finite sample size rather than only eventual convergence.

3. Central negative result for rolling ERM

The paper’s main result is that many standard ERM procedures are not weakly monotonic for any $$S_n = (z_1,\ldots,z_n), \qquad z_i \stackrel{i.i.d.}{\sim} D,$$5. Equivalently, for every sample size $$S_n = (z_1,\ldots,z_n), \qquad z_i \stackrel{i.i.d.}{\sim} D,$$6, there exists a distribution such that the expected risk of the ERM learner at size $$S_n = (z_1,\ldots,z_n), \qquad z_i \stackrel{i.i.d.}{\sim} D,$$7 is worse than at size $$S_n = (z_1,\ldots,z_n), \qquad z_i \stackrel{i.i.d.}{\sim} D,$$8 (Loog et al., 2019).

This result applies to linear models trained with the squared loss, absolute loss, hinge loss, and negative log-likelihood for variance estimation in a normal model. The consequence for rolling ERM is immediate: exact minimization of training loss does not imply stepwise improvement in expected true risk, even when the evaluation criterion is the same loss used in training.

The paper emphasizes that this phenomenon is distinct from the usual interpretation of generalization guarantees. PAC bounds can shrink with $$S_n = (z_1,\ldots,z_n), \qquad z_i \stackrel{i.i.d.}{\sim} D,$$9, but those bounds concern error or excess risk and do not imply that the sequence of expected risks is monotone. A learner can therefore be PAC-learnable and still fail risk monotonicity, or vice versa. This suggests that risk monotonicity is a separate structural property of a learning procedure rather than a corollary of standard sample-complexity analysis.

4. Technical mechanism behind nonmonotonicity

The main technical lens is a two-point construction on $D$0. The paper gives the following criterion: if

$D$1

then ERM is not locally monotonic at $D$2. A simplification used for many losses is

$D$3

which reduces the condition to

$D$4

The proofs exploit a common construction: build a distribution with a dominant common point $D$5 and a rare point $D$6, compare the ERM solutions for samples with $D$7, $D$8, and $D$9 relevant occurrences, show that the loss at $\mathcal{Z}$0 does not decrease quickly enough, and conclude that the expected risk difference is positive (Loog et al., 2019). In this mechanism, the rare point carries the nonmonotonic effect. If the learner’s performance on that rare point fails to improve sufficiently rapidly as $\mathcal{Z}$1 grows, the aggregate expected risk can increase when a single additional sample is incorporated.

The paper explicitly notes that this failure is not due to overfitting in the usual high-complexity sense; it can happen even when the model is quite simple relative to the sample size. A plausible implication is that nonmonotonicity is tied less to model capacity alone than to the interaction among sample composition, ERM geometry, and the loss landscape at finite $\mathcal{Z}$2.

5. Instantiations across regression, classification, and density estimation

The paper provides theorem-level instantiations for several standard losses and model classes (Loog et al., 2019).

Setting	Loss or objective	Result stated in the paper
Linear models	Squared loss	Not weakly monotonic for any $\mathcal{Z}$3
Linear models	Absolute loss	Nonmonotonicity via the same two-point construction
Classification	Hinge loss	Same counterexample structure yields nonmonotonicity
Normal variance estimation	Negative log-likelihood	Not weakly monotonic for any $\mathcal{Z}$4

For linear predictors $\mathcal{Z}$5 with squared loss,

$\mathcal{Z}$6

the paper proves that if the input domain contains an open ball around the origin, then the ERM learner is not weakly monotonic for any $\mathcal{Z}$7. This covers standard least squares and Fisher-style linear regression settings.

For absolute loss,

$\mathcal{Z}$8

the same two-point construction yields nonmonotonicity. In the one-dimensional reduction, the minimizer behaves like a weighted median or threshold rule, and the proof shows that the jump from $\mathcal{Z}$9 to $\mathcal{H}$0 samples can increase expected risk.

For hinge loss,

$\mathcal{H}$1

the paper again uses a two-point distribution with conflicting labels to show that the ERM solution can have worse expected risk after one additional sample. This is especially notable because hinge loss is the canonical SVM surrogate.

For negative log-likelihood, the paper analyzes a one-dimensional normal model with unknown variance and fixed mean $\mathcal{H}$2: $\mathcal{H}$3 It proves that if the domain contains an open ball around the origin, then variance estimation by ERM on negative log-likelihood is not weakly monotonic for any $\mathcal{H}$4. The density-estimation example shows that the phenomenon is not confined to supervised prediction in the narrow sense.

6. Learning-curve behavior and empirical illustrations

The paper includes illustrative learning curves summarized in a figure with four subfigures: (a) squared loss, no intercept; (b) absolute loss, no intercept; (c) squared loss, no intercept, with and without regularization; and (d) squared loss with intercept (Loog et al., 2019). The experiments use simple one-dimensional or low-dimensional discrete distributions.

The empirical takeaways reported in the paper are specific. The absolute-loss curve shows visibly serrated, strongly nonmonotone behavior. The squared-loss example can show dramatic increases in expected risk as sample size grows. Regularization does not automatically fix the problem; in one case it even makes the behavior worse. Adding an intercept does not remove nonmonotonicity.

These experiments serve a narrow but important role. They do not establish the phenomenon by themselves; the theoretical results already do that. Instead, they show that the effect is not merely a pathological proof artifact. This suggests that in rolling ERM, visually irregular learning curves are not necessarily anomalies caused by poor experimental hygiene, but may reflect a genuine property of ERM under finite-sample sampling variation.

7. Conceptual consequences for sequential retraining

In rolling, online, or sequential ERM, the learner is repeatedly retrained as data arrive. The principal implication drawn in the paper is that adding one more sample does not guarantee a decrease in expected risk, even if the learner is minimizing the training loss exactly and even if test-time evaluation uses that same loss (Loog et al., 2019).

This has several conceptual consequences. First, consistency, PAC-learnability, and shrinking excess-risk bounds should not be conflated with monotone learning curves. Second, learning-curve monotonicity becomes an independent object of study rather than an automatic by-product of ERM. Third, the intuition that “more data should never hurt in expectation” is formally invalid for a broad class of textbook ERM procedures.

The broader significance of the paper is therefore methodological. It reframes learning-curve analysis from an asymptotic question to a finite-sample structural question: whether the expected risk sequence generated by retraining at sample sizes $\mathcal{H}$5 is monotone. In that sense, rolling empirical risk minimization is not simply ERM evaluated repeatedly; it is a sequential process whose risk trajectory can exhibit dips and peaks even under the learner’s own loss.