Learning-Curve Monotonicity

Updated 3 July 2026

Learning-curve monotonicity is a property where generalization risk is non-increasing with additional training data, underpinning theoretical guarantees in certain regimes.
Formal definitions include pointwise, expected, and high-probability variants, each specifying different conditions for ensuring improved performance with more data.
Algorithmic wrappers and statistical tests can enforce monotonicity, though practical challenges like model misspecification and rare data events may still cause non-monotonic behavior.

Learning-curve monotonicity is the property that a learner’s generalization performance—typically measured as expected risk or error—does not deteriorate as the number of training examples increases. This concept has become central to both theoretical and empirical investigations of generalization, motivated by the intuitive but frequently violated belief that “more data never hurts.” Formal and empirical studies have revealed that learning-curve monotonicity cannot be universally assumed, but underlies important guarantees in several regimes and can be algorithmically enforced under mild conditions.

1. Formal Definitions and Variants of Monotonicity

A learning curve for an algorithm $A$ maps the training sample size $n$ to expected generalization risk, $L(n) = \mathbb{E}_{S_n}[R(A(S_n))]$ , where $S_n$ is a random training set of size $n$ and $R$ is the population risk functional. Local monotonicity holds at $n$ if $L(n+1) \leq L(n)$ ; global monotonicity requires this for all $n$ beyond some burn-in. Stronger “complete monotonicity” demands that all finite difference sequences alternate sign appropriately, corresponding to $(-1)^k \Delta^k L(n) \geq 0$ for all $n$ 0 (Sellke et al., 11 Dec 2025).

The notion can be instantiated at various levels:

Pointwise monotonicity: For a fixed dataset, $n$ 1 almost surely.
Expected monotonicity: $n$ 2 under the data distribution (Loog et al., 2019, Viering et al., 2021).
High-probability monotonicity: The property holds with probability at least $n$ 3 over all training sets (Mhammedi, 2020, Viering et al., 2019).

For probabilistic predictors or randomized algorithms, population risk is often assessed as the expected loss over both the learner’s randomness and the data.

2. Theoretical Guarantees and Characterizations

Classical PAC learning theory provides the earliest explicit formulation for monotonicity. For hypothesis classes of finite cardinality or finite VC dimension, upper bounds on the risk, such as $n$ 4, decrease monotonically with $n$ 5 (Li et al., 9 Jan 2025). The associated cumulative distribution functions (CDFs) $n$ 6 for the risk random variable are non-decreasing in $n$ 7, ensuring distributional stochastic monotonicity.

In well-specified Bayesian settings, total-evidence arguments guarantee that each new sample does not increase the expected posterior risk—a classical result in Bayesian decision theory (Viering et al., 2021). For Gaussian process regression with a conjugate kernel, the posterior predictive variance at every test point strictly decreases with additional data (Viering et al., 2021). Minimax-optimal learners for realizable problems, as characterized in (Viering et al., 2021), also exhibit monotonic, exponential, or power-law decay of risk.

When maximum likelihood estimators (MLEs) are studied in exponential families, explicit and strict monotonicity under both forward and reverse Kullback–Leibler risk is established. In the multivariate normal model with unknown covariance, Sellke and Yin (Sellke et al., 11 Dec 2025) prove that the expected forward-KL risk decreases strictly when $n$ 8; similar statements hold for Gamma scale estimation and for reverse-KL risk in general exponential families. These results establish “complete monotonicity” in several central parametric settings.

3. Counterexamples and Failure Modes

Despite these positive results, a wide range of standard procedures violate learning-curve monotonicity in expectation or probability. Empirical risk minimization (ERM) for linear regression (squared or absolute loss), SVM/hinge loss, and even variance estimation in the Gaussian family has been shown to produce oscillatory or serrated learning curves: the risk can increase with more data for certain sample sizes or problem instances (Loog et al., 2019, Mhammedi, 2020, Viering et al., 2021).

The mechanism underlying these pathologies typically involves “rare” or underrepresented regions in the feature-label space. Until enough examples from these subsamples have been observed, ERM may improve fit locally while increasing global risk. Well-documented phenomena such as double descent and peaking are also examples of non-monotone behavior, particularly for unstable or unregularized methods (Viering et al., 2021).

Other causal factors identified in (Viering et al., 2021) and related works include mismatch between surrogate and true loss, model misspecification, fixed hyperparameter schedules, label noise, distribution shift, and the specifics of optimization in non-convex regimes.

4. Algorithms for Enforcing Monotonicity

Algorithmic wrappers have been developed that transform any learner—ERM or otherwise—into a monotone learner. The core mechanism is to accept updates to the hypothesis only when statistical evidence shows an improvement, based on holdout sets, hypothesis tests, or empirical-Bernstein concentration inequalities.

Notable approaches include:

MT_SIMPLE / MT_HT / MT_CV Wrappers: Maintain the current best hypothesis and accept a new candidate only if the empirical (or cross-validated) risk on fresh validation data is not worse, optionally using a paired hypothesis test for high-probability guarantees (Viering et al., 2019). The MT_HT method, for instance, achieves per-round strict control over the non-monotone transition probability.
PAC-Bayes Monotonic Wrappers: For bounded losses and under minimal martingale-difference conditions (i.i.d. or certain non-i.i.d. processes), any base learner can be wrapped using tight empirical-Bernstein confidence intervals. New hypotheses are adopted only when a statistically certified reduction in population risk can be made, resulting in high-probability risk-monotonicity and preservation of fast excess-risk rates (Mhammedi, 2020).

In all cases, monotonicity can be enforced without meaningful asymptotic degradation in risk rates. Empirical benchmarks confirm that monotonic wrappers can reduce non-monotone transitions to below 1% on standard datasets such as MNIST (Viering et al., 2019).

5. Monotonicity in Reinforcement Learning and Q-learning

Monotonic update schemes have also been studied in temporal-difference learning. In tabular Q-learning, the Bellman update can be cast as a monotone operator: for appropriate step-size schedules, the Bellman residual and the error with respect to the optimal Q-function exhibit monotonic geometric decay (Yang, 2024). When moving to function approximation (linear or nonlinear), order-preservation may be lost, and non-monotone learning curves (oscillations or divergence) become possible. Enforcing monotonicity requires bounded features, careful step-size selection, or the use of mechanisms such as target networks or update clipping.

6. Empirical Characterization and Influencing Factors

Empirical studies of learning curves indicate that power-law and exponential functional forms often fit well and are monotonic, but exceptions are common. Non-monotonic curves materialize in the presence of peaking, double descent, sample complexity phase transitions, hyperparameter or regularization misalignment, noisy labels, and data distribution shifts (Viering et al., 2021, Loog et al., 2019).

Monotonicity is typically robust in well-specified, regularized, and Bayes-optimal regimes, and less reliable in practice with practical estimators using surrogate losses or suboptimal model selection protocols.

Table: Conditions Under Which Monotonicity Is Guaranteed

Regime / Setting	Type of Monotonicity	Source
Finite $n$ 9 or VC-dim; ERM	PAC-bound, expected	(Li et al., 9 Jan 2025)
Bayesian optimal (well-specified)	Expected, sample-wise	(Viering et al., 2021)
Gaussian MLE: mean/cov, forward-KL	Complete, strict	(Sellke et al., 11 Dec 2025)
Exponential-family MLE: reverse-KL	Expected, strict	(Sellke et al., 11 Dec 2025)
Q-learning (tabular/linear quadratic)	Residual monotonicity	(Yang, 2024)
Monotone wrappers (holdout/test/Bernstein)	High-probability	(Mhammedi, 2020, Viering et al., 2019)

7. Limitations, Open Problems, and Future Directions

Monotonicity is not a universal property and remains subtly dependent on model, loss, data generating process, and algorithm design. Even when enforced algorithmically, some trade-off between speed and certainty of improvement may arise, particularly in finite-sample regimes.

Active questions include:

Characterizing the conditions under which natural estimators (e.g., MLE, ERM) guarantee monotonicity across all losses and models.
Investigating robustness under non-i.i.d., non-stationary, and adversarial processes.
Developing and analyzing monotonicity for deeper or compositional models, such as deep neural networks, where phase transitions and optimization instability are prevalent.
Studying monotonicity for alternative risk functionals (e.g., median or high quantile learning curves), richer statistics, and model selection criteria.
Expanding empirical meta-analysis to quantify the prevalence, magnitude, and causes of non-monotonicity across domains and workflows (Viering et al., 2021, Loog et al., 2019).

Advances in these directions underpin both theoretical guarantees and practical strategies for data acquisition, model evaluation, and curriculum design in increasingly complex learning scenarios.