Monotone Learning Rate Inevitability

Updated 10 June 2026

Monotone Learning Rate Inevitability is the property where risk or learning rates decrease steadily with increasing training data or iterations, ensuring optimal performance in deterministic settings.
Theoretical frameworks like PAC learning and ERM, along with universal monotonization wrappers, mathematically guarantee nonincreasing error bounds for a wide class of learners.
Practical algorithms in SGD and online learning reveal trade-offs, as fixed or fractal learning rate schedules can violate monotonicity, prompting the need for decaying rates for efficiency.

Monotone Learning Rate Inevitability denotes the phenomenon wherein, under precise conditions, learning rates or risk curves must exhibit monotonic (usually nonincreasing) behavior as a function of training data, time, or iteration, either as a mathematical necessity derived from learning theory or as a necessary property for optimal statistical or algorithmic efficiency. Multiple research threads, spanning PAC learning theory, empirical risk minimization (ERM), stochastic optimization, bandit problems, and analysis of learning rate schedules in practical algorithms, have addressed the inevitability or limitations of monotonicity in learning dynamics.

1. Monotone Learning in Theoretical PAC Frameworks

Within classical PAC theory, monotone decrease of generalization risk with growing sample size is not only anticipated but can be formally constructed for a wide class of learners. For any deterministic Empirical Risk Minimization (ERM) algorithm with either a finite hypothesis set or finite VC dimension, it is possible to define, for each sample size $m$ , an explicit, nondecreasing lower-bound cumulative distribution function (CDF) $F_m(\epsilon)$ for the risk. This construction guarantees that the expected risk $r_m$ is stochastically monotone ( $r_{m+1}\le r_m$ for all $m$ ) and converges to zero as sample size diverges, provided the problem is PAC-learnable and the data are i.i.d. (Li et al., 9 Jan 2025).

Explicit sample-complexity bounds in this case yield:

For finite $|\mathcal{H}|$ , $r_m \le (\ln|\mathcal{H}| + 1)/m$ .
For finite VC-dimension $d$ ,

$m(\epsilon,\delta) = O\left( \frac{d + \ln(1/\delta) + \ln\ln m}{\epsilon^2} \right)$

The inevitability of monotonicity is established: any deterministic ERM algorithm under these assumptions admits a performance-lower-bound distribution that moves monotonically leftward with $m$ , ensuring nonincreasing expected risk (Li et al., 9 Jan 2025). Non-monotonic features observed for small sample sizes are artifactual, either due to looseness of PAC bounds or algorithmic non-monotonicity, and not rooted in information-theoretic impossibility.

2. Universal Monotonization of Learning Algorithms

Addressing the lack of monotonicity in some practical algorithms, (Bousquet et al., 2022) constructs a universal "monotonization" wrapper: for any deterministic learner $F_m(\epsilon)$ 0, there is an efficient black-box transformation $F_m(\epsilon)$ 1 such that the monotonicity property is enforced for any distribution $F_m(\epsilon)$ 2 and sample size $F_m(\epsilon)$ 3:

$F_m(\epsilon)$ 4

where $F_m(\epsilon)$ 5 denotes population error under $F_m(\epsilon)$ 6.

This wrapper preserves

Bayes consistency (if $F_m(\epsilon)$ 7 is Bayes-consistent, so is $F_m(\epsilon)$ 8),
asymptotic optimality (matching the PAC learning rate up to negligible $F_m(\epsilon)$ 9 terms),
and competitiveness in risk (error at $r_m$ 0 for $r_m$ 1 nearly matches error at $r_m$ 2 of $r_m$ 3, $r_m$ 4).

Therefore, non-monotonicity of the average learning curve in the standard supervised setting is algorithmic, not fundamental. All learnable multiclass problems admit monotone error decay with increasing sample size, making monotone learning rates unavoidable for deterministic, distribution-free, PAC-learnable problems (Bousquet et al., 2022).

3. Limitations and Open Problems in Monotone Learning

Rigorous monotonicity does not automatically extend beyond deterministic ERM and classical PAC settings:

For randomized (e.g., SGD with nontrivial noise) or regularized procedures, monotonicity may fail, as empirical risk curves can exhibit dips, plateaus, or double descent (Li et al., 9 Jan 2025).
Agnostic settings, where the realizability assumption fails, do not guarantee monotonic decreasing bounds due to the interplay of estimation and approximation errors.
Non-i.i.d. data (e.g., concept drift, adversarial settings) or unstructured hypothesis spaces can violate monotonicity.
Regression and structured output settings (e.g., for surrogate losses other than zero–one loss) require separate analysis (Li et al., 9 Jan 2025).

Active research seeks extensions to monotone PAC learners under weaker or agnostic conditions, as well as for broader algorithmic classes (e.g., stochastic or regularized models). The synthesis of monotonicity with complexity-appropriate low sample-complexity bounds remains an open challenge.

4. Monotone Learning Rates in Optimization and SGD

In stochastic optimization, the inevitability or necessity of monotone (usually decaying) learning rates depends critically on structural properties of the objective.

For homogeneous linear classifiers trained with SGD and strictly monotone, smooth, exponentially-tailed loss (e.g., logistic loss) on linearly separable data, (Nacson et al., 2018) proves that fixed (non-decaying) learning rates—under explicit, data-dependent bounds—suffice for both exact convergence to zero loss and convergence in direction to the $r_m$ 5 max-margin separator. Decreasing the learning rate is unnecessary and, in fact, slows training. These findings also rigorously justify the empirical linear-scaling rule $r_m$ 6 (learning rate proportional to minibatch size) often adopted in large-scale deep network training. However, these results hinge on data being strictly linearly separable and the loss being strictly monotone and smooth; they do not extend to the entire deep net, only to its last linear layer (Nacson et al., 2018).
For general nonconvex objectives, (Shi et al., 2020) establishes via a learning-rate-dependent SDE analysis that the optimal linear convergence rate of SGD decays super-exponentially in the learning rate as $r_m$ 7, for broad classes of nonconvex functions. For such landscapes, fixed (small) rates are inefficient: the mixing time becomes exponentially large, limiting practical convergence. Decaying the learning rate ("annealing") is therefore inevitable for efficient exploration and convergence to low-bias stationary measures in nonconvex problems. By contrast, for strongly convex objectives, the exponential convergence rate is invariant in $r_m$ 8 as $r_m$ 9, and fixed, small learning rates incur no time penalty, making monotonic decay optional (Shi et al., 2020).
In the context of bandit problems with monotone policies (e.g., nondecreasing arm sequences), monotonicity constraints (such as "dose only increases" or "price only rises") impose strict regret penalties: for unimodal/quasiconcave reward functions, the best achievable minimax regret increases from $r_{m+1}\le r_m$ 0 (unconstrained) to $r_{m+1}\le r_m$ 1 (monotone policy), with matching lower and upper bounds (Chen, 2021). The monotonic schedule is not only inevitable, but comes with significant statistical cost.

5. Inevitable Decay for Statistical Estimation in Online Learning Rules

When estimating learning rates in bandit algorithms such as Exp3, (Aubert et al., 2023) shows that a fixed learning rate $r_{m+1}\le r_m$ 2 fundamentally impairs the statistical efficiency of the Maximum Likelihood Estimator (MLE): for any estimator, the worst-case error decays at most logarithmically in sample size, $r_{m+1}\le r_m$ 3, regardless of algorithmic enhancements. Conversely, allowing the learning rate to decrease polynomially with sample size leads to polynomial-rate convergence both in predictive and estimation error (e.g., $r_{m+1}\le r_m$ 4 for certain two-armed bandits). The statistical necessity of decaying learning rates is pronounced in online adaptation and cognitive modeling: non-monotonic schedules can render learning rate estimation intractably slow or impossible for practical purposes (Aubert et al., 2023).

While classical stochastic optimization often presumes monotone decreasing rates for stability or statistical guarantees, counterexamples exist:

Nonmonotone, "fractal" learning rate schedules (constructed from Chebyshev step sizes in a bit-reversal order) break the necessity of monotonicity in certain optimization scenarios. For strongly convex quadratics, such schedules yield accelerated convergence $r_{m+1}\le r_m$ 5—matching Nesterov acceleration—without any momentum term (Agarwal et al., 2021). These schedules allow locally "unstable" steps (learning rates that would individually increase the loss), but the fractal interleaving prevents global blow-up and even improves robustness to noise up to $r_{m+1}\le r_m$ 6.
Empirical results in logistic regression and deep networks (e.g., ResNet-18 on CIFAR-10, convolutional MNIST) show that only fractally-ordered nonmonotonic learning rates accelerate convergence without instability, outperforming both constant and monotone decreasing rates in controlled tests (Agarwal et al., 2021).

This class of results establishes that monotonicity of the learning rate schedule is not an inherent requirement for stability or optimality in all regimes, particularly when using schedule-aware constructions informed by spectral properties of the loss surface.

References

Monotone learning in PAC/ERM: (Bousquet et al., 2022, Li et al., 9 Jan 2025)
Fixed vs. decaying learning rates in SGD: (Nacson et al., 2018, Shi et al., 2020)
Bandits with monotone policies: (Chen, 2021)
MLE estimation of learning rate under monotonic schedules: (Aubert et al., 2023)
Nonmonotonic/fractal learning rate schedules: (Agarwal et al., 2021)