---
title: Horizon-Free Learning-Rate Schedules
url: https://www.emergentmind.com/topics/horizon-free-learning-rate-schedules
type: topic
---

# Horizon-Free Learning-Rate Schedules

Horizon-free learning-rate schedules comprise a family of approaches to learning rate adaptation in stochastic optimization that do not require advance specification of the total training horizon. These methods contrast with conventional horizon-aware schedules (e.g., cosine annealing, exponential decay), which critically depend on knowledge of the final iteration or epoch count. Horizon-free schedules have attracted significant attention due to the rise of continual, open-ended, and large-scale training paradigms, where the training horizon is unbounded, unknown, or subject to dynamic extension. This article provides a rigorous overview of the mathematical foundations, principal algorithmic strategies, theoretical guarantees, limitations, and empirical performance of horizon-free learning-rate schedules.

## 1. Mathematical Foundations of Horizon-Free Schedules

The core mathematical property of a horizon-free schedule is that the learning rate at each step $t$, denoted $\eta_t$, is computable using only information available up to $t$, without reference to a pre-specified $T$ or $N$ (the total planned steps/epochs). Classical horizon-aware schedules are typically parametric functions of $t/T$ (e.g., $\eta_t = \eta_0 \cos(\pi t / T)$), and fail to generalize when $T$ is misestimated or extended.

Horizon-free schedules may be characterized as:

- **Deterministic decay families:** Polynomial decays, e.g., $\eta_t = C t^{-\alpha}$ with $\alpha \in (0,1)$, as shown to be minimax-optimal in the tail-averaged sense for overparameterized linear regression when paired with weight averaging [2602.03702].
- **Adaptive state-based rules:** Schedulers where $\eta_t$ is a function of observable metrics (instantaneous loss, weight norm, validation error, etc.), such as the GreedyLR scheduler which adjusts $\eta_t$ up or down in response to minibatch loss changes [2512.14527], or ABEL which decays the learning rate when "bounces" in total weight norm are detected [2103.12682].
- **Meta-learned context-aware mappings:** Parameterized predictors (e.g., LSTM-based meta-schedulers as in MLR-SNet [2007.14546]) or ODE-driven models which select $\eta_t$ based on learned representations of recent training dynamics [2509.23052].

The theoretical frameworks addressing horizon-freeness clarify the distinction between guarantees for tail-averaged iterates (Polyak–Juditsky averaging) and final-iterate performance, with the former admitting truly anytime minimax rates under horizon-free polynomial decay [2602.03702, 2602.04774] but fundamental barriers for the latter [1904.12838].

## 2. Principal Algorithmic Strategies

Several classes of horizon-free schedules have been systematically instantiated:

**(a) Anytime Polynomial Decay Plus Averaging:**  
SGD with step sizes $\eta_t = \eta\, t^{-\gamma}$, $0<\gamma<1$, combined with tail-weight averaging:
- Achieves near-minimax rates for overparameterized settings if $\gamma$ is tuned to problem regularity (source and capacity exponents) [2602.03702].
- In large LLM pretraining, constant step-size plus EMA averaging ($\mathrm{half\text{-}life} \sim t/10$–$t/100$) or $1/\sqrt{t}$ decay plus EMA matches optimized cosine schedules across 1×–32× Chinchilla scale.
- Pseudocode example (tail-averaged SGD):
  ```python
  for t in range(1, ...):
      eta = base_eta * t**(-gamma)
      g = gradient(...)
      w = w - eta * g
      if t > rho * t:
          A = A + (w - A) / (t - rho * t)
  ```

**(b) Greedy Loss-Adaptive Scheduling (GreedyLR):**  
Dynamically multiplies/divides the learning rate by a factor $F<1$ if the loss increases/decreases from the previous step:
- Requires only per-step loss comparison and one multiplication or division.
- Provably achieves $O(1/T)$ convergence in $L$-smooth convex objectives, with $F^* = 1 - 1/L_{\max}$ optimal under $L_{\max}$-smoothness [2512.14527].

**(c) Weight-Norm–Triggered Decay (ABEL):**  
Monitors the evolution of the total squared weight-norm and executes learning rate decay precisely when a minimum (“bounce”) is detected:
- Fully data-driven; never requires knowledge of $T$.
- In regimes where the bounce is present (e.g., vision with $L_2$ regularization), matches or exceeds finely tuned schedules; otherwise defaults to "decay once at end" [2103.12682].

**(d) Meta-Learned and Online Hypergradient Schedules:**  
Meta-learned parametric maps or ODE models predict $\eta_t$ based on the current loss trajectory and internal recurrent state:
- MLR-SNet uses a single-layer LSTM mapping recent loss and hidden state to $\alpha_t$ [2007.14546].
- Latent ODE schedulers encode recent metrics into latent space, solve a neural ODE, and decode both immediate and long-range schedule recommendations; such models are entirely agnostic to horizon and optimize for long-term generalization rather than pre-specified endpoints [2509.23052].

**(e) Hyperbolic and Epoch-Insensitive Schedules:**  
HyperbolicLR and ExpHyperbolicLR use hyperbolic curve parametrizations to produce schedules whose early and mid-phase decay slopes are asymptotically independent of the epoch budget $N$ [2407.15200]:
- Fitting {init, min, slope} parameters on a small budget and then scaling up $N$ gives consistent performance without retuning, crucial for deployments with unknown or frequently changing horizon.

## 3. Theoretical Properties and Limitations

The minimax convergence of horizon-free schedules is contingent on how performance is measured:

- **Tail-Averaged Guarantees:**  
For overparameterized linear models, polynomial decay with tail or EMA averaging achieves horizon-free minimax excess error rates $N^{-(b-1)/b}$ if $b>a$ (source exponent dominates capacity) [2602.03702]. The exponent $\gamma^* = 1 - a/b$ is theoretically optimal and admits a simple, closed-form schedule.
  
- **Final-Iterate Barriers:**  
For the canonical least squares problem, it is shown that—without explicit horizon-dependent tuning—no horizon-free polynomial decay achieves the minimax final-iterate rate for all $t$; at infinitely many $t$ the excess risk is sub-optimal by a factor of $\kappa / \log \kappa$ (for the strongly convex case), and only geometric (step-decay) schedules with tuned steps per epoch can close the gap, but these require $T$ [1904.12838].

- **Adaptive and Meta-Learned Schedules:**  
When schedules are learned (via bilevel optimization or meta-learning) without any prior on $T$, uniform generalization bounds can be established on the learned schedule vector $(\eta_1, ..., \eta_H)$ over tasks, with sample complexity $\tilde{O}(H^4/\epsilon^2)$ for piecewise-polynomial and Pfaffian objectives. This analysis holds even for non-convex, non-smooth settings and does not require $T$ to enter the optimization as an argument [2512.05084].

- **Loss-Adaptive and State-Based Approaches:**  
GreedyLR and ABEL function independently of the horizon and can adapt instantly to unexpected schedule extensions, restarts, or early stopping—properties not available for any $T$-parametric scheduler [2512.14527, 2103.12682].

## 4. Empirical Performance and Applications

Empirical studies spanning language modeling, computer vision, time series, operator learning, and reinforcement learning indicate the following trends:

- **Anytime Polynomial/EWA Schedules:**  
On LLM pretraining (OLMo 150M/300M, C4), constant plus EMA and $1/\sqrt{t}$ plus EMA track the best-tuned cosine decay envelope within $0.005$–$0.01$ validation loss across 1–32$\times$ Chinchilla scale, also outperforming at very large batch sizes [2602.03702].
- **GreedyLR:**  
Surpasses or matches classic cosine and exponential decay on NLP, vision, and LLM tasks (up to 7B params), is robust to noise, and recovers from loss spikes more quickly [2512.14527].
- **ABEL:**  
In weight-norm bouncing regimes, matches or slightly outperforms step-wise and cosine on ImageNet, CIFAR-10, and large-scale NLP. In "non-bounce" regimes, reduces to optimal one-shot decay at end [2103.12682].
- **Meta-Learned/Learned Schedules:**  
MLR-SNet demonstrates strong transfer to unseen horizons, architectures, and data domains with test performance comparable or superior to the best tuned static schedules, including in robustness to corrupted data scenarios [2007.14546].  
Latent ODE schedulers consistently improve over baseline and hypergradient approaches in test accuracy and find flatter minima on diverse datasets [2509.23052].
- **Hyperbolic/Epoch-Insensitive Schedules:**  
HyperbolicLR and ExpHyperbolicLR deliver near-constant initial decay and stable performance as the epoch budget increases fourfold, reducing the need for retuning and maintaining curve shape across deployments [2407.15200].

## 5. Relationship to Horizon-Aware and Averaging Strategies

A major technical insight is the centrality of iterate averaging for achieving minimax rates in the absence of a horizon.  
- **Polyak–Juditsky averaging** makes polynomial decays anytime-minimax for overparameterized or interpolating models [2602.03702, 2602.04774].
- **Final-instant schemes** cannot, in general, replicate this property: for horizon-freeness with final-iterate metrics, step-decay with epoch boundaries tuned to $T$ is required, but is no longer anytime [1904.12838].
- **Hypergradients, bilevel, and ODE-based methods** can adapt on-the-fly but often trade off interpretability or computational simplicity for flexibility and transfer.  
- **MLR-SNet and related meta-learners** generalize across both horizon and task, parameterizing the schedule as a small recurrent map of local observables, and can be meta-trained to be plug-and-play out-of-the-box for novel deployments [2007.14546].

## 6. Practical Recommendations and Deployment Guidelines

For open-ended, continual, or plug-and-play machine learning deployments, the literature recommends:

- **Avoid horizon-tuned cosine, polynomial, or geometric decays when $T$ is unknown or variable.**
-  
  - For standard SGD: use $\eta_t = \eta_0 t^{-1/2}$ or constant $\eta_0$ (lightly grid-tuned), always with EMA or tail averaging with half-life proportional to $t/10$–$t/100$ [2602.03702].
  - For loss- or state-adaptive approaches: GreedyLR ($F=0.5$–$0.9$), ABEL (bounce-based trigger), or MLR-SNet (meta-learned) provide robust defaults.
  - For tasks requiring rapid scaling of epochs or unpredictable computation budgets: HyperbolicLR/ExpHyperbolicLR, tuned on small $N$ and transferred without retuning [2407.15200].
- Overhead of additional statistics (EMA, loss smoothing, internal RNN state) is $O(d)$ per step, negligible relative to compute- and memory-bound workloads.
- Horizon-free schemes reduce the need for checkpointing, retuning, and early stopping heuristics, and integrate cleanly with contemporary ML pipeline infrastructure.

## 7. Limitations, Open Problems, and Frontiers

- **Final-iterate minimaxity is unattainable in full generality for horizon-free scalar schedules without averaging**: a \emph{suboptimality gap} is impossible to eliminate in strongly convex streaming regression [1904.12838].
- **Extensions to other optimizers, e.g. AdamW or momentum SGD, are empirically validated**, but precise universal theoretical characterization beyond SGD is partially open [2602.03702].
- **Meta-learned and state-based approaches depend on training data diversity:** while robust in benchmark evaluations, their horizon-agnostic property is ultimately empirical in generalization to new domains [2509.23052, 2007.14546].
- **Bilevel and ODE-based algorithms have higher per-iteration cost**, though recent advances suggest these are manageable in practice [2509.23052].
- **Theoretical guarantees for online hypergradient-based schedulers (e.g., MARTHE)** exist for stability under natural conditions but lack formal rate-optimality proofs [1910.08525].

A plausible implication is that the future of horizon-free scheduling will increasingly blend lightweight, minimax polynomial/EMA protocols for core optimization with task-adaptive or meta-learned controllers for long-horizon, heterogeneous, or nonstationary training regimes. Emerging open questions include:
- Can one close the minimax final-iterate gap for horizon-free schemes without averaging, possibly via new algorithmic primitives (e.g., selective checkpointing, hybrid geometric–state triggers)? 
- How robust are ODE/meta-learned schedulers to adversarial or regime-shifting data, and what are the lower bounds on their sample complexity to attain generalization across both model and horizon?

**References:**  
- "Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging" [2602.03702]  
- "Dynamic Learning Rate Scheduling based on Loss Changes Leads to Faster Convergence" [2512.14527]  
- "How to decay your learning rate" (ABEL) [2103.12682]  
- "MARTHE: Scheduling the Learning Rate Via Online Hypergradients" [1910.08525]  
- "HyperbolicLR: Epoch insensitive learning rate scheduler" [2407.15200]  
- "Theory of Optimal Learning Rate Schedules and Scaling Laws for a Random Feature Model" [2602.04774]  
- "Dynamics of Learning: Generative Schedules from Latent ODEs" [2509.23052]  
- "Gradient Descent with Provably Tuned Learning-rate Schedules" [2512.05084]  
- "MLR-SNet: Transferable LR Schedules for Heterogeneous Tasks" [2007.14546]  
- "The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squares" [1904.12838]

Source: https://www.emergentmind.com/topics/horizon-free-learning-rate-schedules