Schedule-Free Learning in Optimization

Updated 3 July 2026

Schedule-free learning is a method that replaces explicit learning rate schedules with implicit decay induced by online averaging and momentum interpolation.
It leverages unified theoretical frameworks to balance scheduling and averaging, ensuring optimal convergence in convex, nonconvex, and reinforcement learning settings.
Empirical results show that schedule-free approaches achieve competitive performance and reduced hyperparameter tuning in deep learning, language models, and continual learning.

Schedule-free learning is a methodology in machine learning optimization that eliminates the use of explicit, hand-designed learning-rate schedules—such as step decay, cosine, or linear annealing—replacing them with mechanisms that induce implicit, self-adaptive decay or prioritization through algorithmic or meta-algorithmic structures. Schedule-free approaches are theoretically grounded, broadly applicable across convex and nonconvex regimes, and have demonstrated competitive or superior empirical performance in convex optimization, large-scale deep learning, continual learning, and reinforcement learning. Key attributes include horizon-independence, minimal additional hyperparameters, applicability to "anytime" training (where training can be stopped at arbitrary times with minimal loss in efficiency), and a broad unification of scheduling and model averaging frameworks (Defazio et al., 2024, Song et al., 14 Jul 2025, Defazio, 18 May 2026).

1. Conceptual Foundations and Motivation

Traditional optimization algorithms for deep learning, such as SGD, Adam, and their variants, universally employ hand-tuned learning-rate schedules that require prior knowledge of the intended training duration (horizon), typically encoded as a fixed number of steps or epochs. Schedule tuning is task-specific, expensive, and brittle; failures to match schedules to the true problem horizon, or to changes in data scale, can cause premature convergence, overfitting, or loss of Pareto-optimality (Defazio et al., 2024, Apte et al., 21 May 2026). Moreover, classical curriculum learning and self-paced learning often require explicit, hand-coded rules governing data presentation sequences.

Schedule-free learning seeks to eliminate these design burdens. The schedule-free perspective is enabled by theoretical insights that unify the roles of learning-rate scheduling and averaging (e.g., Polyak-Ruppert averaging) within a momentum-averaging interpolation framework. This unification enables the construction of algorithms that induce a decayed effective learning rate through the update structure—removing the need to specify or tune a schedule—while maintaining optimal statistical and computational performance (Defazio et al., 2024).

2. Algorithmic Schemes and Theoretical Frameworks

Core Update Structure

Schedule-free algorithms operate by decoupling the learning rate schedule from the optimizer and replacing it with online averaging. A minimal, but general, instance is as follows (Defazio et al., 2024, Morwani et al., 4 Feb 2025, Song et al., 14 Jul 2025):

Maintain three sequences:
- $z_t$ : "fast" iterate updated by the base optimizer with a fixed learning rate,
- $x_t$ : online average of iterates (returned at evaluation),
- $y_t$ : interpolation point for gradient evaluation.
Updates are:

$y_t = \alpha z_t + \beta x_t$

$z_{t+1} = z_t - \gamma g_t$

$x_{t+1} = (1-c_{t+1})x_t + c_{t+1} z_{t+1}$

where $g_t = \nabla f(y_t)$ , $c_{t+1} = 1/(t+1)$ (or a variant), and $\beta \in [0,1]$ is a fixed momentum-like parameter.

This leads to an effective implicit learning-rate decay on $x_t$ , e.g., $x_t$ 0, without explicit schedule specification.

Unification of Scheduling and Averaging

Theoretical results demonstrate that weighted online averaging can substitute for explicit time-varying step-size schedules. In the online-to-batch conversion framework, forming $x_t$ 1 (with $x_t$ 2 constant or increasing) and evaluating gradients at $x_t$ 3 yields minimax-optimal rates for convex and strongly convex stochastic optimization, but also extends to nonconvex domains (Defazio et al., 2024, Ahn et al., 2024). For nonconvex objectives, recent work has established that schedule-free SGD achieves optimal iteration complexity, provided that averaging parameters and step-sizes are set according to theoretical prescriptions specific to the nonconvex regime (Ahn et al., 2024).

Table 1 summarizes key mapping of schedule-free updates to established optimization paradigms:

Averaging parameter β	Special case	Algorithmic behavior
0	Polyak–Ruppert averaging	Worst-case optimal convex
1	Primal averaging	Adaptive online-to-batch
(0,1)	Interpolated averaging	Schedule-free/momentum blend

3. Empirical Performance and Scaling

Schedule-free algorithms match or exceed the best-tuned schedules from convex settings (e.g., logistic regression) to large-scale deep networks and LLMs (Defazio et al., 2024, Song et al., 14 Jul 2025, Apte et al., 21 May 2026, Defazio, 18 May 2026). Key empirical findings include:

In deep learning benchmarks (CIFAR-10, ImageNet, GPT-2, ViT, etc.), schedule-free optimizers track or improve upon hand-tuned step, cosine, or linear decay schedules, converging to similar or better final accuracy and with improved anytime capabilities.
On long-horizon LLM pretraining (i.e., >1000 tokens per parameter), schedule-free methods (e.g., ScheduleFree+) reduce wall-time to target loss by 20–30% and outperform state-of-the-art schedules, particularly for arbitrarily extended training durations (Defazio, 18 May 2026).
In continual learning, schedule-robust predictors built from order-invariant statistics (e.g., nearest-centroid or ridge regression with fixed representations) eliminate performance dependence on the order or schedule of arriving data batches, addressing catastrophic forgetting directly from a schedule-free perspective (Wang et al., 2022).
Weight-averaging-based approaches, including checkpoint merging (WSM), achieve similar benefits. The theoretical equivalence between schedule decay and merging allows any decay (e.g., cosine, linear, inverse-sqrt) to be realized via weighted checkpoint averages, with the merge duration as the critical hyperparameter (Tian et al., 23 Jul 2025).

Several major lines of research build on or extend the schedule-free paradigm:

Spectral geometry optimizers: Schedule-free spectral optimizers (e.g., SF-NorMuon) replace coordinate-wise updates with operator-norm geometry (polar updates), together with row-wise normalization and weight decay at the fast iterate. This closes the gap in anytime performance with hand-tuned AdamW at scale (Apte et al., 21 May 2026).
Curriculum and self-paced learning: Schedule-free learning encompasses not only optimizer-level schedules but also data-selection schedules. Feature-learning regimes in modern deep nets automatically induce a schedule-free curriculum, prioritizing easy examples before hard ones (George et al., 2022). ScreenerNet further demonstrates fully self-paced, schedule-free selection of training weights for each sample via an auxiliary network, accelerating convergence and improving data efficiency (Kim et al., 2018).
Reinforcement learning for schedule adaptation: RL-based meta-schedulers can dynamically adapt the learning rate or replay policies without any hand-designed schedule, using state features and reward signals derived from validation metrics, enabling cross-task generalization (Xu et al., 2019, Klasson et al., 2022).
Checkpoint merging and model soups: Decay-free approaches (e.g., WSM) formalize the connection between decay schedules and post-hoc model averaging, making it possible to emulate any decay curve offline via windowed checkpoint merging (Tian et al., 23 Jul 2025).

5. Practical Guidelines and Implementation Considerations

Hyperparameter selection: Schedule-free optimizers require no additional hyperparameters beyond those of the base optimizer (SGD, AdamW). Typical choices—for AdamW, learning rate γ, weight decay λ, warmup length, momentum β₁≈0.9—carry over directly (Defazio et al., 2024).
Averaging weights: The sequence $x_t$ 4 is typically set to $x_t$ 5 after warmup, or more generally $x_t$ 6 ( $x_t$ 7 for long runs, $x_t$ 8 for short) to bias averaging windows (Defazio, 18 May 2026).
Momentum decoupling and large-batch stability: For large models and batch sizes, explicit momentum buffers and decoupled averaging parameters improve robustness (e.g., ScheduleFree+ uses outer momentum β₁ annealing, r-weighting for averaging, and Polyak step-size adaptation) (Defazio, 18 May 2026, Song et al., 14 Jul 2025).
Evaluation: The recommended evaluation and deployment target is always the averaged sequence $x_t$ 9, not the latest fast iterate $y_t$ 0. For batch normalization, running statistics should be recomputed on $y_t$ 1 (Defazio et al., 2024).
Checkpoint management: For model branching and offline averaging, use the same weighting strategies (e.g., r-weighting) as in online schedule-free training. Merge duration is the key factor for performance in checkpoint-merge methods (Tian et al., 23 Jul 2025).

6. Theoretical Guarantees and Open Directions

Schedule-free algorithms match minimax-optimal rates in convex, strong-convex, and nonconvex regimes:

For convex Lipschitz objectives, optimal $y_t$ 2 rates are attained independently of averaging parameter β (Defazio et al., 2024).
For smooth or strongly convex problems, accelerated rates $y_t$ 3 or $y_t$ 4 (with modifications) are provable (Defazio et al., 2024).
For nonsmooth, nonconvex objectives, schedule-free SGD achieves the optimal $y_t$ 5 complexity for stationarity, matching state-of-the-art online-to-nonconvex reduction results without needing a manual schedule (Ahn et al., 2024).
In the noise-dominated regime, schedule-free optimizers are theoretically equivalent to accelerated SGD variants followed by tail averaging, and thus inherit their variance-reduction and accelerated convergence benefits (Morwani et al., 4 Feb 2025).

Outstanding research challenges include the extension of schedule-free optimization to architectures beyond transformers (e.g., multimodal or MoE), integration within task-free continual learning paradigms, and theoretical characterization in the presence of normalization layers and batch-dependent statistics (Apte et al., 21 May 2026). Empirical scaling for ultra-long horizons and extremely large batch sizes is ongoing, with practical fixes (momentum schemes, r-weighting) continually refined as frontier systems are trained (Defazio, 18 May 2026, Song et al., 14 Jul 2025).

7. Relationship to Data Curriculum and Continual Learning

Schedule-free learning also encompasses automatic ordering or weighting of data, notably in online, feature-learning regimes where neural networks prioritize easy examples before hard ones without explicit curricula (George et al., 2022). In online continual learning, schedule-free or schedule-robust approaches construct predictors that are invariant to orderings and schedule permutations of the data stream, thus eliminating catastrophic forgetting modes tied to class order or memory policy (Wang et al., 2022). Approaches using meta-learned replay schedules (via MCTS or reinforcement learning) offer further schedule-free adaptation, especially in replay-constrained or compute-limited continual learning settings (Klasson et al., 2022). Self-paced curriculum methods such as ScreenerNet dynamically weight or select samples end-to-end, again bypassing hand-crafted or epoch-based curricula (Kim et al., 2018).

References:

(Defazio et al., 2024) "The Road Less Scheduled" (Song et al., 14 Jul 2025) "Through the River: Understanding the Benefit of Schedule-Free Methods for LLM Training" (Defazio, 18 May 2026) "ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to LLMs" (Apte et al., 21 May 2026) "Anytime Training with Schedule-Free Spectral Optimization" (Tian et al., 23 Jul 2025) "WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training" (Ahn et al., 2024) "General framework for online-to-nonconvex conversion: Schedule-free SGD is also effective for nonconvex optimization" (Wang et al., 2022) "Schedule-Robust Online Continual Learning" (Morwani et al., 4 Feb 2025) "Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variants" (George et al., 2022) "Lazy vs hasty: linearization in deep networks impacts learning schedule based on example difficulty" (Kim et al., 2018) "ScreenerNet: Learning Self-Paced Curriculum for Deep Neural Networks" (Klasson et al., 2022) "Learn the Time to Learn: Replay Scheduling in Continual Learning" (Xu et al., 2019) "Learning an Adaptive Learning Rate Schedule"