---
title: Schedule-Free Learning in Optimization
url: https://www.emergentmind.com/topics/schedule-free-learning
type: topic
---

# Schedule-Free Learning in Optimization

Schedule-free learning is a methodology in machine learning optimization that eliminates the use of explicit, hand-designed learning-rate schedules—such as step decay, cosine, or linear annealing—replacing them with mechanisms that induce implicit, self-adaptive decay or prioritization through algorithmic or meta-algorithmic structures. Schedule-free approaches are theoretically grounded, broadly applicable across convex and nonconvex regimes, and have demonstrated competitive or superior empirical performance in convex optimization, large-scale deep learning, continual learning, and reinforcement learning. Key attributes include horizon-independence, minimal additional hyperparameters, applicability to "anytime" training (where training can be stopped at arbitrary times with minimal loss in efficiency), and a broad unification of scheduling and model averaging frameworks [2405.15682, 2507.09846, 2605.19095].

## 1. Conceptual Foundations and Motivation

Traditional optimization algorithms for deep learning, such as SGD, Adam, and their variants, universally employ hand-tuned learning-rate schedules that require prior knowledge of the intended training duration (horizon), typically encoded as a fixed number of steps or epochs. Schedule tuning is task-specific, expensive, and brittle; failures to match schedules to the true problem horizon, or to changes in data scale, can cause premature convergence, overfitting, or loss of Pareto-optimality [2405.15682, 2605.23061]. Moreover, classical curriculum learning and self-paced learning often require explicit, hand-coded rules governing data presentation sequences.

Schedule-free learning seeks to eliminate these design burdens. The schedule-free perspective is enabled by theoretical insights that unify the roles of learning-rate scheduling and averaging (e.g., Polyak-Ruppert averaging) within a momentum-averaging interpolation framework. This unification enables the construction of algorithms that induce a decayed effective learning rate through the update structure—removing the need to specify or tune a schedule—while maintaining optimal statistical and computational performance [2405.15682].

## 2. Algorithmic Schemes and Theoretical Frameworks

### Core Update Structure

Schedule-free algorithms operate by decoupling the learning rate schedule from the optimizer and replacing it with online averaging. A minimal, but general, instance is as follows [2405.15682, 2502.02431, 2507.09846]:

1. Maintain three sequences:
   - $z_t$: "fast" iterate updated by the base optimizer with a fixed learning rate,
   - $x_t$: online average of iterates (returned at evaluation),
   - $y_t$: interpolation point for gradient evaluation.

2. Updates are:
   \[
   y_t = \alpha z_t + \beta x_t
   \]
   \[
   z_{t+1} = z_t - \gamma g_t
   \]
   \[
   x_{t+1} = (1-c_{t+1})x_t + c_{t+1} z_{t+1}
   \]
   where $g_t = \nabla f(y_t)$, $c_{t+1} = 1/(t+1)$ (or a variant), and $\beta \in [0,1]$ is a fixed momentum-like parameter.

This leads to an effective implicit learning-rate decay on $x_t$, e.g., $\gamma_{\text{eff}} \sim \gamma/(t+1)$, without explicit schedule specification.

### Unification of Scheduling and Averaging

Theoretical results demonstrate that weighted online averaging can substitute for explicit time-varying step-size schedules. In the online-to-batch conversion framework, forming $x_T = \sum_{i=1}^T w_i z_i/\sum w_i$ (with $w_i$ constant or increasing) and evaluating gradients at $y_t$ yields minimax-optimal rates for convex and strongly convex stochastic optimization, but also extends to nonconvex domains [2405.15682, 2411.07061]. For nonconvex objectives, recent work has established that schedule-free SGD achieves optimal iteration complexity, provided that averaging parameters and step-sizes are set according to theoretical prescriptions specific to the nonconvex regime [2411.07061].

Table 1 summarizes key mapping of schedule-free updates to established optimization paradigms:

| Averaging parameter β | Special case              | Algorithmic behavior         |
|----------------------|--------------------------|-----------------------------|
| 0                    | Polyak–Ruppert averaging | Worst-case optimal convex    |
| 1                    | Primal averaging         | Adaptive online-to-batch     |
| (0,1)                | Interpolated averaging   | Schedule-free/momentum blend |

## 3. Empirical Performance and Scaling

Schedule-free algorithms match or exceed the best-tuned schedules from convex settings (e.g., logistic regression) to large-scale deep networks and language models [2405.15682, 2507.09846, 2605.23061, 2605.19095]. Key empirical findings include:

- In deep learning benchmarks (CIFAR-10, ImageNet, GPT-2, ViT, etc.), schedule-free optimizers track or improve upon hand-tuned step, cosine, or linear decay schedules, converging to similar or better final accuracy and with improved anytime capabilities.
- On long-horizon LLM pretraining (i.e., >1000 tokens per parameter), schedule-free methods (e.g., ScheduleFree+) reduce wall-time to target loss by 20–30% and outperform state-of-the-art schedules, particularly for arbitrarily extended training durations [2605.19095].
- In continual learning, schedule-robust predictors built from order-invariant statistics (e.g., nearest-centroid or ridge regression with fixed representations) eliminate performance dependence on the order or schedule of arriving data batches, addressing catastrophic forgetting directly from a schedule-free perspective [2210.05561].
- Weight-averaging-based approaches, including checkpoint merging (WSM), achieve similar benefits. The theoretical equivalence between schedule decay and merging allows any decay (e.g., cosine, linear, inverse-sqrt) to be realized via weighted checkpoint averages, with the merge duration as the critical hyperparameter [2507.17634].

## 4. Extensions, Variants, and Related Frameworks

Several major lines of research build on or extend the schedule-free paradigm:

- **Spectral geometry optimizers:** Schedule-free spectral optimizers (e.g., SF-NorMuon) replace coordinate-wise updates with operator-norm geometry (polar updates), together with row-wise normalization and weight decay at the fast iterate. This closes the gap in anytime performance with hand-tuned AdamW at scale [2605.23061].
- **Curriculum and self-paced learning:** Schedule-free learning encompasses not only optimizer-level schedules but also data-selection schedules. Feature-learning regimes in modern deep nets automatically induce a schedule-free curriculum, prioritizing easy examples before hard ones [2209.09658]. ScreenerNet further demonstrates fully self-paced, schedule-free selection of training weights for each sample via an auxiliary network, accelerating convergence and improving data efficiency [1801.00904].
- **Reinforcement learning for schedule adaptation:** RL-based meta-schedulers can dynamically adapt the learning rate or replay policies without any hand-designed schedule, using state features and reward signals derived from validation metrics, enabling cross-task generalization [1909.09712, 2209.08660].
- **Checkpoint merging and model soups:** Decay-free approaches (e.g., WSM) formalize the connection between decay schedules and post-hoc model averaging, making it possible to emulate any decay curve offline via windowed checkpoint merging [2507.17634].

## 5. Practical Guidelines and Implementation Considerations

- **Hyperparameter selection:** Schedule-free optimizers require no additional hyperparameters beyond those of the base optimizer (SGD, AdamW). Typical choices—for AdamW, learning rate γ, weight decay λ, warmup length, momentum β₁≈0.9—carry over directly [2405.15682].
- **Averaging weights:** The sequence $c_{t+1}$ is typically set to $1/(t+1)$ after warmup, or more generally $c_{t+1}\propto t^r$ ($r=1$ for long runs, $r=0$ for short) to bias averaging windows [2605.19095].
- **Momentum decoupling and large-batch stability:** For large models and batch sizes, explicit momentum buffers and decoupled averaging parameters improve robustness (e.g., ScheduleFree+ uses outer momentum β₁ annealing, r-weighting for averaging, and Polyak step-size adaptation) [2605.19095, 2507.09846].
- **Evaluation:** The recommended evaluation and deployment target is always the averaged sequence $x_t$, not the latest fast iterate $z_t$. For batch normalization, running statistics should be recomputed on $x_t$ [2405.15682].
- **Checkpoint management:** For model branching and offline averaging, use the same weighting strategies (e.g., r-weighting) as in online schedule-free training. Merge duration is the key factor for performance in checkpoint-merge methods [2507.17634].

## 6. Theoretical Guarantees and Open Directions

Schedule-free algorithms match minimax-optimal rates in convex, strong-convex, and nonconvex regimes:

- For convex Lipschitz objectives, optimal $O(1/\sqrt{T})$ rates are attained independently of averaging parameter β [2405.15682].
- For smooth or strongly convex problems, accelerated rates $O(1/T^2)$ or $O(1/T)$ (with modifications) are provable [2405.15682].
- For nonsmooth, nonconvex objectives, schedule-free SGD achieves the optimal $O(\varepsilon^{-7/2})$ complexity for stationarity, matching state-of-the-art online-to-nonconvex reduction results without needing a manual schedule [2411.07061].
- In the noise-dominated regime, schedule-free optimizers are theoretically equivalent to accelerated SGD variants followed by tail averaging, and thus inherit their variance-reduction and accelerated convergence benefits [2502.02431].

Outstanding research challenges include the extension of schedule-free optimization to architectures beyond transformers (e.g., multimodal or MoE), integration within task-free continual learning paradigms, and theoretical characterization in the presence of normalization layers and batch-dependent statistics [2605.23061]. Empirical scaling for ultra-long horizons and extremely large batch sizes is ongoing, with practical fixes (momentum schemes, r-weighting) continually refined as frontier systems are trained [2605.19095, 2507.09846].

## 7. Relationship to Data Curriculum and Continual Learning

Schedule-free learning also encompasses automatic ordering or weighting of data, notably in online, feature-learning regimes where neural networks prioritize easy examples before hard ones without explicit curricula [2209.09658]. In online continual learning, schedule-free or schedule-robust approaches construct predictors that are invariant to orderings and schedule permutations of the data stream, thus eliminating catastrophic forgetting modes tied to class order or memory policy [2210.05561]. Approaches using meta-learned replay schedules (via MCTS or reinforcement learning) offer further schedule-free adaptation, especially in replay-constrained or compute-limited continual learning settings [2209.08660]. Self-paced curriculum methods such as ScreenerNet dynamically weight or select samples end-to-end, again bypassing hand-crafted or epoch-based curricula [1801.00904].

---

**References**:  
[2405.15682] "The Road Less Scheduled"  
[2507.09846] "Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Training"  
[2605.19095] "ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models"  
[2605.23061] "Anytime Training with Schedule-Free Spectral Optimization"  
[2507.17634] "WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training"  
[2411.07061] "General framework for online-to-nonconvex conversion: Schedule-free SGD is also effective for nonconvex optimization"  
[2210.05561] "Schedule-Robust Online Continual Learning"  
[2502.02431] "Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variants"  
[2209.09658] "Lazy vs hasty: linearization in deep networks impacts learning schedule based on example difficulty"  
[1801.00904] "ScreenerNet: Learning Self-Paced Curriculum for Deep Neural Networks"  
[2209.08660] "Learn the Time to Learn: Replay Scheduling in Continual Learning"  
[1909.09712] "Learning an Adaptive Learning Rate Schedule"

Source: https://www.emergentmind.com/topics/schedule-free-learning