---
title: 'ScheduleFree+: Learning-Rate-Free Optimization'
url: https://www.emergentmind.com/topics/schedulefree
type: topic
---

# ScheduleFree+: Learning-Rate-Free Optimization

ScheduleFree+ is a schedule-free, learning-rate-free optimization framework designed for efficient, robust, and minimal-tuning training of large language models (LLMs) and deep neural networks at scale. Rooted in the theoretical framework of online-to-batch conversion and Polyak averaging, ScheduleFree+ extends the original Schedule-Free methodology to deal with the unique challenges of LLM-scale training, including large batch sizes, extreme parameter counts, and the requirement for predictable, “anytime” convergence. Empirical evidence demonstrates that ScheduleFree+ achieves faster convergence and greater stability compared to canonical learning-rate schedules, particularly in long-duration, high-budget LLM training scenarios [2605.19095].

## 1. Theoretical Basis and Algorithmic Structure

ScheduleFree+ builds on the unification of iterate averaging and step-size scheduling originally formalized in the Schedule-Free framework [2405.15682], leveraging a general weighted average of “fast” iterates $z_t$ to produce a stable, best-so-far output $x_t$. In the base variant, the AdamW update is cast as

$$
z_{t+1} = z_t - \eta \cdot G_t(y_t),
$$

with $G_t(y_t)$ the stochastic gradient evaluated at an interpolation point $y_t = (1-\beta)z_t + \beta x_t$. The output $x_{t+1}$ is formed via incremental averaging:

$$
x_{t+1} = (1 - c_{t+1}) x_t + c_{t+1} z_{t+1},
$$

with averaging weight $c_{t+1}$ determined by Polyak-based statistics or preset functional forms.

ScheduleFree+ extends this by introducing the following key components:

- **Inner Adam-style momentum $(\beta_1)$** for added stability in high-batch regimes.
- **Polyak-style adaptive step-size $(\eta_t)$** based on real-time estimates of the objective gap and gradient norm via exponential moving averages.
- **Averaging buffer warm-start**, holding $c_t=1$ for an initial phase to prevent norm collapse in early iterations.
- **Annealed outer momentum $(\beta_t)$**: begins with low $\beta_0$ for fast early progress, gradually increased to $\beta_{\max}$ to prioritize smoother convergence in long runs.

The step-size $\eta_t$ is computed per-iteration following

$$
\eta_t = \frac{\max(0, f_t + \beta_t \langle G_t, z_{t-1} - x_{t-1} \rangle)}{\widetilde{E}_t},
$$

where $\widetilde{E}_t$ is the Polyak denominator estimated as the corrected EMA of the $\ell_1$-gradient norm, converting to an $\ell_2$ estimate via a $\sqrt{\pi/2}$ factor.

This structure eliminates all learning-rate and decay hyperparameters except for the moments and decay constants inherited from AdamW [2605.19095].

## 2. Foundations in Averaging and Regret Analysis

ScheduleFree+ arises directly from minimax optimal weighted-regret bounds in stochastic optimization. For convex $F$ with minimizer $x^*$, the method guarantees

$$
\mathbb{E}[F(x_T) - F(x^*)] \leq \frac{\sum_{t=1}^T \mathbb{E} [\langle g_t, z_t - x^* \rangle]}{\sum_{t=1}^T \eta_t},
$$

with optimality (in the canonical SGD setting) achieved when $\eta_t \propto \Vert g_t \Vert^{-2}$. As the true gradient norm is typically unavailable in nonconvex regimes, ScheduleFree+ employs an EMA proxy, as in the Polyak step-size heuristic [2605.19095]. Unlike fixed-schedule step methods, the averaging-driven contraction in the output $x_t$ ensures $\mathcal{O}(1/\sqrt{T})$ rates without horizon-dependent tuning.

On the practical side, ScheduleFree+ implements continuous interpolation between momenta and streaming average checkpoints, which empirically yields “anytime” optimization—$x_t$ is the best available iterate at every step, addressing performance oscillations common in schedule-based methods.

## 3. Scaling and Robustness Enhancements

Despite strong small- and mid-scale performance, the original Schedule-Free AdamW exhibited loss divergence and instability at the scale of LLM training (e.g., batch sizes $\geq 0.5$M tokens, $>1$B parameters). ScheduleFree+ addresses these by:

1. **Restoring inner momentum $\beta_1\approx 0.75$–$0.9$** to suppress large-batch noise and cliff-like instabilities.
2. **Switching to “fully decoupled” AdamC**: Weight decay $\gamma$ is applied via $\gamma \eta_t$ (not just $\gamma$), enforcing stable weight and gradient norm evolution even as $\eta_t$ adapts.
3. **Warm-starting the averaging buffer**: $c_t=1$ for $C_0$ steps (typically $400$–$800$), after which averaging proceeds, preventing early norm collapse in $x_t$ and improving initial loss behavior.
4. **Annealing outer momentum $\beta_t$**: Interpolated from $\beta_0\approx 0.8$ to $\beta_{\max}\approx 0.965$ over the first 10–20% of epochs, balancing fast initial adaptation and later-stage smoothness.
5. **Time-weighted averaging $w_t\propto t^r$** with $r=1$ for long runs (tokens-per-parameter $\geq 30$B), $r=0$ for short runs, empirically matched to the scale of LLM pretraining.

These changes collectively enable near-constant gradient norms, predictable effective step sizes, and robust convergence across a range of batch and model sizes, as demonstrated empirically [2605.19095].

## 4. Empirical Evaluation and Comparative Performance

ScheduleFree+ has been benchmarked on scaling ladders comprising 120M to 1B parameter Llama-3 style transformers, with batch sizes up to 4M tokens and sequence lengths of 2K tokens. The optimizer is consistently competitive with, and often superior to, state-of-the-art Linear-Decay and Warmup-Stable-Decay (WSD) schedules.

Key empirical findings include:

- **Long runs (1000 tokens/parameter):** ScheduleFree+ reaches the same validation loss as tuned Linear-Decay in 31% fewer tokens; Linear-Decay required 45% more tokens to reach parity at 120M scale.
- **Medium/short runs (100/20 tpp):** ScheduleFree+ outperforms or matches WSD, is generally comparable to Linear-Decay except for very short runs on the largest models where prolonged drift in early steps is limiting.
- **Convergence predictability:** The loss $L(t)$ fits the form $a/\sqrt{t + b} + c$ outside the initial 5% burn-in, enabling accurate stopping/horizon planning by early-stage curve fitting.
- **Loss curve behavior:** The adaptive $\eta_t$ schedule provides smooth, monotonic optimization progress with no imposed decay, in contrast to the stepwise fluctuations and plateaux seen in canonical schedules [2605.19095].

## 5. Integration into Foundation Model Fine-Tuning and Physical Modeling

ScheduleFree-style methods have shown practical benefits beyond language model pretraining:

- In atomistic foundation modeling, ScheduleFree with AdamW-style preconditioning and automatic global scaling $s_t$ achieves superior force root mean-squared error (RMSE) and robust molecular dynamics (MD) stability, on par or better than AdamW and LAMB, and strictly superior to SGD, RAdam, and Ranger, as rigorously benchmarked in energy/force accuracy and physical observable fidelity [2512.05489]. 
- The preconditioning framework endows ScheduleFree with nearly perfect Hessian spectrum flattening ($p_i \propto \lambda_i^{-1}$), yielding improved convergence, lower friction along flat modes, and improved stability in strongly anisotropic landscapes, including solid–liquid interface dynamics and phonon spectra recovery.
- ScheduleFree’s adaptation of Polyak averaging renders model averaging and checkpoint merging a theoretically principled procedure rather than a heuristic, supporting empirical gains found in recent “model soups” research [2605.19095].

## 6. Hyperparameterization and Best Practices

ScheduleFree+ eliminates the need for hand-tuned learning-rate schedules, grid searches for decay strategies, or epoch-dependent scheduling:

- **Adam/AdamW parameters:** Retains $\beta_1, \beta_2, \varepsilon, \lambda$ for momentum/decay; no schedule hyperparameters.
- **Polyak EMA coefficient:** $p\approx 0.9$ in $\widetilde{E}_t$ denominator suffices.
- **Average warmup:** $C_0\approx 2\times$ warmup steps (empirically $400-800$).
- **Weight decay $\gamma$:** Used at $5\times$–$50\times$ standard values, always applied as $\gamma \eta_t$.
- **No learning-rate grid search:** Step-size $\eta_t$ is computed online per iteration; users set only optimizer defaults [2605.19095].
- **Separate $x_t$ for evaluation:** $x_t$ yields the best-so-far model; $z_t$ is not used in evaluation.

In fine-tuning workflows, ScheduleFree can be used as a drop-in replacement for AdamW with minimal changes. For strongly heterogeneous or highly anisotropic domains, a brief second-order refinement (e.g., L-BFGS post-processing) after ScheduleFree optimization is recommended but not required for homogeneous tasks [2512.05489].

## 7. Broader Impact and Methodological Significance

ScheduleFree+ exemplifies a principled shift away from heuristic, horizon-dependent schedule design toward theoretically grounded, uniformly convergent, and robust optimizer construction. By unifying Polyak heuristic steps, online-to-batch averaging, and advanced momentum design in a learning-rate-free framework, ScheduleFree+ simplifies practical optimization, reduces parameter-sensitivity, and enhances convergence predictability in both deep learning and scientific modeling contexts.

The approach is especially well-suited for scenarios with unpredictable compute budgets, massive model scales, or environments demanding anytime stop-ability and stability across diverse tasks—including but not limited to LLM pretraining and foundation model fine-tuning across scientific domains [2605.19095, 2512.05489].

Source: https://www.emergentmind.com/topics/schedulefree