ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models

Abstract: Schedule-Free Learning has shown promise as a practical anytime training method for machine learning, showing success across dozens of standard benchmark problems. However, strong performance for LLM training has only been demonstrated at small scales. We identify a number of fixes necessary to scale up Schedule-Free Learning to larger batch sizes and model sizes, and present a learning-rate-free and schedule-free method (ScheduleFree+) for training LLMs which greatly outperforms Warmup-Stable-Decay (WSD) schedules. We also demonstrate that Schedule-Free Learning is most effective for long duration training, and at 1000 tokens per parameter, it outperforms SOTA schedules by 31%. Schedule-Free Learning provides a theoretical foundation for the use of model averaging and checkpoint merging during pretraining.

• The paper introduces a novel learning-rate-free method using adaptive Polyak step sizes, inner momentum, and power-weighted averaging to optimize LLM training.
• It achieves a 31% reduction in time-to-target loss over traditional linear-decay schedules by stabilizing convergence with advanced gradient normalization techniques.
• The study unifies checkpoint averaging with classical optimization theory, enabling anytime evaluation and robust, long-horizon performance in large-scale LLM pretraining.

ScheduleFree+: Advancements in Learning-Rate-Free and Schedule-Free LLM Optimization

Introduction and Motivation

The "ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to LLMs" (2605.19095) technical report addresses a central challenge in LLM optimization: how to achieve convergence and competitive final performance without the extensive grid search, tuning, and instability associated with hand-crafted learning rate schedules. While Schedule-Free Learning has demonstrated stability and simplicity in moderate-scale scenarios, scaling these qualities to large batch and model regimes typical of LLM pretraining has proven problematic. This work systematically identifies and addresses these issues, culminating in the ScheduleFree+ approach, which replaces learning rate schedules with aggressive online averaging, momentum innovations, and a principled learning-rate-free step size, resulting in highly predictable and competitive optimization at LLM scale.

Schedule-Free Foundations and Algorithmic Advances

Traditional optimizers in deep learning, such as AdamW and SGD, rely on carefully designed decaying learning rate schedules for stability and peak performance. Schedule-Free Learning, as introduced in prior work, forgoes explicit learning rate schedules in favor of online averaging of iterates, with gradients evaluated at interpolated mixture points between the current and averaged parameters. However, naive application of this methodology to LLMs exposes several pathologies—especially at large batch sizes—such as convergence instability and suboptimal loss curves.

ScheduleFree+ addresses these issues with several algorithmic innovations:

• Reintroduction of Inner Momentum: While Schedule-Free Learning originally removes inner optimizer momentum to reduce memory, this results in a severe scaling cliff at high batch sizes. The inclusion of modest inner momentum (e.g., AdamW with β₁ ≈ 0.75–0.9) drastically improves large-batch robustness and matches or exceeds the batch size scaling of standard AdamW.
• Polyak-Style Adaptive Step Sizes: The report replaces global learning rates with a Polyak-like adaptive step size estimated from the difference between current and optimal function values, normalized by the (smoothed) L1 gradient norm. This learning-rate-free principle is not only theoretically justified but empirically matches or beats the best hand-tuned schedules in both early and late stages of training.
• Inverse-Gradient Norm Scaling: To stabilize the effective learning rate when gradient magnitudes drift (especially in networks with normalization layers and variable weight norms), the Polyak denominator is approximated using the L1 norm of the gradients, further stabilized by exponential moving averages. This adjustment makes long-horizon training feasible and stable.

Handling Optimization Pathologies in LLM Training

The authors provide a detailed analysis of optimization pathologies exposed when scaling up Schedule-Free Learning. Most notably:

• Gradient Norm Drift and Weight Decay Interplay: Standard AdamW-style weight decay parameters are suboptimal within Schedule-Free regimes because smaller decay is required to avoid rapid model norm shrinkage and corresponding gradient explosions. However, this can lead to large, unstable weight norms. The paper shows that ideal performance is achieved by tuning for flat gradient-norm trajectories, and this necessitates higher decay with explicit gradient norm normalization.
• Learning Rate Adaptation Feedback Loops: In normalized networks, adaptive learning rates and weight decay interact to induce dynamical instabilities. ScheduleFree+ mitigates this with fully decoupled AdamC-style updates and learning rate adaptation schemes that prevent or smooth non-monotonicities in learning rate trajectories.

Practical Modifications for Stability and Fast Convergence

Several further refinements are developed for optimal performance in long-horizon LLM pretraining:

• Warm-Starting and Delayed Averaging: Rather than initiating averaging from the first step, ScheduleFree+ delays averaging until after initial warmup. This accounts for rapid weight norm changes early in training, leading to better loss at early checkpoints.
• Beta Annealing and Averaging Power Weighting: The mixture parameter β, which controls averaging recency versus stability, is annealed from a lower initial value (faster convergence but less smoothing) to a higher value (better late-stage convergence). Additionally, power-weighted averaging (parameter r ≈ 1) accelerates convergence for long runs by overweighting later checkpoints, in line with recent stochastic optimization theory.

Empirical Evaluation and Predictability

ScheduleFree+ exhibits robust empirical superiority over Warmup-Stable-Decay (WSD) and linear-decay baselines, particularly in long-horizon and large-scale regimes relevant for frontier LLMs:

• For LLM pretraining runs at 1000 tokens/parameter, the time to reach target loss is reduced by 31% relative to best-tuned state-of-the-art linear-decay schedules.
• The loss curves fit the theoretical $O(1/\sqrt{t})$ convergence extremely well outside a short burn-in phase, affording predictive modeling of final loss from early-run checkpoints.
• In contrast to WSD and cosine schedules, which require end-of-run annealing phases and careful horizon tuning, ScheduleFree+ provides anytime evaluation: models can be stopped and evaluated at arbitrary checkpoints without loss in performance.

Implications and Theoretical Foundations

A noteworthy contribution is the link between Schedule-Free Learning and recent work on model checkpoint averaging ("souping" and merging). In contrast to post-hoc averaging of late checkpoints, the in-training mixing inherent in ScheduleFree+ provides theoretical guarantees and smoother trajectory convergence. The empirical demonstration that checkpoint averaging and model merging are theoretically justified by the properties of Schedule-Free Learning provides a unifying perspective on multiple strands of current LLM training research.

Moreover, the highly predictable convergence behavior facilitated by ScheduleFree+—mirroring classical stochastic convex theory even in the nonconvex, overparametrized LLM regime—suggests the theory-practice gap in deep learning optimization may be narrower than previously believed, provided correct algorithmic design choices are made.

Future Directions

The report points toward several avenues for further research and development:

• Greater exploitation of schedule-free averaging theory for parallel LLM pretraining architectures, potentially reducing communication and synchronization costs by more frequent checkpoint averaging.
• Integration into open-source LLM training stacks via the provided reference implementation, lowering the barrier for robust, tuning-free training regimes.
• Extension and adaptation of these ideas for other modalities (e.g., vision transformers) and optimization paradigms (e.g., decoupled weight-regularization in normalizer-free architectures).
• Deeper theoretical investigation of the connections between Polyak-style step sizes, normalization-induced effective learning rates, and generalization properties in deep neural networks.

Conclusion

ScheduleFree+ provides a rigorous, empirically validated, and learning-rate-free optimization framework enabling stable and efficient pretraining of LLMs at scale. By integrating adaptive Polyak step sizes, inner momentum reintroduction, delayed and power-weighted averaging, and principled weight-decay correction, it supersedes both classical and recently-introduced schedule-based methods in both stability and convergence time, particularly for long-duration regimes. The approach yields highly predictable loss curves, preserves the practical "anytime" advantage for real-world training regimes, and grounds recent practitioner techniques for checkpoint averaging within a solid optimization-theoretic framework (2605.19095).