---
title: Cosine Learning Rate Decay
url: https://www.emergentmind.com/topics/cosine-learning-rate-decay
type: topic
---

# Cosine Learning Rate Decay

Cosine learning rate decay is a widely used learning rate schedule in large-scale pretraining of deep neural networks. This schedule decays the learning rate according to the upper half of a cosine function, providing a smooth transition from a high learning rate at the beginning of training to a lower value at the end. Its design balances rapid initial progress with slow, stable convergence, but also introduces constraints and limitations that have driven the development of alternative schedules and analytical frameworks.

## 1. Definition and Functional Form

Cosine learning rate decay is defined mathematically as:

\[
\eta_t = \eta_{\min} + \frac{1}{2} (\eta_{\max} - \eta_{\min}) [1 + \cos(\pi t / T)]
\]

where:
- $\eta_{\max}$ is the initial (peak) learning rate,
- $\eta_{\min}$ is the final (floor) learning rate,
- $t \in [0,T]$ denotes the current training step,
- $T$ is the total number of steps over which the decay is applied.

Typical practice is to precede this decay by a linear warmup period, during which the learning rate increases linearly from zero to $\eta_{\max}$ over the initial $W$ steps. At $t = T$, $\eta_t = \eta_{\min}$, generating a monotonic, smooth decay curve from peak to floor learning rate [2410.05192][2503.09411][2602.03702].

## 2. Theoretical and Empirical Properties

Cosine decay is motivated by the goal of enabling aggressive early progress (via high learning rates) while ensuring convergence and fine-tuning (via gradual annealing). Key theoretical results associated with cosine annealing include:

- **Convergence Rates**: For convex objectives, the final suboptimality satisfies $\mathbb{E}[f(x_{T+1})-f(x^*)] \leq O(DL/\sqrt{T}) \cdot O(\rho^{1/5})$, where $\rho$ quantifies learning rate misspecification and $T$ is the training step count. The sublinear $\rho^{1/5}$ scaling (for $p=2$ polynomial decay, as in cosine) reflects robustness absent from fixed-step methods, which degrade linearly with $\rho$ [2503.09411].
- **Loss Curve Characterization**: Empirical studies show that cosine decay yields loss curves consistent with a power-law in the cumulative learning rate, with extra reductions induced by the schedule's continuous decrease [2503.12811]. Predictive models such as the Multi-Power Law (MPL) can fit and forecast the loss trajectory under cosine decay with high accuracy ($R^2 > 0.997$).

Empirically, cosine decay matches or slightly underperforms recent alternatives (e.g., Warmup-Stable-Decay, constant-plus-cooldown) for fixed training budgets, especially when compared at their optimal hyperparameters [2410.05192][2502.15938][2602.03702].

## 3. Comparison with Alternative Schedules

Recent literature contextualizes cosine decay within a broader ecosystem of learning rate schedules. Direct competitors include:

| Schedule Type       | Regime           | Horizon Dependency | Practical Remarks                                                             |
|---------------------|------------------|-------------------|-------------------------------------------------------------------------------|
| Cosine Decay        | Fixed-budget     | Horizon-fixed     | Requires $T$ in advance; not "anytime"; strong for single endpoint [2410.05192][2602.03702]|
| Linear Decay-to-Zero (D2Z) | Fixed-budget     | Horizon-fixed     | Outperforms cosine for high-TPP pretraining, robust to batch/scale [2502.15938]|
| Warmup-Stable-Decay (WSD)   | Compute-agnostic | Horizon-free      | Enables branching to any budget, reusable runs, marginal loss improvement [2410.05192]|
| Checkpoint Averaging (e.g., WSM, SWA) | Horizon-free      | Horizon-free      | Emulates or surpasses cosine via merging, flexible, less compute [2507.17634][2405.18392]|
| 1/$\sqrt{t}$, Const+EMA     | Horizon-free      | Horizon-free      | Matches tuned-cosine across budgets with one hyperparameter set [2602.03702]   |

Cosine decay is suboptimal when repeated runs for scaling law studies or multi-budget checkpointing are desired. Schedules supporting post-hoc merging or anytime training alleviate these limitations, deliver similar final losses, and dramatically reduce compute requirements [2405.18392][2602.03702][2507.17634].

## 4. Limitations and Trade-Offs

Cosine decay exhibits several structural limitations:

- **Horizon Dependence and Non-Anytime Character**: Cosine schedules require pre-specifying $T$ and perform optimally only at this horizon. Intermediate checkpoints underperform compared to separate, appropriately-sized cosine runs [2405.18392][2602.03702].
- **Lack of Flexibility in Continual/Long-Horizon Regimes**: In continual pretraining, repeated cosine restarts induce instability and catastrophic forgetting by injecting large gradient steps. Infinite-style schedules that forgo restarts and decays mitigate these effects [2503.02844].
- **Computational Redundancy**: Scaling law and checkpointing experiments under cosine decay require redundant full runs; constant+cooldown or weight averaging can reach equivalent minima using one run and post-hoc analysis [2405.18392][2507.17634].
- **Sensitivity to Learning Rate and Tuning**: While cosine annealing offers robustness to coarse base learning rate tuning [2503.09411], optimal performance still requires grid search and schedule-dependent tuning, especially outside the high-TPP regime [2502.15938].

## 5. Analytical Frameworks and Loss Curve Modeling

Predictive modeling of pretraining loss under cosine decay has advanced understanding and schedule optimization:

- **Multi-Power Law (MPL)**: Loss at step $t$ is described as $L(t) = L_0 + A(S_1(t)+S_W)^{-\alpha} - LD(t)$, with $S_1(t)$ the sum of learning rates and $LD(t)$ additional reduction from the schedule's decay [2503.12811].
- **Optimization in Schedule Space**: Differentiating the MPL surrogate with respect to the full learning rate schedule allows direct search for schedules outperforming pure cosine. Empirically, the optimal schedule closely resembles a prolonged plateau (stable phase) followed by a power-law decay, slightly different from vanilla cosine [2503.12811].
- **River-Valley Loss Landscape**: Cosine and WSD performance are interpretable via a geometric picture where progress along the "river" direction is achieved with high learning rate (oscillatory), while decay serves to concentrate iterates near the loss basin (reducing the "hill" component) [2410.05192].

## 6. Practical Recommendations and Guidelines

Best practices for cosine decay and its competitors are established across works:

- **Cosine Decay**: Remains a strong choice for fixed-length, production-grade pretraining, provided $T$ (token budget) is pre-specified and both warmup and decay durations are tuned. Common settings include $W\approx2\%$, $D\approx10$–$20\%$ of total steps, $\eta_{\min}\approx0.1\eta_{\max}$ [2410.05192][2405.18392].
- **Anytime or Multi-Checkpoint Regimes**: Prefer WSD-S, WSM, or averaging methods (e.g., constant $+$ cooldown, SWA, model merging) for efficiency and flexibility [2410.05192][2507.17634][2405.18392][2602.03702].
- **Avoiding Cosine in Continual Learning**: For continual (multi-phase) pretraining, avoid cosine restarts; use infinite schedules without rewarm or checkpoint pre-decay [2503.02844].
- **Schedule Selection**: For high-TPP LLM regimes, linear warmup plus decay-to-zero is empirically superior. For low-TPP or when $T$ is unknown, inverse-sqrt or anytime (horizon-free) schedules are preferable [2502.15938][2602.03702].

## 7. Historical and Methodological Context

The cosine learning rate schedule, originally formalized in SGDR (Loshchilov & Hutter, 2016), rose to prominence as the default for LLM pretraining. Its adoption is widespread in major model releases (e.g., LLaMA 3), driven by its empirical effectiveness and simplicity [2602.03702]. Subsequent research has focused on analytical characterizations, empirical benchmarking, and alternatives designed to address its intrinsic horizon dependency and inefficiency in multi-budget, continual, or scaling-law contexts [2410.05192][2507.17634][2405.18392][2503.09411][2502.15938][2503.12811][2503.02844][2602.03702].

A plausible implication is that, while cosine decay remains central to current practice, its monopoly is being eroded by flexible, analytically guided alternatives that permit compute re-use, schedule-free adaptation, and theoretically grounded robustness. The trend in state-of-the-art pretraining increasingly favors schedules compatible with horizon-agnostic training and post-hoc evaluation, especially in resource-constrained or scaling-oriented workflows.

Source: https://www.emergentmind.com/topics/cosine-learning-rate-decay