---
title: 'Lion: Evolved Sign Momentum Optimizer'
url: https://www.emergentmind.com/topics/lion-evolved-sign-momentum
type: topic
---

# Lion: Evolved Sign Momentum Optimizer

Lion (Evolved Sign Momentum) is a sign-based stochastic optimization algorithm developed by symbolic program search, specifically designed for efficient and scalable training of large neural networks. Distinguished by its use of sign normalization with momentum, decoupled weight decay, and extremely low memory overhead, Lion has demonstrated empirical superiority over adaptive optimizers such as AdamW on a diverse array of machine learning tasks, including vision, language, and diffusion models. It achieves provably optimal convergence rates under standard and heavy-tailed noise models, and operates robustly in both centralized and distributed environments.

## 1. Algorithmic Structure and Update Rules

Lion maintains two momentum sequences and leverages a coordinate-wise sign-based update, incorporating decoupled weight decay to solve unconstrained and $\ell_\infty$-constrained problems. The canonical form is as follows:

Let $x_t \in \mathbb{R}^d$ be the parameter vector at iteration $t$; $g_t$ is the (mini-batch) stochastic gradient.

The update rule is:
\[
\begin{aligned}
  v_t &= \beta_1 m_{t-1} + (1-\beta_1) g_t \\
  m_t &= \beta_2 m_{t-1} + (1-\beta_2) g_t \\
  x_{t+1} &= x_t - \eta [\, \mathrm{sign}(v_t) + \lambda x_t \,]
\end{aligned}
\]
where:
- $\eta$ is the learning rate,
- $\lambda$ is the decoupled weight decay,
- $\beta_1, \beta_2 \in (0, 1)$ are the first and second momentum coefficients.

Simplified "single-momentum" variants use only
\[
\begin{aligned}
  v_1 &= g_1; \quad v_t = (1-\beta) v_{t-1} + \beta g_t \\
  x_{t+1} &= x_t - \eta\, \mathrm{sign}(v_t)
\end{aligned}
\]
This asymmetric use of two EMAs is central to Lion's performance for large-scale models [2302.06675], [2602.07425].

The algorithm applies a coordinate-wise sign (i.e., $\{\pm1\}$) to the momentum estimate, producing updates with constant $\ell_2$ norm per parameter, independent of the underlying gradients [2302.06675].

## 2. Theoretical Foundations and Optimization Geometry

Lion can be interpreted as a principled solver for constrained and composite optimization problems. Formally, with decoupled weight decay $\lambda > 0$, it can be viewed as solving:
\[
\min_{x \in \mathbb{R}^d} f(x) \quad \text{subject to} \quad \|x\|_\infty \leq 1/\lambda
\]
More generally, Lion fits into the family of schemes minimizing $f(x) + \kappa^*(\lambda x)$, where $\kappa^*$ is the convex conjugate of a regularization function $\kappa$ [2310.05898].

In continuous time, Lion corresponds to flow dynamics governed by a Lyapunov function $H(x, m)$:
\[
H(x, m) = \alpha f(x) + \frac{\gamma}{\lambda} \phi^*(\lambda x) + c[\phi^*(\lambda x) + \phi(m) - \lambda m^\top x]
\]
where $\phi(x) = \|x\|_1$ and $\phi^*(y)$ is its convex conjugate (indicator for $\|y\|_\infty \leq 1$). Weight decay enforces contraction of iterates to the $\ell_\infty$-ball; monotonicity arises from the Lyapunov descent [2310.05898].

The role of the sign operator is to effect updates in the $\ell_\infty$ geometry, which exhibits increased robustness to heavy gradient noise and ensures non-vanishing update magnitudes even when gradients are sparse or bursty.

## 3. Convergence Analysis and Rates

Lion exhibits optimal convergence rates in the stochastic nonconvex setting. Under $L$-smoothness and bounded-variance stochastic gradients, and for $T$ iterations, the main results are:

- **Centralized Setting:**
  \[
  \frac{1}{T}\sum_{t=1}^T \mathbb{E}\|\nabla f(x_t)\|_1 = O(d^{1/2} T^{-1/4})
  \]
  where $d$ is the parameter count [2508.12327], [2507.12091], [2310.14616]. This matches the lower bound for $\ell_2$-norm convergence rate up to the $\ell_1/\ell_2$ norm gap ($\sqrt{d}$) [2508.12327].

- **Variance Reduced Variant (Lion-VR):**
  \[
  \frac{1}{T} \sum_{t=1}^T \mathbb{E} \| \nabla f(x_t) \|_1 = O(d^{1/2} T^{-1/3})
  \]
  via STORM-style gradient difference correction [2508.12327].

- **Distributed Setting ($n$ nodes):**
  \[
  \frac{1}{T}\sum_t \mathbb{E}\|\nabla f(x_t)\|_1 = O(d^{1/2} (nT)^{-1/4}) \quad \text{(classic Lion)}
  \]
  and $O(d^{1/2}(nT)^{-1/3})$ for Lion-VR [2508.12327].

- **Communication-Efficient 1-Bit Compression:**
  Rates degrade gently, e.g., to $O(\max\{d^{1/4} T^{-1/4}, d^{1/10} (nT)^{-1/5}\})$ when unbiased sign compression is used in both communication directions [2508.12327], [2507.12091].

Importantly, these rates hold under much weaker regularity assumptions than classical methods—only requiring local (weak) first- and second-order smoothness rather than global Lipschitz [2310.14616].

Under a generalized heavy-tailed noise model (with moment exponent $p<2$), Lion attains minimax-optimal stationarity rates and is robust to fat-tailed noise observed in large language models [2602.07425].

## 4. Practical Properties and Implementation Recommendations

Memory efficiency is a primary strength: Lion maintains only two (or even one, in simplified scenarios) momentum vectors, compared to the two-moment structure of AdamW. Its computation per step is simple (no division or square root), yielding marked reductions in memory footprint and improved throughput [2302.06675].

Key hyperparameter regimes are:
- $\eta \sim d^{-1/2} T^{-3/4}$ (standard) or $d^{-1/2} T^{-2/3}$ (variance-reduced) [2508.12327], [2507.12091]
- Momentum coefficients: default $\beta_1 = 0.9, \beta_2 = 0.99$, with tuning toward higher values (e.g., $\beta_1=0.95, \beta_2=0.98$) for improved stability in large-scale LMs [2302.06675]
- Weight decay $\lambda$ set so that $\eta \lambda$ aligns with effective regularization; typically $\lambda \sim 3$--$10\times$ larger than in AdamW [2302.06675].

Lion's larger per-step update norm (due to sign normalization) requires learning rates $3$–$10\times$ smaller than AdamW to match effective step magnitude. Warm-up and cosine decay scheduling are standard [2302.06675].

## 5. Empirical Performance and Benchmarks

Lion surpasses or matches AdamW and Adafactor across domains:
- **ImageNet Classification** (ViT and CoAtNet): up to $+2\%$ top-1 accuracy and $3$–$5\times$ reduction in compute or steps [2302.06675].
- **Vision-Language Contrastive Learning:** +2% in zero-shot ImageNet, with strong gains in transfer datasets [2302.06675].
- **Diffusion Models:** Reaches FID 4.7 with $2.3\times$ fewer steps on ImageNet-256 [2302.06675].
- **Language Modeling:** Reduces perplexity and training step count by $1.5$–$2\times$; exhibits best-in-class performance on nanoGPT pretraining under heavy-tailed gradient noise [2602.07425].
- **Production Deployment:** Used in Google Search Ads CTR model, leveraging bfloat16 momentum for further memory efficiency [2302.06675].

Large-batch performance is a distinguishing feature; Lion's accuracy gains scale with increasing batch size, exceeding AdamW especially at batch sizes $\geq 4096$ [2302.06675].

In distributed experiments using unbiased-sign compression, loss and accuracy remain competitive while greatly reducing communication bandwidth [2508.12327], [2507.12091].

## 6. Theoretical Insights and Generalizations

Lion's update rule emerges naturally from a discrete Lyapunov framework, enforcing descent on a surrogate energy function encoding both loss and regularization constraints [2310.05898]. This interpretation enables principled extensions:
- Replacement of the sign operator by a general subgradient $\nabla \kappa$ to yield Lion-$\kappa$ schemes targeting composite objectives (e.g., group norm, sparsity, entropy, Huber-type regularization) [2310.05898].
- Stability and convergence without any need for global gradient or Hessian Lipschitz constants; i.e., it remains effective even under unbounded curvature far from optima [2310.14616].

The sign-based, $\ell_\infty$-geometry update is adaptively robust to both heavy-tailed and outlier-prone noise, with non-Euclidean concentration effects yielding tighter $\ell_1$-norm stationarity [2602.07425]. Lion's performance is thus particularly pronounced when gradient statistics deviate strongly from Gaussianity.

## 7. Limitations and Deployment Considerations

Lion’s improvements are marginal or statistically insignificant in some regimes:
- Convolutional networks (e.g., ResNet-50) show little gain relative to AdamW [2302.06675].
- In extremely small batch size ($<64$), or when data quality is extremely high, the relative benefits diminish.
- Stronger augmentation or intrinsic robustness of the loss landscape may also dampen the effect.

Grid-search over $\beta\in\{0.9,0.5,0.1,0.01\}$ and $\eta$ in $[5\times10^{-3}, 1\times10^{-4}]$ is advised for empirical tuning [2507.12091].

A plausible implication is that practitioners should scale Lion’s learning rate and weight decay differently than AdamW, and verify stability particularly in new architectures or under extreme data or batch size regimes.


---

**Key References**:
- Symbolic Discovery and Benchmarking: [2302.06675]
- Theoretical Lyapunov and Composite View: [2310.05898]
- Weak Smoothness and Acceleration: [2310.14616]
- Centralized/Distributed Analysis: [2508.12327], [2507.12091]
- Heavy-Tailed Noise Theory: [2602.07425]

Source: https://www.emergentmind.com/topics/lion-evolved-sign-momentum