---
title: Adam-Style Optimizers
url: https://www.emergentmind.com/topics/adam-style-optimizers
type: topic
---

# Adam-Style Optimizers

Adam-style optimizers represent a broad class of adaptive gradient-based optimization algorithms fundamentally rooted in the Exponential Moving Average (EMA) of first- and second-order gradient information. Originating with Adam, these methods extend and generalize the exponential-moment paradigm to modulate step sizes and directions per coordinate, yielding robust convergence across diverse deep learning workloads. Recent works refine, hybridize, and analyze Adam-style schemes, introducing architectural awareness, variance rectification, hybrid preconditioning, learning-to-optimize frameworks, and even automatic discovery. This article synthesizes the current technical landscape on Adam-style optimizers, focusing on principles, mathematical formulations, theoretical properties, empirical findings, newcomer variants, and dynamical artifacts such as the epochal sawtooth effect.

## 1. Core Principles and Mathematical Foundations

Adam-style optimizers combine per-parameter step-size adaptation via EMA of squared gradients with momentum-based smoothing of the raw gradient. The canonical Adam update, excluding bias-correction and weight decay for brevity, is given by:

\[
\begin{aligned}
&m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t \\
&v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2 \\
&\hat m_t = m_t / (1 - \beta_1^{t}),\quad \hat v_t = v_t / (1 - \beta_2^{t}) \\
&\theta_{t+1} = \theta_t - \alpha \frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon}
\end{aligned}
\]

where $g_t = \nabla \ell(\theta_t)$ is the stochastic gradient, $\alpha$ is the base learning rate, $\beta_1,\beta_2 \in (0, 1)$ are the first- and second-moment decay parameters, and $\epsilon$ ensures numerical stability [2410.10056].

Numerous variants alter this template, introducing rectification (RAdam), learning-to-optimize controls (HyperAdam, MADA), isometric or architectural preconditioning (IsoAdam, CaAdam), trend estimation (AdamT), alternative moment statistics (AdaBelief, AdaMomentum), modified batch selection (AdamCB), averaging (Averaged Adam), and alternative second-moment update forms (e.g. Padam, EAdam).

## 2. Extensions, Variants, and Adaptive Mechanisms

### Rectified Adam (RAdam) and Lookahead

RAdam introduces a rectification factor correcting the effective learning rate in the early stages when the variance estimate $\hat v_t$ is unreliable. For $\rho_t > 4$, the step-normalization is scaled by $r_t$, where $\rho_t = \rho_\infty - \frac{2 t \beta_2^t}{1 - \beta_2^t}$, $\rho_\infty = \frac{2}{1 - \beta_2} - 1$, ensuring stable adaptation. For $\rho_t \le 4$, RAdam falls back to momentum SGD [2303.03540].

Lookahead optimizers periodically interpolate ("slow" weights) with "fast" weights after every $k$ steps, smoothing the trajectory and rectifying optimizer oscillations, e.g., Lookahead(RAdam) yields state-of-the-art empirical results in software engineering tasks [2303.03540].

### Architectural/Connection-aware Adaptation

CaAdam introduces proxy-based scaling factors $S$ for the learning rate, based on layer depth, connection counts, or structural parameters:

\[
\theta_t = \theta_{t-1} - \alpha S \frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon}
\]

where $S$ is computed per-parameter as a deterministic function of local connectivity or depth proxies [2410.24216]. Depth-based scaling and connection-based scaling lead to improved convergence and better minima (up to $+4.1\%$ accuracy on CIFAR-10 vs. Adam).

### Hybridization and Meta-optimization

Meta-adaptive algorithms (MADA) parameterize a convex hull of multiple base optimizers (Adam, AMSGrad, Yogi, Adan, Lion) and perform hyper-gradient descent on the optimizer's coefficients $q$ during training [2401.08893]. Learning-to-optimize frameworks such as HyperAdam dynamically ensemble updates produced by parallel Adam sub-processes with task-adaptive decay rates, controlled via an RNN [1811.08996]. Such methods consistently outperform fixed-form Adam schemes, especially under sub-optimal hyperparameter initializations or across task boundaries.

### Full-dimension and Isometric Preconditioning

IsoAdam incorporates isometric preconditioning: it normalizes updates to make them invariant to arbitrary invertible linear transformations of the input or output. The update for a weight matrix $W$ is:

\[
W \leftarrow W - \alpha L_X^{-1/2} M R_G^{-1/2} / (\sqrt{\hat V} + \varepsilon)
\]

where $L_X$ and $R_G$ are empirical covariances of the input and gradient, respectively [2307.12979]. HVAdam introduces a hidden vector $v_t$ estimating the "valley floor" of the loss landscape and preconditions adaptively in the direction of this floor, interpolating between SGD-like and Adam-like regimes via an incremental delay update mechanism [2511.20277].

### Confidence-based and Adaptive Selection

CAdam introduces a confidence gating mechanism: updates are only applied to coordinates where the sign of the momentum $m_{t,i}$ matches that of the gradient $g_{t,i}$ ($c_{t,i} = 1$ if $m_{t,i}g_{t,i}>0$, zero otherwise). This results in improved adaptation to distributional shifts and robustness against noise in online learning [2411.19647].

AdamCB fuses Adam updates with combinatorial semi-bandit batch selection: non-uniform batch selection weights are updated via exponential weights, provably accelerating expected regret convergence by a factor $K^{1/4}$ (where $K$ is batch size) compared to uniform sampling [2512.06795].

## 3. Convergence, Scaling Laws, and Theoretical Guarantees

Adam-style optimizers (including their advanced variants) generally achieve $O(\sqrt{T})$ regret in the online convex setting, and $O(\ln T / \sqrt{T})$ rates in smooth nonconvex settings under mild assumptions [2106.11514, 2401.08893, 2511.20277]. Several advances further tighten or generalize these bounds:

- LaProp separates parameter space momentum from adaptivity, eliminating the classical $(1-\beta_1)/\sqrt{1-\beta_2}$ coupling and providing regret bounds that hold for all $\beta_1,\beta_2$ [2002.04839].
- MADA proves that interpolating among Adam, AMSGrad, Yogi, and Adan can reduce error bounds up to constants, providing an advantage over fixed-rule schemes [2401.08893].
- Higher-order time-stepping schemes (e.g., IMEX-Trapezoidal Adam) convert Adam's dynamics into an ODE and apply second-order implicit-explicit discretizations, yielding lower errors and greater stability in stiff regimes [2403.13704].

The scaling law between batch size $B$ and optimal learning rate $\eta^*$ is non-monotonic for sign-gradient-based Adam variants: $\eta^*(B)$ increases as $\sqrt{B}$ up to $B_{\rm noise}$ (the noise-critical batch size), reaching a maximum, then decreases. This "surge" behavior provides rigorous guidance for batch size and learning rate scheduling distinct from the linear scaling in SGD [2405.14578].

## 4. Empirical Behaviors and Modern Performance Benchmarks

Adam-style optimizers dominate both standard deep learning and emerging scientific computing tasks. Extensive empirical studies demonstrate:

- Adam and its descendants (AdaBelief, Padam, EAdam) are among the fastest for initial objective value reduction but may overfit quickly and exhibit oscillatory validation metrics in certain settings (e.g., EMNIST) [2110.01598].
- RAdam and Lookahead(RAdam) consistently outperform Adam across code-related tasks, achieving median relative improvements exceeding $15\%$ in documentation generation and $10\%$ in method name generation [2303.03540].
- CaAdam, with multiplicative/depth-based scaling, accelerates convergence and achieves up to $+5.97\%$ improvement in CIFAR-100 accuracy relative to Adam, and reduces regression RMSE and training epochs to convergence [2410.24216].
- Averaged Adam (Polyak–Ruppert or exponential moving average of iterates) consistently outperforms plain Adam, reducing test errors $10-30\%$ and boosting generalization in scientific ML, PINNs, deep BSDEs, and image classification [2501.06081].
- Hybrid evolutionary designs (combining sign and adaptive moment terms, e.g., $0.73\, \mathrm{sign}(g) + 3.63\, m/(\sqrt{v}+\epsilon)$) outperform standard Adam, achieving $+7.7\%$ test accuracy gain on CIFAR-10 [2512.11853].

## 5. Dynamical Artifacts: The Epochal Sawtooth Effect

A prominent phenomenon inherent to Adam-style optimizers is the "Epochal Sawtooth Effect" (ESE), identified as a sharp batchwise loss drop at the start of each epoch followed by a gradual intra-epoch increase, yielding a sawtooth-shaped loss trajectory [2410.10056]. The ESE arises from:

- Quadratic accumulation and intra-epoch growth in the second moment $v_t$ leading to shrinking effective step-sizes.
- An initial high momentum $m_t$ spike due to minibatch overlap and data reshuffling, which decays within the epoch.
- The interaction between these momenta causes a sharp initial drop in loss (when the numerator is large and denominator is small), transitioning to a shallower climb as $v_t$ grows.
- Smaller $\beta_2$ amplifies ESE amplitude, larger batch sizes and absence of shuffling/epoch structure suppress the effect.
- ESE is not a mere overfitting artifact but rather an optimizer-induced loss pattern, especially manifest in high-capacity models.

To mitigate ESE, increase $\beta_2$ close to $1$, reduce $\beta_1$ if necessary, use larger batch sizes, or avoid epoch-based data shuffling; conversely, ESE can be exploited for optimization heuristics such as cyclical LR or SWA [2410.10056].

## 6. Practical Guidelines, Limitations, and Design Trade-offs

Practical recommendations, distilled from rigorous benchmarks and large-scale deployment studies, include:

- Defaults: $\alpha=10^{-3}, \beta_1=0.9, \beta_2=0.999, \epsilon=10^{-8}$ work reliably, but advanced architectures or learning-to-optimize schemes may tune these dynamically.
- For code-related deep learning, Lookahead(RAdam) is the recommended default; for image or scientific ML tasks, Averaged Adam or hybrid sign-moment optimizers extend performance margins.
- Confidence-aware (CAdam) or full-dimension (HVAdam) optimizers excel in online learning with nonstationary or noisy data, as evidenced by consistent AUC and GMV improvements in large-scale industrial recommender deployments [2411.19647].
- Architectural scaling (CaAdam), dynamic selection (MADA), and meta-learned adaptation (HyperAdam) indicate that hand-crafted fixed optimizers are being superseded by adaptive, context-aware, and even automatically evolved schemes.
- Limitations: Many new designs require extra memory or compute per iteration (e.g., isometric covariance estimation, ring buffers, meta-optimizer state) or meta-training effort. Such costs are generally offset by improved early convergence, sharper minima, or wall-clock reductions in required training epochs.

## 7. Outlook and Theoretical Trends

Adam-style methods are now a platform on which a wide array of theoretical, algorithmic, and empirical innovations are constructed. Trends include:

- Increasing emphasis on meta-adaptation and hybridization (MADA, evolved optimizers, HyperAdam) to supplant "one-size-fits-all" rule-based schemes.
- The emergence of invariant or structure-aligned preconditioning (IsoAdam, HVAdam) for deep and highly parameterized networks.
- Novel analyses of dynamic loss behaviors (ESE), and their mitigation or exploitation via optimizer configuration.
- Growing impact of adaptive batch selection and non-i.i.d. data handling (AdamCB, CAdam) for variance reduction and rapid adaptation.
- Continued theme that optimizer design space is vast, with evolutionary and learned-search approaches already producing variants that outperform hand-crafted algorithms in challenging settings [2512.11853, 2401.08893].

It is evident that Adam-style optimization continues to provide a fertile foundation for both technical advancement and practical deployment in modern machine learning. The field is characterized by rapid iteration, synergy between empirical discovery and mathematical insight, and by cross-pollination between theory, architecture, and optimizer design.

Source: https://www.emergentmind.com/topics/adam-style-optimizers