ADAM: Adaptive Moment Estimation

• ADAM is an adaptive stochastic gradient optimizer that maintains exponential moving averages of both gradients and squared gradients for per-coordinate updates.
• It employs bias-correction in its moment estimates to provide robust, adaptive learning rates, reducing the need for extensive hyperparameter tuning.
• Recent studies extend ADAM with analyses of convergence, geometry, and generalization, inspiring variants that address scaling and stability challenges.

ADAM, or Adaptive Moment Estimation, is a stochastic optimization method that maintains exponential moving averages of gradients and squared gradients in order to produce coordinate-wise adaptive updates. It is widely used in neural network training because it combines adaptive learning rates with relatively low tuning burden, but it has also been the subject of sustained debate over convergence, implicit geometry, and generalization. The name “ADAM” also appears in later literature as an acronym for unrelated systems in open-world annotation and quantum optimization, so the term is context-dependent (Hwang, 2024, Jin et al., 2024).

1. Canonical update rule

In its standard form, Adam considers a stochastic gradient $g_t$ at iteration $t$, maintains a first-moment estimate $m_t$ and a second-moment estimate $v_t$, applies bias correction, and updates parameters by rescaling the first moment with the inverse square root of the second moment. A common presentation is

$$m_t=\beta_1 m_{t-1}+(1-\beta_1)g_t, \qquad v_t=\beta_2 v_{t-1}+(1-\beta_2)g_t^2,$$

$$\hat m_t=\frac{m_t}{1-\beta_1^t}, \qquad \hat v_t=\frac{v_t}{1-\beta_2^t},$$

$$\theta_t=\theta_{t-1}-\alpha\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon},$$

where all operations are element-wise, $\alpha$ is the global learning rate, $\beta_1,\beta_2\in[0,1)$ are decay rates, and $\epsilon$ is a numerical stabilizer (Liu et al., 2020).

Several papers in the literature restate the same mechanism in slightly different notation. The underlying idea is constant: the first moment acts as momentum, the second moment tracks recent gradient magnitudes, and the ratio $t$0 yields per-coordinate adaptive step sizes. Typical default hyperparameters reported in the literature are $t$1, $t$2, $t$3, and $t$4 (Liu et al., 2020).

This update has also been reinterpreted geometrically. One analysis identifies Adam’s second-moment accumulator with a diagonal empirical Fisher approximation and argues that Adam can be viewed as an approximate natural-gradient method with square-root Fisher scaling rather than full Fisher inversion (Hwang, 2024). That interpretation does not replace the standard algorithmic definition, but it situates Adam within a broader class of curvature-aware and information-geometric methods.

2. Convergence theory and dynamical analyses

The theoretical status of Adam has evolved through several distinct lines of analysis. A continuous-time treatment shows that, in a small-stepsize regime with $t$5, the piecewise-linear interpolation of Adam converges to a non-autonomous ordinary differential equation in the joint variables $t$6. Under the assumptions used there, the ODE has a unique global solution, its trajectories remain bounded, and its long-run behavior is tied to the critical points of the objective function (Barakat et al., 2018).

A later framework establishes asymptotic convergence in the last-iterate sense under assumptions commonly used for SGD rather than the more restrictive bounded-gradient conditions that dominated earlier Adam analyses. In that formulation, the assumptions are $t$7-smoothness, lower boundedness, and the ABC inequality

$t$8

Under a decaying learning-rate schedule and a schedule for $t$9 approaching $m_t$0, Adam attains almost-sure convergence, $m_t$1 convergence, and non-asymptotic sample complexity bounds similar to those of SGD (Jin et al., 2024).

A separate unifying framework, UAdam, treats Adam, NAdam, AMSGrad, AdaBound, AdaFom, and Adan as special cases obtained by choosing different second-moment constructions and interpolation parameters. In the non-convex stochastic setting, that framework proves convergence to a neighborhood of stationary points with rate $m_t$2, with neighborhood size decreasing as $m_t$3 approaches $m_t$4, and it emphasizes that the analysis imposes no restriction on the second-order momentum factor beyond $m_t$5 (Jiang et al., 2023).

More recently, a problem-dependent phase-transition picture has been proposed. In that analysis, Adam converges when $m_t$6 is large and $m_t$7, while for smaller $m_t$8 there exists a region of $m_t$9 combinations in which the iterates can diverge to infinity. The critical boundary is reported as problem-dependent and, in particular, dependent on batch size (Zhang et al., 2 Mar 2026). Taken together, these results do not supply a single universal theorem for all practical regimes; rather, they show that Adam’s convergence behavior depends strongly on the analytical model, the schedule design, and the choice of hyperparameters.

3. Geometry, generalization, and relation to SGD

One of the central criticisms of Adam is not merely that convergence can be delicate, but that the optimizer may generalize worse than SGD on deep networks. A detailed account attributes this to two mechanisms. The first is the “direction-missing problem”: Adam’s per-coordinate scaling alters gradient directions, so its updates can leave the span of historical gradients and move toward sharply curved minima. The second is the “ill-conditioning problem”: in scale-invariant networks, the effect of an update on the network function depends on $v_t$0, and as $v_t$1 grows, the effective learning rate $v_t$2 decreases, shrinking the gradient noise that can help escape sharp minima (Zhang et al., 2017).

Normalized Direction-preserving Adam (ND-Adam) addresses those two issues by grouping each hidden unit’s incoming weights into a vector $v_t$3, projecting raw gradients onto the tangent space of the unit sphere,

$v_t$4

using vector-level moment estimates, and renormalizing after each step so that $v_t$5. In the reported Wide ResNet experiments on CIFAR-10 and CIFAR-100, Adam yields test error $v_t$6 and $v_t$7, whereas ND-Adam yields $v_t$8 and $v_t$9; with BN-Softmax and no per-channel batch-normalization scales, Adam yields $$m_t=\beta_1 m_{t-1}+(1-\beta_1)g_t, \qquad v_t=\beta_2 v_{t-1}+(1-\beta_2)g_t^2,$$0 and $$m_t=\beta_1 m_{t-1}+(1-\beta_1)g_t, \qquad v_t=\beta_2 v_{t-1}+(1-\beta_2)g_t^2,$$1, whereas ND-Adam yields $$m_t=\beta_1 m_{t-1}+(1-\beta_1)g_t, \qquad v_t=\beta_2 v_{t-1}+(1-\beta_2)g_t^2,$$2 and $$m_t=\beta_1 m_{t-1}+(1-\beta_1)g_t, \qquad v_t=\beta_2 v_{t-1}+(1-\beta_2)g_t^2,$$3 (Zhang et al., 2017).

FAdam advances a different geometric claim: Adam should be understood through Riemannian and information geometry as a diagonal empirical Fisher method. On that basis, it identifies several flaws in the original algorithmic form, including invalid EMA of Fisher quantities across changing tangent spaces, over-bias-correction of momentum, static $$m_t=\beta_1 m_{t-1}+(1-\beta_1)g_t, \qquad v_t=\beta_2 v_{t-1}+(1-\beta_2)g_t^2,$$4, lack of built-in clipping, and weight decay that is not Fisher-aware. The proposed corrections include momentum on invariant gradients, adjusted bias correction for the Fisher estimate, adaptive $$m_t=\beta_1 m_{t-1}+(1-\beta_1)g_t, \qquad v_t=\beta_2 v_{t-1}+(1-\beta_2)g_t^2,$$5, in-optimizer clipping, and geometric weight decay. Reported experiments cover a 1B-parameter LLM on C4, a 600M Conformer on LibriSpeech, and a 100M ViT-VQGAN on ImageNet, with improvements in validation perplexity, word error rate, and FID respectively (Hwang, 2024).

The recurring theme across these works is that Adam’s coordinate-wise adaptivity is not a neutral implementation detail. It changes the update geometry, and that geometry affects both optimization dynamics and the flatness or sharpness of the solutions reached.

4. Variants that modify the update mechanism

A large family of Adam-derived optimizers changes the form of the moments, the normalization, or the averaging strategy while keeping the basic adaptive-moment template.

Adam$$m_t=\beta_1 m_{t-1}+(1-\beta_1)g_t, \qquad v_t=\beta_2 v_{t-1}+(1-\beta_2)g_t^2,$$6 removes the second-moment accumulator entirely, evaluates its first-moment estimate at an extrapolated point, and uses a stepsize normalized by the root of the norm of the first moment rather than by $$m_t=\beta_1 m_{t-1}+(1-\beta_1)g_t, \qquad v_t=\beta_2 v_{t-1}+(1-\beta_2)g_t^2,$$7. Its theory emphasizes “adaptive variance reduction,” meaning that the variance of the stochastic-gradient estimator decreases as the algorithm converges. The reported empirical studies span image classification, language modeling, and automatic speech recognition, where Adam$$m_t=\beta_1 m_{t-1}+(1-\beta_1)g_t, \qquad v_t=\beta_2 v_{t-1}+(1-\beta_2)g_t^2,$$8 consistently outperforms vanilla Adam and Adagrad and matches or exceeds well-tuned SGD and momentum SGD (Liu et al., 2020).

EAdam changes only the placement of $$m_t=\beta_1 m_{t-1}+(1-\beta_1)g_t, \qquad v_t=\beta_2 v_{t-1}+(1-\beta_2)g_t^2,$$9. Instead of updating with $$\hat m_t=\frac{m_t}{1-\beta_1^t}, \qquad \hat v_t=\frac{v_t}{1-\beta_2^t},$$0, it adds $$\hat m_t=\frac{m_t}{1-\beta_1^t}, \qquad \hat v_t=\frac{v_t}{1-\beta_2^t},$$1 inside the accumulated second moment and updates with $$\hat m_t=\frac{m_t}{1-\beta_1^t}, \qquad \hat v_t=\frac{v_t}{1-\beta_2^t},$$2. The paper stresses that this modification introduces no extra hyperparameters or computational cost. Reported gains include CIFAR-10 accuracy improvements from $$\hat m_t=\frac{m_t}{1-\beta_1^t}, \qquad \hat v_t=\frac{v_t}{1-\beta_2^t},$$3 to $$\hat m_t=\frac{m_t}{1-\beta_1^t}, \qquad \hat v_t=\frac{v_t}{1-\beta_2^t},$$4 on VGG11 and from $$\hat m_t=\frac{m_t}{1-\beta_1^t}, \qquad \hat v_t=\frac{v_t}{1-\beta_2^t},$$5 to $$\hat m_t=\frac{m_t}{1-\beta_1^t}, \qquad \hat v_t=\frac{v_t}{1-\beta_2^t},$$6 on ResNet18, a CIFAR-100 VGG11 gain from $$\hat m_t=\frac{m_t}{1-\beta_1^t}, \qquad \hat v_t=\frac{v_t}{1-\beta_2^t},$$7 to $$\hat m_t=\frac{m_t}{1-\beta_1^t}, \qquad \hat v_t=\frac{v_t}{1-\beta_2^t},$$8, Penn Treebank perplexity improvement from $$\hat m_t=\frac{m_t}{1-\beta_1^t}, \qquad \hat v_t=\frac{v_t}{1-\beta_2^t},$$9 to $$\theta_t=\theta_{t-1}-\alpha\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon},$$0 for a 3-layer LSTM, and a PASCAL VOC detection mAP improvement from $$\theta_t=\theta_{t-1}-\alpha\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon},$$1 to $$\theta_t=\theta_{t-1}-\alpha\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon},$$2 (Yuan et al., 2020).

PADAM, or parallel averaged Adam, keeps a single underlying Adam trajectory and maintains multiple averaged trajectories in parallel, each with a different EMA schedule. At evaluation intervals, it selects the trajectory with the smallest validation loss or test error. Because all channels share the same underlying gradients, the method requires no more gradient evaluations than usual Adam. Across 13 scientific-machine-learning problems, including PINNs, deep Galerkin, Deep BSDE, Deep Kolmogorov approximations, and optimal control or stopping tasks, PADAM with $$\theta_t=\theta_{t-1}-\alpha\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon},$$3 or $$\theta_t=\theta_{t-1}-\alpha\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon},$$4 channels often attains the smallest test error and in nearly all cases dynamic selection outperforms any fixed averaging schedule (Jentzen et al., 28 May 2025).

An ODE-based re-derivation goes in another direction. By treating Adam as a first-order implicit-explicit Euler discretization of an underlying ODE, one obtains higher-order IMEX-Adam schemes such as IMEX Trapezoidal-Adam. The two-stage trapezoidal method uses two gradient evaluations per iteration, retains $$\theta_t=\theta_{t-1}-\alpha\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon},$$5 memory, and is reported to improve stability and loss reduction on several regression and classification tasks, while remaining comparable to Adam on others such as CIFAR-10 with VGG-16 (Bhattacharjee et al., 2024).

HyperAdam replaces fixed decay hyperparameters with a learned recurrent controller. It generates $$\theta_t=\theta_{t-1}-\alpha\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon},$$6 Adam-style candidate updates with varying decay rates and combines them through learned weights produced by a StateCell, AdamCell, and WeightCell. In the reported experiments, HyperAdam is tested on MLPs, CNNs, and LSTMs and yields lower training loss than Adam or earlier learned optimizers on many tasks, while the ablation studies attribute part of its generalization ability to the recurrent state, preprocessing, “Combination with Convex functions,” and “Random Scaling” (Wang et al., 2018).

5. Adaptive sampling and batch selection

Standard Adam ordinarily assumes uniformly sampled mini-batches. One critique of that regime is that data samples influence model updates to varying degrees, so treating them equally may waste iterations on easy examples and increase the variance of the stochastic-gradient estimate. A prior bandit-based variant, AdamBS, models each training example as an arm, updates sampling weights through an EXP3-style rule, and samples $$\theta_t=\theta_{t-1}-\alpha\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon},$$7 examples with replacement. However, the later analysis reports three shortcomings: a critical algebraic error in the regret analysis, no variance reduction from increasing $$\theta_t=\theta_{t-1}-\alpha\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon},$$8 because of sampling with replacement, and poor stability with inconsistent gains in CNN experiments (Kim et al., 7 Dec 2025).

Adam with Combinatorial Bandit Sampling (AdamCB) replaces that single-arm bandit formulation with a combinatorial semi-bandit. At each iteration it selects a batch $$\theta_t=\theta_{t-1}-\alpha\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon},$$9 of $\alpha$0 distinct arms without replacement, forms an unbiased gradient

$\alpha$1

updates the Adam moments in the usual way, and then updates sample weights using the per-arm losses

$\alpha$2

The algorithm uses DepRound to realize the without-replacement selection with prescribed marginals and exploits semi-bandit feedback because it observes one loss per selected arm (Kim et al., 7 Dec 2025).

Under bounded gradients and bounded iterates, the reported regret analysis gives AdamCB a sublinear regret bound and specifically claims that it is the first Adam variant with adaptive batch selection whose regret improves with batch size $\alpha$3, decaying as $\alpha$4. The same paper contrasts this with a corrected uniform-sampling AdamX bound and with AdamBS, whose regret does not improve with $\alpha$5 (Kim et al., 7 Dec 2025).

The empirical evaluation spans MNIST, Fashion-MNIST, and CIFAR-10, with models including MLP, CNN, VGG-style small net, logistic regression, ResNet-18, ConvNeXt-Base, and ConvNeXt-Large. Across all tasks and models, AdamCB is reported to reduce training loss faster per epoch, yield lower final training loss, and achieve equal or better test loss and accuracy than Adam, AdamX, corrected AdamBS, and AMSGrad. On CIFAR-10 with the CNN model and $\alpha$6, Adam reaches training loss $\alpha$7 after about $\alpha$8 epochs, while AdamCB reaches the same threshold by about $\alpha$9 epochs and achieves approximately $\beta_1,\beta_2\in[0,1)$0–$\beta_1,\beta_2\in[0,1)$1 higher final test accuracy. Averaged across datasets and models, the reported convergence speedup is a $\beta_1,\beta_2\in[0,1)$2–$\beta_1,\beta_2\in[0,1)$3 reduction in epochs to a fixed training-loss threshold, with $\beta_1,\beta_2\in[0,1)$4–$\beta_1,\beta_2\in[0,1)$5 absolute gains in final test accuracy (Kim et al., 7 Dec 2025).

6. Other systems named ADAM

The identifier ADAM is also used outside adaptive optimization. In computer vision, “ADAM: Autonomous Discovery and Annotation Model using LLMs for Context-Aware Annotations” is a training-free, non-parametric pipeline for open-world object labeling. It combines region proposals, CLIP visual embeddings, context-aware prompting of an LLM, an Embedding-Label Repository, nearest-neighbor retrieval with weighted voting, optional cross-modal reranking, and a self-refinement loop based on visual cohesion and $\beta_1,\beta_2\in[0,1)$6-nearest-neighbor relabeling. On COCO, the reported overall Top-1 accuracy improves from $\beta_1,\beta_2\in[0,1)$7 to $\beta_1,\beta_2\in[0,1)$8, a self-refinement gain of $\beta_1,\beta_2\in[0,1)$9; on PASCAL VOC region proposals, macro-precision peaks at $\epsilon$0 for $\epsilon$1. The paper also notes a failure mode in single-object scenes, exemplified by “zebra” without contextual cues (Rouhi et al., 10 Jun 2025).

In quantum optimization, “Adam assisted Fully informed Particle Swarm Optimzation ( Adam-FIPSO )” incorporates Adam-style moment estimation into Fully Informed Particle Swarm Optimization for QAOA parameter prediction. The FIPSO social term is treated as a pseudo-gradient, Adam moments are maintained per particle and dimension, and the resulting corrected term is combined with the PSO inertia weight. The reported experiments on Erdős–Rényi and Watts–Strogatz graphs show improvements over random initialization across QAOA depths $\epsilon$2. Two examples reported in the paper are an approximation-ratio increase from about $\epsilon$3 to about $\epsilon$4, or $\epsilon$5, on an Erdős–Rényi instance at $\epsilon$6, and from about $\epsilon$7 to about $\epsilon$8, or $\epsilon$9, on a Watts–Strogatz instance at $t$00 (Bhat et al., 7 Jun 2025).

These later usages do not alter the technical meaning of Adam as Adaptive Moment Estimation in optimization. They show, however, that by 2025 the label “ADAM” had become polysemous across machine learning, vision, and quantum-computing literatures.