AdamW - Adaptive Weight Decay Optimizer

• AdamW is an adaptive optimization algorithm that combines Adam's exponential averaging with decoupled weight decay, improving the regularization for machine-learning models.
• AdamW is widely used in Transformer-based language models and other deep learning tasks, thanks to its efficient modeling of coordinate-wise scaling, momentum, and decoupled regularization.
• It offers improved stability and performance in settings with heterogeneous gradient scales, making it a preferred choice for large-scale neural network training.

AdamW is an adaptive first-order optimization algorithm that combines Adam’s exponential first- and second-moment estimates with decoupled weight decay. Its defining update applies coordinate-wise adaptive normalization to the loss gradient while shrinking parameters independently by a multiplicative factor. In the standard formulation, with stochastic gradient $g_t$, moments $m_t$ and $v_t$, learning rate $\eta_t$, numerical stabilizer $\epsilon$, and weight-decay coefficient $\lambda$, the update is

$m_t=\beta_1m_{t-1}+(1-\beta_1)g_t,$

$v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2,$

$$\widehat m_t=\frac{m_t}{1-\beta_1^t}, \qquad \widehat v_t=\frac{v_t}{1-\beta_2^t},$$

$$\theta_{t+1} = (1-\eta_t\lambda)\theta_t - \eta_t\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon},$$

where powers, square roots, and divisions are elementwise. The separation between the adaptive gradient step and parameter shrinkage distinguishes AdamW from Adam with a coupled $m_t$0 penalty.

1. Definition and historical development

AdamW was introduced to address the difference between conventional $m_t$1 regularization and direct weight decay under adaptive preconditioning. For ordinary gradient descent, augmenting the objective with $m_t$2 produces

$m_t$3

Thus, in ordinary gradient descent, adding an $m_t$4 penalty and applying multiplicative weight decay are closely related. In Adam, however, the regularized gradient is incorporated into the moving averages and subsequently divided by a coordinate-wise adaptive denominator. The effective regularization strength therefore depends on historical gradient magnitudes.

AdamW instead computes $m_t$5 and $m_t$6 from the unregularized loss gradient and applies decay separately:

$m_t$7

$m_t$8

The decoupled formulation is associated with a more direct interpretation of $m_t$9 as parameter shrinkage and with a reduced dependence between learning-rate and weight-decay tuning. The distinction is analyzed through proximal methods and scale-freeness in "Understanding AdamW through Proximal Methods and Scale-Freeness" (Zhuang et al., 2022).

AdamW is widely used for Transformer-based LLMs and other deep-learning systems. Its practical role reflects the combination of adaptive coordinate-wise scaling, momentum, decoupled regularization, mature implementations, fused kernels, mixed-precision support, and distributed-training infrastructure (Ranganath, 9 May 2026).

2. Algorithmic formulation

The first moment $v_t$0 is an exponentially weighted average of gradients and provides momentum-like smoothing. The second moment $v_t$1 is an exponentially weighted average of coordinate-wise squared gradients and determines adaptive step magnitudes. Bias correction compensates for initialization at zero:

$v_t$2

The adaptive direction is

$v_t$3

Coordinates with historically large squared gradients receive smaller adaptive steps, whereas coordinates with smaller second-moment estimates receive relatively larger steps. The numerical stabilizer $v_t$4 prevents division by zero and can materially affect behavior when second-moment estimates are small.

The decoupled decay term is

$v_t$5

Equivalently, the complete update can be viewed as adaptive optimization combined with direct multiplicative shrinkage:

$v_t$6

Different papers use slightly different conventions for the placement of $v_t$7, including $v_t$8 and $v_t$9. These expressions have different smoothness properties near $\eta_t$0. The local-stability analysis in "Extending AdamW by Leveraging Its Second Moment and Magnitude" (Zhang et al., 2021) uses the denominator $\eta_t$1 in its principal AdamW formulation.

AdamW’s adaptive term can be decomposed coordinatewise into a sign vector and a magnitude vector:

$\eta_t$2

This decomposition motivates extensions that preserve the sign structure while modifying the adaptive magnitude. In the Aida construction, AdamW is recovered by the parameter choice $\eta_t$3 (Zhang et al., 2021).

3. Decoupled decay, proximal methods, and scale-freeness

AdamW can be interpreted as a first-order approximation to a proximal-gradient method for the squared $\eta_t$4 regularizer. For a preconditioning matrix $\eta_t$5 and update direction $\eta_t$6, the proximal-gradient step is

$\eta_t$7

For scalar $\eta_t$8, this becomes

$\eta_t$9

Since

$\epsilon$0

AdamW approximates the proximal contraction to first order, with approximation error of order $\epsilon$1. The exact proximal counterpart, called AdamProx, uses the factor $\epsilon$2 rather than $\epsilon$3 (Zhuang et al., 2022).

AdamW also possesses an approximate scale-freeness property. If gradients in coordinate $\epsilon$4 are rescaled by a positive constant $\epsilon$5, then, ignoring $\epsilon$6,

$\epsilon$7

and hence

$\epsilon$8

The decay term is unaffected by gradient rescaling because it depends only on the parameters. In practical implementations, $\epsilon$9, so exact invariance is lost, but the deviation can be negligible when $\lambda$0 is small relative to $\lambda$1.

Adam with coupled $\lambda$2 regularization does not retain this property. If

$\lambda$3

then rescaling the task gradient does not rescale the additive regularization term. The cancellation between first and second moments consequently fails. This provides one explanation for the stronger performance of AdamW in settings with heterogeneous gradient scales, especially deep networks without Batch Normalization. The experiments in (Zhuang et al., 2022) report larger differences between AdamW and Adam-$\lambda$4 in deep networks without Batch Normalization, while the methods are often comparable when Batch Normalization is present.

4. Stability, implicit bias, and convergence theory

Local stability analysis treats AdamW as an autonomous dynamical system whose state contains the parameters and moment estimates. For a twice-differentiable objective with a strict local optimum at the origin, the Jacobian has eigenvalues determined by $\lambda$5, $\lambda$6, the Hessian spectrum, the learning rate, $\lambda$7, and the decay factor. The stability condition given in (Zhang et al., 2021) is

$\lambda$8

where $\lambda$9 are Hessian eigenvalues and $m_t=\beta_1m_{t-1}+(1-\beta_1)g_t,$0 is the decay parameter in the paper’s notation. The condition shows that larger curvature and larger learning rates make local stability more difficult, while smaller $m_t=\beta_1m_{t-1}+(1-\beta_1)g_t,$1 can reduce the admissible learning rate.

The same work proposes Aida by replacing the squared-gradient second moment with a generalized $m_t=\beta_1m_{t-1}+(1-\beta_1)g_t,$2-power moment and the usual first-power adaptive magnitude with a $m_t=\beta_1m_{t-1}+(1-\beta_1)g_t,$3-power magnitude. For $m_t=\beta_1m_{t-1}+(1-\beta_1)g_t,$4 and $m_t=\beta_1m_{t-1}+(1-\beta_1)g_t,$5, its local Jacobian has eigenvalues $m_t=\beta_1m_{t-1}+(1-\beta_1)g_t,$6, $m_t=\beta_1m_{t-1}+(1-\beta_1)g_t,$7, and $m_t=\beta_1m_{t-1}+(1-\beta_1)g_t,$8. The stated asymptotic-stability theorem therefore requires nonzero weight decay,

$m_t=\beta_1m_{t-1}+(1-\beta_1)g_t,$9

At $v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2,$0, an eigenvalue equals one and the linearization test is inconclusive; this does not establish instability.

A separate characterization identifies an implicit constrained-optimization interpretation. In deterministic full-batch settings, if AdamW converges under a non-increasing learning-rate schedule with divergent cumulative learning rate and $v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2,$1, the limit is a KKT point of

$v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2,$2

For convex $v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2,$3, the limit is a constrained minimizer. The interpretation follows from viewing Adam’s normalized update as a smoothed version of SignGD and recognizing SignGD as normalized steepest descent under the $v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2,$4 norm. Decoupled weight decay then corresponds to a Frank–Wolfe-type update over an $v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2,$5 ball (Xie et al., 2024).

This result is conditional. It does not prove convergence for every nonconvex AdamW trajectory, does not identify AdamW with explicit projection at every iteration, and does not establish that AdamW minimizes $v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2,$6.

For stochastic nonconvex optimization, a 2025 analysis establishes an average expected $v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2,$7-gradient bound of order

$v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2,$8

under smoothness, unbiased stochastic gradients, coordinate-wise bounded conditional variance, particular moment-parameter choices, and additional restrictions on learning rate, weight decay, initialization, and $v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2,$9 (Li et al., 17 May 2025). The theorem omits bias correction and does not cover conventional practical settings such as $$\widehat m_t=\frac{m_t}{1-\beta_1^t}, \qquad \widehat v_t=\frac{v_t}{1-\beta_2^t},$$0, $$\widehat m_t=\frac{m_t}{1-\beta_1^t}, \qquad \widehat v_t=\frac{v_t}{1-\beta_2^t},$$1 in its main asymptotic tuning.

Under heavy-tailed noise with finite conditional $$\widehat m_t=\frac{m_t}{1-\beta_1^t}, \qquad \widehat v_t=\frac{v_t}{1-\beta_2^t},$$2-th moments for $$\widehat m_t=\frac{m_t}{1-\beta_1^t}, \qquad \widehat v_t=\frac{v_t}{1-\beta_2^t},$$3, a complete ordinary $$\widehat m_t=\frac{m_t}{1-\beta_1^t}, \qquad \widehat v_t=\frac{v_t}{1-\beta_2^t},$$4 stationarity theorem remains open. A positive result is obtained only for a weighted stationarity quantity in an equal-memory regime $$\widehat m_t=\frac{m_t}{1-\beta_1^t}, \qquad \widehat v_t=\frac{v_t}{1-\beta_2^t},$$5. The analysis identifies a potential obstruction: a rare large gradient can contribute quadratically to $$\widehat m_t=\frac{m_t}{1-\beta_1^t}, \qquad \widehat v_t=\frac{v_t}{1-\beta_2^t},$$6, remain in the second-moment accumulator, and suppress subsequent updates even when the true gradient remains large (Yu et al., 22 Jun 2026).

5. Empirical behavior and extensions

Experiments in (Zhang et al., 2021) compare AdamW with Aida on ten twice-differentiable toy problems, Transformer and Swin-Transformer tasks, and a local quadratic problem. The results are mixed: AdamW performs better on some toy problems, while $$\widehat m_t=\frac{m_t}{1-\beta_1^t}, \qquad \widehat v_t=\frac{v_t}{1-\beta_2^t},$$7 and $$\widehat m_t=\frac{m_t}{1-\beta_1^t}, \qquad \widehat v_t=\frac{v_t}{1-\beta_2^t},$$8 perform better on others. On WMT16 multimodal translation and CIFAR-10 with a Swin Transformer, the $$\widehat m_t=\frac{m_t}{1-\beta_1^t}, \qquad \widehat v_t=\frac{v_t}{1-\beta_2^t},$$9 configuration improves reported validation accuracy by approximately $$\theta_{t+1} = (1-\eta_t\lambda)\theta_t - \eta_t\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon},$$0 over the $$\theta_{t+1} = (1-\eta_t\lambda)\theta_t - \eta_t\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon},$$1 AdamW configuration under the stated experimental settings. These findings do not establish uniform superiority of the extension.

Several optimizer modifications retain AdamW’s decay mechanism:

• Weight prediction: predicted future weights are used for the forward and backward passes, while the actual AdamW update is applied to the current parameters. The method requires a temporary parameter copy or reversible perturbation and was evaluated on CIFAR-10 and Penn TreeBank (Guan, 2023).
• Adapprox: the dense second moment is replaced with a randomized low-rank approximation using Streamlined Randomized Subspace Iteration and adaptive rank selection. In GPT-2 experiments, it reports memory savings of $$\theta_{t+1} = (1-\eta_t\lambda)\theta_t - \eta_t\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon},$$2–$$\theta_{t+1} = (1-\eta_t\lambda)\theta_t - \eta_t\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon},$$3 for the 117M model and $$\theta_{t+1} = (1-\eta_t\lambda)\theta_t - \eta_t\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon},$$4–$$\theta_{t+1} = (1-\eta_t\lambda)\theta_t - \eta_t\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon},$$5 for the 345M model when the first moment is retained (Zhao et al., 2024).
• Federated AdamW: FedAdamW aggregates second-moment information across clients, initializes local second moments from the aggregated estimate, and adds a global-update correction to reduce client drift. Blockwise second-moment aggregation communicates $$\theta_{t+1} = (1-\eta_t\lambda)\theta_t - \eta_t\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon},$$6 values rather than a full parameter-sized vector (Liu et al., 31 Oct 2025).
• SCSAdamW: the conventional first moment is replaced by a stochastic conjugate-subgradient direction, while AdamW-like second-moment normalization and decoupled decay are retained. The reported experiments use small LSTM models and do not provide a complete convergence theory for the full hybrid method (Zhang et al., 1 Jul 2025).
• HomeAdamW: the square-root-free AdamW variant uses adaptive updates when the second moment is above a threshold and momentum-SGD updates otherwise. Its theoretical purpose is to avoid amplification caused by very small denominators (Huang et al., 3 Mar 2026).

AdamW is also used as a baseline for memory-efficient, matrix-based, and curvature-aware optimizers. Adafactor reduces second-moment storage through row and column factorization; low-rank methods such as Adapprox and GaLore exploit matrix or gradient structure; and matrix-based methods such as Muon alter update geometry rather than using independent coordinate-wise scaling (Ranganath, 9 May 2026).

6. Applications, systems, and broader significance

AdamW remains a principal optimizer for Transformer and large-language-model training because it combines adaptive coordinate-wise scaling with decoupled regularization and extensive systems support. Its main disadvantage is optimizer-state memory: two dense moment tensors are stored in addition to the parameters and gradients. For a parameter matrix in $$\theta_{t+1} = (1-\eta_t\lambda)\theta_t - \eta_t\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon},$$7, the two moment states require $$\theta_{t+1} = (1-\eta_t\lambda)\theta_t - \eta_t\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon},$$8 additional storage. In mixed-precision training, moment states are often maintained in FP32, increasing their practical memory cost (Zhao et al., 2024).

In atomistic foundation-model fine-tuning, AdamW is interpreted as adaptive diagonal preconditioning combined with independent parameter shrinkage. Experiments with MACE-based models report lower force RMSE than Adam on several molecular and out-of-distribution conditions, although AdamW is not uniformly best against ScheduleFree, LAMB, RAdam, or Ranger (Liu et al., 5 Dec 2025).

In federated learning, direct local AdamW can suffer from high second-moment variance, local overfitting, client drift, and repeated moment reinitialization. FedAdamW addresses these issues through global-local correction and second-moment aggregation. In wireless decentralized learning, QEF-GT-AdamW combines AdamW with gradient tracking, biased quantization, error feedback, and packet-loss-aware fallback mechanisms (Thieu et al., 26 Aug 2026).

AdamW also induces a distinctive optimization geometry. Its asymptotic $$\theta_{t+1} = (1-\eta_t\lambda)\theta_t - \eta_t\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon},$$9 interpretation has been used to study optimizer-dependent mode connectivity. In sufficiently wide two-layer ReLU networks, AdamW-induced regularized solution sets can be connected, while at finite width AdamW and Muon can select distinct zero-loss components separated by a provable loss barrier. GPT-2 experiments further report that AdamW and Muon produce different spectral signatures and that cross-optimizer paths can exhibit smooth spectral transitions (Zhang et al., 11 May 2026).

For deep Transformers, AdamW enters continuous-depth analyses through the parameter evolution rather than directly modifying the fixed-parameter hidden-state equation. Under $m_t$00 residual scaling and suitable assumptions, hidden states and adjoint variables converge to a forward–backward ODE system at rate

$m_t$01

where $m_t$02 is depth and $m_t$03 is the number of attention heads. Blockwise AdamW removes explicit dependence on token and head dimensions from the relevant constants, whereas ordinary coordinate-wise AdamW retains dimension dependence (Gibson et al., 11 May 2026).

The optimizer can also be analyzed as a finite-horizon input–state–output dynamical system. A minibatch perturbation affects not only the immediate parameter update but also the first- and second-moment states, which can produce delayed, amplified, attenuated, or sign-reversed effects on future losses. The resulting response operator depends on the perturbation write-in, future Jacobian products, moment persistence, curvature, and the future loss readout (Liu et al., 20 Aug 2026).

Finally, decoupled weight decay directly contributes to the evolution of parameter norms. Expanding the AdamW update yields three leading terms:

$m_t$04

They are respectively an alignment force, an injection force, and a decay force. Measurements on self-trained Pythia-70M models report that alignment contributes approximately $m_t$05–$m_t$06 of the absolute force budget during the rise phase, while alignment and decay approach balance near saturation (Ding, 11 Jun 2026). This analysis concerns weight-scale evolution and does not by itself establish a general causal explanation for AdamW’s optimization or generalization behavior.