Adam with Model EMA Optimization

• Adam with Model EMA is a method that combines adaptive gradient updates with an exponential moving average to enhance optimizer stability and convergence in nonconvex settings.
• The approach utilizes coordinate-wise clipping and adaptive scaling to achieve minimax-optimal rates even under heterogeneous noise and gradient conditions.
• Theoretical analysis shows that employing a clipped Adam update alongside model EMA outperforms vanilla Adam and SGD, providing deterministic convergence guarantees.

Adam with Model Exponential Moving Average (EMA) refers to the combination of the Adam optimizer—a widely used adaptive gradient method in stochastic optimization for deep learning—with an exponential moving average applied directly to model parameters. This pairing, especially in nonconvex settings, merges the benefits of adaptivity, momentum, clipping, and online-to-batch conversion through model weight averaging, achieving minimax-optimal convergence rates for smooth and nonsmooth problems. Recent theoretical advances have established that employing a clipped version of Adam alongside a model EMA enables optimal convergence guarantees in both global and coordinate-wise nonconvex settings, and is superior to vanilla Adam or SGD under heterogeneous coordinate scales (Ahn et al., 2024). The EMA component can itself be understood via a physical analogy to damped harmonic motion, motivating further algorithmic generalizations and improved parameter schedules (Patsenker et al., 2023).

1. Algorithmic Foundations

The joint method is designed for stochastic nonconvex optimization. Let $F:\mathbb{R}^d \to \mathbb{R}$ be the objective, with stochastic gradients $g_t \approx \nabla F(x_t)$. The optimization proceeds by alternating between clipped Adam updates and maintenance of a model EMA shadow sequence.

Clipped Adam Update (coordinate-wise):

• Exponential gradient and squared-gradient averages:

$$m_{t,i} = \sum_{s=1}^t \beta_1^{t-s} g_{s,i}, \quad v_{t,i} = \sqrt{\sum_{s=1}^t \beta_2^{t-s} g_{s,i}^2}$$

• Scalar clip operator:

$\clip_D(u) = \min\Big(\frac{D}{|u|}, 1\Big) u$

• Parameter update with smoothing constant $\varepsilon>0$:

$z_{t,i} = -\clip_D\Big(D\,\frac{m_{t-1,i}}{v_{t-1,i}+\varepsilon}\Big), \quad x_t = x_{t-1} + z_t$

Model Exponential Moving Average:

• EMA parameter with discount $\beta$:

$$\tilde x_t = \frac{\beta-\beta^t}{1-\beta^t}\tilde x_{t-1} + \frac{1-\beta}{1-\beta^t} x_t, \quad \tilde x_0 = x_0$$

Or, equivalently,

$$\tilde x_t = \frac{1-\beta}{1-\beta^t} \sum_{s=1}^t \beta^{t-s} x_s$$

At algorithm termination, $\tilde x_T$ is returned as the solution.

In practice, parameters are typically chosen as $g_t \approx \nabla F(x_t)$0, $g_t \approx \nabla F(x_t)$1 (with theory favoring $g_t \approx \nabla F(x_t)$2), small $g_t \approx \nabla F(x_t)$3, and a large EMA decay $g_t \approx \nabla F(x_t)$4 (Ahn et al., 2024).

2. Convergence Theory and Optimality

Convergence is quantified using a generalized $g_t \approx \nabla F(x_t)$5–Goldstein stationarity notion:

$g_t \approx \nabla F(x_t)$6

2.1 Global Nonconvex Convergence

Under $g_t \approx \nabla F(x_t)$7-Lipschitz $g_t \approx \nabla F(x_t)$8, stochastic gradients with variance $g_t \approx \nabla F(x_t)$9, and initial gap $$m_{t,i} = \sum_{s=1}^t \beta_1^{t-s} g_{s,i}, \quad v_{t,i} = \sqrt{\sum_{s=1}^t \beta_2^{t-s} g_{s,i}^2}$$0, the algorithm achieves:

• With $$m_{t,i} = \sum_{s=1}^t \beta_1^{t-s} g_{s,i}, \quad v_{t,i} = \sqrt{\sum_{s=1}^t \beta_2^{t-s} g_{s,i}^2}$$1:

$$m_{t,i} = \sum_{s=1}^t \beta_1^{t-s} g_{s,i}, \quad v_{t,i} = \sqrt{\sum_{s=1}^t \beta_2^{t-s} g_{s,i}^2}$$2

This rate is minimax-optimal for smooth and nonsmooth nonconvex problems (Ahn et al., 2024).

2.2 Coordinate-wise Adaptive Rates

For per-coordinate Lipschitz and noise parameters $$m_{t,i} = \sum_{s=1}^t \beta_1^{t-s} g_{s,i}, \quad v_{t,i} = \sqrt{\sum_{s=1}^t \beta_2^{t-s} g_{s,i}^2}$$3, $$m_{t,i} = \sum_{s=1}^t \beta_1^{t-s} g_{s,i}, \quad v_{t,i} = \sqrt{\sum_{s=1}^t \beta_2^{t-s} g_{s,i}^2}$$4, the coordinate-wise result is:

• Stationarity in $$m_{t,i} = \sum_{s=1}^t \beta_1^{t-s} g_{s,i}, \quad v_{t,i} = \sqrt{\sum_{s=1}^t \beta_2^{t-s} g_{s,i}^2}$$5:

$$m_{t,i} = \sum_{s=1}^t \beta_1^{t-s} g_{s,i}, \quad v_{t,i} = \sqrt{\sum_{s=1}^t \beta_2^{t-s} g_{s,i}^2}$$6

An $$m_{t,i} = \sum_{s=1}^t \beta_1^{t-s} g_{s,i}, \quad v_{t,i} = \sqrt{\sum_{s=1}^t \beta_2^{t-s} g_{s,i}^2}$$7 speedup arises if only a few coordinates are “hard” ($$m_{t,i} = \sum_{s=1}^t \beta_1^{t-s} g_{s,i}, \quad v_{t,i} = \sqrt{\sum_{s=1}^t \beta_2^{t-s} g_{s,i}^2}$$8).

3. Comparative Analysis: EMA-Augmented Adam vs. Alternatives

Optimizer	Requires Model EMA	Rate (Smooth Nonconvex)	Key Limitation
Adam (no EMA)	No	$$m_{t,i} = \sum_{s=1}^t \beta_1^{t-s} g_{s,i}, \quad v_{t,i} = \sqrt{\sum_{s=1}^t \beta_2^{t-s} g_{s,i}^2}$$9	Suboptimal, random iterate
Clipped Adam + EMA	Yes	$\clip_D(u) = \min\Big(\frac{D}{|u|}, 1\Big) u$0	Optimal, deterministic EMA
SGD + Polyak averages	No	$\clip_D(u) = \min\Big(\frac{D}{|u|}, 1\Big) u$1	No coordinate adaptation
Adagrad (scale-free FTRL)	No	$\clip_D(u) = \min\Big(\frac{D}{|u|}, 1\Big) u$2	Step-size tuning needed

Without model EMA, Adam attains suboptimal rates, and analysis typically forces random iterate selection, which increases output variance and is rarely used in practice. SGD with averaging also reaches optimal rates, but lacks per-coordinate scaling, effectively slowing progress on heteroskedastic objectives. Adaptive methods such as scale-free FTRL can be integrated but still require additional tuning (Ahn et al., 2024).

4. Structural Roles: Momentum, Discounting, and Adaptivity

• Momentum ($\clip_D(u) = \min\Big(\frac{D}{|u|}, 1\Big) u$3): Interpreted as discount factors in the sequence of observed gradients, implementing an online-to-nonconvex reduction where each step minimizes a discounted history of linearized losses.
• EMA Discounting ($\clip_D(u) = \min\Big(\frac{D}{|u|}, 1\Big) u$4): Both the loss-regret framework and model EMA employ the same discount parameter to weight history, making the optimizer both responsive and stable in nonstationary or varying regimes.
• Coordinate-wise Adaptivity ($\clip_D(u) = \min\Big(\frac{D}{|u|}, 1\Big) u$5 and normalization): Each coordinate adaptively tunes its own effective step size via per-coordinate second-moment normalization, crucial when gradient or noise scales are nonuniform across coordinates, leading to dimension-dependent speedup.
• Gradient Clipping ($\clip_D(u) = \min\Big(\frac{D}{|u|}, 1\Big) u$6): Added robustness to occasional large-magnitude gradients, enforcing bounded iterates.

5. Physical and Dynamical Analogies for EMA

Model EMA can be viewed, in continuous time, as a first-order low-pass filter, governed by the ODE

$\clip_D(u) = \min\Big(\frac{D}{|u|}, 1\Big) u$7

with $\clip_D(u) = \min\Big(\frac{D}{|u|}, 1\Big) u$8 as the time constant set by the EMA decay (Patsenker et al., 2023). A refined analogy introduces the framework of damped harmonic motion: the EMA weights (“mass” $\clip_D(u) = \min\Big(\frac{D}{|u|}, 1\Big) u$9) are attracted to the model weights (“mass” $\varepsilon>0$0) via a spring ($\varepsilon>0$1) and experience damping ($\varepsilon>0$2). This analogy provides an interpretation for stability and smoothness in the optimization trajectory and inspires generalizations such as BELAY, where feedback between the EMA and the model further enhances stability and convergence, especially under large learning rates or ill-conditioning.

A direct implication is that the EMA decay parameter $\varepsilon>0$3 should scale inversely with training length, ensuring the averaging horizon is appropriate:

$\varepsilon>0$4

where $\varepsilon>0$5 is the expected number of optimization steps (Patsenker et al., 2023).

6. Proof Techniques

The convergence proof proceeds in three main steps (Ahn et al., 2024):

1. Discounted-online-to-nonconvex reduction: Small discounted regret of the clipped Adam optimizer implies bounds on the stationarity of the EMA output, controlling both gradient norm and output variance.
2. Scale-free FTRL regret bound: Using per-coordinate adaptive step sizes, the regret is bounded as $\varepsilon>0$6 globally or coordinate-wise.
3. Parameter balancing: Careful algebraic selection of discount, clipping, and total steps, balancing bias, stochastic variance, and regret, yields the minimax-optimal sample complexity.

7. Practical Implementation and Extensions

In practice, Adam with model EMA is implemented trivially: maintain a second set of “shadow” parameters that are updated according to the EMA recursion after every Adam step. In frameworks such as PyTorch, this consists of a single additional memory buffer and an update rule involving the current decay constant. BELAY extends this further by treating both the model and the EMA as coupled but independently governed masses, offering additional robustness and tunable feedback.

Theoretical insights indicate the widespread convention of using both Adam and model EMA in large-scale machine learning pipelines is well-justified, particularly in large, nonconvex, and heteroskedastic optimization problems (Ahn et al., 2024, Patsenker et al., 2023).