Inverse Adam (InvAdam) Optimizer

• InvAdam is an adaptive optimization method that inverts Adam’s use of the second moment by multiplying instead of dividing to amplify updates in high-curvature directions.
• It enables larger parameter updates in steep areas, promoting escape from sharp minima and favoring flatter regions that typically generalize better.
• InvAdam underpins DualAdam, which smoothly transitions from the exploration dynamics of InvAdam early in training to the stable convergence of Adam later.

Inverse Adam, usually abbreviated InvAdam, is an adaptive optimization method introduced as an inverse counterpart to Adam in "Combining Adam and its Inverse Counterpart to Enhance Generalization of Deep Learning Optimizers" (Shi et al., 7 Mar 2026). It preserves Adam’s exponential moving averages of first- and second-order moments, but reverses the role of the second moment in the update rule: instead of dividing by a second-moment term, it multiplies by one. The stated objective is to alter the optimizer’s interaction with curvature so that it can more readily escape sharp minima and remain in flatter minima, which the paper treats as a widely accepted explanation for improved generalization. Because pure InvAdam may fail to converge in practice, the same work introduces DualAdam, which interpolates from InvAdam early in training to Adam later in training (Shi et al., 7 Mar 2026).

1. Adam’s limitation and the rationale for inversion

The starting point is the standard Adam mechanism. For parameters $\boldsymbol{\theta}_t$ and mini-batch gradient $$\boldsymbol{g}_t = \nabla_{\boldsymbol{\theta}} L(\boldsymbol{\theta}_t)$$, Adam maintains biased first- and second-moment estimates,

$$\boldsymbol{m}_t = \beta_1 \boldsymbol{m}_{t-1} + (1-\beta_1)\boldsymbol{g}_t,\qquad \boldsymbol{v}_t = \beta_2 \boldsymbol{v}_{t-1} + (1-\beta_2)\boldsymbol{g}_t^2,$$

with bias-corrected moments

$$\hat{\boldsymbol{m}}_t = \frac{\boldsymbol{m}_t}{1-\beta_1^t},\qquad \hat{\boldsymbol{v}}_t = \frac{\boldsymbol{v}_t}{1-\beta_2^t}.$$

Its defining adaptive feature is that large entries of $\hat{\boldsymbol{v}}_t$ reduce the effective per-coordinate step size, while small entries increase it (Shi et al., 7 Mar 2026).

The paper frames Adam’s generalization deficit through the sharp-minima versus flat-minima distinction. Around sharp minima, gradient magnitudes fluctuate more and Hessian eigenvalues are larger; this produces large entries in the second-moment estimate. Since Adam updates coordinates according to

$$u_{t,i} = \frac{\hat{m}_{t,i}}{\sqrt{\hat{v}_{t,i}+\epsilon}},$$

it shrinks steps precisely in high-curvature directions. In the paper’s interpretation, this makes Adam likely to slow down and remain near sharp minima rather than transition toward flatter basins (Shi et al., 7 Mar 2026).

This motivation is not presented as a claim that Adam is ineffective as an optimizer. Rather, the specific claim is that Adam is fast and stable for optimization, but often exhibits worse test performance than SGD or appropriately tuned variants because of the geometry of the minima to which it converges. InvAdam is proposed as a direct alteration of that curvature response.

2. InvAdam update rule and local dynamics

InvAdam retains exactly the same moment estimates as Adam:

$$\boldsymbol{m}_t = \beta_1 \boldsymbol{m}_{t-1} + (1-\beta_1)\boldsymbol{g}_t,\qquad \boldsymbol{v}_t = \beta_2 \boldsymbol{v}_{t-1} + (1-\beta_2)\boldsymbol{g}_t^2,$$

and the same bias corrections,

$$\hat{\boldsymbol{m}}_t = \frac{\boldsymbol{m}_t}{1-\beta_1^t},\qquad \hat{\boldsymbol{v}}_t = \frac{\boldsymbol{v}_t}{1-\beta_2^t}.$$

The change is exclusively in the parameter update. Adam divides by a second-moment term, whereas InvAdam uses element-wise multiplication:

$$\tilde{u}_{t,i} = \hat{m}_{t,i}\sqrt{\hat{v}_{t,i}},\qquad \boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta\,\tilde{\boldsymbol{u}}_t.$$

The paper therefore characterizes InvAdam as “Adam with inverted use of the second moment” (Shi et al., 7 Mar 2026).

In hyperparameter terms, pure InvAdam introduces no new parameter beyond the Adam family defaults: the learning rate $\eta$, the momentum coefficients $$\boldsymbol{g}_t = \nabla_{\boldsymbol{\theta}} L(\boldsymbol{\theta}_t)$$0, and the small $$\boldsymbol{g}_t = \nabla_{\boldsymbol{\theta}} L(\boldsymbol{\theta}_t)$$1, which the paper notes is not needed in the main multiplicative formula but can be added if desired. This is significant because the method’s novelty lies in the sign of the curvature response rather than in auxiliary scheduling or regularization machinery.

The step-size scaling is the central local property. Ignoring bias correction and $$\boldsymbol{g}_t = \nabla_{\boldsymbol{\theta}} L(\boldsymbol{\theta}_t)$$2, the paper summarizes the coordinate-wise behavior as

$$\boldsymbol{g}_t = \nabla_{\boldsymbol{\theta}} L(\boldsymbol{\theta}_t)$$3

Hence, as $$\boldsymbol{g}_t = \nabla_{\boldsymbol{\theta}} L(\boldsymbol{\theta}_t)$$4 grows, Adam scales like $$\boldsymbol{g}_t = \nabla_{\boldsymbol{\theta}} L(\boldsymbol{\theta}_t)$$5 and InvAdam scales like $$\boldsymbol{g}_t = \nabla_{\boldsymbol{\theta}} L(\boldsymbol{\theta}_t)$$6. In the paper’s interpretation, large second moments are associated with sharp curvature, so InvAdam amplifies motion where Adam damps it. Conversely, in flatter regions, where gradients and second moments are smaller, InvAdam takes smaller steps and thus tends to remain there (Shi et al., 7 Mar 2026).

The paper’s qualitative simulations on two-dimensional landscapes, including a custom function and the Eggholder function, are aligned with this account: Adam converges quickly to a nearby sharp minimum and stays there, whereas InvAdam explores more widely and ends at flatter minima. This suggests that InvAdam is best understood not merely as a modified preconditioner, but as a deliberate bias toward aggressive escape dynamics in high-curvature directions.

3. Diffusion-theoretic interpretation and escape-time analysis

The theoretical analysis is formulated through diffusion theory and the Kramers escape problem. Under a second-order Taylor approximation around a point $$\boldsymbol{g}_t = \nabla_{\boldsymbol{\theta}} L(\boldsymbol{\theta}_t)$$7, a quasi-equilibrium assumption near minima, and a low-temperature approximation, the paper treats mini-batch SGD dynamics as approximately governed by the stochastic differential equation

$$\boldsymbol{g}_t = \nabla_{\boldsymbol{\theta}} L(\boldsymbol{\theta}_t)$$8

where $$\boldsymbol{g}_t = \nabla_{\boldsymbol{\theta}} L(\boldsymbol{\theta}_t)$$9 is the gradient-noise covariance and $$\boldsymbol{m}_t = \beta_1 \boldsymbol{m}_{t-1} + (1-\beta_1)\boldsymbol{g}_t,\qquad \boldsymbol{v}_t = \beta_2 \boldsymbol{v}_{t-1} + (1-\beta_2)\boldsymbol{g}_t^2,$$0 is a Wiener increment. Following Xie et al., it further states that near critical points

$$\boldsymbol{m}_t = \beta_1 \boldsymbol{m}_{t-1} + (1-\beta_1)\boldsymbol{g}_t,\qquad \boldsymbol{v}_t = \beta_2 \boldsymbol{v}_{t-1} + (1-\beta_2)\boldsymbol{g}_t^2,$$1

with batch size $$\boldsymbol{m}_t = \beta_1 \boldsymbol{m}_{t-1} + (1-\beta_1)\boldsymbol{g}_t,\qquad \boldsymbol{v}_t = \beta_2 \boldsymbol{v}_{t-1} + (1-\beta_2)\boldsymbol{g}_t^2,$$2 and Hessian $$\boldsymbol{m}_t = \beta_1 \boldsymbol{m}_{t-1} + (1-\beta_1)\boldsymbol{g}_t,\qquad \boldsymbol{v}_t = \beta_2 \boldsymbol{v}_{t-1} + (1-\beta_2)\boldsymbol{g}_t^2,$$3. The corresponding Fokker–Planck equation is written as

$$\boldsymbol{m}_t = \beta_1 \boldsymbol{m}_{t-1} + (1-\beta_1)\boldsymbol{g}_t,\qquad \boldsymbol{v}_t = \beta_2 \boldsymbol{v}_{t-1} + (1-\beta_2)\boldsymbol{g}_t^2,$$4

with diffusion matrix

$$\boldsymbol{m}_t = \beta_1 \boldsymbol{m}_{t-1} + (1-\beta_1)\boldsymbol{g}_t,\qquad \boldsymbol{v}_t = \beta_2 \boldsymbol{v}_{t-1} + (1-\beta_2)\boldsymbol{g}_t^2,$$5

(Shi et al., 7 Mar 2026).

The escape analysis considers a sharp minimum $$\boldsymbol{m}_t = \beta_1 \boldsymbol{m}_{t-1} + (1-\beta_1)\boldsymbol{g}_t,\qquad \boldsymbol{v}_t = \beta_2 \boldsymbol{v}_{t-1} + (1-\beta_2)\boldsymbol{g}_t^2,$$6, a saddle point $$\boldsymbol{m}_t = \beta_1 \boldsymbol{m}_{t-1} + (1-\beta_1)\boldsymbol{g}_t,\qquad \boldsymbol{v}_t = \beta_2 \boldsymbol{v}_{t-1} + (1-\beta_2)\boldsymbol{g}_t^2,$$7, and a flatter minimum $$\boldsymbol{m}_t = \beta_1 \boldsymbol{m}_{t-1} + (1-\beta_1)\boldsymbol{g}_t,\qquad \boldsymbol{v}_t = \beta_2 \boldsymbol{v}_{t-1} + (1-\beta_2)\boldsymbol{g}_t^2,$$8. Let $$\boldsymbol{m}_t = \beta_1 \boldsymbol{m}_{t-1} + (1-\beta_1)\boldsymbol{g}_t,\qquad \boldsymbol{v}_t = \beta_2 \boldsymbol{v}_{t-1} + (1-\beta_2)\boldsymbol{g}_t^2,$$9 denote the Hessian eigenvalue at $$\hat{\boldsymbol{m}}_t = \frac{\boldsymbol{m}_t}{1-\beta_1^t},\qquad \hat{\boldsymbol{v}}_t = \frac{\boldsymbol{v}_t}{1-\beta_2^t}.$$0 along escape direction $$\hat{\boldsymbol{m}}_t = \frac{\boldsymbol{m}_t}{1-\beta_1^t},\qquad \hat{\boldsymbol{v}}_t = \frac{\boldsymbol{v}_t}{1-\beta_2^t}.$$1, let $$\hat{\boldsymbol{m}}_t = \frac{\boldsymbol{m}_t}{1-\beta_1^t},\qquad \hat{\boldsymbol{v}}_t = \frac{\boldsymbol{v}_t}{1-\beta_2^t}.$$2 denote the corresponding eigenvalue at $$\hat{\boldsymbol{m}}_t = \frac{\boldsymbol{m}_t}{1-\beta_1^t},\qquad \hat{\boldsymbol{v}}_t = \frac{\boldsymbol{v}_t}{1-\beta_2^t}.$$3, and let $$\hat{\boldsymbol{m}}_t = \frac{\boldsymbol{m}_t}{1-\beta_1^t},\qquad \hat{\boldsymbol{v}}_t = \frac{\boldsymbol{v}_t}{1-\beta_2^t}.$$4 be the barrier height. For Adam, prior work attributed to Xie et al. yields an approximate mean escape time $$\hat{\boldsymbol{m}}_t = \frac{\boldsymbol{m}_t}{1-\beta_1^t},\qquad \hat{\boldsymbol{v}}_t = \frac{\boldsymbol{v}_t}{1-\beta_2^t}.$$5 satisfying

$$\hat{\boldsymbol{m}}_t = \frac{\boldsymbol{m}_t}{1-\beta_1^t},\qquad \hat{\boldsymbol{v}}_t = \frac{\boldsymbol{v}_t}{1-\beta_2^t}.$$6

For InvAdam, the paper derives a more complex expression in Theorem 1 and then isolates the dominant dependence on sharpness as

$$\hat{\boldsymbol{m}}_t = \frac{\boldsymbol{m}_t}{1-\beta_1^t},\qquad \hat{\boldsymbol{v}}_t = \frac{\boldsymbol{v}_t}{1-\beta_2^t}.$$7

Thus,

$$\hat{\boldsymbol{m}}_t = \frac{\boldsymbol{m}_t}{1-\beta_1^t},\qquad \hat{\boldsymbol{v}}_t = \frac{\boldsymbol{v}_t}{1-\beta_2^t}.$$8

As $$\hat{\boldsymbol{m}}_t = \frac{\boldsymbol{m}_t}{1-\beta_1^t},\qquad \hat{\boldsymbol{v}}_t = \frac{\boldsymbol{v}_t}{1-\beta_2^t}.$$9 increases, both escape times decrease, but the InvAdam dependence decays much more rapidly with sharpness (Shi et al., 7 Mar 2026).

Within the paper’s framework, this formalizes the geometric intuition behind the update rule. High-curvature minima are precisely the places where InvAdam is predicted to be substantially more likely than Adam to cross the barrier and leave the basin. Since the analysis depends on diffusion approximations, a plausible implication is that its strongest force is explanatory rather than universal: it clarifies why an inverse second-moment mechanism may favor flat-minimum selection, but it does not eliminate the need for empirical validation.

4. DualAdam: combining inverse exploration with Adam convergence

The same work treats pure InvAdam as unstable enough that it should primarily be used as a component rather than as a standalone production optimizer. The proposed solution is DualAdam, designed to use InvAdam in the early phase of training and Adam in the later phase. The optimizer computes the same gradient and moment quantities as Adam and InvAdam, forms the Adam update

$\hat{\boldsymbol{v}}_t$0

and the InvAdam update

$\hat{\boldsymbol{v}}_t$1

then mixes them with a time-dependent coefficient

$\hat{\boldsymbol{v}}_t$2

where $\hat{\boldsymbol{v}}_t$3 is the switching rate. The combined update is

$\hat{\boldsymbol{v}}_t$4

followed by

$\hat{\boldsymbol{v}}_t$5

When $\hat{\boldsymbol{v}}_t$6, DualAdam behaves approximately like InvAdam; when $\hat{\boldsymbol{v}}_t$7, it behaves approximately like Adam (Shi et al., 7 Mar 2026).

The motivation is explicitly sequential. Early iterations are used for exploration and for escaping sharp minima through the large high-curvature steps induced by InvAdam. Later iterations shift to Adam’s more stable convergence behavior. The paper emphasizes that the transition is linear in time, which is intended to avoid the abrupt hard-switch behavior used in methods such as SWATS or MIAdam. The only new hyperparameter beyond Adam’s is the switching rate $\hat{\boldsymbol{v}}_t$8.

The switching schedule is empirically important. In the CIFAR experiments, $\hat{\boldsymbol{v}}_t$9 is used, corresponding to an InvAdam-like phase of roughly the first 12,500 iterations out of approximately 78k, or about 16% of training. An ablation on CIFAR-100 with ResNet-18 reports that $$u_{t,i} = \frac{\hat{m}_{t,i}}{\sqrt{\hat{v}_{t,i}+\epsilon}},$$0, corresponding to pure InvAdam, leads to non-convergence with training loss stuck at about 4.58 and validation accuracy around 2%. Very small or very large $$u_{t,i} = \frac{\hat{m}_{t,i}}{\sqrt{\hat{v}_{t,i}+\epsilon}},$$1 also degrades performance, while the best validation performance appears around $$u_{t,i} = \frac{\hat{m}_{t,i}}{\sqrt{\hat{v}_{t,i}+\epsilon}},$$2 (Shi et al., 7 Mar 2026).

The paper also compares switching mechanisms. On CIFAR-100 with ResNet-18, linear switching yields the best test accuracy, reported as around 75.1–75.3%; exponential switching is slightly worse, at 71–73%; and fixed-epoch hard switching is worst, at 66–71%. This is used to support the claim that a smooth transition is important. The result is notable because it ties the method’s performance not only to the inverse update itself, but also to the temporal manner in which that update is withdrawn.

5. Empirical performance and loss-landscape evidence

The empirical evaluation spans synthetic landscapes, standard vision benchmarks, and LLM fine-tuning. On two-dimensional loss landscapes, including a custom function and the Eggholder function, Adam rapidly falls into a sharp minimum and remains there, while InvAdam explores more broadly and ends at flatter minima. These visualizations provide an intuitive counterpart to the diffusion-based argument (Shi et al., 7 Mar 2026).

On image classification benchmarks, DualAdam is reported to outperform Adam and several Adam-family variants. Selected CIFAR-100 results are summarized below.

Model / dataset	Adam	DualAdam
ResNet-18 / CIFAR-100	$$u_{t,i} = \frac{\hat{m}_{t,i}}{\sqrt{\hat{v}_{t,i}+\epsilon}},$$3	$$u_{t,i} = \frac{\hat{m}_{t,i}}{\sqrt{\hat{v}_{t,i}+\epsilon}},$$4
ResNet-50 / CIFAR-100	$$u_{t,i} = \frac{\hat{m}_{t,i}}{\sqrt{\hat{v}_{t,i}+\epsilon}},$$5	$$u_{t,i} = \frac{\hat{m}_{t,i}}{\sqrt{\hat{v}_{t,i}+\epsilon}},$$6
VGG-16 / CIFAR-100	$$u_{t,i} = \frac{\hat{m}_{t,i}}{\sqrt{\hat{v}_{t,i}+\epsilon}},$$7	$$u_{t,i} = \frac{\hat{m}_{t,i}}{\sqrt{\hat{v}_{t,i}+\epsilon}},$$8
ViT-Small-4 / CIFAR-100	$$u_{t,i} = \frac{\hat{m}_{t,i}}{\sqrt{\hat{v}_{t,i}+\epsilon}},$$9	$$\boldsymbol{m}_t = \beta_1 \boldsymbol{m}_{t-1} + (1-\beta_1)\boldsymbol{g}_t,\qquad \boldsymbol{v}_t = \beta_2 \boldsymbol{v}_{t-1} + (1-\beta_2)\boldsymbol{g}_t^2,$$0

The same section reports corresponding baselines from RAdam and MIAdam, with DualAdam remaining best in these examples. The paper characterizes the typical gains over Adam as 2–3 percentage points, while gains over strong variants such as RAdam, Adan, and MIAdam are smaller but consistent (Shi et al., 7 Mar 2026).

The reported Tiny ImageNet and ImageNet-1k results extend the same pattern.

Model / dataset	Adam	DualAdam
ResNet-18 / Tiny ImageNet	$$\boldsymbol{m}_t = \beta_1 \boldsymbol{m}_{t-1} + (1-\beta_1)\boldsymbol{g}_t,\qquad \boldsymbol{v}_t = \beta_2 \boldsymbol{v}_{t-1} + (1-\beta_2)\boldsymbol{g}_t^2,$$1	$$\boldsymbol{m}_t = \beta_1 \boldsymbol{m}_{t-1} + (1-\beta_1)\boldsymbol{g}_t,\qquad \boldsymbol{v}_t = \beta_2 \boldsymbol{v}_{t-1} + (1-\beta_2)\boldsymbol{g}_t^2,$$2
VGG-16 / Tiny ImageNet	$$\boldsymbol{m}_t = \beta_1 \boldsymbol{m}_{t-1} + (1-\beta_1)\boldsymbol{g}_t,\qquad \boldsymbol{v}_t = \beta_2 \boldsymbol{v}_{t-1} + (1-\beta_2)\boldsymbol{g}_t^2,$$3	$$\boldsymbol{m}_t = \beta_1 \boldsymbol{m}_{t-1} + (1-\beta_1)\boldsymbol{g}_t,\qquad \boldsymbol{v}_t = \beta_2 \boldsymbol{v}_{t-1} + (1-\beta_2)\boldsymbol{g}_t^2,$$4
ViT-Small-8 / Tiny ImageNet	$$\boldsymbol{m}_t = \beta_1 \boldsymbol{m}_{t-1} + (1-\beta_1)\boldsymbol{g}_t,\qquad \boldsymbol{v}_t = \beta_2 \boldsymbol{v}_{t-1} + (1-\beta_2)\boldsymbol{g}_t^2,$$5	$$\boldsymbol{m}_t = \beta_1 \boldsymbol{m}_{t-1} + (1-\beta_1)\boldsymbol{g}_t,\qquad \boldsymbol{v}_t = \beta_2 \boldsymbol{v}_{t-1} + (1-\beta_2)\boldsymbol{g}_t^2,$$6
ResNet-18 / ImageNet-1k	$$\boldsymbol{m}_t = \beta_1 \boldsymbol{m}_{t-1} + (1-\beta_1)\boldsymbol{g}_t,\qquad \boldsymbol{v}_t = \beta_2 \boldsymbol{v}_{t-1} + (1-\beta_2)\boldsymbol{g}_t^2,$$7	$$\boldsymbol{m}_t = \beta_1 \boldsymbol{m}_{t-1} + (1-\beta_1)\boldsymbol{g}_t,\qquad \boldsymbol{v}_t = \beta_2 \boldsymbol{v}_{t-1} + (1-\beta_2)\boldsymbol{g}_t^2,$$8
ResNet-50 / ImageNet-1k	$$\boldsymbol{m}_t = \beta_1 \boldsymbol{m}_{t-1} + (1-\beta_1)\boldsymbol{g}_t,\qquad \boldsymbol{v}_t = \beta_2 \boldsymbol{v}_{t-1} + (1-\beta_2)\boldsymbol{g}_t^2,$$9	$$\hat{\boldsymbol{m}}_t = \frac{\boldsymbol{m}_t}{1-\beta_1^t},\qquad \hat{\boldsymbol{v}}_t = \frac{\boldsymbol{v}_t}{1-\beta_2^t}.$$0

For ImageNet-1k, the paper also reports AdamW baselines of $$\hat{\boldsymbol{m}}_t = \frac{\boldsymbol{m}_t}{1-\beta_1^t},\qquad \hat{\boldsymbol{v}}_t = \frac{\boldsymbol{v}_t}{1-\beta_2^t}.$$1 for ResNet-18 and $$\hat{\boldsymbol{m}}_t = \frac{\boldsymbol{m}_t}{1-\beta_1^t},\qquad \hat{\boldsymbol{v}}_t = \frac{\boldsymbol{v}_t}{1-\beta_2^t}.$$2 for ResNet-50, both below DualAdam. In large-scale vision training, the paper therefore presents DualAdam as improving over both Adam and AdamW (Shi et al., 7 Mar 2026).

The language-model evidence comes from fine-tuning the 1B-parameter Chinese LLM OpenPangu-Embedded-1B on Alpaca-GPT4-CN. Here the comparison is against AdamW. AdamW achieves lower training loss, but its validation perplexity initially decreases and then starts to rise, which the paper interprets as overfitting. DualAdam shows higher training loss but lower and more stable validation perplexity, with a generalization gap close to zero throughout training. This supports the paper’s flat-minima interpretation: reduced optimization aggressiveness on the training objective can coincide with better validation behavior (Shi et al., 7 Mar 2026).

Further geometric evidence is obtained from Hessian and loss-landscape analysis using PyHessian, attributed to Yao et al., on CIFAR-100 with ResNet-18. DualAdam solutions show Hessian eigenvalue distributions more concentrated around zero, with significantly lower maximum eigenvalues and lower trace than Adam solutions. One-dimensional visualizations of $$\hat{\boldsymbol{m}}_t = \frac{\boldsymbol{m}_t}{1-\beta_1^t},\qquad \hat{\boldsymbol{v}}_t = \frac{\boldsymbol{v}_t}{1-\beta_2^t}.$$3 along random directions $$\hat{\boldsymbol{m}}_t = \frac{\boldsymbol{m}_t}{1-\beta_1^t},\qquad \hat{\boldsymbol{v}}_t = \frac{\boldsymbol{v}_t}{1-\beta_2^t}.$$4 show wider, flatter valleys for DualAdam and narrower, steeper valleys for Adam. This is presented as empirical confirmation that DualAdam finds flatter minima.

6. Relation to other methods, computational profile, and limitations

The paper situates InvAdam and DualAdam within the Adam-family literature. AdamW is described as improving generalization via decoupled weight decay but not explicitly changing curvature response in the way InvAdam does. RAdam focuses on variance rectification for warm-up stability rather than sharp minima. NAdam introduces Nesterov momentum without fundamentally changing the sharp-versus-flat mechanism. Adan is described as a stronger but more complex adaptive method. SWATS switches from Adam to SGD at a heuristic time, and MIAdam uses multiple-integral moments early and then switches to Adam via a hard switch, which the paper says can cause instability. Against this background, DualAdam’s conceptual novelty is identified as the explicit inverse second-moment mechanism combined with a linear, smooth switching schedule (Shi et al., 7 Mar 2026).

The paper also distinguishes DualAdam from flat-minima methods such as SAM and SWA. It does not provide detailed empirical comparisons with those methods, but it notes a conceptual difference: InvAdam and DualAdam induce a bias toward flat minima through the optimizer step rule itself, without requiring the extra forward/backward passes of SAM or the explicit weight averaging of SWA. A plausible implication is that the method occupies an intermediate position between classical adaptive preconditioning and explicitly geometry-aware regularization.

The computational overhead is reported as modest. Per parameter per iteration, the paper gives approximately $$\hat{\boldsymbol{m}}_t = \frac{\boldsymbol{m}_t}{1-\beta_1^t},\qquad \hat{\boldsymbol{v}}_t = \frac{\boldsymbol{v}_t}{1-\beta_2^t}.$$5 FLOPs for Adam and approximately $$\hat{\boldsymbol{m}}_t = \frac{\boldsymbol{m}_t}{1-\beta_1^t},\qquad \hat{\boldsymbol{v}}_t = \frac{\boldsymbol{v}_t}{1-\beta_2^t}.$$6 FLOPs for DualAdam. The additional $$\hat{\boldsymbol{m}}_t = \frac{\boldsymbol{m}_t}{1-\beta_1^t},\qquad \hat{\boldsymbol{v}}_t = \frac{\boldsymbol{v}_t}{1-\beta_2^t}.$$7 FLOPs per parameter are characterized as negligible relative to the forward/backward cost of about $$\hat{\boldsymbol{m}}_t = \frac{\boldsymbol{m}_t}{1-\beta_1^t},\qquad \hat{\boldsymbol{v}}_t = \frac{\boldsymbol{v}_t}{1-\beta_2^t}.$$8 FLOPs, so with typical batch sizes $$\hat{\boldsymbol{m}}_t = \frac{\boldsymbol{m}_t}{1-\beta_1^t},\qquad \hat{\boldsymbol{v}}_t = \frac{\boldsymbol{v}_t}{1-\beta_2^t}.$$9 the overhead is an $$\tilde{u}_{t,i} = \hat{m}_{t,i}\sqrt{\hat{v}_{t,i}},\qquad \boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta\,\tilde{\boldsymbol{u}}_t.$$0 fraction of total compute. The empirical timing example on CIFAR-100 with ResNet-18 for 200 epochs and batch size 128 is 2197s for Adam versus 2214s for DualAdam, and another example reports 36m37s versus 36m54s, respectively. The implementation change is correspondingly simple: add the InvAdam update, compute $$\tilde{u}_{t,i} = \hat{m}_{t,i}\sqrt{\hat{v}_{t,i}},\qquad \boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta\,\tilde{\boldsymbol{u}}_t.$$1, and linearly mix the Adam and InvAdam updates (Shi et al., 7 Mar 2026).

The limitations are explicit. Pure InvAdam is unstable: with $$\tilde{u}_{t,i} = \hat{m}_{t,i}\sqrt{\hat{v}_{t,i}},\qquad \boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta\,\tilde{\boldsymbol{u}}_t.$$2, training fails to converge on CIFAR-100. Performance depends on the switching schedule, and poor choices of $$\tilde{u}_{t,i} = \hat{m}_{t,i}\sqrt{\hat{v}_{t,i}},\qquad \boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta\,\tilde{\boldsymbol{u}}_t.$$3 or switching mechanism can negate the benefits. The theory is incomplete: the paper does not provide a full convergence proof for InvAdam, and its convergence account for DualAdam is indirect, relying on the fact that DualAdam eventually becomes Adam. The scope of evaluation, while spanning several vision models and one LLM, does not include domains such as reinforcement learning, graph networks, or diffusion models. The future directions suggested in the paper therefore include more sophisticated switching mechanisms, combining InvAdam with other base optimizers such as SGD, RMSProp, or Adan, deeper non-convex convergence analysis, and broader application to other large-scale NLP, multimodal, and reinforcement-learning settings (Shi et al., 7 Mar 2026).