DualAdam: Hybrid Optimizer for Neural Networks

• DualAdam is a hybrid optimizer that blends Adam’s fast convergence with InvAdam’s mechanism for escaping sharp minima.
• It uses a linear, time-decaying interpolation controlled by a switching rate to transition from exploration in early training to stable convergence later.
• Empirical studies on vision tasks and LLM fine-tuning show that DualAdam achieves better generalization and produces flatter minima compared to standard Adam.

Searching arXiv for the main paper and closely related optimizer references to ground the article in current arXiv-indexed sources. Dual Adam, usually written DualAdam, is an adaptive optimization method for neural network training that combines the update rules of Adam and an “inverse” counterpart called InvAdam in order to improve generalization while retaining stable convergence (Shi et al., 7 Mar 2026). The method is motivated by the observation that Adam typically converges fast but often exhibits suboptimal generalization performance because it tends to converge to sharp minima; InvAdam is introduced to counter this tendency by reversing Adam’s use of the second-order moment, and DualAdam integrates both mechanisms through a time-decaying interpolation that is intended to preserve InvAdam’s ability to escape sharp minima early in training and Adam’s convergence behavior later (Shi et al., 7 Mar 2026).

1. Origin, objective, and defining idea

DualAdam was introduced in the paper "Combining Adam and its Inverse Counterpart to Enhance Generalization of Deep Learning Optimizers" (Shi et al., 7 Mar 2026). Its immediate context is the long-standing contrast between Adam’s fast optimization behavior and its weaker generalization relative to methods that more readily locate flatter minima. The paper adopts a widely accepted explanation for this defect in generalization: Adam often tends to converge to sharp minima (Shi et al., 7 Mar 2026).

The proposed remedy begins with InvAdam, which keeps Adam’s first- and second-order exponential moving averages but changes the parameter update mechanism from element-wise division by the square root of the second-order moment to element-wise multiplication by it. In the formulation given in the paper, this modification increases the step size when the elements in the second-order moments are large and decreases it when they are small, with the explicit aim of helping the optimizer escape sharp minima and remain at flat ones (Shi et al., 7 Mar 2026).

InvAdam alone, however, may face challenges in convergence. DualAdam is therefore defined as a hybrid method that integrates Adam and InvAdam, using InvAdam behavior early and transitioning smoothly to Adam later. The stated purpose of this integration is to ensure convergence while enhancing generalization performance (Shi et al., 7 Mar 2026).

2. Mathematical formulation and update dynamics

The paper considers neural network training with parameters $\boldsymbol{\theta} \in \mathbb{R}^p$ and empirical objective

$$\min_{\boldsymbol{\theta}} \mathcal{L}(\boldsymbol{x}, \boldsymbol{y}; \boldsymbol{\theta}).$$

For a mini-batch of size $b$, the batch loss is

$$L(\boldsymbol{\theta}) = \frac{1}{b} \sum_{k=1}^{b} \mathcal{L}(\boldsymbol{x}_k,\boldsymbol{y}_k; \boldsymbol{\theta}),$$

and the stochastic gradient at iteration $t$ for parameter index $i$ is

$$\boldsymbol{g}_{t,i} = \frac{\partial L(\boldsymbol{\theta}_{t,i})}{\partial \boldsymbol{\theta}_{t,i}}.$$

The first- and second-order exponential moving averages are

$$\begin{cases} \boldsymbol{m}_{t,i} = \beta_1 \boldsymbol{m}_{t-1,i} + (1 - \beta_1)\boldsymbol{g}_{t,i}, \ \boldsymbol{v}_{t,i} = \beta_2 \boldsymbol{v}_{t-1,i} + (1 - \beta_2)\boldsymbol{g}_{t,i}^2, \end{cases}$$

with bias-corrected forms

$$\hat{\boldsymbol{m}}_{t,i} = \frac{\boldsymbol{m}_{t,i}}{1-\beta_1^t}, \qquad \hat{\boldsymbol{v}}_{t,i} = \frac{\boldsymbol{v}_{t,i}}{1-\beta_2^t}.$$

The learning rate is $\eta$ and $$\min_{\boldsymbol{\theta}} \mathcal{L}(\boldsymbol{x}, \boldsymbol{y}; \boldsymbol{\theta}).$$0 is a small constant (Shi et al., 7 Mar 2026).

For Adam, the element-wise update term is

$$\min_{\boldsymbol{\theta}} \mathcal{L}(\boldsymbol{x}, \boldsymbol{y}; \boldsymbol{\theta}).$$1

and the parameter update rule is

$$\min_{\boldsymbol{\theta}} \mathcal{L}(\boldsymbol{x}, \boldsymbol{y}; \boldsymbol{\theta}).$$2

For InvAdam, the corresponding update term is

$$\min_{\boldsymbol{\theta}} \mathcal{L}(\boldsymbol{x}, \boldsymbol{y}; \boldsymbol{\theta}).$$3

with update

$$\min_{\boldsymbol{\theta}} \mathcal{L}(\boldsymbol{x}, \boldsymbol{y}; \boldsymbol{\theta}).$$4

Ignoring $$\min_{\boldsymbol{\theta}} \mathcal{L}(\boldsymbol{x}, \boldsymbol{y}; \boldsymbol{\theta}).$$5, the contrast is

$$\min_{\boldsymbol{\theta}} \mathcal{L}(\boldsymbol{x}, \boldsymbol{y}; \boldsymbol{\theta}).$$6

Thus, large $$\min_{\boldsymbol{\theta}} \mathcal{L}(\boldsymbol{x}, \boldsymbol{y}; \boldsymbol{\theta}).$$7 implies a small effective step under Adam and a large effective step under InvAdam (Shi et al., 7 Mar 2026).

DualAdam introduces a switching rate $$\min_{\boldsymbol{\theta}} \mathcal{L}(\boldsymbol{x}, \boldsymbol{y}; \boldsymbol{\theta}).$$8 and the coefficient

$$\min_{\boldsymbol{\theta}} \mathcal{L}(\boldsymbol{x}, \boldsymbol{y}; \boldsymbol{\theta}).$$9

then forms the combined update

$b$0

with parameter rule

$b$1

When $b$2 and $b$3 is small, $b$4, so the method is close to pure InvAdam; as $b$5 increases, $b$6 linearly decays to $b$7, and after $b$8, DualAdam is exactly Adam (Shi et al., 7 Mar 2026).

Computationally, the paper states that DualAdam adds only a small overhead relative to Adam, at about $b$9 versus $$L(\boldsymbol{\theta}) = \frac{1}{b} \sum_{k=1}^{b} \mathcal{L}(\boldsymbol{x}_k,\boldsymbol{y}_k; \boldsymbol{\theta}),$$0 FLOPs per iteration, and that after $$L(\boldsymbol{\theta}) = \frac{1}{b} \sum_{k=1}^{b} \mathcal{L}(\boldsymbol{x}_k,\boldsymbol{y}_k; \boldsymbol{\theta}),$$1 becomes $$L(\boldsymbol{\theta}) = \frac{1}{b} \sum_{k=1}^{b} \mathcal{L}(\boldsymbol{x}_k,\boldsymbol{y}_k; \boldsymbol{\theta}),$$2, its cost is essentially the same as Adam (Shi et al., 7 Mar 2026).

3. Mechanism for escaping sharp minima

The optimizer’s central intuition is expressed through the dependence of step size on the second-order moment. In Adam, large $$L(\boldsymbol{\theta}) = \frac{1}{b} \sum_{k=1}^{b} \mathcal{L}(\boldsymbol{x}_k,\boldsymbol{y}_k; \boldsymbol{\theta}),$$3 reduces the update magnitude, whereas in InvAdam large $$L(\boldsymbol{\theta}) = \frac{1}{b} \sum_{k=1}^{b} \mathcal{L}(\boldsymbol{x}_k,\boldsymbol{y}_k; \boldsymbol{\theta}),$$4 increases it. The paper interprets this difference geometrically: in sharp minima, curvature and second-order moments tend to be large, so Adam takes cautious steps that can keep the iterate inside narrow basins, while InvAdam pushes harder in such directions and therefore promotes escape from sharp wells toward flatter regions (Shi et al., 7 Mar 2026).

This contrast is summarized in the paper’s qualitative account of optimizer behavior. Adam tends to stabilize near high-curvature regions and not leave them; InvAdam tends to enlarge steps in high-curvature or high-variance directions and reduce motion in very flat directions, thereby biasing the trajectory toward flatter basins. In 2D toy experiments, Adam quickly sinks into a sharp minimum, whereas InvAdam continues exploring and ends in a flatter basin (Shi et al., 7 Mar 2026).

DualAdam’s role is to exploit this asymmetry temporally. Early in training, when $$L(\boldsymbol{\theta}) = \frac{1}{b} \sum_{k=1}^{b} \mathcal{L}(\boldsymbol{x}_k,\boldsymbol{y}_k; \boldsymbol{\theta}),$$5 is large, the optimizer inherits InvAdam-like exploration and sharp-minima escape. Later, as $$L(\boldsymbol{\theta}) = \frac{1}{b} \sum_{k=1}^{b} \mathcal{L}(\boldsymbol{x}_k,\boldsymbol{y}_k; \boldsymbol{\theta}),$$6, it becomes Adam and acquires stable settling behavior. The paper emphasizes that the transition is smooth and linear, in contrast to hard switches that may destabilize training (Shi et al., 7 Mar 2026).

A common misconception would be to interpret DualAdam as merely another regularized Adam variant. The description in the source instead distinguishes it from approaches such as AdamW: AdamW decouples weight decay, whereas DualAdam changes the optimization dynamics themselves by modifying how the second-order moment influences the step (Shi et al., 7 Mar 2026).

4. Diffusion-theoretic analysis

The paper introduces a diffusion-theoretic analysis to explain InvAdam’s ability to escape sharp minima (Shi et al., 7 Mar 2026). The analysis adopts the setup of Xie et al. (2020, 2022), beginning with three assumptions: a second-order Taylor approximation near $$L(\boldsymbol{\theta}) = \frac{1}{b} \sum_{k=1}^{b} \mathcal{L}(\boldsymbol{x}_k,\boldsymbol{y}_k; \boldsymbol{\theta}),$$7, quasi-equilibrium near minima so that the parameter distribution behaves like a Boltzmann distribution, and a low-temperature approximation in which gradient noise is small and escape is dominated by the loss barrier and curvature (Shi et al., 7 Mar 2026).

For SGD-like dynamics, the paper writes the Langevin-type stochastic differential equation

$$L(\boldsymbol{\theta}) = \frac{1}{b} \sum_{k=1}^{b} \mathcal{L}(\boldsymbol{x}_k,\boldsymbol{y}_k; \boldsymbol{\theta}),$$8

where $$L(\boldsymbol{\theta}) = \frac{1}{b} \sum_{k=1}^{b} \mathcal{L}(\boldsymbol{x}_k,\boldsymbol{y}_k; \boldsymbol{\theta}),$$9 is the gradient noise covariance and $t$0 is a Wiener process. Near critical points, it states that empirically $t$1. The corresponding Fokker–Planck equation for the density $t$2 is

$t$3

with diffusion matrix $t$4 (Shi et al., 7 Mar 2026).

Momentum is modeled through a heavy-ball analogy. In discrete time,

$t$5

with $t$6 and $t$7. In continuous time,

$t$8

After reduction to position space, the effective diffusion in Hessian eigen-direction $t$9 becomes

$i$0

This expression is used to characterize how momentum changes diffusion along curvature directions (Shi et al., 7 Mar 2026).

The escape-time comparison is framed through Kramers analysis. For escape from a minimum $i$1 through a saddle $i$2 along eigen-direction $i$3, with barrier $i$4, the dominant scaling reported in the paper is

$i$5

for Adam, while for InvAdam

$i$6

Equivalently,

$i$7

As the sharpness $i$8 increases, the escape time for InvAdam decreases faster than for Adam. The paper’s interpretation is therefore that InvAdam is better at escaping sharp minima, and DualAdam uses this advantage in its early training phase before reverting to Adam for convergence (Shi et al., 7 Mar 2026).

5. Empirical behavior across vision and LLM fine-tuning

The reported experiments cover image classification and LLM fine-tuning (Shi et al., 7 Mar 2026). The image datasets are CIFAR-10, CIFAR-100, Tiny ImageNet, and ImageNet-1k; the architectures include ResNet-18, ResNet-50, VGG-16, and ViT-Small variants. Tiny ImageNet is described as having 200 classes, 64×64 images, and 500 train / 50 val / 50 test per class; ImageNet-1k is described as having 1.28M training images, 50k test images, and 224×224 resolution. For CIFAR-10 and CIFAR-100, the paper uses 200 epochs and batch size 128, yielding about 78k iterations (Shi et al., 7 Mar 2026).

The paper reports that DualAdam outperforms Adam and several Adam variants in generalization performance. Selected numerical results are as follows.

Setting	Baseline	DualAdam
CIFAR-100, ResNet-18	Adam: 72.56 ± 0.08%	75.29 ± 0.21%
CIFAR-100, ResNet-50	Adam: 74.53 ± 0.50%	76.84 ± 0.16%
CIFAR-100, VGG-16	Adam: 68.66 ± 0.13%	72.66 ± 0.21%
CIFAR-100, ViT-Small-4	Adam: 55.43 ± 0.09%	58.54 ± 0.24%
CIFAR-10, ResNet-18	Adam: 93.66 ± 0.17%	94.99 ± 0.13%
Tiny ImageNet, ResNet-18	Adam: 50.98 ± 0.42%	57.19 ± 0.21%
Tiny ImageNet, VGG-16	Adam: 48.57 ± 0.10%	54.42 ± 0.11%
Tiny ImageNet, ViT-Small-8	Adam: 30.72 ± 0.21%	37.90 ± 0.22%
ImageNet-1k, ResNet-18	Adam: 69.55%	72.61%
ImageNet-1k, ResNet-50	Adam: 70.10%	73.32%

The paper also compares with RAdam, AdamW, MIAdam, and other baselines, and states that DualAdam is best among the compared methods in the listed image-classification settings (Shi et al., 7 Mar 2026). Training time is described as essentially unchanged: for ResNet-18 on CIFAR-100, Adam takes 36m37s and DualAdam 36m54s (Shi et al., 7 Mar 2026).

For LLM fine-tuning, the model is OpenPangu-Embedded-1B and the dataset is Alpaca-GPT4-CN, described as a Chinese instruction-following dataset. The comparison is against AdamW, using training loss, validation perplexity, and generalization gap defined as validation loss minus training loss. The reported behavior is qualitatively asymmetric: AdamW achieves lower training loss, but DualAdam achieves lower and more stable validation perplexity, while AdamW’s perplexity eventually increases and its generalization gap grows significantly over time; DualAdam’s generalization gap stays close to zero (Shi et al., 7 Mar 2026). This suggests that DualAdam’s flat-minima bias is not restricted to vision models.

6. Ablations, practical configuration, and operating regime

The central additional hyperparameter in DualAdam is the switching rate $i$9. On CIFAR-100 with ResNet-18, the ablation reported in the paper shows that $$\boldsymbol{g}_{t,i} = \frac{\partial L(\boldsymbol{\theta}_{t,i})}{\partial \boldsymbol{\theta}_{t,i}}.$$0, corresponding to pure InvAdam, fails to converge: the training loss remains at $$\boldsymbol{g}_{t,i} = \frac{\partial L(\boldsymbol{\theta}_{t,i})}{\partial \boldsymbol{\theta}_{t,i}}.$$1 and validation accuracy at $$\boldsymbol{g}_{t,i} = \frac{\partial L(\boldsymbol{\theta}_{t,i})}{\partial \boldsymbol{\theta}_{t,i}}.$$2 (Shi et al., 7 Mar 2026). This is presented as empirical evidence that InvAdam alone is not a practical optimizer for deep networks without an additional mechanism.

When $$\boldsymbol{g}_{t,i} = \frac{\partial L(\boldsymbol{\theta}_{t,i})}{\partial \boldsymbol{\theta}_{t,i}}.$$3 lies in $$\boldsymbol{g}_{t,i} = \frac{\partial L(\boldsymbol{\theta}_{t,i})}{\partial \boldsymbol{\theta}_{t,i}}.$$4, training is convergent, with best validation accuracy around $$\boldsymbol{g}_{t,i} = \frac{\partial L(\boldsymbol{\theta}_{t,i})}{\partial \boldsymbol{\theta}_{t,i}}.$$5 at $$\boldsymbol{g}_{t,i} = \frac{\partial L(\boldsymbol{\theta}_{t,i})}{\partial \boldsymbol{\theta}_{t,i}}.$$6 on the held-out validation split. The main image experiments use $$\boldsymbol{g}_{t,i} = \frac{\partial L(\boldsymbol{\theta}_{t,i})}{\partial \boldsymbol{\theta}_{t,i}}.$$7. With 200 epochs and batch size 128, corresponding to about 78,200 iterations, $$\boldsymbol{g}_{t,i} = \frac{\partial L(\boldsymbol{\theta}_{t,i})}{\partial \boldsymbol{\theta}_{t,i}}.$$8 reaches $$\boldsymbol{g}_{t,i} = \frac{\partial L(\boldsymbol{\theta}_{t,i})}{\partial \boldsymbol{\theta}_{t,i}}.$$9 at approximately $$\begin{cases} \boldsymbol{m}_{t,i} = \beta_1 \boldsymbol{m}_{t-1,i} + (1 - \beta_1)\boldsymbol{g}_{t,i}, \ \boldsymbol{v}_{t,i} = \beta_2 \boldsymbol{v}_{t-1,i} + (1 - \beta_2)\boldsymbol{g}_{t,i}^2, \end{cases}$$0, so only the early 16% of training uses the InvAdam blend (Shi et al., 7 Mar 2026).

The paper further compares switching mechanisms. Linear switching,

$$\begin{cases} \boldsymbol{m}_{t,i} = \beta_1 \boldsymbol{m}_{t-1,i} + (1 - \beta_1)\boldsymbol{g}_{t,i}, \ \boldsymbol{v}_{t,i} = \beta_2 \boldsymbol{v}_{t-1,i} + (1 - \beta_2)\boldsymbol{g}_{t,i}^2, \end{cases}$$1

is reported as the best and most stable. On CIFAR-100 with ResNet-18, linear switching attains top-1 accuracy of approximately $$\begin{cases} \boldsymbol{m}_{t,i} = \beta_1 \boldsymbol{m}_{t-1,i} + (1 - \beta_1)\boldsymbol{g}_{t,i}, \ \boldsymbol{v}_{t,i} = \beta_2 \boldsymbol{v}_{t-1,i} + (1 - \beta_2)\boldsymbol{g}_{t,i}^2, \end{cases}$$2; exponential switching with bases $$\begin{cases} \boldsymbol{m}_{t,i} = \beta_1 \boldsymbol{m}_{t-1,i} + (1 - \beta_1)\boldsymbol{g}_{t,i}, \ \boldsymbol{v}_{t,i} = \beta_2 \boldsymbol{v}_{t-1,i} + (1 - \beta_2)\boldsymbol{g}_{t,i}^2, \end{cases}$$3, $$\begin{cases} \boldsymbol{m}_{t,i} = \beta_1 \boldsymbol{m}_{t-1,i} + (1 - \beta_1)\boldsymbol{g}_{t,i}, \ \boldsymbol{v}_{t,i} = \beta_2 \boldsymbol{v}_{t-1,i} + (1 - \beta_2)\boldsymbol{g}_{t,i}^2, \end{cases}$$4, and $$\begin{cases} \boldsymbol{m}_{t,i} = \beta_1 \boldsymbol{m}_{t-1,i} + (1 - \beta_1)\boldsymbol{g}_{t,i}, \ \boldsymbol{v}_{t,i} = \beta_2 \boldsymbol{v}_{t-1,i} + (1 - \beta_2)\boldsymbol{g}_{t,i}^2, \end{cases}$$5 yields $$\begin{cases} \boldsymbol{m}_{t,i} = \beta_1 \boldsymbol{m}_{t-1,i} + (1 - \beta_1)\boldsymbol{g}_{t,i}, \ \boldsymbol{v}_{t,i} = \beta_2 \boldsymbol{v}_{t-1,i} + (1 - \beta_2)\boldsymbol{g}_{t,i}^2, \end{cases}$$6, $$\begin{cases} \boldsymbol{m}_{t,i} = \beta_1 \boldsymbol{m}_{t-1,i} + (1 - \beta_1)\boldsymbol{g}_{t,i}, \ \boldsymbol{v}_{t,i} = \beta_2 \boldsymbol{v}_{t-1,i} + (1 - \beta_2)\boldsymbol{g}_{t,i}^2, \end{cases}$$7, and $$\begin{cases} \boldsymbol{m}_{t,i} = \beta_1 \boldsymbol{m}_{t-1,i} + (1 - \beta_1)\boldsymbol{g}_{t,i}, \ \boldsymbol{v}_{t,i} = \beta_2 \boldsymbol{v}_{t-1,i} + (1 - \beta_2)\boldsymbol{g}_{t,i}^2, \end{cases}$$8, respectively; fixed-epoch hard switching at epochs $$\begin{cases} \boldsymbol{m}_{t,i} = \beta_1 \boldsymbol{m}_{t-1,i} + (1 - \beta_1)\boldsymbol{g}_{t,i}, \ \boldsymbol{v}_{t,i} = \beta_2 \boldsymbol{v}_{t-1,i} + (1 - \beta_2)\boldsymbol{g}_{t,i}^2, \end{cases}$$9, $$\hat{\boldsymbol{m}}_{t,i} = \frac{\boldsymbol{m}_{t,i}}{1-\beta_1^t}, \qquad \hat{\boldsymbol{v}}_{t,i} = \frac{\boldsymbol{v}_{t,i}}{1-\beta_2^t}.$$0, and $$\hat{\boldsymbol{m}}_{t,i} = \frac{\boldsymbol{m}_{t,i}}{1-\beta_1^t}, \qquad \hat{\boldsymbol{v}}_{t,i} = \frac{\boldsymbol{v}_{t,i}}{1-\beta_2^t}.$$1 yields $$\hat{\boldsymbol{m}}_{t,i} = \frac{\boldsymbol{m}_{t,i}}{1-\beta_1^t}, \qquad \hat{\boldsymbol{v}}_{t,i} = \frac{\boldsymbol{v}_{t,i}}{1-\beta_2^t}.$$2, $$\hat{\boldsymbol{m}}_{t,i} = \frac{\boldsymbol{m}_{t,i}}{1-\beta_1^t}, \qquad \hat{\boldsymbol{v}}_{t,i} = \frac{\boldsymbol{v}_{t,i}}{1-\beta_2^t}.$$3, and $$\hat{\boldsymbol{m}}_{t,i} = \frac{\boldsymbol{m}_{t,i}}{1-\beta_1^t}, \qquad \hat{\boldsymbol{v}}_{t,i} = \frac{\boldsymbol{v}_{t,i}}{1-\beta_2^t}.$$4 (Shi et al., 7 Mar 2026). The paper’s conclusion is that linear decay of the InvAdam proportion is the most effective and stable mechanism among those tested.

In practical use, the recommended configuration largely preserves standard Adam hyperparameters: the same $$\hat{\boldsymbol{m}}_{t,i} = \frac{\boldsymbol{m}_{t,i}}{1-\beta_1^t}, \qquad \hat{\boldsymbol{v}}_{t,i} = \frac{\boldsymbol{v}_{t,i}}{1-\beta_2^t}.$$5, $$\hat{\boldsymbol{m}}_{t,i} = \frac{\boldsymbol{m}_{t,i}}{1-\beta_1^t}, \qquad \hat{\boldsymbol{v}}_{t,i} = \frac{\boldsymbol{v}_{t,i}}{1-\beta_2^t}.$$6, $$\hat{\boldsymbol{m}}_{t,i} = \frac{\boldsymbol{m}_{t,i}}{1-\beta_1^t}, \qquad \hat{\boldsymbol{v}}_{t,i} = \frac{\boldsymbol{v}_{t,i}}{1-\beta_2^t}.$$7, and $$\hat{\boldsymbol{m}}_{t,i} = \frac{\boldsymbol{m}_{t,i}}{1-\beta_1^t}, \qquad \hat{\boldsymbol{v}}_{t,i} = \frac{\boldsymbol{v}_{t,i}}{1-\beta_2^t}.$$8 are used in most experiments, and only $$\hat{\boldsymbol{m}}_{t,i} = \frac{\boldsymbol{m}_{t,i}}{1-\beta_1^t}, \qquad \hat{\boldsymbol{v}}_{t,i} = \frac{\boldsymbol{v}_{t,i}}{1-\beta_2^t}.$$9 is added. For CIFAR experiments, the paper states that $\eta$0 is usually $\eta$1, $\eta$2 usually $\eta$3, and $\eta$4 for example $\eta$5 (Shi et al., 7 Mar 2026). It also recommends starting from Adam hyperparameters already known to work and choosing $\eta$6 so that the InvAdam phase covers roughly 10–20% of total iterations. If training is unstable or diverges early, the stated rules-of-thumb are to increase $\eta$7 or reduce $\eta$8; if generalization gains are small, one may slightly reduce $\eta$9, while avoiding values so small that behavior approaches pure InvAdam (Shi et al., 7 Mar 2026).

7. Interpretation, limitations, and relation to adjacent methods

DualAdam is positioned in the paper against several adjacent optimizers. Relative to vanilla Adam, its distinctive feature is the temporary inverse use of the second moment early in training. Relative to AdamW, it changes optimization dynamics rather than regularization. Relative to SWATS and MIAdam, the paper emphasizes that DualAdam uses a simple linear schedule rather than a hard switch, and states that abrupt transitions can be unstable (Shi et al., 7 Mar 2026).

The paper also reports post hoc evidence that the solutions reached by DualAdam are flatter than those reached by Adam. Using PyHessian on ResNet-18 trained on CIFAR-100, it evaluates top Hessian eigenvalues, Hessian trace, and eigenvalue density; the DualAdam solutions have eigenvalues more centered near zero and smaller top eigenvalues and Hessian trace than Adam’s solutions. Using loss-landscape visualization of the form $$\min_{\boldsymbol{\theta}} \mathcal{L}(\boldsymbol{x}, \boldsymbol{y}; \boldsymbol{\theta}).$$00 along a random direction $$\min_{\boldsymbol{\theta}} \mathcal{L}(\boldsymbol{x}, \boldsymbol{y}; \boldsymbol{\theta}).$$01, it reports that the curve around the DualAdam solution for ResNet-18 on CIFAR-10 is wider and flatter than around the Adam solution (Shi et al., 7 Mar 2026). These observations are presented as consistent with the paper’s flat-minima account of generalization.

Several limitations are stated explicitly. First, InvAdam alone is unstable and does not converge on realistic tasks. Second, performance depends on the switching rate $$\min_{\boldsymbol{\theta}} \mathcal{L}(\boldsymbol{x}, \boldsymbol{y}; \boldsymbol{\theta}).$$02: too small a value yields excessive InvAdam behavior and poor convergence, whereas too large a value makes the method close to pure Adam and limits generalization gains. Third, the convergence argument for DualAdam is indirect: because the method becomes Adam later in training and Adam’s convergence has been studied in prior work, the authors argue that DualAdam’s convergence is guaranteed, but they do not provide a full non-asymptotic convergence theorem for the full trajectory (Shi et al., 7 Mar 2026).

The future directions proposed in the source include more effective switching mechanisms, combinations of InvAdam-like mechanisms with other base optimizers such as inverse SGD or inverse Adan, and broader theoretical analysis of dual-optimizer schemes (Shi et al., 7 Mar 2026). A plausible implication is that DualAdam serves not only as a specific optimizer but also as a template for temporally staged combinations in which early sharp-minima escape and late stable convergence are assigned to different update geometries.