Singularity-aware Adam Optimizers

• The paper on strongly convex SAdam demonstrates a data-dependent O(log T) regret bound by replacing the sqrt(v) scaling with v⁻¹ and tuning β₂ dynamically.
• Sadam calibrates adaptive learning rates using a softplus function, which stabilizes extreme coordinate updates and enhances convergence and generalization in various settings.
• S-Adam for non-smooth optimization employs a Local Geometric Instability score to dampen updates near Clarke singularities, effectively reducing gradient chattering.

Singularity-aware Adam, variously written SAdam, Sadam, or S-Adam, denotes several distinct Adam-derived optimization methods that alter Adam’s adaptive denominator or effective step size in order to address specific failure modes of the original algorithm. In the literature represented here, the name has been used for a strongly convex online-learning variant that replaces Adam’s $v_t^{-1/2}$ scaling by $v_t^{-1}$ and attains a data-dependent $O(\log T)$ regret bound (Wang et al., 2019), a softplus-calibrated adaptive method intended to regularize highly anisotropic coordinatewise learning rates and improve convergence and generalization (Tong et al., 2019), and a non-smooth optimization method that damps Adam updates according to a randomized estimate of local geometric instability near Clarke-singular regions (Xu et al., 28 May 2026).

1. Nomenclature and scope

A persistent source of confusion is that “singularity-aware Adam” is not a single standardized optimizer. The label has been attached to different modifications of Adam, each targeting a different pathology. One line of work focuses on strongly convex online convex optimization, another on adaptive-learning-rate calibration in smooth or weakly structured stochastic optimization, and a third on non-smooth deep-learning regimes involving ReLU kinks, quantization, and Clarke-subdifferential effects.

Variant	Core modification	Primary setting
SAdam (Wang et al., 2019)	Uses $\hat v_t^{-1}$ instead of $\hat v_t^{-1/2}$ and controls decay through time-varying $\beta_{2t}$	Strongly convex online convex optimization
Sadam (Tong et al., 2019)	Replaces $\sqrt{v_t}+\epsilon$ by $\mathrm{softplus}(\sqrt{v_t})$	Nonconvex, convex, P-L, and strongly convex stochastic optimization
S-Adam (Xu et al., 28 May 2026)	Multiplies Adam’s step by $\exp(-\lambda \rho_t)$ using a Local Geometric Instability score	Non-smooth stochastic optimization, QAT, and small-batch learning

The shared theme is modification of Adam’s adaptivity near problematic regimes. The problematic regime, however, differs substantially across papers: strongly convex geometry in (Wang et al., 2019), anisotropic adaptive learning rates and $\epsilon$-sensitivity in (Tong et al., 2019), and non-smooth singularities characterized through the Clarke subdifferential in (Xu et al., 28 May 2026).

2. Strongly convex SAdam

In the 2019 paper “SAdam: A Variant of Adam for Strongly Convex Functions,” SAdam is the strongly-convex adaptation of Adam (Wang et al., 2019). The motivating question is whether the strong-convexity structure can be exploited to improve on the data-dependent $v_t^{-1}$0 regret bounds available for Adam-type methods under general convexity. The answer given is affirmative: SAdam is designed so that the overall effective learning rate decays approximately as $v_t^{-1}$1, matching the classical strongly convex online-gradient regime while preserving diagonal adaptivity and momentum.

The setting is online convex optimization over a convex decision set $v_t^{-1}$2. At round $v_t^{-1}$3, the learner selects $v_t^{-1}$4, observes $v_t^{-1}$5, and incurs loss $v_t^{-1}$6, with regret

$v_t^{-1}$7

The analysis assumes that every $v_t^{-1}$8 is $v_t^{-1}$9-strongly convex, that gradients satisfy

$O(\log T)$0

and that the domain is bounded in $O(\log T)$1 diameter:

$O(\log T)$2

The method also uses a weighted projection

$O(\log T)$3

for positive definite $O(\log T)$4.

Relative to standard Adam,

$O(\log T)$5

with update

$O(\log T)$6

SAdam modifies two components. First, it removes the square root from the preconditioner. Second, it makes $O(\log T)$7 time-varying so that the inverse preconditioner grows in a controlled manner. The paper’s Algorithm 1 is

$O(\log T)$8

$O(\log T)$9

$\hat v_t^{-1}$0

$\hat v_t^{-1}$1

$\hat v_t^{-1}$2

A typical choice is

$\hat v_t^{-1}$3

The central analytic device is control of the effective $\hat v_t^{-1}$4-like step through conditions on the diagonal preconditioner. The key monotonicity-style requirement is:

There exists $\hat v_t^{-1}$5 such that for any $\hat v_t^{-1}$6, for all $\hat v_t^{-1}$7 and coordinates $\hat v_t^{-1}$8, $\hat v_t^{-1}$9

A second technical condition bounds the cumulative effect of the EMA denominator relative to $\hat v_t^{-1/2}$0. Under these assumptions, Theorem 1 yields a regret bound whose characteristic term is

$\hat v_t^{-1/2}$1

so the worst-case rate becomes $\hat v_t^{-1/2}$2 (Wang et al., 2019). The bound is explicitly data-dependent and becomes tighter when coordinatewise gradient histories are small or sparse.

This SAdam also contains SC-RMSprop as a special case. When $\hat v_t^{-1/2}$3 and $\hat v_t^{-1/2}$4 for suitable constants, the method reduces to SC-RMSprop, and the paper provides what it describes as the first data-dependent logarithmic regret bound for that method. In the $\hat v_t^{-1/2}$5 case, Corollary 3 gives a bound of the form

$\hat v_t^{-1/2}$6

up to the exact constants and normalization used in the paper.

Empirically, this strongly convex SAdam is evaluated on $\hat v_t^{-1/2}$7-regularized softmax regression over MNIST, CIFAR-10, and CIFAR-100, where it achieves the lowest regret among the compared methods; strongly convex methods such as SC-Adagrad, SC-RMSprop, and SAdam outperform Adam, AMSgrad, and AdamNC, while plain OGD performs worst overall. On a 4-layer CNN trained on the same datasets, SAdam yields the lowest training loss and strong test accuracy (Wang et al., 2019).

3. Softplus-calibrated Sadam

The 2019 paper “Calibrating the Adaptive Learning Rate to Improve Convergence of ADAM” uses Sadam to denote a different algorithmic idea (Tong et al., 2019). Its diagnosis is that Adam’s adaptive learning rate,

$\hat v_t^{-1/2}$8

is often highly anisotropic across coordinates and over training time. The authors report that some coordinates of

$\hat v_t^{-1/2}$9

can become extremely large, even reaching about $\beta_{2t}$0 when $\beta_{2t}$1. This motivates what the paper calls a “small learning-rate dilemma”: choosing the base learning rate $\beta_{2t}$2 small enough to prevent unstable large updates in some coordinates may cause other coordinates to update too slowly later in training.

Sadam retains Adam’s first- and second-moment recursions,

$\beta_{2t}$3

$\beta_{2t}$4

but replaces Adam’s denominator $\beta_{2t}$5 by

$\beta_{2t}$6

where

$\beta_{2t}$7

The update becomes

$\beta_{2t}$8

For large $\beta_{2t}$9, $\sqrt{v_t}+\epsilon$0; for small $\sqrt{v_t}+\epsilon$1, it lifts the denominator away from zero. The paper recommends the Adam-like defaults $\sqrt{v_t}+\epsilon$2 and $\sqrt{v_t}+\epsilon$3, and reports that $\sqrt{v_t}+\epsilon$4 is typically chosen from $\sqrt{v_t}+\epsilon$5, with $\sqrt{v_t}+\epsilon$6 recommended based on observed performance (Tong et al., 2019).

The companion method SAMSGrad applies the same calibration to AMSGrad’s monotone second-moment construction:

$\sqrt{v_t}+\epsilon$7

followed by

$\sqrt{v_t}+\epsilon$8

The conceptual distinction from Adam is that Adam stabilizes through a fixed additive $\sqrt{v_t}+\epsilon$9, whereas Sadam stabilizes by smoothly bounding and compressing the denominator via softplus. According to the paper, this reduces extreme A-LR values in coordinates where $\mathrm{softplus}(\sqrt{v_t})$0 is tiny and removes the explicit need for $\mathrm{softplus}(\sqrt{v_t})$1 as a tuning knob. When $\mathrm{softplus}(\sqrt{v_t})$2 is large, $\mathrm{softplus}(\sqrt{v_t})$3 and the method approaches Adam-like behavior; when $\mathrm{softplus}(\sqrt{v_t})$4 is small, the behavior becomes more like momentum SGD (Tong et al., 2019).

The theoretical contribution is organized around explicit bounds on the adaptive learning rate. For Adam, the A-LR bounds are denoted by $\mathrm{softplus}(\sqrt{v_t})$5; for Sadam, by $\mathrm{softplus}(\sqrt{v_t})$6. A central claim is that Adam’s convergence constants depend materially on $\mathrm{softplus}(\sqrt{v_t})$7, a dependence said to have been largely overlooked in earlier analyses. Under $\mathrm{softplus}(\sqrt{v_t})$8-smoothness, bounded gradients, and bounded stochastic-gradient variance, the paper derives:

• in the nonconvex setting with $\mathrm{softplus}(\sqrt{v_t})$9, Adam has a convergence rate with explicit $\exp(-\lambda \rho_t)$0 dependence, whereas Sadam has leading dependence summarized as $\exp(-\lambda \rho_t)$1;
• in the convex, non-strongly convex setting for the averaged iterate $\exp(-\lambda \rho_t)$2, Adam satisfies

$\exp(-\lambda \rho_t)$3

while Sadam satisfies

$\exp(-\lambda \rho_t)$4

• under the Polyak–Łojasiewicz condition

$\exp(-\lambda \rho_t)$5

and in the strongly convex case, with $\exp(-\lambda \rho_t)$6, the paper gives linear-contraction-style expressions multiplied by an $\exp(-\lambda \rho_t)$7 term, with Sadam constants determined by $\exp(-\lambda \rho_t)$8 and $\exp(-\lambda \rho_t)$9 (Tong et al., 2019).

Empirical studies are conducted on MNIST with a CNN; CIFAR-10 with ResNet-20, ResNet-56, and DenseNet-40; CIFAR-100 with VGG and ResNet-18; and Penn Treebank and WikiText-2 with 3-layer LSTMs. The reported findings are that Sadam and SAMSGrad improve test accuracy, often close the gap with momentum SGD, improve perplexity curves and convergence behavior in language modeling, and compress the extreme range of the A-LR distribution.

4. S-Adam for non-smooth optimization

The 2026 paper “Singularity-aware Optimization via Randomized Geometric Probing: Towards Stable Non-smooth Optimization” introduces S-Adam in a third, more explicitly geometric sense (Xu et al., 28 May 2026). The motivating claim is that modern deep networks are not globally smooth because of ReLU activations, quantization operators in QAT, and sparsity or regularization terms. At such singular points, the gradient is modeled not as a single vector but as a set-valued Clarke subdifferential $\epsilon$0.

In this framework, Adam is said to suffer from gradient chattering: violent oscillations in the update direction caused by conflicting signals within the Clarke subdifferential. The paper attributes to this phenomenon a mismatch between Adam’s momentum buffer and local geometry, overshooting across kinks or discrete boundaries, poor convergence, and worse generalization. The method is presented as a drop-in Adam variant that detects geometric singularity and damps updates only when necessary (Xu et al., 28 May 2026).

The core diagnostic is the Local Geometric Instability (LGI) score, intended to estimate the diameter of the Clarke subdifferential,

$\epsilon$1

At iterate $\epsilon$2, one samples random directions

$\epsilon$3

forms directional probes

$\epsilon$4

and defines

$\epsilon$5

The appendix also writes the denominator as $\epsilon$6. The paper proves

$\epsilon$7

so the damping factor $\epsilon$8 lies in $\epsilon$9. A finite-sample guarantee states

$v_t^{-1}$00

using

$v_t^{-1}$01

The appendix also gives a randomized-smoothing interpretation. For

$v_t^{-1}$02

LGI approximates a relative-curvature quantity

$v_t^{-1}$03

where

$v_t^{-1}$04

Within the paper’s interpretation, high curvature or high nonsmoothness implies $v_t^{-1}$05, whereas flat or smooth regions imply $v_t^{-1}$06 (Xu et al., 28 May 2026).

S-Adam keeps Adam’s moments,

$v_t^{-1}$07

but uses the LGI-modulated step size

$v_t^{-1}$08

leading to the update

$v_t^{-1}$09

The experiments use perturbation scale $v_t^{-1}$10, probe count $v_t^{-1}$11, damping intensity $v_t^{-1}$12, stabilization constant $v_t^{-1}$13, and LGI score cap $v_t^{-1}$14.

The convergence analysis is developed under local Lipschitzness, lower boundedness, path differentiability, almost-sure boundedness of iterates, a Robbins–Monro-style step-size condition

$v_t^{-1}$15

and consistency conditions for the moment estimators. The algorithm is written as a stochastic approximation whose mean-field limit is a differential inclusion,

$v_t^{-1}$16

with

$v_t^{-1}$17

Using this framework, the paper proves

$v_t^{-1}$18

so every limit point is Clarke stationary, and it establishes the rate

$v_t^{-1}$19

equivalently $v_t^{-1}$20 iterations to reach an $v_t^{-1}$21-stationary point (Xu et al., 28 May 2026).

5. Empirical regimes and reported behavior

The three S-Adam lines are empirically oriented toward different regimes, and their reported behavior should be read in that context.

For the strongly convex SAdam of (Wang et al., 2019), experiments emphasize online-style regret and regularized convex learning. On $v_t^{-1}$22-regularized softmax regression over MNIST, CIFAR-10, and CIFAR-100, SAdam achieves the lowest regret among the compared methods. Strongly convex methods—SC-Adagrad, SC-RMSprop, and SAdam—outperform Adam, AMSgrad, and AdamNC, while OGD performs worst overall. On a 4-layer CNN over the same datasets, SAdam obtains the lowest training loss and strong test accuracy.

For the softplus-calibrated Sadam of (Tong et al., 2019), experiments target convergence and generalization under standard deep-learning workloads. The paper reports that Adam often has lower test accuracy than SGD with momentum, that Sadam and SAMSGrad improve test accuracy and often narrow that gap, and that on CIFAR-10 the calibrated methods often outperform Adam, AMSGrad, Yogi, PAdam, PAMSGrad, AdaBound, and AmsBound. On Penn Treebank and WikiText-2, the methods improve perplexity curves and convergence behavior. A key empirical diagnostic is the A-LR distribution: softplus calibration is reported to compress the extreme range and make coordinatewise learning rates more regular.

For the non-smooth S-Adam of (Xu et al., 28 May 2026), the experiments are organized around QAT and high-noise small-batch training. In QAT, using a custom QATNet CNN, S-Adam is compared against AdamW and Prox-SGD on CIFAR-100, TinyImageNet, Imagewoof2-160, and ImageNet. Reported accuracies for S-Adam with $v_t^{-1}$23 are 18.67 on CIFAR-100, 23.18 on TinyImageNet, 35.19 on Imagewoof2-160, and 11.24 on ImageNet; these exceed the corresponding AdamW values 15.94, 18.91, 33.24, and 8.63. In high-noise small-batch learning with a ResNet18 pretrained on ImageNet and batch size 2, S-Adam with $v_t^{-1}$24 reaches 56.11 on CIFAR-100, 86.13 on CIFAR-10, and 75.87 on Imagewoof2-160, versus AdamW at 51.12, 80.48, and 51.18. The paper emphasizes that baselines often descend early and then rebound or collapse due to chattering, whereas S-Adam remains more stable and maintains a lower loss floor (Xu et al., 28 May 2026).

The ablations in (Xu et al., 28 May 2026) reinforce the central role of the LGI brake. Setting $v_t^{-1}$25 reduces accuracy; using $v_t^{-1}$26 performs much worse because the sample variance vanishes and effectively disables the brake; and making $v_t^{-1}$27 too small, such as $v_t^{-1}$28, degrades performance because the probes fail to detect quantization boundaries or singular regions.

6. Conceptual distinctions and recurrent misconceptions

The phrase “singularity-aware” has different meanings across these papers. In (Wang et al., 2019), the paper explicitly does not discuss singularity in a geometric or manifold sense; the term is consistent instead with the addition of $v_t^{-1}$29 to the diagonal preconditioner and with control of inverse step sizes when second-moment estimates are small. In (Tong et al., 2019), singularity-awareness refers to the danger of a potentially tiny denominator $v_t^{-1}$30 and the resulting extreme coordinatewise adaptive learning rates. In (Xu et al., 28 May 2026), singularity-awareness is genuinely geometric: it concerns instability near non-differentiable points, quantified through randomized probes of Clarke-subdifferential variability.

A second misconception is that the theoretical guarantees of these methods are directly comparable. They are not stated in the same framework. The first paper studies online regret for strongly convex losses and proves a data-dependent logarithmic bound. The second studies stochastic optimization convergence in nonconvex, convex, P-L, and strongly convex regimes, with constants expressed through $v_t^{-1}$31 or $v_t^{-1}$32. The third studies non-smooth stochastic approximation and convergence to Clarke stationary points at the standard $v_t^{-1}$33 rate. This suggests that headline rates such as $v_t^{-1}$34 and $v_t^{-1}$35 should not be compared without reference to the underlying objective class, stationarity notion, and proof framework.

A third misconception is that all three methods simply “stabilize Adam” in the same way. Their mechanisms are structurally different. The strongly convex SAdam changes the preconditioner from $v_t^{-1}$36 to $v_t^{-1}$37 and tunes $v_t^{-1}$38 for controlled $v_t^{-1}$39-like decay. The softplus Sadam leaves the overall Adam structure intact but replaces the denominator by a smooth calibration function. The non-smooth S-Adam keeps the standard Adam denominator and instead multiplies the learning rate by a geometry-dependent brake $v_t^{-1}$40.

Taken together, these works show that S-Adam is best understood as a family of Adam modifications rather than a unique algorithmic object. Their common concern is Adam’s fragility under mismatched geometry—strongly convex structure, anisotropic adaptive scaling, or non-smooth singular behavior—but each paper formalizes that fragility differently and responds with a distinct modification of the adaptive update rule.